Planning a Data Warehouse - Power of Meta Data
(Page 2 of 6 )
Typical relational databases which were designed for on-line transactional processing (OLTP) do not meet the requirements for effective on-line analytical processing (OLAP). As a result, data warehouses are designed differently than traditional relational databases. As the value of data warehouses and their associated OLAP capabilities have increased, comprehensive Meta data management has proven to be a vital element to the success or failure of data warehouses. Without accurate Meta data, a data warehouse rapidly becomes unmanageable and ineffective.
Meta data is literally "data about data". It describes the kind of information in the warehouse, where it stored, how it relates to other information, where it comes from, and how it is related to the business. The topic of standardizing meta data across various products and applying a systems engineering approach to this process in order to facilitate data warehouse design is what this project intends to address.
We intend to create a design process for creating a data warehouse through the development of a metadata system. The Meta data system we intend to create will provide a framework to organize and design the data warehouse. Using a systems engineering approach, this process will be used initially to help define the data warehouse requirements and it will then be used iteratively during the life of the data warehouse to update and integrate new dimensions of the warehouse. We will illustrate our process with a case study using an advertising/product-based business as the data warehouse application.
In order to be effective, the user of the data warehouse must have access to Meta data that is accurate and up to date. Without a good source of Meta data to operate from, the job of the analyst is much more difficult, and the work required for analysis is compounded significantly.
Understanding the role of the enterprise data model relative to the data warehouse is critical to a successful design and implementation of a data warehouse and its Meta data management. Some specific challenges which involve the physical design trade-offs of a data warehouse include:
- Granularity of data - refers to the level of detail held in the unit of data
- Partitioning of data - refers to the break-up of data into separate physical units that can be handled independently
- Performance issues
- Data structures inside the warehouse
- Migration
Meta data itself is arguably the most critical element in effective data management. Effective tools must be used to properly store and use the Meta data generated by the various systems. This paper outlines a Systems Engineering approach to designing and maintaining a data warehouse, emphasizing the key element of metadata. This approach is then used as a template for a case study of the development of an actual data warehouse in a product based business. Finally, a metadata system is created and explained as a tool for designing and updating a data warehouse.
Next: Systems Planning >>
More ASP.NET Articles
More By Jagadish Chaterjee
/ 18 
