Using Data Mining for Business Intelligence - When did data mining begin?
(Page 2 of 4 )
Data mining techniques are the result of a long process of research and product development. This evolution began when business data was first stored on computers, continued with improvements in data access, and more recently, generated technologies that allow users to navigate through their data in real time. Data mining takes this evolutionary process beyond retrospective data access and navigation to prospective and proactive information delivery. Data mining is ready for application in the business community because it is supported by three technologies that are now sufficiently mature:
- Massive data collection
- Powerful multiprocessor computers
- Data mining algorithms
The core components of data mining technology have been under development for decades, in research areas such as statistics, artificial intelligence, and machine learning. Today, the maturity of these techniques, coupled with high-performance relational database engines and broad data integration efforts, make these technologies practical for current data warehouse environments.
Data mining techniques can yield the benefits of automation on existing software and hardware platforms, and can be implemented on new systems as existing platforms are upgraded and new products developed. When data mining tools are implemented on high performance parallel processing systems, they can analyze massive databases in minutes. Faster processing means that users can automatically experiment with more models to understand complex data. High speed makes it practical for users to analyze huge quantities of data. Larger databases, in turn, yield improved predictions.
The Enterprise Data Warehouse as a Data Mining Source
An enterprise data warehouse is an excellent source for locating data to mine. Because of the nature of a data warehouse, most pertinent data that has been selected by analysts and business users should be located within the warehouse structure. In addition, this data is organized and stored for the explicit purpose of reporting. Through the data warehouse, further processing of OLAP data can occur. This processing can take the form of additional aggregations into multidimensional cubes (i.e., SQL Server 2000 Analysis Services Cubes) or undergo further segregation into organizational data marts.
The data mining process will utilize the data in the enterprise data warehouse, based on user selection and location of pertinent data, to test and validate a data mining model. It is important that the data be granular enough to analyze. Data that is characterized by significant aggregations beyond the original grain of the data will not produce significant results when used to create or test against a mining model.
An enterprise data warehouse is a prime source for data mining data because the data housed within the warehouse has already undergone significant data additions, modifications and cleansing based on business rules and processes. Refined Extraction Transformation and Loading (ETL) processes are required for reliable OLAP and enterprise data warehouse reporting. It is the ETL process which is responsible for cleansing bad data from the OLTP source, reclassifying or aggregating granular transactions from the operational system, and enriching the data with more readable and comprehensible data as opposed to the operational codes and abbreviations used in an OLTP system. Once the data has been sufficiently cleansed and refined, it is ripe for data mining.
Typical data warehousing implementations in organizations will allow users to ask and answer questions such as “How many sales were made, by territory, by sales person between the months of May and June in 1999?” Data mining will allow business decision makers to ask and answer questions, such as “Who is my core customer that purchases a particular product we sell?” or “Geographically, how well would a line of products sell in a particular region and who would purchase them, given the sale of similar products in that region?”
Next: How does data mining work? >>
More MS SQL Server Articles
More By Jagadish Chaterjee
