The Process of Data Mining In data mining, data is acquired and prepared for what is known as a data warehouse through the following steps:

Transcription

1 Data Mining A personal database, such as the address list of friends you have on your microcomputer, is generally small. But some databases are almost unimaginably vast, involving records for millions of households and trillions of bytes of data. Some recordkeeping activities require the use of so-called massively parallel database computers that cost $1 million or more. These machines gang together scores or even hundreds of the fastest microprocessors around giving them the oomph to respond in minutes to complex database queries, says one description These large-scale efforts go under the name data mining. Data mining (DM) is the computer-assisted process of sifting through and analyzing vast amounts of data in order to extract hidden patterns and meaning and to discover new knowledge. The purpose of DM is to describe past trends and predict future trends. Thus, datamining tools might sift through a company s immense collections of customer, marketing, production, and financial data and identify what s worth noting and what s not. The Process of Data Mining In data mining, data is acquired and prepared for what is known as a data warehouse through the following steps: 1. DATA SOURCES Data may come from a number of sources: (a) point-of-sale transactions in files (flat files) managed by file management systems on mainframes, (b) Databases of all kinds and (c) other for example, news articles or online sources such as the internet. To the mix may also be added (d) data from data warehouses, as we describe below. Flat file refers to data files that contain records with no structured relationships. Flat files may contain only basic formatting, have a small fixed number of fields, and it may or may not have a file format. It contains a single table of data. Fig (1) flat file 2. DATA FUSION & CLEANSING Data from diverse sources, whether from inside the company (internal data) or purchased from outside the company (external data), must 1

2 be fused together and then put through a process known as data cleansing, or scrubbing. Even if the data comes from just one source, such as one company s mainframe, the data may be of poor quality, full of errors and inconsistencies. Therefore, for data mining to produce accurate results, the source data has to be scrubbed that is, cleaned of errors and checked for consistency of formats. 3. DATA & META-DATA The cleansing process yields both the cleaned-up data and a variation of it called meta-data. Meta-data is essentially data about data; it describes how and when and by whom a particular set of data was collected and how the data is formatted. (The data in a data dictionary, for example, is meta-data.) Meta-data is essential for understanding information stored in data warehouses. Meta-data shows the origins of the data, the transformations it has undergone, and summary information about it, which makes it more useful than the cleansed but unintegrated, unsummarized data. The meta-data also describes the contents of the data warehouse. 4. DATA TRANSPORT TO THE DATA WAREHOUSE Both the data and the metadata are sent to the data warehouse. A data warehouse is a special database of cleanedup data and meta-data. It is a replica, or close reproduction, of a mainframe s data. The data warehouse is stored on disk using storage technology such as RAID (redundant arrays of independent disks). Small data warehouses may hold 100 gigabytes of data or less; large ones may store terabytes of data. Fig (2) the data mining process 2

3 Mining the data: searching for patterns & interpreting the results Data in the warehouse is usually analyzed or mined using one of two popular algorithms, or step-by-step problem-solving procedures: Regression analysis: Basically, regression analysis takes a particular set of numerical data and develops a mathematical formula that fits the data. This formula is then applied to new sets of data of the same type to predict future situations. Classification analysis: Classification analysis is a statistics pattern recognition process that is applied to data sets with more than just numerical data. Some Applications of Data Mining Some short-term payoffs from data mining can be dramatic. o One telephone company, for instance, mined its existing billing data to identify 10,000 supposedly residential customers who spent more than $1,000 a month on their phone bills. When it looked more closely, the company found that these customers were really small businesses trying to avoid paying the more expensive business rates for their telephone service. o The Internal Revenue Service has a program designed to catch tax cheaters by locating inconsistencies between mortgage payments and income. However, the payoffs in the long term could be truly astonishing. o Sifting subatomic-particle information may reveal new insights into the nature of the universe. o Mining medical research data may reveal new knowledge about diseases. o Text-mining programs, which build on data-mining principles, may scour 250,000 pages an hour of articles, automatically categorizing information, making links between unconnected documents, and providing visual maps. DATABASES & THE DIGITAL ECONOMY: E-Business & E-Commerce At one time there was a difference between the Old Economy and the New Economy. The first consisted of traditional companies car makers, retailers, publishers. The second consisted of computer, telecommunications, and internet companies (Amazon, ebay, and a raft of dot-com firms). Now, however, most Old Economy companies have absorbed the new internet-driven technologies, and the differences between the two sectors have dwindled. In other words, most companies are now engaged in e-business, using the internet to facilitate every aspect of running a business. As one article puts it: The internet is a tool that dramatically lowers the cost of communication. That means it can radically alter any industry or activity that depends on the flow of information. One sign of growth is that the number of internet host computers has been almost doubling every year. But the mushrooming of computer networks and the booming popularity of the World Wide Web are only the most obvious signs of the digital economy. Behind them lies something equally important: the growth of vast stores of 3

4 4 information in databases. How are databases underpinning the digital economy? Let us consider two aspects: e-commerce and types of e-commerce systems B2B, B2C, and C2C. E-Commerce: Online Buying & Selling The arrival of the web, aided by the growth and the increasing sophistication of enormous databases, has led to e-commerce, or electronic commerce, the buying and selling of products and services through computer networks. E-commerce is reshaping entire industries and revamping the very notion of what a company is. Indeed, online shopping is growing even faster than the increase in computer use, which has been fueled by the falling price of personal computers. Two well-known e- firms are Amazon.com, which sells books and many other goods, and priceline.com, which lets you name the price you re willing to pay for airline tickets and hotel rooms. (ebay is considered an online shopping bazaar, or auction place, rather than an online retailer.) Probably the foremost example of e-commerce is Amazon.com. AN EXAMPLE OF E-COMMERCE: AMAZON.COM In 1994, seeing the potential for electronic retailing on the World Wide Web, Jeffrey Bezos left a successful career on Wall Street to launch an online bookstore called Amazon.com. Why the name Amazon? Earth s biggest river, Earth s biggest bookstore, said Bezos in a 1996 interview. The Amazon River is ten times as large as the next largest river, which is the Mississippi, in terms of volume of water. Twenty percent of the world s fresh water is in the Amazon River Basin, and we have six times as many titles as the world s largest physical bookstore. A more hardheaded reason is that, according to consumer tests, words starting with A show up on search engine lists first. Still, Bezos realized that no bookstore with four walls could possibly stock the more than 2.5 million titles that are now active and in print. In addition, he appreciated that there would be opportunities to obtain demographic information about customers in order to offer personalized services. For example, Amazon could let customers know of books that might be of interest to them. Such personalized attention is difficult for traditional large bookstores. Finally, Bezos saw that there could be a good deal of online interaction: customers could post reviews of books they read and could reach authors by to provide feedback. All this was made possible on the web by the recording of information on giant databases. Amazon.com sold its first book in July 1995, and by 2008 was No. 1 in the Internet Retailer Top 500 Guide, with sales of $4.06 billion. What began as Earth s biggest bookstore also has rapidly become Earth s biggest anything store, offering CDs, DVDs, videos, electronics, toys, tools, home furnishings, clothing, prescription drugs, and filmprocessing services. Types of E-Commerce Systems: B2B, B2C, & C2C Three main types of e-commerce systems are business-to-business (B2B), business-toconsumer (B2C), and consumer-to-consumer (C2C).

5 BUSINESS-TO-BUSINESS (B2B) SYSTEMS In a business-to-business (B2B) system, a business sells to other businesses, using the internet or a private network to cut transaction costs and increase efficiencies. Besides selling products, a business might sell advertising, employee training, market research, technical support, banking services, and other business support services. Online B2B exchanges have been developed to serve a variety of businesses, from manufacturers of steel and airplanes to convenience stores to olive oil producers. B2B exchanges help business by moving beyond pricing mechanisms and encompassing product quality, customer support and shipping reliability, which often count for more than price. The name given to this system is the business web, or b-web, in which suppliers, distributors, customers, and e-commerce service providers use the internet for communications and transactions. In addition, b-webs are expected to provide extra revenue from additional services, such as financing and logistics. None of these innovations is possible without databases and the communications lines connecting them. B2B used to be conducted mostly over leased communications lines; now, however, many B2B businesses use the internet, an extranet, and/or a virtual private network. BUSINESS-TO- CONSUMER (B2C) SYSTEMS In a business-to-consumer (B2C) system, a business sells goods or services to consumers, or members of the general public. This kind of e-commerce system essentially removes the middleman and often even the need for a physical store. Example of B2C systems is Amazon.com as well as many financial institutions and the U.S. government. Fig () Multistage Model for E-Commerce (B2B and B2C) CONSUMER-TO-CONSUMER (C2C) SYSTEMS In a consumer-to-consumer (C2C) system, consumers sell goods or services directly to other consumers, often with the help of a third party, such as ebay, the online auction company. Such intermediaries mediate between consumers who want to buy and sell, taking a small percentage of the 5

6 seller s profit as a fee. Examples of C2C exchanges are music and file sharing sites, career and job websites and social websites (facebook.com). The advantage of C2C e- commerce is most often the reduced costs and a smaller but profitable customer base. It also gives many small business owners a way to sell their goods without running a costly physical store. 6