Finding Hidden Intelligence with Predictive Analysis of Data Mining

Transcription

1 Finding Hidden Intelligence with Predictive Analysis of Data Mining Rafal Lukawiecki Strategic Consultant, Project Botticelli Ltd

2 Objectives Show use of Microsoft SQL Server 2008 Analysis Services Data Mining Tantalise you with the power of DM This seminar is based on a number of sources including a few dozen of Microsoft-owned presentations, used with permission. Thank you to Marin Bezic, Kathy Sabourin, Aydin Gencler, Bryan Bredehoeft, and Chris Dial for all the support. Thank you to Maciej Pilecki for assistance with demos. The information herein is for informational purposes only and represents the opinions and views of Project Botticelli and/or Rafal Lukawiecki. The material presented is not certain and may vary based on several factors. Microsoft makes no warranties, express, implied or statutory, as to the information in this presentation. Portions 2009 Project Botticelli Ltd & entire material 2009 Microsoft Corp. Some slides contain quotations from copyrighted materials by other authors, as individually attributed or as already covered by Microsoft Copyright ownerships. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Project Botticelli Ltd as of the date of this presentation. Because Project Botticelli & Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft and Project Botticelli cannot guarantee the accuracy of any information provided after the date of this presentation. Project Botticelli makes no warranties, express, implied or statutory, as to the information in this presentation. E&OE. 2

3 Agenda Data Mining and Predictive Analytics Server and Process Considerations Scenarios & Demos 3

4 What does Data Mining Do? Explores Your Data Finds Patterns Performs Predictions 4

5 Typical Uses Correct Data During ETL Seek Profitable Customers Understand Customer Needs Detect and Prevent Fraud Data Mining Anticipate Customer Churn Build Effective Marketing Campaigns Predict Sales & Inventory 5

6 Server Mining Architecture Deploy BIDS Excel Visio SSMS Excel/Visio/SSRS/Your App OLE DB/ADOMD/XMLA App Data Analysis Services Server Mining Model Data Mining Algorithm Data Source 6

7 Mining Process Training data Mining Model Data to be predicted DM Engine Mining Model Mining Model With predictions 7

8 Concepts Case set of Columns (attributes) you want to analyse Age, Gender, Annual Spending Column Usage Input: We analyse them Predict: Build a model for them Nested Case case containing a table column Age, Gender, Annual Spending, Products, Purchases Case Key unique ID of a case Data Mining Model container of patterns discovered by a DM algorithm in your data 8

9 Who are our customers? Are there any relationships between their demographics and their buying power? SCENARIO: CUSTOMER CLASSIFICATION & SEGMENTATION 9

10 Microsoft Decision Trees Use for: Classification: churn and risk analysis Regression: predict profit or income Association analysis based on multiple predictable variable Builds one tree for each predictable attribute Fast 10

11 Decision Trees for Classification of Customers Buying Potential 11

12 Who are our most profitable customers? Can I predict profit of a future customer based on demographics? Are they creditworthy? How much should I charge them to give a good loan and protect against losses? SCENARIO: PROFITABILITY AND RISK 12

13 Profitability and Risk Finding what makes a customer profitable is also classification or regression Typically solved with: Decision Trees (Regression), Linear Regression, and Neural Networks or Logistic Regression Often used for prediction Important to predict probability of the predicted, or expected profit Risk scoring Logistic Regression and Neural Networks 13

14 Neural Network & Logistic Regression Applied to Classification Regression Great for finding complicated relationship among attributes Difficult to interpret results Gradient Descent method LR is NNet with no hidden layers Input Layer Hidden Layers Output Layer Loyalty Age Education Sex Income 14

15 1. Neural Networks for Profitability Analysis 2. Predicting Lending Risk with Neural Networks 15

16 How do they behave? What are they likely to do once they bought that really expensive car? Should I intervene? SCENARIO: CUSTOMER NEEDS ANALYSIS 16

17 Sequence Clustering Analysis of: Customer behaviour Transaction patterns Click stream Customer segmentation Sequence prediction Mix of clustering and sequence technologies Groups individuals based on their profiles including sequence data 17

18 Analysis Customer Behaviour with Sequence Clustering 18

19 What are my sales going to be like in the next few months? Will I have credit problems? Will my server need an upgrade in the next 3 months? SCENARIO: FORECASTING 19

20 Time Series Uses: Forecast sales Inventory prediction Web hits prediction Stock value estimation Regression trees with extras 20

21 Forecasting Using Time Series 21

22 TECHNIQUE SUMMARY 22

23 Algorithms Algorithm Decision Trees Association Rules Clustering Naïve Bayes Sequence Clustering Time Series Neural Nets Linear Regression Logistic Regression Description Finds the odds of an outcome based on values in a training set Identifies relationships between cases Classifies cases into distinctive groups based on any attribute sets Clearly shows the differences in a particular variable for various data elements Groups or clusters data based on a sequence of previous events Analyzes and forecasts time-based data combining the powerof ARTXP (developed by Microsoft Research) for short-term predictionswith ARIMA (in SQL 2008) for long-term accuracy. Seeks to uncover non-intuitive relationships in data Determines the relationship between columns in order to predict an outcome Determines the relationship between columns in order to evaluate the probability that a column will contain a specific state 23

24 Association Rules Clustering Decision Trees Linear Regression Logistic Regression Naïve Bayes Neural Nets Sequence Clustering Time Series 24

25 Summary Data Mining is a powerful, predictive technology Turns data into valuable, decision-making knowledge SQL Server 2008 Analysis Services support Predictive Analytics Mine your mountains of data for gems of intelligence today! 25

26 2009 Microsoft Corporation & Project Botticelli Ltd. All rights reserved. The information herein is for informational purposes only and represents the opinions and views of Project Botticelli and/or Rafal Lukawiecki. The material presented is not certain and may vary based on several factors. Microsoft makes no warranties, express, implied or statutory, as to the information in this presentation. Portions 2009 Project Botticelli Ltd & entire material 2009 Microsoft Corp. Some slides contain quotations from copyrighted materials by other authors, as individually attributed or as already covered by Microsoft Copyright ownerships. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Project Botticelli Ltd as of the date of this presentation. Because Project Botticelli & Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft and Project Botticelli cannot guarantee the accuracy of any information provided after the date of this presentation. Project Botticelli makes no warranties, express, implied or statutory, as to the information in this presentation. E&OE. 26