Decision Analytic Thinking. Pekka Malo, Assist. Prof. (statistics) Aalto BIZ / Department of Information and Service Economy

Size: px

Start display at page:

Download "Decision Analytic Thinking. Pekka Malo, Assist. Prof. (statistics) Aalto BIZ / Department of Information and Service Economy"

Hester Woods
5 years ago
Views:

1 Decision Analytic Thinking Pekka Malo, Assist. Prof. (statistics) Aalto BIZ / Department of Information and Service Economy

2 Agenda Data-driven Decision Making - Predictive analytics as a process (CRISP-DM) - Predictive analytics as a story - Measuring Value from Predictive Analytics - The Role of Analytics Culture Decision Analytic Thinking: What is a good model? - Expected Value Framework for model building - Visualizing Model Performance Profit curves ROC curves Cumulative response and lift charts 2

3 Learning objectives for this week Knows CRISP-DM approach to predictive analytics Able to use Expected Value Framework to guide evaluation and comparison of predictive models Knows common techniques used to visualize model performance (e.g., Profit curves, ROC curve) 3

Recent studies suggest: More data driven, more productive Correlated with higher ROE, asset

4 Data-driven Decision making (DDD) Practice of basing decisions on the analysis of data rather than purely on intuition. Recent studies suggest: More data driven, more productive Correlated with higher ROE, asset utilization and market value (causal?) Source: Provost, F. and Fawcett, T.: Data Science and its Relationship to Big Data and Datadriven Decision making 4

in the past that strawberry Pop-Tarts increase in sales, like seven times their normal

5 Wal-Mart: It s a Hurricane! start predicting what's going to happen, instead of waiting for it to happen We didn't know in the past that strawberry Pop-Tarts increase in sales, like seven times their normal sales rate, ahead of a hurricane And the pre-hurricane top-selling item was beer Source: 5

6 Predictive Analytics as a Process CRISP-DM

7 CRISP-DM = Cross-Industry Standard Process for Data Mining 7

8 CRISP-DM as a Proposal Review Guide Understanding Business: Recognize the business objectives: What is the status quo (current business processes and costs)? What problem of the client are we trying to solve? What are the performance/success measures? Who owns the problem? Map the objectives to several smaller data mining tasks (e.g. classification problems) Analyze constraints (incl. resources, legal, and ethical issues) 8

9 Understanding data: What data sources are available? How scattered is the data? Are there e.g. legacy system problems? Is the problem a supervised or unsupervised problem? Is target variable defined? What values can it take? Is the definition precise? Does modeling of target variable improve business problem? Are features defined precisely? What values can they take? 9

10 Data Preparation: Is it practical to get values for attributes, create feature vectors, and put into table? If not, is there a clearly defined alternative data format? Is this taken into account in later stages? (Remember: many techniques assume feature-vector format ) Is modeling supervised? Can you get values for the target variable and put them into table? How do you get these values? Are there costs involved in the process? Does the data represent the actual population where model is applied? Are there selection biases? Can they be compensated? 10

11 Modeling: Is the choice of model appropriate for the choice of target variable? Does the model meet the other requirements (e.g., generalization performance, comprehensibility, speed, missing values, type and amount of data) Should several models be tried and compared? 11

12 Evaluation and deployment: Are you going to need domain experts and stakeholders to validate the model before deployment? Do they understand it? Is the evaluation setup and metric appropriate for the tasks (depends on modeling approach) Can you use hold-out data for evaluation? What models do you use as baselines? What about the measurement of final business impact (after deployment)? 12

13 Breakdown of CRISP-DM methodology Source: 13

14 Data Mining as the analysis step of knowledge discovery process Source: Tan, Steinbach & Kumar: Introduction to data mining 14

15 Predictive Analytics as a Story How to explain your findings?

16 Telling a Story with Data Challenge: Meaning in numbers is usually difficult to grasp! Important message gets only partially understood, misunderstood, or completely ignored Importance of data stories Data story ~ narrative that includes analysis Attention to business goals and how data can help to achieve them Persuade, influence, motivate 16

17 Data Storytelling? Source: Gilbert, M.: Storytelling with Numbers 17

18 Source: Gilbert, M.: Storytelling with Numbers 18

19 8 Best Practices NUMBER ONE FRAME THE STORY What problem are we solving; e.g., why are we losing customers NUMBER TWO UNDERSTAND THE KIND OF STORY YOU WANT TO TELL NUMBER THREE KNOW YOUR AUDIENCE One time story (e.g., what caused last month s shortage) vs. Updated, ongoing story (e.g., weekly rise and fall of sales, fraud detection ) What knowledge your audience brings to the story? What kind of preconceptions does the audience have? NUMBER FOUR INCLUDE THE CRITICAL ELEMENTS OF A TRADITIONAL STORY STRUCTURE Point of view: someone has to ask the question that s answered with data Empathy: need to have human protagonist who is solving the problem An antagonist: confusion or misunderstanding that makes achievement of solution difficult An explicit narrative: this happened, then happened, and then Source: Cuzzillo,T and Harper, F: Telling a Story with Data: Eight Best Practices 19

20 NUMBER FIVE DEVELOP THE RIGHT HOOK What helps to grab the attention of the managers; newspaper lead opening; startling statistics; teaser? NUMBER SIX A PICTURE IS PRICELESS People like visuals, but good ones are really difficult to create NUMBER SEVEN WHAT S YOUR POINT? RESOLVE AND CLOSE What does your story advice to do? (a call to action) ITERATE NUMBER EIGHT Some stories need to be retold continuously when new data arrives; good stories live on Source: Cuzzillo,T and Harper, F: Telling a Story with Data: Eight Best Practices 20

21 Measuring Value from Predictive Analytics

22 What drives measurable value? Companies that were able to measure value from predictive analytics were more likely to have a process in place for it. ytics for Business Advantage Measured Value (n=73) Didn t Measure Value (n=124) Satisfaction (completely satisfied or satisfied) 47% 27% Standardized analytics 26% 5% Measured ROI 41% 10% Disparate data types 41% 22% Statisticians building models 87% 67% Table 1. Responses between two groups for certain characteristics. Satisfaction and value are linked. Not surprisingly, those who measured value were generally more Companies that were able to measure value from predictive analytics were more likely to have a process in place for it. à repeatability Utilizing disparate data types can add value to predictive analytics. satisfied with their predictive analytics efforts than those who didn t. That is not to say that those who are more deliberate about predictive analytics cannot be dissatisfied. Many were, especially Utilizing disparate data regarding budget. However, this group was more satisfied (47% versus 27%) than those who could types can add value to predictive analytics. not measure value. See Table 1. This (see makes below good chan A standard as successfu measure t Given the standardi impact put did not (2 companies decision-m (see below) return on it undersc A standardiz measure to standardiz Those wh did theynotare(26n decision-m and so on return on i experime it undersco to five yea Those who also sugg they are no Adding diff and so on. disparate experiment and to fivebotto year also sugges be furthe was gener Adding diffe part of thd disparate thatbottom can h and be further was genera This suggests that it is not enough to simply perform predictive analytics and hope that you get part of the some value out of it. It is important to put a program in place to actually measure this value, which 24 TDWI RESE A RCH 22 can ha that includes defining the project candidates where predictive analytics is most likely to have an impact.

23 Which statement best describes the value you ve seen from your predictive analytics efforts? We have measured positive top- and bottom-line impact 36% We have measured top-line impact only 7% We believe that we have become more effective, but can t measure top-line impact 18% We have measured a cost reduction only 2% We believe that we have become more efficient, but cannot measure impact 12% We have gained more insight 25% Figure 7. Based on 126 active respondents. Source: Halper, F. (2014): Predictive Analytics for Business Advantage 23

24 How satisfied are you with the following aspects of your predictive analytics deployment? (Please rate on a scale of 1 5, where where 1 is completely dissatisfied and 5 is completely satisfied.) Software and tools 3.37 Executive support 3.25 Ability to support multiple data sources 3.25 Others satisfaction level 3.15 Organizational support 3.13 Infrastructure 3.12 Skills in organization 2.96 Analytics culture 2.96 Funding 2.76 Cultural issues can impede progress with predictive analytics. Figure 6. Based on 126 active respondents. Source: Halper, F. (2014): Predictive Analytics for Business Advantage 24

25 The Role of Analytics Culture

26 Learn from Your Analytics Failures While the computational resources and techniques for prediction may be novel and astonishingly powerful, many of the human problems and organizational pathologies appear depressingly familiar. - Michael Schrage (HBR, ) 26

27 What makes an exceptional data scientist? X factor = curiosity Ask a series of questions Accept analytic failures Bigger questions, better insights, more valuable decisions The best way to predict the future is to learn from failed predictive analytics 27

28 The Need for Culture Source: Kiron et al. (2014) 28

29 Data-informed culture ~ the secret sauce Behavior Values Analytics as part of strategy Collaborative use of data Promotion of best analytics practices Invest into tech, talent, training Management is data-driven Data is a core asset Analytics is a mandate driven by executives Decision-making norms Analytical insights guide future strategy Analytics outweighs management experience in key issues Openness to ideas, ability to challenge current practices Data-informed culture 29

30 Does your company have a datainformed culture? Few questions to check your company s current standing Do operational decision makers have clear business rules*? Do you create and revise business rules on the basis of business analytics? Do you provide high quality coaching to employees who make decisions on a regular basis? Have business leaders accepted ownership of key data? Do you have an undisputed source of performance data? Do individuals receive daily feedback on performance? 30 Source: Ross et al. (2013) *) business rule = mechanism for specifying what actions should be taken in a given circumstance

31 Adopting a data-informed culture is often difficult Highly regarded, high performing data skeptics Problem: Afraid that performance measures will not capture the true value of their contribution Remedy: Need to be involved early and have a say in development of metrics Source: Bladt, J. and Filbin, B., HBR, May 16,

32 Adopting a data-informed culture is often difficult The data antagonists Problem Coworkers love them, but they are really afraid to be spotted out! Sometimes nice ideas, but a lot of shooting in the dark Remedy? Source: Bladt, J. and Filbin, B., HBR, May 16,

33 Decision Analytic Thinking: What is a good model? Model performance and visualization

34 From models to decisions When using predictive models, we need to evaluate the goodness of the set of decisions made by a model when applied to held-out data - Is our data driven model better than the hand-crafted model proposed by marketing group? - Which algorithm is better? Does a classification tree perform better than a linear discriminant model? - Is any of the models better than a baseline (e.g., random prediction) - How much value do we expect to get? What is the expected value (profit) under different models? 34

35 Why data mining needs caution? Bonferroni s Principle (in human readable form): If you look harder than the quantity of data supports, you will find a pattern that fits. Examples: Sun spots & stock markets Regressions with close to perfect fit - Too many explanatory variables - Not enough observations 35

Reminder: Evaluating classifier performance Targeted marketing example TN (true negatives): number of prospects who are correctly predicted to be nonrespondents TP (true positives): number of

36 Reminder: Evaluating classifier performance Targeted marketing example TN (true negatives): number of prospects who are correctly predicted to be nonrespondents TP (true positives): number of prospects who are correctly predicted to be respondents FN (false negatives): prospects who were predicted to be non-respondents but would have actually been respondents FP (false positives): prospects who were assumed to be respondents but did not respond 36

37 Why accuracy of a model can be misleading? Unbalanced or skewed class distributions What can easily happen when modeling a problem (e.g., fraud detection) where one class is rare? Problems with unequal costs and benefits Accuracy doesn t see a difference between false positive and false negative errors (i.e., it assumes that both errors are equally significant) Example: cancer diagnosis (FP: extra tests given to patient; FN: person with cancer misses early detection) 37

Expected Value Framework Helps to decompose data-analytic thinking into - Structure of the problem - Elements of analysis that can be extracted or learned from data - Elements

38 Expected Value Framework Helps to decompose data-analytic thinking into - Structure of the problem - Elements of analysis that can be extracted or learned from data - Elements that need to be acquired from other sources (e.g., business knowledge) EV = X i2i p(o i )v(o i ) = p(o 1 )v(o 1 )+p(o 2 )v(o 2 )+ Probability of an outcome Value of an outcome 38

39 Using Expected Value to Frame Classifier Use Example: Targeted marketing campaign Outcomes = {response (R), no response (NR)} x = feature vector describing a customer Our knowledge of the customer Expected benefit of targeting = p R (x)v R +[1 p R (x)]v NR Model based estimate for the probability of response Benefit in case of response Benefit in case of no response 39

40 Example (cont d): What customers should we target? price of product is $200, production costs are $100 cost of making the marketing offer is $1 v R = $200 $100 $1 = $99 v NR = $1 Expected benefit = p R (x) $99 [1 p R (x)] $1 > 0 p R (x) $99 > [1 p R (x)] $1 p R (x) >

Using Expected Value Framework in Classifier Evaluation Training data

b(y,p) c(y,n) N c(n,p) b(n,n) Predicted( class Y N Actual(class p n

Confusion matrix Normalize to rates Y N p tp%rate% p(y,p) fn%rate%

41 Using Expected Value Framework in Classifier Evaluation Training data Evaluation data Learning algorithm Cost-benefit information p n Y b(y,p) c(y,n) N c(n,p) b(n,n) Predicted( class Y N Actual(class p n True( False( positives positives False( True( negatives negatives Confusion matrix Normalize to rates Y N p tp%rate% p(y,p) fn%rate% p(n,p) n fp%rate% p(y,n) tn%rate% p(n,n) Expected rates (probabilities) Expected Value 41

42 Example: Confusion matrix Actual4class p n Predicted4 Y 56 7 class N 5 42 Total 110 Normalize to rates Estimated probabilities Actual2class p n Predicted2 Y class N p(h, a) = count(h, a)/t p n Y b(y,p) c(y,n) N c(n,p) b(n,n) Cost-benefit information p n Y 99 %1 N 0 0 Expected benefit = p(y, p) b(y, p)+p(n, p) b(n, p)+p(n, n) b(n, n)+p(y, n) b(y, n) 50 Average profit per consumer 42

43 Common pitfalls in cost-benefit approach Be careful with the signs of quantities in cost-benefit matrix Watch out for double counting, which happens if you put a benefit in one cell and a negative cost for the same thing in another cell (one should be set zero!) 43

44 Expected Value Framework in case of unbalanced class distributions Suppose you have two analysts working on the same problem: - Analyst 1: reports model performance statistics over a representative but unbalanced population - Analyst 2: reports statistics for a class-balanced population - How do you compare the results? Solution: Factor out the probabilities of seeing each class (i.e. class priors) à separates the influence of class imbalance from fundamental predictive power of the models 44

45 Expected Values Re-expressed Recall that p(x, y) =p(y) p(x y) Use the rule of basic probability to factor out p and n in expected benefit formula Expected benefit = p(p) [p(y p) b(y, p)+p(n p) c(n, p)] + p(n) [p(n n) b(n, n)+p(y n) c(y, n)] Class priors (how likely it is to see positive and negative instances; show class imbalance) Model performance (confusion matrix) and costbenefit data 45

46 Example Predicted0 class Actual0class p n Y N p(p)0= 0.55 (=61/110) p(n)0= 0.45 (=49/110) p(y p)0= 0.92 (=56/61) p(n p)0= 0.08 (=5/61) p(y n)0= 0.14 (=7/49) p(n n)0= 0.86 (=42/49) Cost-benefit information p n Y 99 %1 N 0 0 p(y p) = tp rate ( specificity ) p(n p) = fn rate p(y n) = fp rate p(n n) = tn rate ( sensitivity ) Expected benefit = p(p) [p(y p) b(y, p)+p(n p) c(n, p)] + p(n) [p(n n) b(n, n)+p(y n) c(y, n)] 50 46

47 Example (cont d) Using the previous framework we can now compare two models even though one analyst tested using representative distribution and the other tested using a class-balanced distribution In calculating expected value, you just have to replace the priors! A balanced distribution corresponds to priors p(p)=p(n)=0.5 The other factors in the equation will not change if the test-set priors change 47

48 Choice of baseline models The signal and the noise What is a reasonable baseline model to compare against? Random baseline: - Often easy to create and easy to beat too! Majority classifier baseline: - A naïve classifier that always chooses the majority class of the training set - Useful for avoiding a lot of mistakes, especially with unbalanced data Models that use very small amount of feature information: - E.g., models that use just one (perhaps most important) variable - Decision stumps (decision trees with only 1 internal node) are a good example of reduced data models 48

49 Ranking Instead of Classifying Many models yield scores that represent the likelihood of a certain outcome (e.g., the probability of each particular customer to respond to a campaign) Instead of expected values, we could rank the cases by their scores and consider the ones above a certain threshold - Useful in many practical scenarios; e.g., in case you have a tight budget for decisions Questions: - How to choose a proper threshold? - How to compare different rankings? 49

50 Thresholding a list of instances Instance True+class Score 1 p p n n p n p p n p Assume+total+of+200+obs TP FP FN TN threshold: 0.96 p n Y 2 1 N threshold: 0.99 p n Y 1 0 N threshold: 0.98 p n Y 2 0 N Each choice of threshold corresponds to a different confusion matrix! 50

51 Profit curves Given a threshold and a confusion matrix, we can compute the expected value (profit) corresponding to the threshold While moving down the list and computing expected profit after each instance, we get a graph known as the profit curve Interpretation of profit curve in the case of targeted marketing: A profit curve shows the expected cumulative profit for a classifier as progressively larger proportions of the consumer base are targeted 51

52 Uses for profit curves: Classifier 1 Choice of threshold: How many people to target? How much profit do you expect to make? Classifier 2 Which model is better? (may in some cases depend on budget) 52

53 Critical conditions for profit calculations Class priors (i.e. proportion of positive and negative instances) in the target population need to be known and they are assumed to be stable Expected profit is really sensitive to the way costs and benefits are specified 53

54 ROC Curves: Hit rate vs. False Alarms In many situations (with uncertain conditions), it is difficult to specify the cost-benefit information needed for profit curves Also priors may change over time (e.g., in fraud detection) Receiver Operating Characteristic (ROC) curves are more tolerant to uncertainties ROC graph = two-dimensional plot with false positive rate (i.e. false alarms) on x-axis and true positive rate (i.e. hit rate) on y-axis 54

55 ROC graph for discrete (non-ranking) classifiers 1.0 (0,1): Perfect prediction (0,0): All n predicted (1,1): All p predicted N-W of ROC space is preferred: Higher TP rate, lower FP rate True positive rate D A B E C Dashed line ~ random baseline False positive rate Source: T. Fawcett / Pattern Recognition Letters 27 (2006)

56 Conservative - predicts positive only with strong evidence D B Liberal predict positive with weak evidence Conservative vs. Liberal classifiers True positive rate A E C False positive rate Worse than random classifier Source: T. Fawcett / Pattern Recognition Letters 27 (2006)

57 ROC curve for ranking classifiers Inst# Class Score Inst# Class Score 1 p.9 11 p.4 2 p.8 12 n.39 3 n.7 13 p.38 4 p.6 14 n.37 5 p n.36 6 p n.35 7 n p.34 8 n n.33 9 p p n n.1 Each point on ROC curve corresponds to different confusion matrix (threshold) True positive rate Infinity False positive rate Source: T. Fawcett / Pattern Recognition Letters 27 (2006)

58 The Area Under the ROC Curve (AUC) Sometimes it is convenient to summarize the information in ROC curve by single statistic AUC = area under a classifier s curve as a fraction of the unit square Equivalent to Gini coefficient and Mann-Whitney-Wilcoxon measure (which represent the probability that a randomly chosen positive instance will be ranked ahead of a randomly chosen negative instance) 58

59 C Convex hull of classifiers 868 T. Fawcett / Pattern Recognition Letters 27 (200 CH 1.0 B A CH B 0.6 True positive rate D, B are sub-optimal 0.6 True positive rate A classifier is potentially optimal, if it sits on the convex hull of points in the ROC space True positive rate C 0.8 D A D True positive rate 0.8 α A (a) (a) positiverate rate FalseFalse positive (b) (b) 00 Fig. 7. (a) The ROC convex hull identifies potentially optimal classifiers. (b) Lines a and b sho Fig. 7. (a) The ROC convex hull identifies potentially optimal classifiers. (b) Lines a a 59

60 Cumulative response curve Gains charts Plots the gain (percentage of positives correctly classified by model; tp rate; hit rate) as a function of the percentage of the population that is targeted (decreasing by score) Diagonal x=y represents a random classifier; any classifier that is above the diagonal is providing some advantage 60

61 Random model Wide gap indicates performance advantage How large part of population is targeted (decreasing by score) 61

62 Lift curve How much better is the model compared to a random classifier? Lift ~ advantage of a classifier over random guessing Represents the degree to which the model pushes up positive instance in a list above the negative instances Lift curve ~ value of cumulative response curve at a given point x divided by the diagonal line (y=x) value at that point 62

63 Random model 63

64 Moral and ethical issues Privacy Increasingly important What kind of data can be used and to what purposes? Confidentiality e.g. non-disclosure agreements Consent Permissions to collect and use the data Informing subjects on the purposes to which the data is used See Kenneth Goodman s paper for further discussion: 64

65 Best practices for mining Data is the foundation for analytics Quality & quantity concerns Bonferroni s Principle Careful model evaluation needed generalization vs. over-fitting Remember the human side of analytics Communication of benefits Model understanding CRISP-DM 65

Introduction to Analytics and Data Science. Pekka Malo, Assist. Prof. (statistics) Aalto BIZ / Department of Information and Service Economy

Introduction to Analytics and Data Science Pekka Malo, Assist. Prof. (statistics) Aalto BIZ / Department of Information and Service Economy 2 Agenda ADS seminar Administrative information Schedule and