My Five Predictive Analytics Pet Peeves

Transcription

1 My Five Predictive Analytics Pet Peeves Dean Abbott Abbott Analytics, Inc. Predictive Analytics World, San Francisco, CA (#pawcon) April 16, Blog: 1

2 Topics Why Pet Peeves? A call for humility for Predictive Modelers The Five Pet Peeves 1. Machine Learning Skills > Domain Expertise 2. Just Build the Most Accurate Model! 3. Significance? What do you mean by Significance? 4. My Algorithm is better than Your Algorithm 5. My classifier calls everything 0 time to resample! 2

3 Peeve 1 Which is Better: Machine Learning Expertise or Domain Expertise? Question: who is more important in the process of building predictive models: The Data Scientist / Predictive Modeler / Data Miner The Domain Expert / Business Stakeholder? Photo from 3

4 Which is Better: 2012 Strata Conference Debate? From Strata Conference: machine-learning-expertise-googleanalytics.html I think you can get pretty far with some common sense, maybe Google-ing the basic information you need to know about a domain, and a lot of statistical intuition 4

5 Formula for Success? 5

6 Conclusion: Frame the Problem First Mark Driscoll: Moderator of Strata Debate could you currently prepare your data for a Kaggle competition? If so, then hire a machine learner. If not, hire a data scientist who has the domain expertise and the data hacking skills to get you there. But even this may not work, which brings me to the second pet peeve 6

7 Peeve 2 Just Build Accurate Models The Problems with Model Accuracy: 1. There s More to Success than Accuracy 2. Which Accuracy? 7

8 The Winner is Best Accuracy 8

9 Why Model Accuracy is Not Enough: Netflix Prize netflix-recommendations-beyond-5-stars.html 9

10 Why Data Science is Not Enough: Netflix Prize netflix-recommendations-beyond-5-stars.html There s more to a solution than accuracy you have to be able to use it! 10

11 Peeve 3 The Best Model Wins We select the winning model, but is there a significant difference in model performance? 12

12 KDD Cup 98 Results Calculator from 13

13 Example: Statistical Significance without Practical Significance Measure Control Campaign (based on model) Number Mailed 5,000,000 4,000,000 Response Rate 1% 1.011% outside margin of error? yes i.e., statisticall significant? yes expected responders 50,000 40,000 actual responders 50,000 40,440 difference Revenue Per Responder $100 Total Revenue Expected $4,000,000 Total Revenue Actual $4,044,000 Difference Revenue $44,000 Significance based on z=2 (95.45% confidence) Cost per contact: negligible ( ) Cost for analysts to build model: $80,000 14

14 Peeve 4 My Algorithm is Better than Your Algorithm From 2011 Rexer Analytics Data Mining Survey Data-Miner-Survey- Results-2011.html 15

15 Every Algorithm Has its Day Elder, IV, J. F., and Lee, S. S. (1997), Bundling Heterogeneous Classi ers with Advisor Perceptrons, Technical Report, University of Idaho, October,

16 PAKDD Cup 2007 Results: Look at all them Algorithms! 18 Different Algorithms Used in Top 20 Solutions; Modeling) Par6cipant) Par6cipant) AUCROC) (Trapezoi AUCROC) (Trapezoidal) Top)Decile) Top)Decile) Response) Modeling)Technique)/>) Implementa6on)/>) Affilia6on)Loca6on) Affilia6on)Type) dal)rule)) Rule))Rank)) Response)Rat) Rate)Rank) TreeNet&+&Logis-c&Regression& Salford&Systems& Mainland&China& Prac--oner& 70.01%& 1& 13.00%& 7& Probit&Regression& SAS& USA& Prac--oner& 69.99%& 2& 13.13%& 6& MLP&+&nHTuple&Classifier& Brazil& Prac--oner& 69.62%& 3& 13.88%& 1& TreeNet& Salford&Systems& USA& Prac--oner& 69.61%& 4& 13.25%& 4& TreeNet& Salford&Systems& Mainland&China& Prac--oner& 69.42%& 5& 13.50%& 2& Ridge&Regression& Rank& Belgium& Prac--oner& 69.28%& 6& 12.88%& 9& 2HLayer&Linear&Regression& USA& Prac--oner& 69.14%& 7& 12.88%& 9& Log&Regr+&Decision&Stump&+&AdaBoost&+&VFI& Mainland&China& Academia& 69.10%& 8& 13.25%& 4& Logis-c&Average&of&Single&Decision&Func-ons& Australia& Prac--oner& 68.85%& 9& 12.13%& 17& Logis-c&Regression& Weka& Singapore& Academia& 68.69%& 10& 12.38%& 16& Logis-c&Regression& Mainland&China& Prac--oner& 68.58%& 11& 12.88%& 9& Decision&Tree&+&Neural&Network&+&Log.Regression& Singapore& 68.54%& 12& 13.00%& 7& Scorecard&Linear&Addi-ve&Model& Xeno& USA& Prac--oner& 68.28%& 13& 11.75%& 20& Random&Forest& Weka& USA& 68.04%& 14& 12.50%& 14& Expanding&Regression&Tree&+&RankBoost&+&Bagging& Weka& Mainland&China& Academia& 68.02%& 15& 12.50%& 14& Logis-c&Regression& SAS&+&Salford& India& Prac--oner& 67.58%& 16& 12.00%& 19& J48&+&BayesNet& Weka& Mainland&China& Academia& 67.56%& 17& 11.63%& 21& Neural&Network&+&General&Addi-ve&Model& Tiberius& USA& Prac--oner& 67.54%& 18& 11.63%& 21& Decision&Tree&+&Neural&Network& Mainland&China& Academia& 67.50%& 19& 12.88%& 9& Decision&Tree&+&Neural&Network&+&Log.&Regression& SAS& USA& Academia& 66.71%& 20& 13.50%& 2& 17

17 Peeve 5 You Must Stratify Data to Balance the Target Class For example, 93% non-responders (N), 7% responders (R) What s the Problem? (The justification for resampling) Sample is biased toward responders Models will learn non-responders better Most algorithms will generate models that say call everything a non-responder and get 93% correct classification! (I used to say this too) Most common solution: Stratify the sample to get 50%/50% (some will argue that one only needs 20-30% responders) 18

18 Neural Network Results on Same Data Distribution of Target NOTE: all models built using JMP 10, SAS Institute, Inc. 19

19 Sample Decision Tree Built on Imbalanced Population Distribution of Target But.ROC Curve Looks like this All Rows Count G^2 LogWorth AVG_DON>=12.6 AVG_DON<12.6 Count G^2 LogWorth Count G^2 LogWorth REC_DON_AMT>=22 REC_DON_AMT<22 REC_DON_AMT>=15 REC_DON_AMT<15 Count G^2 LogWorth Count G^2 LogWorth Count G^2 LogWorth Count G^2 LogWorth RFA_2(L3F, L2F, L3G) RFA_2(L4G, L2G, L1F, L1G, L1E, L2E, CARDPM12>=8 CARDPM12<8 MAX_DON_AMT>=21 MAX_DON_AMT<21 MAX_DON_AMT>=20 MAX_DON_AMT<20 L4F) Count G^2 Count G^2 Count G^2 Count G^2 Count G^2 Count G^2 LogWorth Count G^ Count G^2 LogWorth CARDGIFT_LIFE<4 CARDGIFT_LIFE>=4 MAX_DON_DT<9110 MAX_DON_DT>=9110 Count G^2 Count G^2 Count G^2 Count G^ Predictions of Target Variable Sensitivity Specificity Why do we get a ROC Curve that looks OK, but the confusion matrix says everything is N (No)? 20

20 So What Happened? Note: no algorithm predicts decisions (N or R): they all produce probabilities/likelihoods/confidences Every data mining tool creates decisions (and by extension, forms confusion matrices) by thresholding the predicted probability by 0.5 (i.e., assuming equal likelihoods is the baseline) When the imbalance is large, algorithms will not produce probs/likelihoods > 0.5 a score this large is far too unlikely for an algorithm to be that sure 21

21 What the Predictions Looks Like 22

22 Confusion Matrices For the Decision Tree: Before and After Decision Tree: Threshold at 0.5 Decision Tree: Threshold at Response_ STR) N) R) Total) N& 5,002& 0& 5,002& R& 386& 0& 386& Total& 5,388& 0& 5,388& Response_ STR) N) R) Total) N& 2,798& 2,204& 5,002& R& 45& 341& 386& Total& 2,843& 2,545& 5,388& 24

23 Conclusions The Rant is Done! The Five Pet Peeves 1. Machine Learning Skills > Domain Expertise Be humble; we need both data science and domain experts! 2. Just Build the Most Accurate Model! Select the model that addresses your metric 3. Significance? What do you mean by Significance? Don t get hung up on best when many models will do well Learn from difference in patterns found by these models 4. My Algorithm is better than Your Algorithm Don t stress about the algorithm; learn to use a few very well 5. My classifier calls everything 0 time to resample! Don t throw away 0s needlessly; only do it when there are enough of them that you won t miss them. 25