The Power and Peril! of Predictive Analytics

Transcription

1 The Power and Peril! of Predictive Analytics John F. Elder, 300 West Main Street, Suite 301 Charlo5esville, Virginia

2 Outline Exponentially-growing power of the computer will it eventually think? Harness its power with Predictive Analytics trend examples with visualization Analytics Success Stories Fight fraud, judge disability, discover a new drug Features of successful projects The difficulty of thinking rationally How to succeed with our flawed but vital assistant - the computer 2

3

4

5

6

7 Of course machines can think. After all, humans are just machines made of meat. - MIT CS professor My view: Human and computer strengths are more complementary than alike

8 Computer can't: Create Contextualize Common sense Comprehend Care

9 What is Predictive Analytics? Discovering patterns in known data! that are useful for new data. The computer finds the model structure! not just its parameters. Requires business knowledge, sophisticated statistics, and powerful computing. -> Studies the past to generate new ideas. 9

10 Data Mining and the Hype Cycle Source: Gartner Group 10

11 Data -> Model -> Scores for new Data Data Table Model Build Model Code ID Outc ome Input1 Input 2 1 N 1.2 h 0 2 Y 7.5 m 1 3 Y 6.2 h 0 4 N 0.1 l 0 Input 3 Regression Decision Tree Neural Net Y_hat = 3*x1 + sqrt(x7) /x32 New Real Cases ID Input1 Input2 Input m l h l 0 Organizational Constraints Implementation Model Decisions Overlays Schedules IT requirements Bus. framework If (v AND Y_hat >0.5) then action = work; ID Y_hat Action Work No work Work No work

12 4 Series Pairs: (X,Y1) (X,Y2) (X,Y3) (X4,Y4) X Y 1 Y 2 Y 3 X 4 Y r xy = 0.85 y LS = x MSE = 1.25 R 2 =

13 Anscomb s Quartet (1973, American Statistician) Y Y X X Y 3 8 Y

14 Case Studies Tables, Text, Links 3 Major ways Predictive Analytics helps: 1) Eliminate the bad: Capital One, IRS, HP, FINRA 2) Discover the good: Westwind Foundation, Pfizer 3) Streamline & Automate: Anheuser-Busch, Lumidigm, Peregrine, SSA 14

15 Case 1) IRS Fraud Detection No Action No false alarm (b) No Score return Score exceeds threshold? Yes Classify Examined? Adjustment? Yes No A non-compliant return here is a miss or false dismissal (c) No Action Prediction Truth Not Flagged Flagged Refund due to taxpayer No Adjustment Positive? Yes Compliant a b Non- Compliant c d hit (d) Additional tax due from taxpayer detection rate = d/(c+d) workload = b+d hit:scan = 1 : (b+d)/d 15

16 Case 2) HP - Service Fraud Detection Tips indicated fraud exists Goal: Learn from known cases to find unknown Automate current process, build model on known, score data, investigate top Recovered $11M in 9 months; $67M in 5 years Awards + promotions + growth 16

17 Case 3) Pharmacia & Upjohn (Pfizer)! New Compound an Effective Treatment? Placebo Drug Density surfaces enclose ascending quartiles of data 17

18 Case 4) Breakout Fraud! found via Network (Link) Analysis Networks can reveal critical relationships (links),! or key agents (nodes) Multiple types of relationships can be visualized in networks Predictive Link Analysis to model network to find fraud rings Birds of a feather flock together :! fraud is essentially contagious among connected individuals 18

19 Gain Expected: either: Leverageable - an incremental improvement matters Low-hanging fruit - nobody s (dared) attack problem Interdisciplinary Team: experts needed in! business area, statistics, algorithms, databases Data Vigilance: capture and maintain the accumulating information stream; feedback results Time: learning occurs over multiple cycles Business Champion is essential! Lessons Learned:! Necessary Ingredients for Analytic Project Success then Analytics can add extraordinary value 19

20

21

22 Model of 2 Systems: System 1 fast, intuitive, unconscious System 2 slow, analytical, effortful!! The analytical system is hard work,! so we much prefer the intuitive system! (especially when we re happy)! but it is easily fooled 22

23 Anchoring

24 Halo Effect 4% helped 88% helped

25 from Mindset of an Analyst - Elisabeth Fosslien

26 To succeed with Analytics, we need: Data (knowledge of the past and its results) Variety of skills Business knowledge, statistics, computers people who want to work in teams Performance orientation identify true measures of quality for stakeholders Foster a culture that tolerates then destroys mistakes Mutual respect and humility people will do amazing things when they re valued -> Close the loop! tell people & computer and what they re getting right and wrong 26

27 John F. Elder IV Founder & CEO, Elder Research, Inc. Dr. John Elder heads the USA s largest and most experienced data mining consulting team. Founded in 1995, Elder Research, Inc. has offices in Charlottesville, Virginia, Washington DC, and Baltimore Maryland ( ERI focuses on Federal, commercial, and investment applications of advanced analytics, including text mining, credit scoring, process optimization, cross-selling, drug efficacy, market timing, and fraud detection. John earned Electrical Engineering degrees from Rice University, and a PhD in Systems Engineering from the University of Virginia, where he s an adjunct professor teaching Optimization or Data Mining. Prior to 19 years at ERI, he spent 5 years in aerospace defense consulting, 4 heading research at an investment management firm, and 2 in Rice's Computational & Applied Mathematics department. Dr. Elder has authored innovative data mining tools, is a frequent keynote speaker, and has chaired International Analytics conferences. John was honored to serve for 5 years on a panel appointed by President Bush to guide technology for National Security. His book with Bob Nisbet and Gary Miner, Handbook of Statistical Analysis & Data Mining Applications, won the PROSE award for top book in Mathematics for His book with Giovanni Seni, Ensemble Methods in Data Mining, was published in 2010, and his book with colleague Andrew Fast and 4 others on Practical Text Mining won the 2012 PROSE award for Computer Science. John is grateful to be a follower of Christ and father of 5. 27