How to do practical Data Science?

Size: px

Start display at page:

Download "How to do practical Data Science?"

Isaac Wilkins
5 years ago
Views:

1 How to do practical Data Science? Artur Suchwałko, QuantUp code::dive Conference 2018, , Wroclaw

2 Introduction

3 Introduction Several real-world Data Science projects + business results, methods, and findings: prediction of fraudulent insurance claims for an insurance company (with Machine Learning, generic) customer churn prediction for a bank (with Machine Learning, generic) prediction of pre-paid customers registration for a telecom (with Deep Learning, specific but important) Wrapping up and structuring these conclusions + adding recommendations regarding executing Data Sciences projects. (Traditional (and boring) businesses where AI/ML can give tons of money.)

4 Data frame Duration Amount Age Gender Car loan? Installment loan? Target (example: German credit dataset, 1994)

5 Detection of insurance frauds (ML)

6 Business case There was a car crash An insurance claim is reported It could be a fraud attempt We need to decide in a limited time: accept or decline the claim We have only some information Usually the claims are analysed by people

7 Goal & results Client: Insurance company Goal: Improving of detection of probable frauds for further investigation Probable / doubtful / suspicious claim: suspected to be a fraud but not proven to be one Finding and proving are two different things Result: Savings of order of 30% (comparing to past simple models)

8 Important business questions How to choose claims for investigation to: detect highest number of fraud attempts? detect highest amount of fraudulent claims? detect highest amount with limited resources and time for detection? be able to prove highest number / amount of fraud attempts? This requirement is translated into a suitable goal function for a model and should affect the optimization criterion.

9 How to build a model? Preparation of the predictors (can be complex because of aggregation of data from many sources) Having the target variable in the historical data Build a predictive model

10 Important Checking if modeling is possible (the process influences the historical data) Definition of new predictors Detection of false predictors Data enhancement: historical aggregates, textual, external Technicalities Using PCs

11 Claim case 1: Rules / human Description: A driver hit the rear side of a victim s car. The car was pushed to the crossroads area and there was a collision with a third car (Mercedes). The police was called. Rules: airbags inflated similar age of both drivers difference of cars age >=11 years historical loss coefficient >=5 Result: Not refused to pay because of fraud attempt: the description was consistent with the damages

12 Claim case 2: Model Description: I (victim) was driving a left lane. The second driver (a culprit) was driving a right lane (the same direction). He wanted to change the lane, haven t seen my car and hit my car. Its rear left side damaged my car s right front side. Analysis: no clear evidence only one year of cars age difference no age information for the second driver insurance policy was not new no claim history for drivers and cars Result: Refused to pay because of fraud attempt: no correlation between description and damages not possible to be a real claim

13 Example: Sample 119 top scored claims given for experts to verify if they are frauds 82 were confirmed as doubtful nearly 70% percent of positive verification

14 Pure analytics vs. business AUC: boosted trees: 0.85, log. regr.: 0.77 Figure 1: An example of simple / complex model difference in business / analytical terms

15 Inside Boosted trees with non-standard goal function Reduction of number of predictors

16 Findings & conclusions Really complex process of handling claims and detecting doubtful ones Uncertainty if the prediction is possible (sample representativeness) Great results An excellent ROI: 2 months including cost of third party software

17 Churn prediction (ML)

18 Business case Retaining existing customers is cheaper and easier than acquiring new ones Being able to predict churn means being able to prevent or reduce it It is predictable

19 Goal & results Client: Bank Goal: The best churn prediction for accounts (stated as) Result: Improvement of 10% (comparing to simpler models) Improving model actionability

20 Important The best churn prediction Reduction of churn? Activities Target Finding the optimal strategy vs. prediction of behavior assuming today s strategy (as in bad debt collection models) Explaining reasons of churn strategy Contract (termination) / no contract (reduced activity) Target definition Definition of predictors Money fransfers (+ descriptions) Stopping salary transfering: almost false predictor Avoid aggregates (especially long-term) Actionability Effect: prediction effectiveness (known) + action effectiveness (unknown) They are not splittable Theoretical effectiveness vs. actionability (improving models in a lab doesn t help)

21 Inside Boosted trees with suitable modifications and optimization Careful hyperparameter optimization Laptops, desktops

22 Findings & conclusions Definition of predictors and target is really hard if you want to have a good and actionable model We destroyed predictive power of a model by using correct predictors but this improved actionability

23 Prediction of pre-paid customers registration (DL)

24 Business case Pre-paid customers of a telecom Anonymous SIM card owners: no registration obligation Anti-terrorism law regulation (heavy historical usage for some of SIM cards!) Obligatory registration before Otherwise loosing phone number

25 Goal & results Client: Telecom Goal: Prediction of number of pre-paid clients who will register themselves after the registration obligation is implemented (February, 2017): Result: knowing budget for registration prediction of traffic in points of sales prediction of cash flow from pre-paids after the change Correct forecast (error of 3%) Well planned activities and savings Hard to give exact amounts or percentages

26 Important Registration obligation implemented for the first time: no historical data Proxy variable for registration: how to use the historical data? Just one number an expert method? Desktop PCs + GPUs

27 Inside I We tried a whole bunch of methods Just guess the percentage (not that stupid as it seems!) Vintage analysis (cohorts of clients) One-class classification (geometrical) One-class classification (SVM) Density estimation Anomaly detection methods (distance based) Deep Learning / representation learning (recurrent neural network) Deep learning turned out to be the best method

28 Inside II Assumptions possible to be validated after implementation of the registration obligation: How to define an active user? What does it mean that users are similar? Who does need to use a phone? Does does using need imply a registration? What is a threshold probability? A proxy measure for registration: making a pre-payment! 400 features x millions of clients

29 Results

30 Findings & conclusions Surprisingly good forecast There was always a magical parameter to set or a missing link Instead of guessing the parameter value we could directly guess the registration percentage because the parameter was directly tied to it in a complex and unknown way Guessing or not guessing the percentage is a matter of our comfort It doesn t influence forecasting quality Using wrong method could result in a really big error It was crucial to find a righ proxy variable

31 Recommendations

32 Data-driven decisions It is crucial to take data-driven decisions, regardless we use a model or not Business people often tell that: using AI requires only putting any data inside improving business decisions by models is impossible Don t trust both these groups!

33 Setting business goal + translating business into analytics Clearly state and work on the business goal Translate it really carefully into analytics not loosing much (usually you loose something) Usually custom goal function helps us a lot

Process & data The present process decides if the modeling is doable or not Is it possible to build a model if all the cases are currently analysed by people?

34 Process & data The present process decides if the modeling is doable or not Is it possible to build a model if all the cases are currently analysed by people? Prepare a big set of potential predictors Need to understand both, process and data Carefully check the data quality Beware of false predictors (leaks from future)

35 Causing and measuring business influence Ensure that the model can be and will be used for decision making Actionability What are possible actions? Example: churn predictors Time to action for churn: not too soon Clear success criteria (not the best model possible!) Compare to no model

36 Project execution Resources people + skills (not just using the tools!) time (!) software hardware Management (CRISP?) Process of modeling Project risk management

37 Modeling tools Adequately complex What is an improvement by a better model? What are risks and costs associated? How to implement the solution? Use open source

38 There are no fully automatic tools! When you have the business goal translated into analytics then that s easy Mistakes in analytics itself (overfit, suboptimal model) are relatively easy to avoid The most painful mistakes are besides of it The automatic solutions simplify the analytics only

39 Implementation & keeping it working Planned implementation + planned tests / checks Cold running (without taking actions) Running on samples Initial validation + ongoing monitoring

40 Summary

41 Summary I (Deep Learning) Figure 3:

42 Summary II (practical modeling) Figure 4:

43 Contact

44 QuantUp We improve data-driven business decisions applying Data Science, Artificial Intelligence / Machine Learning We serve the following industries: banking, insurance, collection agencies, telecoms, retail, bio/med.

45 QuantUp We deliver end-to-end ML solutions Problem analysis, Studies on model impact on business processes, Solution definition & development, Maintenance services, Support in continuous model development, if required, Trainings & knowledge sharing

46 Contact Artur Suchwałko, Ph.D., CEO e: artur [at] quantup [dot] pl m: w: quantup.pl

The usage of Big Data mechanisms and Artificial Intelligence Methods in modern Omnichannel marketing and sales

The usage of Big Data mechanisms and Artificial Intelligence Methods in modern Omnichannel marketing and sales Today's IT service providers offer a large set of tools supporting sales and marketing activities