Best practices in risk model development

Transcription

1 Best practices in risk model development

2 Introducing: Keith Tanaka Experian Jeff Meli Experian

3 Risk modeling landscape Market conditions Regulatory compliance More and better data Stronger tools 3 Experian

4 Marketing efforts Risk management Model applications Account acquisition Account management Account loss Front-end risk Prescreen risk Fraud Bankruptcy First payment default Front-end risk Behavioral CLI Roll rate Delinquency Transactional fraud Early stage collections Late stage collections Recovery Response Activation Conversion Purchase propensity Profitability Usage Activation / reactivation Balance transfer Cross-sell Attrition/churn Pre-payment Retention 4 Experian

5 Risk modeling Drivers of success Strong project team Preparation Execution excellence Best-in-class On time Seamless 5 Experian

6 Model development process Project inception Data prep Inference Segmentation Model development Documentation Implementation 6 Experian

7 Project inception Assemble project team Project inception Data prep Inference Segmentation Model development Documentation Implementation 7 Experian

8 Project inception Key participants Project sponsor Line of business Risk managers Analysts IT Compliance Project manager Third party consultants Comprehensive Collaborative Committed 8 Experian

9 Project inception Assemble project team Establish clear objectives Evaluate lending environment Define model development parameters Project inception Data prep Inference Segmentation Model development Documentation Implementation 9 Experian

10 Project inception Define model development parameters Sample parameters Target population Sample timeframe Performance flag Performance window Exclusion Data sources and attribute candidates Model design Inference Segmentation Attribute selection Modeling technique Model performance measurement Implementation considerations 10 Experian

11 Project inception Example parameters by model application Prescreen solicitation Account acquisition Credit line authorizations reissue Attrition Collections Tool Prescreen scoring New applicant scoring Behavior scoring Behavior scoring Collection scoring Data sources at observation Credit bureau demographic Application credit bureau customer Master file credit bureau Master file credit bureau demographic Master file credit bureau Performance window 2 3 months months 6 12 months 3 6 months 3 6 months Performance flags Response / nonresponse profitable / unprofitable Good / bad Good / bad Stay / attrite $ collected / no collection 11 Experian

12 Gini Coefficient Gini Coefficient Project inception Model robustness across performance flags 60+ DPD Performance Flag 90+ DPD Performance Flag Auto Bankcard 0 Auto Bankcard Model 60+ DPD Model 90+ DPD Model 60+ DPD Model 90+ DPD 12 Experian

13 Gini Coefficient Gini Coefficient Project inception Model robustness across performance windows 12-Month Performance Window 24-Month Performance Window Auto Bankcard 0 Auto Bankcard Model 12 Months Model 24 Months Model 12 Months Model 24 Months 13 Experian

14 Project inception Define model development parameters Sample parameters Target population Sample timeframe Performance flag Performance window Exclusion Data sources and attribute candidates Model design Inference Segmentation Attribute selection Modeling technique Model performance measurement Implementation considerations 14 Experian

15 Project inception Roadblocks and potential risks Progress impeded Potential risks Lack of representation from key stakeholders Poor stakeholder communication and / or competing objectives Weak project management is counterproductive Don t move forward with design decisions without stakeholder consensus Poorly designed samples may limit model shelf life Consider implementation planning now Best-in-class On time Seamless 15 Experian

16 Data preparation Data extraction Standard data integrity checks and multi-dimensional EDA Portfolio reports Population Stability Index (PSI) Characteristic analysis Attribute treatment Project inception Data prep Inference Segmentation Model development Documentation Implementation 16 Experian

17 % of Distribution Data preparation Population stability Externally Sourced Attribute PSI = Reveals attribute stability concerns Missing 1+ Is marginal added predictive value justified? Development Out of Time 17 Experian

18 Data preparation Characteristic analysis: Revolving balance ($) all trades* Bad Good ,500 2,501-5,000 5,001 + * Originations model Good / bad index 18 Experian

19 Data preparation Characteristic analysis: Revolving balance ($) one account* Bad Good ,000 1,001-2,000 2, Good / bad index * Behavioral line management model for a store credit card 19 Experian

20 Data preparation Data extraction Standard data integrity checks and multi-dimensional EDA Portfolio reports Population Stability Index (PSI) Characteristic analysis Attribute treatment Waterfall statistics Sample selection Project inception Data prep Inference Segmentation Model development Documentation Implementation 20 Experian

21 Data preparation Roadblocks and potential risks Progress impeded Potential risks Lack of specifications for data extraction Insufficient attention given to EDA Lack of sufficient attribute vetting Delaying extraction and preparation of out-of-time validation sample Best-in-class On time Seamless 21 Experian

22 Inference Assess the need for inference Determine appropriate technique Evaluate impact on booked population performance Project inception Data prep Inference Segmentation Model development Documentation Implementation 22 Experian

23 Inference The need for inference To correct for sample bias Example: Approved applications Modeled population Score 640 cut-off Application population Declined applications Without inference on declined applications the scorecard would only be valid for a small proportion of the application universe 23 Experian

24 Inference Inference techniques Re-weighting Based on a score interval, weight-up accounts with known performance to the through the door distribution Reclassification High risk applications, typically based on a score or high risk profile, are assigned a bad performance Parceling Rejected applications are assigned good and bad performance based on a risk score Bureau inference Using bureau data assign good and bad performance based on performance of similar trades with other creditors 24 Experian

25 Gini Coefficient Inference Pre-diction vs. post-diction 100 Outcome Window 90 Observation Point (Pre-diction Attributes) Outcome Point (Post-diction Attributes) Pre-diction and post-diction model development and assessment: Booked trade suppressed from post-diction attributes Models built on booked accounts only 10 Non-booked performance assigned based on performance of a similar trade on the bureau 0 Known Performance Inferred Performance Model applied to non-booked applicants Prediction Postdiction 25 Experian

26 Gini Coefficient Inference Inferred applicant weighting and impact on booked account performance Existing benchmark BG model (100% known) BGI model 60/40 known/inf) BGI model (85/15 known/inf) Model 26 Experian

27 Inference Roadblocks and potential risks Progress impeded Potential risks Gaps in sample design and data preparation Overcoming the inference design Too little inference may increase risk exposure Too much inference compromises performance of traditional book of business Best-in-class On time Seamless 27 Experian

28 Segmentation Determine candidate schemes Build master model Build out niche models for each candidate scheme Compare master vs. niche model performance Assess trade-offs of candidate schemes Project inception Data prep Inference Segmentation Model development Documentation Implementation 28 Experian

29 Gini Coefficient Segmentation Segment evaluation Niche Master Segment 1 Segment 2 Segment 3 Overall 29 Experian

30 Segmentation Roadblocks and potential risks Progress impeded Potential risks Lack of attribute vetting during project initiation Potential segment schemes omitted from consideration Segmentation overkill Failure to review simple statistics to help eliminate non-viable segments Best-in-class On time Seamless 30 Experian

31 Model development Finalize attribute selection and modeling approach Develop preliminary models Conduct stakeholder review, and test alternate attributes Refine and finalize models For each iteration Test model vs. benchmarks Conduct in- and out-of-time validations Project inception Data prep Inference Segmentation Model development Documentation Implementation 31 Experian

32 Model development Roadblocks and potential risks Progress impeded Potential risks Lack of stakeholder sign-off on all prior phases Insufficient stratification of estimation and in-sample validation samples during data preparation Insufficient attribute vetting Science only gets you so far- expert intuition is key Delaying out-of-time validations Best-in-class On time Seamless 32 Experian

33 Documentation Primary focus on compilation rather than construction Compliance and governance Document key decisions alongside the rationale Project inception Data prep Inference Segmentation Model development Documentation Implementation 33 Experian

34 Documentation Roadblocks and potential risks Progress impeded Potential risks Decisions made that were not recorded Insufficient documentation from prior phases Do not delay! Avoid large gaps end-to-end documents Ensure document is tailored to complete audience Best-in-class On time Seamless 34 Experian

35 Implementation Coding / testing / auditing Strategy design Compliance and governance Monitoring Project inception Data prep Inference Segmentation Model development Documentation Implementation 35 Experian

36 Implementation Roadblocks and potential risks Progress impeded Potential risks Unrealistic implementation timeframe Key participants not consulted System limitations not explored Use scorecard for designed purpose Monitor regularly Do not neglect model maintenance Best-in-class On time Seamless 36 Experian

37 Rules of the road Chose the right partners for the journey Use the right fuel Plan your route and your milestones Look ahead and stay focused Maintain a travel log Be prepared for the unexpected Project inception Data prep Inference Segmentation Model development Documentation Implementation 37 Experian 37

38 Risk modeling Drivers of success Strong project team Preparation Execution excellence Best-in-class On time Seamless 38 Experian

39 Questions and answers Experian contact: Jeff Meli 39 Experian

40 Share your thoughts about Vision 2017! Please take the time now to give us your feedback about this session. You can complete the survey at the kiosk outside. How would you rate both the Speaker and Content? 40 Experian

41