Exploiting full potential of predictive analytics on small data to drive business outcomes

Size: px

Start display at page:

Download "Exploiting full potential of predictive analytics on small data to drive business outcomes"

Jade Fisher
5 years ago
Views:

1 Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data Science Expert TM Forum, Nice 15 May 2018

2 HelloFresh breaks the dinner routine by continuously innovating both service and product No Planning No Shopping No Waste 1 Box Delivered Weekly To The Door Perfectly Portioned Ingredients For 3-5 Meals Per Week Personalised Fresh Food, Locally Sourced Easily Managed Via Subscription Platform

3 Disrupting the supply chain by cutting middlemen, ensuring higher margins and fresher products

4 HelloFresh global footprint 5

5 How we use data science / machine learning Fraud detection Deep learning CNNs Recommendation engines Demand forecasting Support Vector Regressions Random Forests ARIMA & other time series models Generalized Additive Models DATA HF Bayesian networks Graph databases Hidden Markov Models Extreme Gradient Boosting Collaborative filtering Lifetime / churn prediction minimize cost Marketing attribution maximize revenue

6 Myself: a drift between consulting and data science Quant methods and computational psychoacoustics Demand forecasting Market research & business intelligence Data Science in strategic consulting Data Science in-house

7 Why bother about small data? Isn t it dead by now? Not all areas of business produce big data Several business functions use / produce both small and big data, e.g. : Demand planning / forecasting Attribution & ROI of marketing activities As a result, top-down and bottom-up models appear Top-down focuses on trends of phenomena, predictors, aggregated values Bottom-up drills down to individual customer / transaction etc. Both model types require validation and could provide it for each other.

8 Demand forecasting challenges in a subscription meal kit business Huge cost impact Short time series High variance of pausing, strong seasonal (holiday) effects New conversions highly dependent on marketing plans Certain predictors difficult to track (e.g. errors in delivery) Large set of numbers to be delivered Splits into box types, recipes, delivery day, delivery time, Tech legacy What can we cook from these ingredients?

9 First decision: forecast top-down or bottom-up? Sales (boxes)** ~ Outlook of actives + Outlook of pauses +.. ` Customer ID Weeks from activati on Weeks from last pause Weeks from last meal swap No. of meal swaps total No. of boxes in total Box type Total Probability of getting a box ** Dummy data in all charts

10 Why do we need a top-down forecasting model? Y Should I stay (or should I go)? Y N Shall I take a break? N Y Do I care to see my options? Y N Do I swap my meals? N Each decision increases variance of final output In a bottom-up model those variances could mitigate each other or could explode Top-down model (aggregate number of boxes) is much more stable CANCEL? PAUSE? TRUST DEFAULT MEAL CHOICE? SWAP MEALS?

Methodological challenges of short time series forecasting Possible lack of time series effects no significant autocorrelation of sales still, burning need to control for trend in data / baseline

11 Methodological challenges of short time series forecasting Possible lack of time series effects no significant autocorrelation of sales still, burning need to control for trend in data / baseline Dynamic business growth introduces sales disruptions Lots of predictors are inter-correlated particularly true in the case when some outlooks (early trends) of activeness, pausing and cancelling are available Forecasting like cooking is a mixture of art and science

12 To our surprise, often times we found lack of time-related effects in forecast target Boxes shipped in one of countries ** ` Autocorrelation of differenced time series Partial autocorrelation of differenced ts ` ` Standard time series models cannot be the only tool we use. ** Dummy data in all charts

13 We needed to introduce dummies for disruptions in sales time series Histogram of weekly sales in country 1 ** Time series of weekly sales in country 2 ** ` ` Mixture of 2 distributions due to major change in marketing spend Price cut effect = shift of entire sales ** Dummy data in all charts

Feature selection is limited by size of input data and requires regularization ` Lots of predictors are highly correlated, even after controlling for trend / baseline

14 Feature selection is limited by size of input data and requires regularization ` Lots of predictors are highly correlated, even after controlling for trend / baseline This calls for regularization in both feature selection process and then in the model We use Lasso in feature selection and cubic shrinkage in Generalized Additive Models `

15 Addressing non-linearity: Generalized Additive Models Introduced by Hastie and Tibsharani in 1990 a step taken from GLM towards non-parametric models Instead of estimating parameters, for each variable GAM estimates a function composed of smoothing elements Future sales ie. no. of boxes to be delivered X weeks from now = b 0 + f 1 (marketing spend) + f 2 (planned pauses to date) + f 3 (planned cancellations to date) + f 4 (holiday effect) + weekday effect + The functions f 1,..., f p can be natural splines, smoothing splines, local regressions, technically even polynomials (not used) Parametric terms and 2D-smoothers are also allowed

Neither standard time series nor average ensemble forecast work Best forecast method selected by progressive

16 Model error based on 16-week progressive cross-validation Model error based on 16-week progressive cross-validation Ensemble approach and root cause analysis Backtest in country 1 Backtest in country 2 ` ` Neither standard time series nor average ensemble forecast work Best forecast method selected by progressive cross validation is better (final.forecast) Frequent review based on backtesting and root-cause analysis is even better

17 Next step: predicting user-level demand with deep learning CNNs Factorization / Word2Vec

18 Marketing attribution problem Marketing spend for a company within multiple years, typically measured daily or weekly Question: how many conversions / how much revenue / how much CLV can be attributed to activity in each channel?

Marketing attribution challenges in most e-commerce businesses Huge cost impact Short time series Varying granularity of input data Very often marketing data stored by multiple

19 Marketing attribution challenges in most e-commerce businesses Huge cost impact Short time series Varying granularity of input data Very often marketing data stored by multiple people in obscure ways (sheets, docs, no standardization) Lack of full attribution models to cross check results Dealing with counter intuitive results Is there a silver / golden bullet?

Here we go again: top-down or bottom-up approach? Number of boxes from newly acquired customers ~ Activity in TV** + Activity in PaidSocial ** +.. ` Customer ID Touch point Paid- Social Touchpt.

20 Here we go again: top-down or bottom-up approach? Number of boxes from newly acquired customers ~ Activity in TV** + Activity in PaidSocial ** +.. ` Customer ID Touch point Paid- Social Touchpt. Affiliates Touchpt. Bloggers Touchpt. Likelihood of outdoor exposure Likelihood of TV exposure Total 9.2. Number of boxes over first year (CLV) ** Dummy data in all charts

21 Balancing business insight and simulation / prediction power Typically, statistics used doesn t align exactly to desired business outcomes There is usually an inverse relationship between how well the model predicts and how interpretable are its components In marketing attribution, forcing intuitive constraints (nonnegative contribution of channels, convex shape of response = saturation etc.) often affect fit and predictive strength Hitting sweet spot requires an iterative process of refining the model against business assumptions and usability / actionability

22 Simulator for marketing attribution & ROI purposes based on a PCA + Bayesian network + GAM model

Conclusions: Do s for small data Combine bottom-up and top-down Experiment if no history available Ensemble your models wisely Back-test and root-cause-analyze Factor in iterations

23 Conclusions: Do s for small data Combine bottom-up and top-down Experiment if no history available Ensemble your models wisely Back-test and root-cause-analyze Factor in iterations with business and make it part of model building Keep calm and explain discrepancies. Forecasting / simulating is the art of saying what will happen and then explaining why it didn t

24 We re hiring at HelloFresh! Data Scientists Python, R, Spark, Scala, ML + computer vision / NLP / other deep learning experience Machine Learning Engineers Python, Hadoop, Spark, Kafka, ML productionizing expertise Data Engineers Python, Hadoop, Spark, Kafka, Airflow, ETL experience

25 Thanks! Any Questions?

ANALYZING CUSTOMER CHURN USING PREDICTIVE ANALYTICS

ANALYZING CUSTOMER CHURN USING PREDICTIVE ANALYTICS POWERED BY ARTIFICIAL INTELLIGENCE JANUARY 2018 Authors - Dr. Cindy Gordon & Sai Krishna Design - Bharat Tahiliani 2017 SalesChoice Inc. 1 OVERVIEW CUSTOMER