Predictive analytics, forecasting and marketing Advances in promotional modelling and demand forecasting. Nikolaos Kourentzes

Transcription

1 Predictive analytics, forecasting and marketing Advances in promotional modelling and demand forecasting Nikolaos Kourentzes Lancaster University Stockholm School of Economics 16/11/2016

2 About the speaker Associate Professor at Dept. Management Science Lancaster University Management School, UK Lancaster Centre for Forecasting Editorial Board of the International Journal of Forecasting, the leading academic journal in the field of forecasting Research interests and consulting experience in various fields of forecasting, including: Business Forecasting and Demand Planning Promotional Modelling Supply Chain Forecasting and Bullwip effect Long experience in applied research projects with industry in various sectors, including: retailing, FMCG manufacturers, pharmaceuticals, media among others. Research blog: 2/75

3 Agenda 1. Business forecasting and uncertainty 2. Forecasting challenges from a modelling view 3. Innovations: 3a. Temporal Hierarchies 3b. Promotional Modelling 3c. Using online search information 3/75

4 Forecasting & decision making Decision making in organisations has at its core an element of forecasting Accurate forecasts lead to reduced uncertainty better decisions Forecasts maybe implicit or explicit Forecasts aims to provide information about the future, conditional on historical and current knowledge Company targets and plans aim to provide direction towards a desirable future. Forecast Target Present Forecast Difference between targets and forecasts, at different horizons, provide useful feedback 4/75

5 Forecasting & decision making Forecasts match targets Great! Do they also match organisational strategy and objectives? Forecast Future` Target Forecasts to do match targets Revise plans/target to reflect probable (forecasted) outcomes. Act to invalidate forecasts change market to make your target the most probably outcome Our forecasts will always be wrong: we took actions since we produced them! Market 5/75

6 Forecasting & decision making Forecasts are central in decisions relating to: Inventory management Promotional and marketing activities Logistics Human resource planning Purchasing and procurement Cash flow management Building new production/storage unit Entering new markets Accurate forecasts can support - Decision making - Identifying and capitalising on opportunities - Cost saving 6/75

7 Forecasting & uncertainty Statistical forecasts are (often) able to provide a point forecast and a degree of uncertainty, reflected in the prediction intervals Probability that true value will lie within bounds. Additional useful information will lead to tighter prediction intervals Less uncertainty about the future 7/75

8 Forecasting & uncertainty Therefore, it is crucial to identify and capture useful information to introduce in our forecasts. Such information may be: Past univariate information (trend, season, ) Past exogenous information (events, price, competition, ) Future exogenous information ( ) The nature of this information does not have to be well-structured or even quantitative (initially). Often it comprises of judgemental forecasts, though targets should be kept separate. Companies often have a wealth of available information, but do not use it effectively (or at all!). Advances in forecasting allow us to take steps in using it reducing uncertainty about the future taking better decisions. 8/75

9 Forecasting & uncertainty Let us assume we improved our forecast accuracy by 20%, so what? Consider two products, one with low variability and one with high and some statistical forecasts Let s calculate how much stock we should have 9/75

10 Forecasting & uncertainty For this example lets us always keep 1 week of stock Stock = Forecast + Days of Supply Great! Do I really need all that stock? I always sell about 300. High service level More variability, I probably need all this stock 10/75

11 Forecasting & uncertainty Can we do any better? Given a statistical forecast we can estimate the uncertainty of its predictions this is essentially how accurate are our forecasts! This uncertainty is invaluable for decision making. The point forecast will most probably be wrong, but the prediction intervals tell us within which values the future values will probably be. With this we can calculate the inventory 11/75

12 Forecasting & uncertainty The correct safety stock is associated with the forecast error (accuracy) Now the stock levels reflect how accurate are our forecasts and allow us to achieve the desired service level Stock = Forecast + Service Level * Uncertainty Stock automatically adjusts to forecasting accuracy/uncertainty The previous topics focused on increasing accuracy and automation Impact on inventory! Impact on financial position! 12/75

13 Agenda 1. Business forecasting and uncertainty 2. Forecasting challenges from a modelling view 3. Innovations: 3a. Temporal Hierarchies 3b. Promotional Modelling 3c. Using online search information 13/75

14 How do we build & models now? This is by no means a resolved question, but there are some reliable approaches Take the example of exponential smoothing family: Considered one of the most reliable and robust methods for automatic univariate forecasting. It is a family of methods: ETS (error type, trend type, seasonality type) Error: Additive or Multiplicative Trend: None or Additive or Multiplicative, Linear or Damped/Exponential Seasonality: None or Additive or Multiplicative Adequate for a most types of time series. Within the state space framework we can select and fit model parameters automatically and reliably. 14/75

15 How do we build & models now? None Additive Multiplicative None Additive Additive Damped Multiplicative Multiplicative Damped Trend Seasonality t h t t t t L F F a aa L ) (1 1 1 t s t t t L a S A L s t t t t S L A S ) (1 ) ( k s t t k t S L F 1 1 ) (1 ) ( t t t t T L L T ( 1 ) 1 t t t t T L A S s t S ) 1 ( h s t t h i i t h t S T L F 1 ) )( 1 ( 1 1 t t s t t t T L a S A a L Use likelihood to find smoothing and initial values. Use some information criterion (AICc, BIC, etc) to choose appropriate model per series. 15/75

16 Any issues with current practice? Issues with automatic modelling: Model selection How good is the best fit model? How reliable? Sampling uncertainty Identified model/parameters stable as new data appear? Model uncertainty Appropriate model structure and parameters? Transparency/Trust Practitioners do not trust systems that change substantially ETS(M,A d,a) - AIC: ETS(A,A d,m) - AIC: Months /75

17 Any issues with current practice? What can go wrong in parameter and model selection: Business time series are often short Limited data Estimation of parameters can fail miserably (for monthly data optimise up to 18 parameters, with often no more than 36 observations) Model selection can fail as well (30 models over-fitting?) Both optimisation and model selection are myopic Focus on data fitting in the past, rather than forecastability Special cases: Sales Demand Fit Forecast Month True model: Additive trend, additive seasonality Identified model: No trend, additive seasonality Why? In-sample variance explained mostly by seasonality 17/75

18 Characteristics of a good solution Provide accurate forecasts (i.e. narrow prediction intervals). Being consistent: the nature of the forecast should not change, unless there are substantial changes in the market increase trust to users. Be relatively transparent increase trust to the users and insights in the market. Be automatic: it is common to have to deal with several thousand of time series cannot model in detail each one separately. Underlying specification of the forecasting model should match the desired business objective. 18/75

19 Agenda 1. Business forecasting and uncertainty 2. Forecasting challenges from a modelling view 3. Innovations: 3a. Temporal Hierarchies 3b. Promotional Modelling 3c. Using online search information 19/75

20 Forecasting model & objective Decisions need to be aligned: Operational short-term decisions Tactical medium-term decisions Strategic long-term decisions Shorter term plans are bottom-up and based mainly on statistical forecasts & expert adjustments. Longer term plans are top-down and based mainly on managerial expertise factoring in unstructured information and organisational environment. Given different sources of information (and views) forecasts will differ plans and decisions not aligned. Objective: construct a framework to reconcile forecasts of different levels and eventually align decisions less waste & costs, agility to take advantage of opportunities. 20/75

21 Long term forecasting We know that different forecasting models are better for different forecast horizons We also know that it helps to forecast long horizons using aggregate data Forecasting a quarter ahead using daily data is `adventurous (90 steps ahead) Forecasting a quarter ahead using quarterly data is easier (1 step ahead) At different data frequencies different components of the series dominate ETS(M,A d,a) - AIC: x ETS(A,A,N) - AIC: d Months Years These forecasts often do not agree, which one is `correct? 21/75

22 A different take on modelling: temporal tricks! Traditionally we model time series at the frequency that we sampled them or take decisions. However, a time series can be view in many different ways, adapting the notion of product hierarchies to temporal hierarchies: The advantages of temporal hierarchies can be highlighted by examining the data at both time and frequency domains. 22/75

23 How temporal aggregation changes the series As an example let us use the following time series: 23/75

24 How temporal aggregation changes the series Seasonal diagrams 24/75

25 The Idea Temporal aggregation strengthens and attenuates different elements of the series: at an aggregate level trend/cycle is easy to distinguish at a disaggregate level high frequency elements like seasonality typically dominate. Modelling a time series at a very disaggregate level (e.g. weekly) shortterm forecast. The opposite is true for aggregate levels (e.g. annual) Propose Temporal Hierarchies that provide a framework to optimally combine information from various levels (irrespective of forecasting method) to: reconcile forecasts avoid over-reliance on a single planning level avoid over-reliance on a single forecasting method/model 25/75

26 Temporal aggregation and forecasting It is not new, but the question has been at which single level to model the time series. Econometrics have investigate the question for decades inconclusive Supply chain applications: ADIDA beneficial to slow and fast moving items forecast accuracy (like everything not always!): Step 1: Temporally aggregate time series to the appropriate level Step 2: Forecast Step 3: Disaggregate forecast and use Selection of aggregation level No theoretical grounding for general case, but good understanding for AR(1)/MA(1)/ARMA(1,1) cases. 26/75

27 Temporal aggregation and forecasting Recently there has been a resurgence in using temporal aggregation for forecasting. Non-overlapping temporal aggregation is a moving average filter. Filters high frequency components: noise, seasonality, etc. Reduces intermittency. But can increase complexity: Loss of estimation efficiency Complicates dynamics of underlying (ARIMA) process Identifiable process converge to IMA - often IMA(1,1) What is the best temporal aggregation to work on? Traditionally try to identify a single best level. 27/75

28 Multiple temporal aggregation What if we do not select an aggregation level? use multiple 200 Aggregation level 1 ETS(A,N,A) 200 Aggregation level 3 ETS(A,M,A) Demand Demand Period Aggregation level 7 ETS(A,M,N) Period Aggregation level 12 ETS(A,A,N) Issues: Different model Different length Combination Demand Demand Period Period 28/75

29 Forecast combination: Issue: Multiple temporal aggregation Forecast combination is widely considered as beneficial for forecast accuracy Simple combination methods (average, median) considered robust, relatively accurate to more complex methods If there are different model types to be combined then the resulting forecast does not fit well at any component! Demand Demand Period Period 29/75

30 y t [1] y t [2] y t [3] Aggregate Fit state space ETS Save states Level y t [10] y t [11] y t [12] Trend Season /75

31 Transform states to additive and to original sampling frequency Combine states (components) Produce forecasts 31/75

32 Multiple Aggregation Prediction Algorithm (MAPA) Step 1: Aggregation Step 2: Forecasting Step 3: Combination 1 Y ETS Model Selection l 1 1 b 1 s + 1 Yˆ k 2 k 3... k K 2 Y 3 Y... Y K ETS Model Selection ETS Model Selection... ETS Model Selection l 2 b 2 s l 3 b s... l b s K K K 1 K 1 K 1 K l b s Strengthens and attenuates components Estimation of parameters at multiple levels Robustness on model selection and parameterisation 32/75

33 Multiple Aggregation Prediction Algorithm (MAPA) 33/75

34 Multiple Aggregation Prediction Algorithm (MAPA) MAPA was developed to take advantage of temporal aggregation and hierarchies: MAPA provides a framework to better identify and estimate the different time series components better forecasts On average outperforms ETS, one of the most widely used, robust and accurate univariate forecasting models in supply chain forecasting It provides reconciled forecasts across planning levels and forecast horizons Robust against model selection and parameterisation issues Shown to be useful for fast moving items, promotional modelling and intermittent time series forecasting. MAPA is available for R, in the MAPA package: Its intermittent demand counterpart is available in the tsintermittent package: Examples and interactive demos for both are available at my blog: 34/75

35 Temporal Hierarchies: A modelling framework MAPA demonstrated the strength of the approach, but it is not general: How to incorporate forecasts from any model/method? How to incorporate judgement? We can introduce a general framework for temporal hierarchies that borrows many elements from cross-sectional hierarchies Eventually we will get to cross-temporal hierarchies, also touted as the onenumber forecast, i.e. a reconciled forecast across planning horizons and product/customer/location groups. 35/75

36 Cross-sectional and Temporal Hierarchies We know how to do cross-sectional hierarchies Top-down, bottom-up, middle-out Optimal combinations Total UK Spain Product A Product B Product A Product B We know how to do this! then we know how to do this as well, with some small-print! but we have to correct for the different scales at each aggregation level, which is not that difficult due to the imposed temporal structure. For that we need to calculate the covariance matrix between the forecast errors at different aggregation levels. There are easy and not so easy ways to do this. 36/75

37 Some evidence that it actually works! Comparison with other M3 results (symmetric Mean Absolute Percentage Error): Monthly dataset Temporal (ETS based): 13.61% ETS: 14.45% [Hyndman et al., 2002] MAPA: 13.69% [Kourentzes et al., 2014] Theta: 13.85% (best original performance) [Makridakis & Hibon, 2000] Quarterly dataset Temporal (ETS based): 9.70% ETS: 9.94% [Hyndman et al., 2002] MAPA: 9.58% [Kourentzes et al., 2014] Theta: 8.96% (best original performance) [Makridakis & Hibon, 2000] Detailed results available if you are interested at the end of the presentation! 37/75

38 Application: Predicting A&E admissions Collect weekly data for UK A&E wards. 13 time series: covering different types of emergencies and different severities (measured as time to treatment) Span from week (7 th Nov 2010) to week (7 th June 2015) Series are at England level (not local authorities). Accurately predict to support staffing and training decisions. Aligning the short and long term forecasts is important for consistency of planning and budgeting. Test set: 52 weeks. Rolling origin evaluation. Forecast horizons of interest: t+1, t+4, t+52 (1 week, 1 month, 1 year). Evaluation MASE (relative to base model) As a base model auto.arima (forecast package R) is used. 38/75

39 Application: Predicting A&E admissions Total Emergency Admissions via A&E Red is the prediction of the base model (ARIMA) Blue is the temporal hierarchy reconciled forecasts (based on ARIMA) Observe how information is `borrowed between temporal levels. Base models for instance provide very poor weekly and annual forecasts 39/75

40 Application: Predicting A&E admissions Accuracy gains at all planning horizons Crucially, forecasts are reconciled leading to aligned plans 40/75

41 Hierarchical forecasting & decision making Hierarchical (or grouped) forecasting can improve accuracy, but their true strength lies in the reconciliation of the forecasts aligning forecasts is crucial for decision making. Is the reconciliation achieved useful for decision making? Cross-sectional Reconcile across different items. Units may change at different levels of hierarchy. Suppose an electricity demand hierarchy: lower and higher levels have same units. All levels relevant for decision making. Suppose a supply chain hierarchy. Weekly sales of SKU are useful. Weekly sales of organisation are not! Needed at different time scale. Temporal Reconcile across time units/horizons. Units of items do not change. Consider our application. NHS admissions short and long term are useful for decision making. Suppose a supply chain hierarchy. Weekly sales of SKU is useful for operations. Yearly sales of a single SKU may be useful, but often not! Operational Tactical Strategic forecasts. 41/75

42 Cross-temporal hierarchies Temporal hierarchies permit aligning operational, tactical and strategic planning, while offering accuracy gains useful for decision making BUT there can be cases that strategic level forecasts are not required for each item, but at an aggregate level. Let us consider tourism demand for Australia as an example. Local authorities can make use of detailed forecasts (temporal/spatial) but at a country level weekly forecasts are of limited use. Temporal: tactical strategic Cross-sectional: local country Cross temporal can support decisions at both dimensions: Tactical/local; strategic/local; tactical/country; strategic/country 42/75

43 56 (bottom level) quarterly tourism demand series MAPE % 6 years in-sample 3 years out-of-sample horizon: up to 2 years rolling origin evaluation Cross-temporal hierarchical forecasts: Most accurate Most complete reconciliation (one number forecast) Flexible decision making support 43/75

44 Production ready? Multiple Aggregation Prediction Algorithm (MAPA) Kourentzes, N.; Petropoulos, F. & Trapero, J. R. Improving forecasting by estimating time series structural components across multiple frequencies. International Journal of Forecasting, 2014, 30, (Details) Petropoulos, F. & Kourentzes, N. Improving forecasting via multiple temporal aggregation. Foresight: The International Journal of Applied Forecasting, 2014, 2014, (Easier introduction!) Petropoulos, F & Kourentzes, N. Forecast combinations for intermittent demand. Journal of the Operational Research Society, 2014, 66.6, (Intermittent) Kourentzes, N. & Petropoulos, F. Forecasting with multivariate temporal aggregation: The case of promotional modelling. International Journal of Production Economics, (Promotional modelling) R package on CRAN: MAPA (and tsintermittent for slow movers) All papers, code and examples available at my website ( Temporal Hierarchies Working paper at my blog! R package on CRAN: thief 44/75

45 Agenda 1. Business forecasting and uncertainty 2. Forecasting challenges from a modelling view 3. Innovations: 3a. Temporal Hierarchies 3b. Promotional Modelling 3c. Using online search information 45/75

46 What is the promotional modelling problem? 4000 SKU demand 5 - Product: Trend: 0 - Season 13: 0 - Season 52: 1 Demand Promocode Product: Adcode: 401 Adcode: 406 Adcode: 408 Adcode: 410 Adcode: 412 Discount % Price Discount What is the future demand? Normal demand? With promotion(s)? 46/75

47 A practical approach Use statistics for structured tasks that can be automated Use human expertise for events Statistical Baseline Forecast Expert Judgement Final Forecast Time series methods: e.g. exponential smoothing Special events: e.g. promotions, calendar events, etc Demand planning forecast Promotional and advertising activity is one of the main reasons for adjusting statistical forecasts 47/75

48 Why do companies adjust forecasts? % indicating important or extremely % indicating Reason for using judgment important importance Promotional & advertising activity 51.4 Price change 45.5 Holidays 36.0 Insufficient inventories 33.3 Changes in regulations 32.7 Government policy 27.9 Substitute products produced by your company 22.5 Activity by competitors (promotions, advertising etc) 20.7 International crises 18.0 Weather 16.7 Insufficient inventories of competitors 16.2 Sporting events 11.8 Strikes 9.9 Other /75

49 Promotional modelling basics What is the best forecast for this time series? 550 Out-of-stock: Promotion wasted Units Safety stock Overstocking erroneously Aug Jan May Oct Feb Jul-2009 No additional information? Use univariate modelling; e.g. exponential smoothing Lets assume a company goes on with this forecast Take a look at the inventory side of things (Also, how do you forecast the next promotion?) 49/75

50 Promotional modelling basics 550 What is the best forecast for this time series? Enough stock, but... Units Safety stock Wrong! Aug Jan May Oct Feb Jul-2009 Unstructured information Use univariate modelling + judgemental adjustments (S&OP meetings = planners + marketers) Promotional modelling requires building formal statistical models that capture promotions in their forecasts (systematically elasticities = decision making) and therefore provide correct safety stocks 50/75

51 Promotional modelling basics If inventory management does not account for the uncertainty (due to events/promotions) correctly, then 51/75

52 A statistical approach We can build a forecast using statistical models, enhanced by: Promotions Price Store Events Competition Actuals Fit Prediction y(t) = 6.91 * Promo 1 (t) 0.55 * Promo 2 (t) 0.69 * Promo 3 (t) 0.17 * Promo 2 (t-1) * Promo 3 (t-1) ; adj. R 2 = Sales Period 25 Histogram of standardised residuals 3500 Scatterplot of residuals vs actuals Promotional impact 52/75

53 A statistical approach What can go wrong? A lot! Effects across SKUs, categories, Cannibalisation, how to identify it? Multiple effects multicollinearity (i.e. things happening together and then it is difficult to identify which causes what) Modelling with limited data (or no data!) Scalability run the model for multiple time series within reasonable times 53/75

54 Promotional forecasting: a case study Used a subset of 60 SKUs: Sales One-step-ahead system forecasts (SF) One-step-ahead adjusted or final forecasts (FF) Promotional information: 1. Price cuts 2. Feature advertising 3. Shelf display 4. Product category (22 categories) 5. Customer (2 main customers) 6. Days promoted in each week Data withheld for out-of-sample forecast evaluation Some products do not contain promotions in the parameterisation (historical) sample 54/75

55 Promotional forecasting: a case study Periods highlighted in grey have some type of promotion 55/75

56 Promotional forecasting: a case study The advanced promotional model contains the following elements: Principal component analysis of promotional inputs: The 26 promotional inputs are combined a new composite inputs that are no longer multicollinear Only a few new inputs are needed now, simplifying the model. Pooled regression: When an SKU does not have promotions in each history, we pool together information from other SKUs (the product category information is available to the model) to identify an average promotional effect. When enough promotional history for an SKU is available we do not pool information from other products Dynamics of promotions: Carry-over effects of promotions, after they are finished, are captured Dynamics of the time series: Demand dynamics are modelled for both promotional and non-promotional periods. 56/75

57 Promotional forecasting: a case study A series of benchmark models are used to assess the performance of the proposed promotional model System Forecast (SF): The baseline forecast of the case study company Final Forecast (FF): This is the SF adjusted by human experts in the company Naïve: A simple benchmark that assumes no structure in the demand data Exponential Smoothing (SES): An established benchmark that has been shown to perform well in supply chain forecasting problems. It cannot capture promotions Last Like promotion (LL): This is a SES forecast adjusted by the last observed promotion. When a promotion occurs the forecast is adjusted by the impact of the last observed promotion Note that only FF and LL can capture promotions from our benchmarks 57/75

58 Promotional forecasting: a case study High error Forecast accuracy under promotions DR: Advanced Promotional Model Low error Number of forecasts Size of adjustment Negative adjustments Positive adjustments 58/75

59 Promotional forecasting: a case study High error Forecast accuracy under promotions Human judgment performs inconsistently LL has overall poor accuracy Low error DR is always better under promotions DR is more accurate than human experts and statistical benchmarks in promotional periods 59/75

60 Promotional forecasting: a case study High error Forecast accuracy in non-promotional periods Human judgment performs poorly No reason for positive adjustments? LL has overall poor accuracy Low error DR is models accurately carryover promotional effects DR marginally worse that SF DR accurately captures carry-over promotional effects. SF is marginally better because the forecasts are based on adjusted historical demand. DR is modelled in raw data. 60/75

61 Best Best Best Promotional forecasting: a case study High error No promotions SF FF Naïve SES LL DR Promotions SF FF Naïve SES LL DR Overall SF FF Naïve SES LL DR Low error Advanced promotional model is consistently the most accurate Improvement of DR over FF Overall +14.4% Substantially outperforms current case study company practice (FF) Promotions +34.6% Major improvements during promotional periods No promotions +7.8% 0.0% 10.0% 20.0% 30.0% 40.0% 61/75

62 Promotional forecasting: a case study 1. Advanced statistical promotional model significantly outperform human experts and statistical benchmarks 2. Company can benefit from increased automation of the forecasting process and reduced effort of experts to maintain forecasts, while accuracy is increased 3. Can produce promotional forecasts when there is no or limited promotional history for an SKU 4. Overcomes limitations of previous promotional models 5. Paper available: Trapero, J R., Kourentzes N. and Fildes R. On the identification of sales forecasting models in the presence of promotions. Journal of the Operational Research Society, 2014, 66.2, (available at my blog) 62/75

63 Temporal hierarchies & promotions Multivariate MAPA allows capturing the effects of external variables at different temporal scales (short vs. long-term effects). However, temporal aggregation may increase further the multicollinearity between variables, making the estimation of c i problematic. Consider the example of binary dummies, which will tend to become similar at high levels of temporal aggregation (e.g. promotional dummies at annual level). Use principal components to mitigate this problem Principal components construct composite orthogonal inputs Useful also when various promotional campaigns happen simultaneously 63/75

64 Case study Case study: major UK manufacturer of cider drinks. 12 SKUs of a popular cider brand Promotions at customer/store level multiple at the same time Weekly data: 86 observations for estimation, 18 observations for test Predict t+4, t+8 and t+12 periods ahead (1, 2 and 3 months). Rolling origin evaluation. 64/75

65 Case study Bias sme Method Horizon: 4 Horizon: 8 Horizon: 12 Naive ETS MAPA Regression ETSx MAPAx smae scaled Mean Absolute Error Accuracy - smae Method Horizon: 4 Horizon: 80.9 Horizon: 12 Naive ETS MAPA Regression ETSx MAPAx Superior bias and error variance 0.55 safety stock implications smae t+4 t+8 t+13 Horizon Naive ETS MAPA Regression ETSx MAPAx 0.5 t+4 t+8 t+13 Horizon 65/75

66 Agenda 1. Business forecasting and uncertainty 2. Forecasting challenges from a modelling view 3. Innovations: 3a. Temporal Hierarchies 3b. Promotional Modelling 3c. Using online search information 66/75

67 Using consumer search behaviour Another way to manage uncertainty is to use additional external information this must be leading sales, so that it is available for the forecasting model to use. Nowadays there is abundant information about how people search online, share posts, consume media, etc. Can this information help us produce better forecasts? Operational forecasting: evidence suggests that there is little gain, if any, given the model complexity. New product launches and lifecycle modelling promising! 67/75

68 Peak Scale Peak Scale Case study: forecasting computer games Call of Duty 3 Actual Sales Google Trends Week since Launch Wii Sports Actual Sales Google Trends Week since Launch 68/75

69 Different generations of games Sales for Assassin's Creed Sales Peak scaled search traffic j=1 j=2 j=3 j=4 j=5 j=6 j=7 Google Trends j=1 j=2 j=3 j=4 j=5 j=6 j=7 69/75

70 The idea Gamers may search online for a game for multiple reasons. Pre-launch: news, pre-views, buy After-launch: reviews, updates, walkthroughs, buy, We are interested in information that can be known well in advance, so that it can support meaningful decisions. Enhance lifecycle models (Bass) with pre-launch online search information (and from previous generations) increase confidence & accuracy, given minimal data availability. Focus on market size parameter. 70/75

71 Modelling process Sales Sales Peak scaled search traffic Actuals Forecast p j = p j 1 p j = p j 1 q j = q j 1 q j = q j 1 m j = Search m j = Search Traffic Traffic j=1 j=2 j=3 j=4 j4 Search traffic t t t j=1 j=2 j=3 j=4 71/75

72 Case study: evaluation Use 6 games series (43 games in total) Bass model with alternative estimations for market size: Naïve (same parameter as last generation) Naïve + difference between generations Linear trend for the parameter AR(1) process for the parameter Parameter is a function of google search data Optimal : estimate model once all data is available. Use relative mean absolute errors (denominator is with google search), error <1 means that it is improving. 72/75

73 Case study: results Horizon: 1 week Naïve Naïve Diff. Linear Trend AR(1) Optimal <1 = better Horizon: 6 weeks Naïve Naïve Diff. Linear Trend AR(1) Optimal <1 = better In all comparable cases there are gains in using online search information! This is ongoing work 73/75

74 Conclusions Managing uncertainty is one of the main objectives of business forecasting leads to superior decision making. We can do that by: Better capturing historical structure of sales data Using additional information (e.g. promotions, discounts, online search) Temporal hierarchies joins operational, tactical and strategic decision making by reconciling forecasts `one-number forecast superior decision making. Statistical promotional modelling accurate and automatic at SKU level leaves human experts to focus on unstructured information. Online search behaviour gains in modelling sales using pre-launch information superior forecasts, but can we shape future demand? 74/75

75 Thank you for your attention! Questions? Published, working papers and code available at my blog! Nikolaos Kourentzes blog: 75/75

76 Appendix Detailed M3 results for temporal hierarchies

77 Some evidence that it actually works! M3 quarterly dataset % error change over base % error change over base ETS ARIMA BU: Bottom-Up; WLS H : Hierarchy scaling; WLS v : Variance scaling; WLS s : Structural scaling 756 series, forecast t+1 - t+8 quarters ahead

78 Some evidence that it actually works! M3 monthly dataset % error change over base % error change over base ETS ARIMA BU: Bottom-Up; WLS H : Hierarchy scaling; WLS v : Variance scaling; WLS s : Structural scaling 1453 series, forecast t+1 - t+18 months ahead

79 Appendix Calculation details for temporal hierarchies

80 Temporal Hierarchies - Notation Non-overlapping temporal aggregation to k th level: Annual Semi-annual Quarterly Observations at each aggregation level

81 Temporal Hierarchies - Notation Collecting the observations from the different levels in a column: We can define a summing matrix S so that: Lowest level observations Annual Semi-annual Quarter

82 Example: Monthly Aggregation levels k are selected so that we do not get fractional seasonalities

83 Temporal Hierarchies - Forecasting We can arrange the forecasts from each level in a similar fashion: The reconciliation model is: Reconciled forecasts Summing matrix Reconciliation errors how much the forecasts across levels do not agree Unknown conditional means of the future values at lowest level The reconciliation error has zero mean and covariance matrix

84 Temporal Hierarchies - Forecasting If was known then we can write (GLS estimator): But in general it is not know, so we need to estimate it. It can be shown that is not identifiable (you need to know the reconciled forecasts, before you reconcile them), however: Reconciliation errors So our problem becomes: Covariance of forecast errors

85 Temporal Hierarchies - Forecasting All we need now is an estimation of W Sample covariance of in-sample errors In principle this is fine, but its sample size is controlled by the number of toplevel (annual) observations. For example 104 observations at weekly level, results in just 2 sample points (2 years). So the estimation of is typically weak in practice.

86 Temporal Hierarchies - Forecasting We propose three ways to estimate it, with increasing simplifying assumptions. Using as example quarterly data the approximations are: Hierarchy variance scaling Diagonal of covariance matrix less elements to estimate Series variance scaling Structural scaling Assume within level equal variances. This is what conventional forecasting does. Increases sample size. Assume proportional error variances. No need for estimates can be used when unknown (e.g. expert forecasts).