Combining Decision Analysis and Analytics. John Busbice, Managing Partner Keen Strategy

Transcription

1 Combining Decision Analysis and Analytics John Busbice, Managing Partner Keen Strategy (804)

2 2 About my perspective Background in marketing analytics Management consultant and practitioner of statistical methods Introduced to decision analysis about 5 years ago Now combining statistics and DA Founded firm that combines marketing mix modeling and DA

3 What I want to do when I grow up 3

4 Good old days surfing decisions 4 Where to Surf? Surfing! Waves Wind

5 100 ft wave!... Garrett McNamara, January 2013 in Portugal 5

6 6 Big wave surfing: A transformation from available information. Where to Surf? Big Wave Surfing! Surf Forecasts Surfline.com Stormsurf.com Waves Wind

7 Data is exploding 7 1 exabyte = 1,000,000,000 gigabytes 1 gigabyte = 1,000,000,000 bytes All words ever spoken by humans ~ 5 exabytes Source:

8 How will Big Data impact decision quality? 8 Components Data required Judgment required Frame? Yes Alternatives? Yes Decision Quality Information Yes Yes Logic? Yes Values? Yes Commitment? Yes

9 Getting information from data is a search process. 9 You always find something the last place you look Two methods of search Human guided: Analyst or researcher runs iterative analyses, gets feedback and learning and continues down the path, stopping point is guided by the person. Computer guided: Algorithm runs iterative analyses, calculates metrics and then continues according to stopping rules developed by the engineer of the algorithm.

10 Analysis requires judgment and depends on the quality factors governing the decision. 10 Decision Quality Framework Applied to Analytical Quality Decision Quality Components Frame Logic Judgments in Analysis Variables selected to represent considerations within the problem frame Defines the relationships explored or considered valid. Values Information is viewed through the lens of a valued outcome.

11 Manager s Monty Hall problem 11 Finding #1 Finding #2 Finding #3 Where is the prized information?

12 A partnership is required by the analyst and decision maker to promote decision quality 12 Manager needs: What s behind the analysis? Transparency to frame, logic, values and path of search? Apply their understanding of the context Analyst needs: Understanding of the frame, logic, values Provide transparency to the manager Bayesian methods provide an avenue Manager provides prior assessments based on their logic, context Analyst improves upon the managers assessments through the analysis DATE: 9/13/2013 Finding the next leap forward for accelerating growth

13 Managers benefit by applying their prior knowledge 13 Data & Statistics Managerial Judgment Strength Synthesizing information from patterns Context and Logical reasoning Weakness Limited by data and governed by the analyst Limited by biases and the depth of the manager s knowledge

14 CASE STUDY 14

15 Managers are faced with many difficult questions when making marketing investment decisions: 15 Objectives: Are we spending too much or too little? What is the risk of missing revenue targets? Are we maximizing value? cash flow? Are we building the brand? What is the optimal mix across sales and marketing channels? What should we forecast given our sales and marketing budget? Pain points: Complexity: Multiple channels, interactions, non-linear impact, long-term effects, uncertain returns Conflict: No two analyses converge, different people responsible for different marketing efforts Cost: Marketing mix modeling and Big Data lead analysts to digging through imperfect information for months on end. DATE: 9/25/2013 Finding the next leap forward for accelerating growth

16 Model structure and estimation 16 Price Sales volume = f(x,β) Pr(B) from assessments Sales Force Pr(B X) = Pr(X B) * Pr(B) / Pr(X) Offline (TV & Magazine) Sales Volume Online (Display & Search) Price-based promotion

17 Assessments are converted to a quantitative input for analysis 17 What portion of sales would be lost within one year if you were to eliminate investment in marketing? Answers in form of a range: 25% chance below 50% chance between 25% chance above 25th percentile 75th percentile

18 The data and assessments are combined in a statistical analysis 18 Combined Result Assessment Data

19 8/16/ /16/ /16/2009 2/16/2010 4/16/2010 6/16/2010 8/16/ /16/ /16/2010 2/16/2011 4/16/2011 6/16/2011 8/16/ /16/ /16/2011 2/16/2012 4/16/2012 6/16/2012 8/16/ /16/ /16/2012 Sales Volume The model fits well to the data and can be validated in holdout samples , ,000 Actual vs. Fitted (in sample) And Forecast (validation) 200, , ,000 50,000 Actual Fitted Forecast Residual 0 50,000 -

20 Revenue $ Marketing Investment Response 20 Sales and Marketing Activity 95th Percentile_Revenue 75th Percentile_Revenue 50th_Percentile_Revenue 25th Percentile_Revenue 5th Percentile_Revenue Investment $

21 Revenue $ Price response 21 Price 95th Percentile 75th Percentile Median 25th Percentile 5th Percentile Price

22 1/14/2011 3/14/2011 5/14/2011 7/14/2011 9/14/ /14/2011 1/14/2012 3/14/2012 5/14/2012 7/14/2012 9/14/ /14/2012 1/14/2013 3/14/2013 5/14/2013 7/14/2013 9/14/ /14/2013 1/14/2014 3/14/2014 5/14/2014 7/14/2014 9/14/ /14/2014 Marketing NPV is calculated based on the contribution and longterm effects 22 40, Contributions Over Time by Short vs. Long-term 35, , , , , , Short-term Impact 2014 Long-term Impact 2013 Long-term Impact 2012 Long-term Impact 2011 Long-term Impact 5,

23 Online Display Free trial TV Activity #4 Activity #5 Trade shows Paid search Sales force Renewal notices Public relations Magazines Total ROI and Marginal ROI (1.0 = Break-even) Each activity is evaluated in financial terms ROI and Marginal ROI by Channel Valuation Parameters: 1 future years, 10% discount rate $6.00 $5.3 $5.3 $ $4.00 $3.00 $2.00 $2.3 $2.8 $2.2 $2.6 $2.0 $1.8 $1.5 $1.5 $1.4 $1.9 ROI Marginal ROI $1.00 $0.9 $1.2 $0.8 $0.5 $0.7 $0.6 $0.8 $0.5 $0.5 $0.3 $0.2 $0.7 $0.00

24 Marketing NPV Millions The NPV-maximizing investment is calculated 24 $45 Sales and Marketing Activity $40 $35 $30 $25 $20 $15 95th Percentile_Revenue 75th Percentile_Revenue 50th_Percentile_Revenue 25th Percentile_Revenue 5th Percentile_Revenue $10 $5 $-

25 Revenue Comparing multiple alternatives motivates trade-offs 25 Scenarios NPV max Increase cash Constraints / Trade-offs Status quo NPV = Bubble size Cash Flow

26 As a result of this analysis the company experienced renewed revenue growth even while reducing overall ad investment 26 Quite frankly we [were wasting] a lot of money. - Company CEO Recommendation vs. Implemented Actual Results 10% Recommendation & Forecast Implementation and Results 5% 0% - 5% - 10% - 15% Investment Change Revenue Change NPV Change (Est.) - 20% - 25% - 30% DATE: 9/13/2013 Finding the next leap forward for accelerating growth

27 The approach is applicable in many ways outside of marketing 27 Extending the approach Sign Posts for Investments: Often valuation models are used extensively to support an investment decision, but without continuity after the fact. It s possible to specify sign posts as observable metrics that signal the state of as assessment, then make assessments of model parameters conditional on sign posts and assess the probability of the sign post, perhaps with statistical analysis. With the probability model as the state of the sign post changes, the assessment will too, and the state of the investment can be monitored. This would enable better evaluation of ongoing embedded options such as further investment, buy-sell, or abandonment. DATE: 9/13/2013 Finding the next leap forward for accelerating growth

28 Decision Analysis vs. Analytics: Heart of the debate about the role of people and information. 28 Decision Analysis Analytics Information Data Subjective probability Frequency Human emphasis Machine emphasis KEY Decision Decision People Computers

29 Key points 29 Data is exploding and will transform the way we do things. In a data rich environment, human judgment is still required. People are getting lost in the process of mechanization and measurement in the conversation around Big Data. Data and judgment should be blended by people and for people to make better decisions.

30 30 Monty Hall Problem DATE: 9/13/2013 Finding the next leap forward for accelerating growth

31 The contestant picks a door, but the host does not yet open it... 31? Contestant first picks the door without a prize 2/3 rd of the time.

32 The host opens another door Zonk! 32? X Contestant assessment: Is this random or intentional?

33 If you believe that the host will never reveal the prize 33 The un-chosen door will contain the prize 2/3 rd of the time Better off switching? X Happens 2/3 rd of the time, if the host always opens the door without the prize.