Market Simulations. A New Approach to Marketing Strategy. Simulate Your Market. Copyright 2016 Concentric Inc.

Transcription

1

2 Market Simulations A New Approach to Marketing Strategy Simulate Your Market Copyright 2016 Concentric Inc.

3 THE CHANGING SITUATION With the proliferation of new media types, the explosion of available data, the rising influence of social networks, a general desire to reduce advertising spend, and the increasing number of new product failures, marketers are finding it increasingly difficult to pick strategies that produce desired outcomes. The reaction to this challenge has been a massive investment in data and insights, which has also failed to provide a solution to the problem. A recent survey from the CMO council identified the following: 92% of CIOs are under pressure to integrate various data sets 71% of CMOs do not use the data they have purchased 61% of CEOs struggle to evaluate the impact of marketing initiatives on business goals 66% of CFOs are unable to validate ROI of marketing investments In this turbulent and fragmented environment, simulation has emerged as a new technology that solves the challenges of the C-suite by delivering the following benefits: This white paper defines the simulation concept as it applies to marketing, explains how our product works, and identifies applications to a wide variety of business questions.

4 THE SIMULATION CONCEPT When pilots practice flying new airplanes, they use flight simulators. When doctors train, they use models of human bodies. When engineers design computer chips, they test their ideas in silico before sending them to the factory. When marketers test ideas, strategies, and tactics about consumer markets, they use Concentric s simulation application, Concentric Market. Concentric Market simulates consumer markets with accuracy that matches or exceeds the highest statistical standards of any existing marketing analytic tool, while providing insights that go further. Our users diagnose the performance of offline, online, and social marketing vehicles. They explore strategies to develop and launch new products and evaluate the risk of new competitive entrants. They forecast consideration, perceptions, word-of-mouth, sales, and return on investment for each of the brands competing in the market and for each of their target consumer segments. Ultimately, Concentric simulations allow users to build better marketing strategies faster because they recreate market dynamics, validate to multiple metrics, and learn with artificial intelligence. The following sections go into more detail on each of these three capabilities. Recreates market dynamics The two most widely used marketing analytics approaches are brand tracking and marketing mix modeling. The former measures the consumer preferences in the market, but has no predictive power to forecast what would happen if consumers or the marketing change. The latter attributes ROI per media per brand, but does not take into account consumer preference changes. Simulation offers the same capability of brand tracking reports and marketing mix modeling and combines them to allow analysis of what-if scenarios about consumers and marketing strategies. In this way, simulation models the entirety of the market, rather than approaching the consumer and the marketing strategy as two separate systems to be analyzed. Validates to multiple metrics Since simulation has a broader ambition than other marketing analytic tools, i.e., to replicate the entirety of the market, it requires a more stringent validation to ensure its accuracy. Concentric Market uses the market data in two ways to validate the model: for calibration and for a hold-out test. Data from the calibration period is used to fit the simulation, whereas data from the holdout period is only used after running the simulation to evaluate how well it forecasts. Users commonly forecast from one quarter to two years, but some users have forecasted up to 10 years in the future. While a holdout comparison is a good basis for judging a forecasting model, a single comparison on sales for a particular brand is too small a set of validating measures for the scope of a simulation. Concentric Market users validate to multiple brands, and to outcomes beyond sales such as perceptions, word-ofmouth, and awareness. Additionally, users also evaluate all metrics by segment, a statistical benefit that ensures appropriate model specification.

5 Learns with Artificial Intelligence As part of Concentric s conceptualization of a simulation, the application includes various forms of machine learning. We define machine learning as a type of artificial intelligence that provides computers with the ability to learn without being explicitly programmed. Machine learning focuses on the development of algorithms that can teach themselves to grow and change when exposed to new data. This capability is embedded into a pre-defined process for human and machine interaction: The system enables the user to repeat this process within minutes, creating a cycle of learning and exploration that could not be achieved through an individual or even a team. This leads to broader exploration of alternatives and learning through feedback, a process that in turn identifies risks and rewards.

6 HOW IT WORKS Machine learning on its own does not deliver the benefits of simulation. In order to create a working model of a market, Concentric combines additional methodologies and scientific concepts. Included in Concentric Market tm are principles from marketing analytics, behavioral economics, network science, and agent-based modeling. Agent-based modeling is the core engine in Concentric Market tm and understanding its components reveals how simulation works. The components of our Agent-Based Model are: 1) Variables and Rules 2) Initial Conditions 3) Calibration Algorithms 4) What-if Scenario Analysis VARIABLES AND RULES Our algorithms start by describing the interactions of the key components (or agents) of the system with a few simple individual-level rules that lead to aggregate behavior, matching what we observe in reality. To begin with, the model accounts for the current marketing strategy of the brand and its competitors. All activities of the brand communicate to the consumer and the system allows for all marketing investments to be analyzed simultaneously. Consumers have properties, including consideration and perceptions of different brands. These properties are the drivers that govern what is important to each consumer when making decisions. Consumers are also connected in a social network the fabric of a consumer market. Finally, consumers engage in behaviors related to products: they shop, choose among alternatives, experience the products, and share their experiences with their social network. A few rules define how consumer properties drive the behaviors. For example, the more consumers consider a product, the more likely they are to make a purchase. We use variables to change the impact of properties on behaviors. For example, the greater the social density, the more likely information will spread about the alternatives in the market. While behaviors are intentional, consumers are also affected by events, such as overhearing a friend talk about a new product or reading an online review. These events are likely to change consideration and perceptions. In addition to people in their social network, consumers are affected by outside factors, such as marketing, influencers, the category seasonality, or the product itself. These external factors create additional events that change the properties of consumers, and can be influenced by media investment and creative quality. So what we have now described is the entirety of the Concentric Market tm model. The diagram below shows all the variables, rules, and intervening factors.

7 The most important point here is that unlike most marketing analytics approaches, which have dependent variables (like sales numbers) and independent variables (like marketing investment), an agent-based model is a system dynamics model in which many variables are interdependent. This means that everything affects everything else. Sales are not a final outcome: the more people buy a product, the more people experience it, and the more people change or solidify their perceptions and consideration. INITIAL CONDITIONS All variables and rules of the above schema are informed through a diverse set of information and data that forms the initial conditions of the agent-based model: 1) Consumer research like brand trackers, focus groups, or panels, to inform consumer properties 2) Media and sales data, which are usually used in any marketing effectiveness analysis 3) Information about the actual performance of the product, like product specs and industry reports 4) Social tracking information about the volume and content of conversations These disparate data sources, which were previously analyzed in isolation, are now integrated into initial conditions for Concentric simulations. There are two types of initial conditions: On one hand we have inputs over time, like marketing investment (from a media plan), product changes which may occur during the year, category seasonality, and some market events. On the other hand, we

8 have settings that we are only interested in inputting at the first moment of the simulation, because we want to forecast their evolution over time. Here is the key to the system dynamics approach: necessary assumptions are limited to the rules of the market and the initial conditions (which are both verifiable by measuring simulation output). The forecasts result from the logical evolution of the variables over time based on the interaction of these conditions and rules. CALIBRATION ALGORITHMS Users are interested in four key metrics: consideration, perceptions, sales, and word of mouth. The first thing to do when running Concentric Market tm is to make sure the simulation forecasts match the real world measures of these KPI s. To do so, Concentric Market compares the simulated results with the real data and measures the error between them. To improve the match between the simulation and the real world, a user identifies a set of parameters to vary. Perhaps these are chosen because the user did not have the best information or simply had no information at all. An automatic calibrator systematically adjusts parameters, runs Monte Carlo simulations, calculates the error margin between the updated simulation and the real results, and continuously narrows the search until it finds the variable settings or the rules that best fit the simulation to the observed real world results.

9 WHAT-IF SCENARIO ANALYSIS Once the user is satisfied with the validity of the diagnostics and the forecasting accuracy of the simulation model, the user creates what-if scenarios. There are plenty of options to explore in Concentric Market. Users change the media budget or mix for their own brand or for a competitors. They introduce new products or competitors into the market or change features of the product. They prepare for unanticipated PR or viral events. They change the nature of the consumer segments and anticipate trends before they happen. All simulations in Concentric Market are probabilistic, which means that when a user re-runs the same scenario twice, they may get different results. As Concentric Market runs more and more simulations using a process called Monte Carlo analysis, it forms a distribution of outcomes. This distribution is easily represented through a box plot, which shows the minimum and maximum of outcomes, the top and bottom 25%, the middle 50%, and a median outcome. The box plots help the user understand the confidence to have in simulation forecasts: the wider the box plot the lower the confidence of the baseline scenario, which is the scenario used to calibrate to past performance.

10 Box-Plot Box plots of Monte Carlo outcomes also help in comparing scenarios. Adding a new scenario, called Messaging Shift, we see that it performs better than the baseline, as the median outcome is higher. By showing the two scenarios side by side on four different metrics, the user can identify trade-offs. Although the messaging shift scenario increased sales, it has a potential downside for awareness and word of mouth. The most useful part of Concentric Market is the comparison of multiple scenarios over multiple metrics there is no limit to the number of ideas and strategies that one can test and explore.

11 SUMMARY Concentric Market tm provides a system that combines social science and marketing science to solve specific marketing challenges and enhance executive processes. Simulation reinvigorates marketing innovation by empowering marketers to evaluate and optimize outside the box programs that they may have been uncomfortable executing in the real world by safely evaluating them in a virtual environment. What makes Simulation unique for solving challenges is that it can do all of the following simultaneously: Business Strategy Comparison Campaign/Channel Attribution Media Optimization Creative Content Optimization CRM Program Optimization Corporate Social Responsibility In-Store Strategy Pricing Analytics Product Development Product Launch Support Trade-dollar Assessments Targeted Media Buying Additional benefits of a simulation approach stem from the implementation of a unifying process to: Improve resource allocation Make real-time/near-time decisions Identify new markets Communicate insights Facilitate change Encourage collaboration Enhance transparency Accelerate the development of new products/services With a more complete answer, at lower costs, and at faster speeds, simulations are getting broad adoption and are set to transform marketing strategy. We hope you join Concentric on our quest to better marketing strategy.

12 Thoughts on Simulation A gathering of our technical blogs Simulate Your Market Copyright 2016 Concentric Inc.

13 WHAT DOES A SIMULATION LOOK LIKE? While simulation is a commonly used tool in areas like social science, operations, physics, and engineering, it is not well established in the marketing community. Often times, we are asked what is going on under the hood. For those looking to see the nuts and bolts of a marketing simulation, we have created a learning map pictured above. Below we offer some key definitions of the elements of our simulation system. Perceptions At the core of an agent-based model is the recreation of consumer behaviors. How they gather information, influence other, and use the product all interact to change their perceptions. Changing perceptions combined with Behavioral Economics is the core forecast engine of the system. Consumer-centric forecasting is a central difference between traditional media-centric models. Drivers With consumers at the center of the model, simulation accounts for diverse consumer preferences by detailing the relative importance of attributes by segment. Brands that are perceived to best meet drivers win more market share than other brands.

14 Connectivity In addition to including consumers, simulation accounts for the effect social networks, online behavior, product usage and influencers have on changing perceptions. Capturing the impact this earned media is a unique feature of simulation that allows marketers to consider strategies that go well beyond typical measures of paid media impacts. Marketing No model of marketing would be complete without a clear understanding of how media investments stimulate consumers in the context of their social setting. Simulations allow for the inclusion of marketing mix models or digital attribution results as an empirical foundation of the system. Additionally, simulations can be used as an alternative to traditional models of attribution because they account for all touchpoints in the marketing plan simultaneously. In summary, the above map of a simulation of a consumer market shows how simulation fuses social and marketing science to account for the entire competitive environment and to keep consumer behavior at the center of analysis.

15 FRAMING A MARKET BRINGING STRUCTURE TO A SIMULATION Simulation is flexible and somewhat unconstrained by nature. This open-endedness makes simulation useful for understanding real-world markets in a broad variety of contexts. But given this open-endedness, it becomes important to remain focused on a specific objective and to bring some structure to the simulation. Accordingly, the first step in building any Concentric Market simulation is to construct a simulation framework. This framework should be built based on the consumer decision we are seeking to better understand, in line with some business objective. The framework consists of the following elements: Alternatives: What set of brands or options are in the consumers choice set? Attributes: What factors do consumers consider when selecting among alternatives? Touchpoints: What marketing activities or events influence consumers decision or behavior? Segments: What different types of consumers exist in the market? Most importantly, these elements should be selected according to the specific consumer decision you wish to simulate. Here are some additional guidelines when building a simulation framework, along with the typical number of framework elements in simulations: Alternatives (3-10): Consider the market share of the competitors. It s usually appropriate to include the dominant competitors in the market and to exclude brands with very small share. Factor in how directly the alternative competes with your brand. It s a good practice to consider not only the size of the competitor, but also whether or not they present a threat or an opportunity for your brand s market share. You may choose to include a brand with low market share if they are an emerging threat to your brand. Think about if there are any substitutes in adjacent categories that would be helpful to consider.

16 Attributes (2-6): Consider the factors that matter most to a consumer when making a decision. Typically this list can be summarized into a few key drivers of choice. Observe what consumers are talking about and what messages marketers are broadcasting. This can provide insight into what factors matter the most. Think about both tangible attributes (physically experienced by the consumer) and intangibles (more ethereal in nature, i.e. cool, brand reputation). Touchpoints (5-150): Consider anything that can change consumer perceptions or awareness or stimulate conversation in the marketplace. Touchpoints may include traditional media channels, and also other types of interactions with the brand (for example, in-store touchpoints, events, PR, salespeople, online blogs). Remember to consider not only your brand s marketing activities, but also the activities of the competitors. Think about the level of detail at which marketing strategy decisions will be made when determining how granular to make the list of touchpoints. For example, if the simulation will only be used to determine broadly the mix of television and online advertising, then it would probably not be helpful to expand the touchpoint list to include splits for different TV programs, 15 vs. 30 second ads, etc. Segments (1-6) Consider how the population can be divided into groups based on differences in behavior, media habits, drivers of choice, and perceptions/awareness about the alternatives. Select a set of segments that are mutually exclusive and encompass the relevant population. Segments may be demographic, attitudinal, psychographic, or based on other factors so long as that holds. Think about if there are any key non-purchasing influencer groups (i.e. doctors/pharmacists, children) that should be included in the simulation. Even if an influencer group does not purchase any of the alternatives, they may still be active in the social network and have an impact on outcomes.

17 HOW DATA FITS INTO CONCENTRIC SIMULATION Building a simulation does not require any data. However, validating a simulation does. Building a simulation only requires that we make assumptions. We assume initial conditions, constraints, and behaviors within the system we seek to understand. Qualitative or quantitative data may be used to inform some of these assumptions, but often times we just apply insights or judgment. Once we have made assumptions and built a simulation, we run the simulation and see if it accurately reflects real world dynamics. This validation is what requires data. Calibrating a valid simulation is an iterative process. The test-and-learn nature of the process is what frees us from being bound by data when initially building the simulation. However, the validation points by which we judge the accuracy of our simulation are based on data.

18 BUILDING CONFIDENCE IN ATTRIBUTION FROM A SIMULATION APPROACH Simulation, like any other modeling approach, is an imperfect representation of the real world. A question that commonly comes up when performing analysis and forecasting is how can we be confident that what this model tells us is right? Of course, all modeling results have some uncertainty, but attribution is a particularly difficult finding to validate. In a nutshell, attribution is the decomposition of a variable into the elements driving changes in its value. Attribution results in a measurement of the impact of each element. For example, marketing mix analysis often decomposes sales into base (an unexplained contribution) and individual marketing channel contributions. The difficulty in building high confidence about attribution is that the numbers reported are virtually impossible to measure in the real world. If we say that TV contributes 30% to the increase in sales, we are making an accounting statement. There is no way to go in the field and truly isolate that 30% figure. Getting to that 30% requires assumptions about how we measure effects and how we quantify contribution. Yet, while attribution is in this sense more art than science, there are ways to increase our confidence in the attribution results that come from a simulation. More often than not, increasing the confidence requires investing more time into validation of the model against metrics that are actually observable and measurable in the real world. Only then, by proxy, is our confidence boosted that our latent results are as valid as the results we can compare with measured observations. Below we outline 4 routes that can be taken to increase the confidence in the simulation attribution: 1. Calibrate the simulation to multiple metrics at an aggregate level. 2. Perform a holdout forecast. 3. Compare individual consumer journeys to simulated agent journeys. 4. Conduct in-market tests to assess impacts. CALIBRATING THE SIMULATION TO MULTIPLE OBJECTIVES Simulations are often built with a relatively large number of parameters and initial conditions. This often raises a question of how one can be confident that they have not over-fit the simulation. One way to avoid over-fitting risk and build confidence is to calibrate to more data points. Typically, regression approaches fit to one time series, for example, sales over time for one brand. With a market simulation, one may calibrate to sales for the primary brand of interest as well as sales for its competitors. The number of data points that build confidence in the fit is then quickly scaled up by the number of competing brands.

19 In addition to calibrating to sales across brands, one may also calibrate to other metrics beyond sales. Simulations can be calibrated to brand metrics such as awareness, perceptions, and word-of-mouth volume. The number of calibration points can be scaled even further by the number of KPIs that are fit. Fitting a simulation to data from multiple brands across KPIs builds confidence that it is not over-specified and that it recreates dynamics of the real-world. This topic is explored further in the next blog post. PERFORMING A HOLDOUT FORECAST Often stakeholders are interested in the predictive power of a model or simulation. In this case, the evaluation of the approach goes beyond analytics to explain the past and into how well it projects future outcomes. A holdout forecast is one approach for assessing predictive capabilities. When conducting a holdout, the historical data is split between a calibration time frame and a holdout period. The model or simulation parameters are tuned to the calibration time frame only. This calibrated model is then used to make a forecast during the holdout period.

20 Using a holdout, we see how effectively the simulation was able to forecast past outcomes and this provides a better sense of what sort of predictive accuracy may be anticipated in forecasting future outcomes. ANALYZING CONSUMER JOURNEYS One of the advantages of simulation is that it recreates the dynamics that take place at an individual level in the real world. Aggregate sales outcomes emerge from many individual consumer decisions. One may be interested in seeing not only how well macro outcomes are fit by a simulation or model, but also how well the approach recreates individual-level data. These journeys can give us a better sense of how all the marketing activities and earned touchpoints interact to lead to sales. If data on individual consumer journeys is available, this can serve as another way to assess the validity of a simulation. Simulated agent journeys can be recorded and compared to the data on individual consumer journeys. If the simulated journeys align with the actual consumer journeys, then the simulation is tuned to the micro-level dynamics of the simulation.

21 CONDUCTING IN-MARKET TESTS Ultimately, an audience may choose that the best way to assess the impact of a marketing strategy is to actually put it into practice in the real-world. An in-market test allows for data to be gathered that can help measure impacts. Although running a test in the actual market allows one to see how an outcome pans out in the complex real-world, it is difficult to run a controlled experiment when the many variable factors of the real world come into play. Even a real-world experiment has its limitations in terms of building confidence in a particular finding.

22 HOW TO COUNTER THE RISK OF OVER-SPECIFICATION WHEN CALIBRATING AN AGENT- BASED MODEL Practiced statisticians are very familiar with the problem of over-fitting when building traditional econometric models. As the number of parameters approaches the number of data points fitted two things will happen: 1. The model will fit the history progressively more precisely. 2. The model will become progressively more over-specified, eventually to the point of losing all statistical and predictive value. This is why experienced model-builders place such high value on parsimony and generally avoid complicating a regression model with extra variables unless they contribute significant explanatory power. The classical degrees of freedom problem can eliminate a model s credibility. To summarize: As the number of regression parameters increases with a number of data points, over-fitting risk also increases Similar concerns may arise when building a simulation. Sure, having the ability to adjust a variety of initial conditions and settings in an agent-based model may provide a lot of flexibility, but may we encounter the problem of overspecification when calibrating an agent-based model? The answer is yes, absolutely. With all the various model settings in Concentric, one could simulate an annual sales figure that matches with historical values with an array of input assumptions. Many of these assumption sets would not align with reality and result in simulations that are misleading and lacking in insight. There are two ways to avoid the problem of over-specification in agent-based modeling. 1. Reduce input uncertainty: Decrease the number of freely floating inputs when calibrating your model by building confidence in your input settings via research, data analysis, expertise, or norms. This will cut down the model calibration search space to a more manageable size and reduce your risk of over-fitting. 2. Increase the number of validation points: Calibrate to more metrics and trends based on what you know to be true. Go beyond sales to assess awareness, word-of-mouth volume, and brand equity outputs. Analyze the sales and brand metrics over time to see if the trends are sensible. Look at brand equity attribution to see if the relative impacts of each touchpoint are realistic. Carry out sensitivity analyses to see if the simulation mimics marketing responses you would expect.

23 By implementing these two approaches, you are attacking the problem of over-fitting risk from both sides: You are reducing the number of parameters in your search space and increasing the number of data points in your validation. Your efforts should always be geared towards the business question you are ultimately trying to answer. This should help guide where to focus your energy in reducing input uncertainty and selecting your validation points. Concentric is meant to be used as a thinking system. The more thought that goes into the system inputs and validation efforts, the more confidence you will have in the insights the system generates. So keep this simple rule in mind: As Thinking, Over-fitting risk Always.

24 HOW DO MARKETING MIX MODELING AND SIMULATION ACCOUNT FOR UNCERTAINTY I can calculate the motion of heavenly bodies but not the madness of people said Isaac Newton and almost four centuries later, the sentiment is still strong among practitioners and consumers of predictive analytics that forecast human behavior. It is true we lack theories and empirical evidence to come close to predicting human behavior with accuracy found in the physical sciences, and making choices based on models of human behavior is risky. But we have identified ways of quantifying and analyzing that risk. One of my projects back when I worked in actuarial science was to quantify a range around the expected paid losses (i.e., the portion of losses that the insurer actually pays). I set to work analyzing the historical data and began attempting to quantify the range. Process Risk After fitting a model to the data, my first approach was to estimate how much variability seemed to be inherent in the process. I quantified the range by estimating the error term of the model i.e., the volatility observed in the actual data around the model fit and assuming a distribution for the error term. This type of variability is often referred to by actuaries as Process Risk. Parameter Risk I proudly shared my results with the group, but then another issue was raised. In order to compute an error term, I had to assume a model with specific, fixed parameters. What if those parameters, which had been calibrated to a sample of data, were not accurate? On top of the Process Risk, there is additional risk due to uncertainty surrounding the model parameters Parameter Risk. Adding a bit more statistical rigor to my analysis, I came up with an estimate of this term. Model Risk With both process and parameter risk taken into consideration, I was confident in the range around the expected loss estimate. Then I was asked a final question. What about Model Risk? What do we do if the assumed functional form or structure of the model does not reflect the reality of the underlying process? This one had me stumped. After months of analysis and research, I could not solve this puzzle. Actually, it turns out the solution to the Model Risk predicament had alluded the entire group.

25 In statistical models, the different types of risk can be quantified in a formulaic manner. In simulations different types of risk can be assessed in an empirical manner based on more iterative approaches. So how does simulation deal with the various types of risk? Monte Carlo Analysis Assesses Process Risk This involves running multiple iterations of a simulation with the same initial conditions. The simulation can evolve differently each time due to the probabilistic and path-dependent nature of the approach. We can then observe the inherent risk in a given strategy. Sensitivity Analysis Assesses Parameter Risk This entails building a series of simulations with inputs that are tweaked in each run. Variation in simulated outcomes may be observed as the inputs are adjusted. We can then assess how sensitive outcomes are to shifts in initial conditions or assumptions for which we are uncertain. Testing Different Rules of the Simulation Assesses Model Risk Assessing model risk requires critical thinking about the fundamental assumptions made in the simulation framework. Have we considered all the relevant competitors or drivers of choice? Do we understand the consumer segments and all the touchpoints that affect them? We should also consider the rules of the simulation. Are the underlying algorithms relevant in our real world market? Are our assumptions about how consumer decision making, paid media effects, and the social network accurate? These questions raise challenges that cannot be addressed overnight. Model Risk is best mitigated through collaborative, creative, and strategic thinking. So Newton is still right: We cannot account for all types of risk and uncertainty. This is not just because of the limitations of our analytical methods, but due to the shear unpredictability of a complex, ever-changing world. The best we can do is to decipher patterns and build insights through data analysis and statistics, research and observation of the world around us, and our intuition. Simulation can help in our efforts to better understand the uncertainties and risks involved in markets and business strategies. And these efforts will be most successful with a team that is open, adaptive, knowledgeable, persistent, and creative.

26 OVER-SPECIFICATION: COMPARING AGENT-BASED AND MARKETING MIX MODELS A common objection that we hear during discussions about agent-based modeling is that agent-based models are over-specified and therefore their findings are invalid. Although it is possible with ABM as with any other modeling technique to encounter the issue of over-fitting, building the simulation diligently will minimize the risk of over-specification (more details on how to minimize the risk of over-specifying are given in the preceding blog post). In fact, agent-based models often have a lower risk of over-specification than marketing mix models. In this post we will explore an illustrative example. A metric that is commonly used to assess whether or not a model is over-specified is the ratio of calibration data points to model parameters. If there are too few data points per model parameter, the model is over-specified. In an extreme case where there is one parameter for every calibration data point, the model can achieve a perfect fit. The parameters, in a sense, would just be a transformed restatement of what happened historically. This model would not be very useful for making any forward-looking projections or drawing any conclusions about what happened in the past. Let s compute the ratio of data points per model parameter for an example agent-based model and marketing mix model below: MARKETING MIX MODEL Calibration Data Points: Marketing mix models are often fit to a time series of a brand s sales. In this example, let s assume the model is fit to two years of historical weekly sales data. That gives us: 2 years x 52 weeks = 104 calibration data points Calibrating Model Parameters: Marketing mix models are typically fit by linking changes in sales to different marketing activities. Let s assume there are 8 touchpoints that are being analyzed. There will be a coefficient that defines the impact of each touchpoint on sales. Additionally, each touchpoint s marketing activities can also be modulated by parameters that define the point of saturation and decay rate of the marketing impact. Taken together, there are 3 parameters defining each of the 8 touchpoint effects plus a y-intercept so a total of: 8 touchpoints x 3 parameters per touchpoint + 1 = 25 calibrating model parameters Ratio of Data Points to Model Parameters: 104/25 = 4.2 data points per model parameter

27 AGENT-BASED MODEL Calibration Data Points: Concentric simulations are often fit to multiple time series. These time series correspond to KPIs across competing brands. Let s assume the simulation is fit to one year of historical weekly sales data, but not just for one, but for six competing brands. Let s also assume that the simulation is fit to tracking data on awareness and perceptions for each of the six brands. However, in this case, let s say the tracking study was only run quarterly. The total number of calibrating data points is then: 6 brands x 52 weeks of sales + 6 brands x 4 quarters of awareness + 6 brands x 4 quarters of perception = 360 calibration data points Calibrating Model Parameters: Concentric simulations are built by integrating data from various sources and calibrating a set of parameters to available calibration data. Let s assume that the settings defining consumer behaviors and initial preferences and awareness levels are provided from a consumer survey or tracking study and are therefore constrained. The parameters that are used to fit the simulation are then based on three types of settings: 1. Touchpoint impacts on awareness and perceptions: 4 per touchpoint 2. Consumer awareness and perception decays: 2 in total 3. Decision-making parameters: 2 in total 8 touchpoints x 4 parameters per touchpoint = 36 calibrating model parameters Ratio of Data Points to Model Parameters: 360/36 = 10 data points per model parameter In the example above, the risk of over-specifying the ABM is lower than the risk of over-specifying the marketing mix model. Despite the fact that the ABM is fitting a time period that is half the length, the number of calibrating data points per parameter is over twice as high as the marketing mix model. This is because an ABM fits an entire market as opposed to one particular brand in isolation. Ultimately, there are a number of approaches that may be taken to build confidence in the validity and usefulness of a given model. These approaches include alternative specification formulas like a log function, calibration against multiple time series, holdout forecasts, and in-market tests. By diligently taking such approaches, our users consistently build valid and useful market simulations with agent-based modeling.

28 ATTRIBUTION TO WHAT? UNDERSTANDING CROSS-CHANNEL INTERACTIONS Cross-channel effects occur in situations where the sum of the parts is not equal to the whole. These effects can be positive (synergistic) or negative (inefficient). For instance, imagine that you have two media channels at your disposal: TV and Facebook. Should you run them separately or together? The answer to that question depends on the nature of cross-channel effects between these two channels: whether there is a positive or negative interactive effect from running these two channels together. If the sales lift you get from running these two channels together is different from what you would get if you ran each channel separately and added the sales lift you got from each, then there is a cross-channel effect. The types of cross-channel effects that can occur are defined as follows: Synergistic Cross-Channel Effect A case when the cross-channel effect results in a synergy In this case, the sum of the parts is less than the two considered as a whole. There is some extra lift in sales that results from a synergistic interplay between both channels. Inefficient Cross-Channel Effect A case when the cross-channel effect results in an inefficiency

29 In this case, the sum of the parts is greater than the two channels working together, which has created some inefficiencies. The cross-channel effect is therefore defined as the difference between the individual lifts and the total lift: [Lift in Sales from TV] + [Lift in Sales from Facebook] + [Cross-Channel Effect] = [Lift in Sales from TV & Facebook]. The effect of marketing activity on sales is often analyzed using regression techniques using sales as a dependent variable. Cross-channel interactions may be quantified with regression by including an interaction term in the model specification. This analytical technique is useful for measuring the effects of media, i.e., allowing for sales attribution and cross-channel interactions to be computed. But the task of understanding what mechanism actually causes the cross-channel effects remains a mystery. To fully understand the mechanism that drives cross-channel interactions we need to recognize that the linkage between marketing activity and sales lies in the mind of a consumer. The consumer s decision-making behavior is what ultimately drives sales. So how do consumers choose? Consumer Decision-Making is the Key to Cross-Channel Effects The growth and acceptance of behavioral economics as a science behavior has been instrumental in allowing us to take some simple rules to create simulated models of consumer decision-making that take into account heuristics and rationality. This knowledge has allowed us to build models to measure consumer behavior that go beyond notions of perfect equilibrium and neo-classical economics that were never originally designed for this purpose. In simple terms, consumers make decisions based on options they would consider, and select the option they perceive most highly on attributes that are most important to them (based on the behavioral principle of utility maximization). This view assumes a path in the consumer-decision making process as follows: The components of the consumer decision-making process lead to the different cross-channel effects

30 This approach means that both brand awareness and brand perceptions are important in driving sales. Some touchpoints may be more effective at building awareness and others at building perceptions. It won t do the marketer much good to focus on building brand awareness if the product is perceived very poorly a perception or satisfaction bottleneck that may not be solved by throwing more media at the problem. Likewise, it might be unhelpful to attempt to build deeper engagement for a brand no one recognizes an awareness bottleneck. Synergies and inefficiencies may result from the awareness stages that exist in the complex non-linear path to consumer decisions. Inefficiency may occur when activities focused on awareness-building or perception-building continue when brand awareness or brand perceptions are already maximized. We may saturate awareness and observe that any additional marketing yields no incremental benefit. A synergy may occur when touchpoints that build awareness and others that build perceptions are used in conjunction. We will get more out of our perceptionbuilding media campaign, for instance, if more of the consumers are made aware of the brand. In short, inefficiencies may occur when we concentrate too much on one step in the consumer-decision making process, and synergies may occur when we balance our efforts between different steps. See below for a mapping of TV and Facebook from our example. A mapping of two channels on two criteria: how well they affect awareness and perceptions Building understanding of what drives synergies and inefficiencies within a marketing plan can bring valuable additional insight to strategic decision-making. Thinking about the path in the non-linear consumer-decision making process and the various bottlenecks and saturation points therein may allow a team to consider new creative strategies that balance awareness building and perception building activities. Simulations may then serve as an aid to testing these hypothetical strategies prior to implementation to include interim metrics such as brand awareness, brand perceptions, and WOM activity. So as you think about the attribution of all of your brand activities, ask yourself: attribution to what?

31 FROM OPTIMIZATION TO STRATEGY: DEEPER INTO THE ATTRIBUTION STACK Sales attribution is a common diagnostic used in evaluating marketing effectiveness. This involves decomposing a brand s sales into Base and Media components. Media is often further divided to attribute sales by distinct touchpoints. The result can be visualized as a stacked attribution chart as follows: Stacked Attribution Chart This chart provides an indication of how much touchpoints are lifting sales and therefore can be used to assess the Return on Investment (ROI). A planner may use these outcomes to change the mix, putting more weight into the touchpoint with greater ROI. Simulation makes it possible to dive deeper into attribution analysis to provide more insights and further guidance on marketing strategy. Here are four more detailed views of the stack that simulation provides: 1. Understanding how creative and messaging impact the effectiveness of each touchpoint 2. Identifying the synergies and inefficiencies that exist between channels 3. Decomposing the base to understand the role of earned media and brand history 4. Placing the attribution into a consumer-focused competitive context CREATIVE EXECUTION AND MESSAGING EMPHASIS The concepts of creative execution and messaging emphasis allow us to go beyond analysis of levels of investment and reach to analyze the effectiveness of media. Creative execution defines how well the ad is created the production value and how effectively it communicates the message. Messaging emphasis defines what attributes of the product are the focus of the ad. For example, is the ad part of a promotional campaign or a branding campaign? Unlike most marketing mix modeling, simulation allows us to test the effects of different creative executions and messaging emphasis and decompose each individual touchpoint contribution further.

32 Creative Execution and Messaging Emphasis CROSS-CHANNEL EFFECTS Touchpoint impacts may interact with one another through duplication or social network effects. Inefficiencies occur when reaching the same audience via multiple touchpoints leads to no gain. Synergies occur when messaging on two touchpoints yields better results than using each touchpoint on its own. Simulation can serve as a guide to better understanding and replicating these interactions in the consumer s mind and provide a platform to test different approaches to fully leverage synergies and avoid inefficiencies. Cross-Channel Effects DECOMPOSING THE BASE The Base is often the forgotten piece of the attribution stack an unexplained variance that does not yield any actionable insights. Running a simulation that incorporates the effects of earned media social networks and product experiences allows for a better understanding of what drives base sales. Often times, word-of-mouth (new restaurant opens up) and the actual experience of the product (the joy of driving your automobile) play a major role in driving a brand s performance. Sometimes the base is simply a matter of consumer inertia (think

33 about insurance), seasonality (canned soup), or a brand s history (Coca-cola vs. Pepsi). Better understanding the base may provide strategic ideas on how to avoid erosion of the base or how to keep it growing strong. Decomposing the Base CONSIDERING THE COMPETITION Sales outcomes are the aggregate result of many individual consumer decisions. Consumers make choices among a set of alternatives and so in many cases a brand s performance is just as much dependent on what its competitors do as what the brand does itself. Building a simulation of an entire market where simulated consumers choose among competitive alternatives allows for an assessment of how a brand will fare in different competitive environments. If a brand positions itself appropriately, it may rapidly grow by cannibalizing competitive share. If not positioned appropriately, the entire attribution stack may sink into oblivion (think about Blockbuster Video with the emergence of Netflix and Redbox). Considering competition is key to evaluating strategies to boost and maintain market share over the long run. Considering the Competition

34 CONCLUSION MOVING BEYOND OPTIMIZATION TO STRATEGY The initial view of the attribution stack allows for an assessment of ROI that can guide decisions around how much to invest in touchpoints. However, assessing the stack from the lens of creative messaging, cross-channel synergies and inefficiencies, earned media and brand history, and the competitive environment allows for a richer understanding of how a brand is likely to perform in its market. Simulation provides flexibility in aperture to plan and react strategically in a variety of ways.

35 MARKETING EFFECTIVENESS MEASURES: KNOW YOUR OPTIONS There are a variety of approaches used to quantify the impact of marketing activities and attribute sales to touchpoint activities. Such methods include: Marketing Mix Modeling Test and Control Group Experiments Attribution Modeling Agent-based modeling In marketing mix models, sales or some other KPI of interest are regressed against marketing activities to understand how changes in the KPI are related to changes in marketing activities. The resulting coefficients define the sensitivity of the KPI to touchpoint activities. From this model specification, the quantity of sales to attribute to a given marketing activity may be derived. In test and control experiments, one group (the test group) is exposed to a marketing treatment while another group (the control group) is left unexposed. Ideally, these two groups are otherwise completely the same. The difference in outcomes (sales, response rate, or other KPIs) between the two groups is then attributed to the marketing treatment. In attribution modeling, individual level data is used to assess what individual touchpoint activities (commonly digital ads) led to a sale or some other action. These models typically link a KPI (such as a sale or website visit) to one or some combination of activity that preceded it. These accounting approaches include last touch, first touch, or some other weighting across touchpoint activities. In Concentric agent-based models, the dynamics of a marketplace are recreated by simulated consumers who decide between alternatives, interact in a social network, and experience marketing. In this way, the impacts of marketing activities are considered in the context of all interactions a consumer may have with brands in a market including not just paid marketing for a single brand, but competitive actions and earned media. From these simulations, an attribution to sales and brand metrics may be derived by comparing differences in simulated outcomes when touchpoint activities are added and removed. The decomposition of sales and other metrics goes beyond paid media to include earned media and competitive impacts.