Measure Marketing Effectiveness: A Guide to Implementing Incrementality

Transcription

1 Measure Marketing Effectiveness: A Guide to Implementing Incrementality Incrementality is a relatively new term, but it s not a new concept. Incrementality is the measure of the true value created by any business strategy, determined by isolating and measuring the results it caused, independent of other potential business factors. Incrementality is calculated by comparing differences in outcomes between two separate groups of people: those who ve been exposed to the strategy and those who haven t. For years, businesses have been able to effectively measure how choices, in areas such as product testing or direct mail, incrementally influence their business outcomes. This same method of measurement is possible with digital advertising. In this guide, you ll find advice created in partnership with some of the industry s leading advertisers on how to bring incrementality measurement to your marketing. We will walk you through some of the essential questions and considerations to make along the path to adopting incrementality. These best practices can help make your strategies both successful and impactful. Incrementality is the cornerstone for our business cases both in product and marketing. Matthew Gerrie, Booking.com Measure Marketing Effectiveness 1

2 Contents Introduction Before you get started Getting started Choose a business question Choose a method Questions to ask at every step to ensure quality experiments Step 1: A hypothesis that isolates a single variable Step 2: Precision Step 3: Stability of treatment assignment Step 4: Realistic exposure Step 5: Comparability between treatment groups and control groups Questions to ask to ensure quality observational methods Conclusion Contributors Measure Marketing Effectiveness 2

3 Before you get started Is your organization ready to embrace incrementality? Incrementality measurement allows you to make smarter business decisions by helping you understand how and where marketing is contributing to your business outcomes. Before shifting your measurement models to incrementality, it s important to ensure that your organization is ready to make the investment and do the work needed to embrace a data-driven, test-and-learn approach. Tony Flanery-Rye of ebay provides some practical advice on how to think about this: You are committing to continually learning and continually challenging the status quo. Everything that you thought was true may not be true. You are committing to continuing to evolve your company. Is the culture you are working with ready for this amount of rigor? People are used to making decisions based on gut, yet you will have these experiments coming in that may counter those. Will that be accepted? Does your culture value people bringing data into the conversation? Investing in incrementality means that we believe that more robust measurement will make us, in the long run, a more efficient marketing business. Alok Gupta, Lyft When thinking about where to start, consider beginning with your most important marketing channels. As you evaluate how comfortable your organization would be with embracing a testand-learn mentality, it may make sense to start with smaller tests in areas where you know you can prove the value quickly, and then build up over time. Measure Marketing Effectiveness 3

4 Getting started After you have determined that incrementality is right for your business, this guide can help you in evaluating the right questions to set up your organization for success. Choose a business question Your first step is to decide which business question you want to answer. Once that is established, take time to consider your options and be deliberate about what you want to measure and how you d like to measure it. Proper planning up front will help guide you and your key stakeholders to make the best business decision, and help ensure a smooth decision-making process. Decision-making process Decide business question to answer Consider measurement options Decide how to measure The question that always comes first is how are we going to measure the impact of any change? Sometimes it isn t possible to measure the change, but 90% of the time we will measure it, and we will always ask the question. Alok Gupta, Lyft Measure Marketing Effectiveness 4

5 Choose a method There are multiple techniques you can use, which fall into two main categories: experimental and observational. + The most effective way to measure incrementality is to run an experiment in which you tightly manage the strategy, or treatment, to which people are exposed. Experimental Begin by developing a hypothesis about the effect your change in strategy will have. Then designate a group (or groups) of people who will be exposed to the treatment, and a control group who will not. By isolating the exposure of a variable, such as creative or audience, and then comparing it to the control group, you can understand the true incremental value of the strategy. The quality of experiments may vary, but they are still the ideal and most accurate way to measure incrementality. True experiments are often the benchmark for other methodologies. Despite their benefits, experiments do require up-front setup, as well as the opportunity cost of withholding treatment from the control group. Hypothesis Test group Exposed to variable Compare conversions between the two groups to determine the true incremental value of the strategy Control group Not exposed to variable Tony shares his thoughts on the value of choosing to run an experiment: Experiments are preferred because you have a higher level of certainty that the results you are seeing represent the actions you are trying to measure. Measure Marketing Effectiveness 5

6 If conducting an experiment would be too costly or not possible, you can use observational techniques. Observational Begin with an existing set of data that resulted from exposing people to a certain ad or ad variable, and then apply a model or statistics to estimate how much value a treatment may have had. Common methods involve using synthetic experiments to attempt to replicate a real experiment by finding a control group within a group of people who were not exposed to the ad or ad variable you are trying to evaluate. For example, you could evaluate the effect of a technical issue that only impacted some users by finding a similar group of people who were unaffected. This method does not require up-front work, but it may be less accurate and subject to bias on unknown factors. It does also require advanced methods and support from data scientists later in the process. Use synthetic experiments to find a control group within a group of people exposed Exposed to variable Data results from exposure Alok offers perspective on when it may be necessary to choose observational methods: Considering observational methods makes sense when upside or resourcing is slim. Oftentimes it is quick and easy. As a marketing channel becomes more important for the business, it becomes important to move towards an experiment. Measure Marketing Effectiveness 6

7 Incrementality measurement does require time and resources and, for experiments, the opportunity cost of withholding exposure from a control group. But as you become more of an expert and improve your infrastructure to automate processes, the process will likely become less costly. Incrementality measurement requires: + + Time Resources Opportunity Cost Before getting started, consider matching the potential impact of your hypothesized outcome with the resources that would be needed to accomplish the measurement. Comparing impact to investment may help you decide how you want to run your experiment or if you need an experiment at all. In our business, if an experiment is able to be run, we will always run one. Where no experiment can be run, we will choose between an observational method or a hybrid of the two. Business decisions come down to the confidence you have in your observational and experimental methods. Matthew Gerrie, Booking.com Now that you have a general understanding of the two possible approaches to measuring incrementality, we re going to go in depth into each of them to help you plan your process and prevent issues along the way. We ll begin by discussing experiments because we consider them to be the most accurate and effective approach. Measure Marketing Effectiveness 7

8 Questions to ask at every step to ensure quality experiments Once you have decided to run an experiment and determined what business question you want to answer we recommend that you start by asking questions about what and how you re measuring. Alok shares the benefits of approaching an experiment thoughtfully: Asking these questions is best practice because things can change on the publisher side and the marketer side that we might not be aware of or plan for. It s like experimentation within a lab we want to keep making sure the tools are calibrated. To guide your experiment, establish each of the following: 1 2 A hypothesis that isolates a single variable: Have I isolated the question I want to answer? Precision: Will my test provide enough data to accurately answer the question I m asking? 3 Stability of treatment assignment: When someone is assigned to a treatment group, can I be sure they ll stay in that group for the entire test? 4 Realistic exposure: Does my experiment act like it would in the real world? 5 Comparability between treatment and control groups: Do the test groups and control groups have the same characteristics and propensity to take action? Although these questions apply specifically to experiments, it s a good idea to apply a similar set of thinking when evaluating observational methods. These questions should be asked at three times: up front when you are making your hypothesis, during the test as a check-in and at the analysis point. You want to make sure all of these points are adhered to. Just because you answer these questions in the beginning doesn t mean that is enough to guarantee the results are good. Matthew Gerrie, Booking.com Measure Marketing Effectiveness 8

9 1 A hypothesis that isolates a single variable To truly understand the effect of a treatment such as the difference in performance between two different campaigns it s important to determine up front what you want to test, and then to isolate that variable by ensuring it is the only difference between your test and control groups. This will allow you to confidently conclude that it was indeed the variable that caused the effect on ad performance. Matthew Gerrie of Booking.com offers perspective on choosing a hypothesis: Without a hypothesis, testing will only be for testing s sake, meaning you may not have an actionable result. If the test is not in service to the hypothesis, it shouldn t be done. Ways to achieve this: Plan your campaign to execute tests in which there is only one test variable. Don t fish for results after a test. Make sure you re not drawing conclusions based on variables you didn t isolate. Ways to check this: Pre-test: Validate that your setup isolates the variable you want to test. Post-test: Ensure the campaign(s) delivered as designed and the test variable remained isolated. When you implement the test results, don t extrapolate too broadly beyond the intended scope of the test. For example, don t apply learnings outside of the tested channel. If you re testing on digital, don t apply the learnings to TV campaigns. Examples: If you are trying to test the effect of budget on ad campaign performance, only change the budget allotted to each treatment group, while leaving targeting, bid value, creative, timing and execution the same. If you are trying to test the effect of delivery frequency for direct mail, only change the frequency of delivery, while leaving creative, product advertised and messaging the same. Measure Marketing Effectiveness 9

10 Isolating a single variable is important so you can be very confident in what changed. If you vary multiple things at once, you risk conflating interactions. By focusing on a single variable, you can learn more quickly on one hypothesis, and you can take those wins as you expose other ones in subsequent campaigns and learnings. Stephan McBride, Netflix Organizational advice: At this point in the process, it s a good time to set expectations and get early buy-in across your organization and the publishers you are working with. If this is the first incrementality test your teams will undertake, consider spending additional time up front to educate the cross-functional groups that may be impacted such as finance, marketing and product. Some teams may have outsized expectations about the broad applicability of experimental results, especially around additional segmentation that was not planned. Also, we generally see more success by beginning with smaller and more frequent experiments, which produce learnings quickly, versus larger and less frequent experiments, which involve a bigger investment of time and money before yielding results. Stephan McBride of Netflix shares advice on what to prioritize as you approach your experiment: As a tactical strategy, I generally suggest to focus on your largest channels because of the impact. But as an alternative, I would recommend that you prioritize types of tests or testing a particular hypothesis from which you can learn quickly and deliver wins quickly to your stakeholders, so that you can help have buy-in in the approach and build up over time. Measure Marketing Effectiveness 10

11 2 Precision Experiments, as with any statistical measure, come with some level of variance. And, while the variability in experiments does reflect the variability present in the real world, it can affect your ability to learn the true effect of the treatment you re testing as well as your stakeholders perception of your experiments reliability. When running experiments, make sure that your test is set up with enough precision to measure what you re trying to test. It will take more data to reveal smaller differences, and less data to see results proving bigger differences. Be sure your test is planned and executed in such a way that you can confidently answer the question you re trying to address. Ways to achieve this: Set expectations for what metric will determine success for the test, and plan around it. Plan up front to determine if your test will provide enough data to answer your question, considering the potential effect of your test variable. Way to check this: Use statistical methods to understand how effective the ads are, what conclusions you can draw and how confident you can be in your results. Avoid slicing the results in too many ways to help maintain precision and prevent problems caused by multiple comparisons. We are very concerned about two areas: where we don t collect enough data spending money on media but not getting a reliable result or where we overpower our test by buying too many impressions wasting money and making incorrect decisions based on small effects. We use statistical power analysis to ensure that we have the correct number of observations as to not over- or under-power our intended analysis. Matthew Gerrie, Booking.com Measure Marketing Effectiveness 11

12 Typically a data scientist can do experiment power calculations to provide the decision maker with a suite of options. If you want to detect at least X size of effect change, in maximum Y amount of time, Z is how much you must spend. It s really the size of the effect how small of an effect do you want to detect, and how quickly you want to detect it that tells you how much you have to spend. Alok Gupta, Lyft There are business reasons why you may choose to make a split in the data and make an immediate business decision. If you do need to split data in this way, we would advise a followup test in which you specifically manipulate that variable as part of another experiment that way you will ensure the business decision was made properly. There is not much point to carefully designing an experiment that is tightly controlled, and then running splits and analyses that compromise that tight control. Matthew Gerrie, Booking.com Measure Marketing Effectiveness 12

13 3 Stability of treatment assignment A key component of any experiment is making sure that each test audience stays in the treatment group to which they are assigned throughout the length of the test and across whichever devices and platforms you re measuring. That way, the people who aren t supposed to see a treatment don t see it, and the people who are supposed to see a treatment do see it, and at your intended cadence. In some cases, the duration of the test may exceed the limits of a testing platform. For example, the longer your experiment, the more difficult it will be to ensure that your control group is not exposed to a treatment. If so, consider revising the campaign s duration to maintain the stability of your treatment assignment. Stephan shares the importance of maintaining the stability of your treatment assignment: In the real world there will be substantial interactions between all people. The stable separation between treatment and control is a balance. We recognize that there is a trade-off, but I would discourage seeing that as a barrier to using experiments to get better. Stability is something that can be used, adapted or adjusted for by data scientists on your team, but it is important to avoid violations of stable treatment as much as possible. Way to achieve this: The easiest way to ensure stable treatment assignment is to establish a persistent knowledge of the participants in your test audience. Depending on what you re testing, this could be as simple as an address or , or as complex as a collection of different forms of identity. If your product is bought or used at the household level, it is necessary to isolate the treatment assignment not just to a person, but to the entire household. Way to check this: Evaluate where there is potential for crossover and how consistent these identifiers will stay over time. For example, does your test depend on people staying signed in to a website? Do you anticipate their mailing addresses will stay consistent over the length of your test? Examples: Household-based measurement in direct mail (mailing address) Digital communication address ( or phone number) Digital platform login IDs Device data (app and browser) Geographies (DMA/Postal Code) Measure Marketing Effectiveness 13

14 4 Realistic exposure Your testing environment should mimic the real world as closely as possible. From a platform perspective, ensure that the platform is treating your test campaigns like your normal ad campaigns. From an execution perspective, deliver media that is representative of what you d typically use. You also want to make sure that the people in your test are exposed to outside media and campaigns with the same frequency as they would be under normal circumstances. Some test designs might make this difficult due to technical limitations, which could cause interaction effects between your ads. For example, not withholding a campaign from a control group for one round of testing could cause your other campaigns to over-deliver to that group. While this is difficult to prevent on digital platforms, you may be able to validate this during or after the test. You may have non-test treatments that are a result of actions taken by people during a test, like retargeting campaigns aimed at users who visit your site. Exposure to these treatments doesn t need to be equal across groups, as long as the events that trigger them and the treatment people receive afterward aren t affected by the test treatment. Ways to achieve this: Check in with your various marketing departments to let them know about your test, and make sure you re aware of any upcoming campaigns that could impact your study. Ensure the method used in your test doesn t alter how people view or interact with your ad. Ways to check this: Run tests that compare the performance of two otherwise identical campaigns in two different treatment groups. This should show no statistical difference. Compare delivery metrics for normal and test campaigns. Measure your campaigns. Don t design your campaigns to be measured. To ensure this, run campaigns like you normally would, and make sure to adjust budgets to be proportionate to the size of the audience you re testing. For example, a 50% holdout means you should reduce your budget by 50%. We work across all marketing departments to do our best to make sure that the marketing is as close to ceteris paribus as possible. We make the assumption that you will always have marketing, so the interactions that exist should be considered in the test. Tony Flanery-Rye, ebay Measure Marketing Effectiveness 14

15 5 Comparability between treatment groups and control groups The best experiments make sure that the groups of people being compared are statistically similar. When evaluating this, look at a few dimensions: Characteristics: Are the people you re comparing similar across dimensions like demographics, product engagement and utilization? Outcomes: Do the groups you re comparing have a similar propensity to use or to purchase products, as measured by pre-treatment conversion rates? Outside and prior exposure: Have these groups historically been exposed to treatments or ads at the same rate? Is the media to which they re exposed the same across channels outside of the platform you re testing? Way to achieve this: Randomization: Up-front randomization for treatment assignment is the easiest way to achieve comparability among groups, because it implicitly balances all factors known and unknown, given a large enough sample. Way to check this: Check that the groups have a balance of various metrics that are known up front and will remain unaffected by the test, like demographics or pre-campaign conversion rate. Measure Marketing Effectiveness 15

16 Questions to ask to ensure quality observational methods Sometimes high-quality experiments are either unavailable or too onerous to run, so you may need to use observational methods. When using observational methods to develop a test for incrementality, your first question should be: Can I validate this against a true experiment? Observational techniques are meant to mirror the results of an experiment, so the simplest and most effective way to evaluate the quality of observational techniques is to compare them to experiments you ve run in the past. Broadly, there are two categories by which you can evaluate the observational method model against an experiment: 1 Accuracy: How close am I to the true incrementality of 2 the treatment represented by an experiment? Decision-making: How often would my observational model choose the actual winner, like a true experiment would? If it s not possible to validate your observational method against an experiment, and you re using a synthetic control, focus on making sure that a balance exists between your treatment group and synthetic control group. Even if you re making optimization decisions using observational metrics, you should see your KPIs improve over time based on your decisions. Ways to achieve this: Use quality modeling techniques that are intended to be unbiased and incremental. Train observational models with experiments. Ways to check this: Run test campaigns, like those for PSA placebo ads, and validate that the model doesn t show incrementality for that group. Validate observational models against experiments. Measure Marketing Effectiveness 16

17 We use observational methods to help interpret data or correlational patterns within experiments and to help guide hypotheses for future tests. If we see a change we didn t anticipate, that might guide our future testing priorities to focus on that dimension. Stephan McBride, Netflix Measure Marketing Effectiveness 17

18 Conclusion Now that you understand what implementing incrementality measurement entails, you can use these guidelines to help your cross-functional partners, publishers and teams understand the importance of adopting a test-and-learn measurement model. Determine what business problem you re trying to solve, design the best experiment you can and then continue to act on what you ve learned within your business. This process will continue to evolve over time. You will know that your organization has successfully embraced incrementality when it becomes core to all business cases that require key decisions. 1 Determine problem to solve 2 Design experiment A few final thoughts as you consider the importance of moving your business toward incrementality: 3 Act on what you ve learned Stephan McBride, Netflix Our general approach is to be transparent. Internally, we make our stakeholders understand that, over time, experiments and measurement on platforms whose experiments are trustworthy can make marketing better. We invest in testing so we can learn and get better. Alok Gupta, Lyft One piece of advice I would give to anyone starting on this path: Don t go into it for the sake of going into it. Keep pushing, prodding and asking questions until you re convinced you have to do it or you won t be able to sleep at night. This mindset of getting to the truth is the mindset behind experimentation. Matthew Gerrie, Booking.com For me, it s a purely financial decision. If I m a business with a limited marketing budget, I want to make sure my marketing dollars are being used in the most powerful way possible. For me, the only way of knowing that is through incrementality. I want to know that my marketing dollars are spent in the absolute best way possible, and the only way to do that currently is through experiments or tight attributional control. Measure Marketing Effectiveness 18

19 Contributors Alex Esber Product Marketing, Measurement, Facebook Alok Gupta Director of Data Science, Head of Marketing Science, Lyft Carolyn Bao Product Marketing, Measurement, Facebook Catherine Oddenino Advertising Insights Marketing Strategist, Facebook Jesse Goranson Director of Client Measurement, Facebook Maggie Burke Client Council Lead, Facebook Matthew Gerrie Senior Director of Marketing Science & Communication, Booking.com Sarah Hartnett Product Marketing Communications, Facebook Sophia Lin Project Manager, Facebook Stephan McBride Director of Science and Analytics, Marketing and Economics, Netflix Tony Flanery-Rye Senior Director of Growth Analytics, ebay