Pre-test advertising proposing a new validity project.

Transcription

1 Pre-test advertising proposing a new validity project. Rachel Kennedy, University of South Australia Haydn Northover, University of South Australia Jane Leighton, Mountainview Learning Geoff Bird, Birkbeck, University of London Sebastien Lion, Mars Inc. Abstract Advertising can be pre-tested in many varied ways ranging from expert panels to consumer tests (based on verbal response; implicit or explicit measures; and physiological measurement such as galvic skin and fmri). We introduce key alternatives and outline an approach to validate how competing pre-test approaches perform in identifying sales effective advertising. We briefly introduce our pilot study that compares the performance of different pre-test approaches on a sample of 7 ads with known sales differences. We conclude that there appears to be variation in the predictive ability of the approaches but stress that further validation of pretesting is required. Keywords: Advertising, pre-testing, copy testing, validation, single source Track: Advertising, Promotion and Marketing Communication

2 1. Introduction Advertising pre-testing (copy-testing) is market research that evaluates advertising before it goes to air (often before production). It is an activity that generates controversy. There is a lack of consensus amongst academics and practitioners whether such testing can shed light on the sales success of executions in-market. Amongst those supportive of pre-testing, the controversy concerns how to test (i.e. which approaches and measures are most effective). Global advertising expenditure represents about 2% of world GDP (Nayaradou, 2006). While the recent economic downturn will temporarily reduce spend, advertising continues to represent a considerable expenditure for firms. But not all campaigns are effective at driving sales (Jones, 1995a). Further to this, it is acknowledged that copy quality is more important to effectiveness than media weight (Tellis, 2009) magnifying the importance of selecting the best ad(s) at an early stage. With ad expenditure this significant - and pre-testing an opportunity to minimise wastage through the elimination of inefficient campaigns - there is a need for quality pre-testing. But the question remains as to what is a quality pre-test. 2. Pre-testing and past validations Pre-testing has been conducted in many varied ways ranging from expert panels to more typically, consumer tests. Expert panels evaluate ads against set criteria such as the level of branding, consistency with past campaigns and the potential of the ad to capture attention. They can take the form of a Management Judgement Ad Test where the management team confirms the ad meets the creative brief (Rossiter & Bellman, 2005). They represent a very different approach to pre-testing that is potentially quick and cost effective. Consumer tests in contrast have been mostly based on verbal response to measures of persuasion, awareness, recall, communications, ad liking, and ad diagnostics; but newer consumer tests tap into a range of physiological measures such as galvanic skin, brain wave and heart rate measures, just to mention a few. Testing has ranged from: on air testing (i.e. single exposure, split cable) to forced exposure (online, in theatres and laboratories). The options are vast. Practitioners have typically collected reactions to ads from category users and judged effectiveness by comparing the results to past tests (via norms or benchmarks) or against a competing execution. A successful ad is one that is at or above norm, or scores more than competing executions. While pre-testing is effective at distinguishing if the execution is better than norm or a competing execution for the measures, there has been little work to determine whether pre-testing can determine how sales effective the execution will be in-market. Yet the ultimate goal of brand advertising is to provoke a response in-market. This might be an immediate sale (as with direct response ads), a delayed sale some time after exposure (for a car or chocolate purchase). Nearly 30 years ago it was already known that in copy testing, money was wasted on techniques of unknown reliability and that there appears an unusual degree of practitioner apathy regarding the quality of methodologies used (Ostlund, Clancy, & Sapra, 1980). Unfortunately not much has changed. Ideally a pre-test should be valid (predictive of in-market sales), reliable (close agreement between tests and retests of ads on the method), and sensitive (able to discriminate between ads where true differences in effectiveness exist).

3 Unfortunately few studies have independently validated the effectiveness of competing pre-test measures or approaches (exceptions include Haley and Baldinger 1991; (Faulkner & Kennedy, 2008; Haley & Baldinger, 2000). Even testing the same ad twice (by the same or a different method) has been uncommon (Ostlund, et al., 1980). Many factors like controlling the testing environment; matching respondent characteristics; timing issues and measurement error make validation studies complex. But a key reason for the lack of validation studies is the difficulty in determining what success looks like. From the marketing perspective, success is measured in profitable sales, but advertising plays only a part in the mix, hence linking specific ads to sales has been difficult. Confounding factors such as distribution, price, and competitor activity play a major role but can t be controlled easily. We believe that the availability of Single Source data presents a new opportunity for pre-test validations that over come some of the past hurdles. 3. Our proposed validation and the role of Single Source Our proposed pre-test validation study will build on the ARF Copy Research Validity Project (Haley & Baldinger, 1991) as this is the most signification pre-test validation to date. In the 80/90s the ARF devoted considerable resources and years of effort to attempt to validate the commonly used copy tests in the USA. Their methodology tested variations of pre-tests including on-air versus off-air, recruited versus self-selected audiences, in-program versus naked contexts, and single versus multiple exposures. Their key objective was to see how well competing copy tests could identify known sales winners from pairs of ads for specific brands. We replicate many features of their design including a focus on 30-second ads for established brands, in packaged good markets, that predominantly use television advertising. But being 20 years later we wish to include competing approaches such as expert systems; and implicit recognition and functional Magnetic Resonance Imaging (fmri), which hold the potential to measure very different things. Recent developments in behavioural sciences have changed our understanding of how the brain processes and stores information and how we make decisions (Damasio, 1994). This knowledge clashes with traditional advertising theories and pre-testing approaches. We acknowledge that our competing approaches are likely to measure different things, but we want to see what best predicts sales. Our initial focus is European advertising for global brands owned by Mars Inc across a range of varied categories. We have the benefit of Single Source data for identifying ads of varying in-market sales performance. Single Source data collects data for individuals (or household) on both advertising exposure and brand buying (Abraham & Lodish, 1990; Jones, 1995b; McDonald, 2000). It is the gold standard data to allow marketers to identify sales effective advertising (and control for factors such as price offers). The ARF saw value in looking at databases of past pre-tests. For our ads we have access to such databases for standard verbal response consumer testing and expert testing of key elements of the ads (i.e. measuring ads emotion, memory and attention). For other approaches, we need to test the ads with the different approaches and measures. Our studies initial focus is with pre-testing measures and approaches that are publicly available and transparent i.e. likeability or implicit recognition, as opposed to proprietary tools, such as Millward Brown s LINK test. We have selected some of the most discussed and used measures, in addition to new approaches. These include:

4 1. Likeability (5-point scale): This is the measure as used in the ARF s Copy Research Validity Project (Haley & Baldinger, 1991). 2. Explicit recall and recognition: (Recall What do you remember from the adverts that you watched two days ago? and What products / brands did you see advertised?; Recognition Which of the following products / brands did you see adverts for? ). The effectiveness of recall as a pre-testing metric is much debated with the opinion of Jones reminiscent of many detractors, the inability of pre-testing based on recall to predict marketplace effectiveness was demonstrated as long ago as 1964 (he was referring to Hoskin s study Factual recall as a measure of advertising effectiveness ). This is despite day-after-recall remaining a common measure in the US (Kastenholz, Kerr, & Young, 2004). Its inclusion is therefore important due to sheer use. 3. Persuasion (5 point scale) scores were also collected for the test ads. 4. Self-reported attention, memory, engagement and word of mouth (Memory = How likely are you to remember this ad ; Attention = How likely is it that this ad would grab your attention? ; Engagement = If you saw the beginning of this ad, how likely is it that you would want to watch it to the end? ; WOM How likely are you to tell friends and family about this ad? ). 5. Emotions (self-reported, 60 item scale - Panas-X). Panas-X assesses all aspects of emotion, and has been validated across numerous psychology studies (Watson & Clark, 1999). 6. Implicit recognition and accuracy. Participants were shown brand and pack images and asked to choose which one they recognised as being the ad they were exposed to in the test. Accuracy and response time were recorded. This technique is a proxy to assess the strength of connection produced by the ad in the memory of the respondent. 7. fmri. Spurning the self-report nature of much of the above, physiological measures of emotion and engagement have grown in recent years. While intuitively appealing for their unbiased response, little work has been done to adequately test: 1. Whether the responses are different, and 2. What they are measuring. This latter complaint is commonly directed at neurological practice, which is still a comparatively young field, while some are more dismissive simply feeling it s a technique in search of a solution (Cramphorn, 2004). Perhaps the most discussed neuro-physiological technique is the functional Magnetic Resonance Imaging (fmri) with over 150,000 peer-reviewed papers published as of 2005 (Quartz and Asp 2005). The power of fmri is that it can observe levels of blood flow to specific regions of the brain. Importantly, work is being conducted that maps the brain response, as observed in an fmri, to a range of advertising metrics including attention, emotion and memory (Quartz & Asp, 2005) (Quartz and Asp 2005). For our investigation, we believe activity in three sites is relevant. The hippocampus as it is considered the primary memory system for humans, the parietal for its association with attention and the amygdala for its association with emotion (Phelps, 2004). 3. Pilot study An index of the sales impact of relevant ads was calculated from the single source data. The index is essentially the sales differentials from people exposed to the ad compared to sales

5 amongst those not exposed with controls for confounding factors such as promotions. Seven ads with varying sales effectiveness were identified for the initial pilot. The verbal response consumer tests and expert tests conducted prior to their airing were tagged in the database. Four hundred category users aged between years were recruited for alternative pre-testing on the target ads. A two-phased design was employed; respondents were sent a DVD and instructed to watch the program. The test ads were embedded in the material but respondents were not alerted to the research interest in the advertising. Two days after exposure, respondents were contacted and completed a questionnaire. Twelve respondents were recruited for the fmri component of the study with each viewing up to 16 commercials. 4. Pilot Results These initial results only focus on 7 ads with known sales performance; hence we are not making claims about validity of the different approaches. Instead we seek feedback on the research methodology before we scale this important project. The following outlines the rank order of correlations between each pre-test measure, and the sales index for the tested ads: Measure Correlation Measure Correlation order order Memory (Hippocampus) 1 st Positive emotion (Panas-X) 8 th Reaction time (Implicit) 2 nd Aroused emotion (Panas-X) 9 th Attention (Parietal) 3 rd Word of Mouth (self-reported) 10 th Memory (self-reported) 4 th Engagement (self-reported) 11 th Happy emotion (Panas-X) 5 th Recall 12 th Emotion (Amygdala) 6 th Attention (self-reported) 13 th Accuracy (Implicit recognition) 7 th Likeability 14 th Recognition The initial results indicated that activity in the hippocampus, while watching ads, had the highest correlation to sales effectiveness (ranked 1 st amongst the 15 measures). The Parietal activity also achieved a high correlation (3 rd ). Of the non-neurological measures, Reaction time achieved the highest correlation (2 nd ) suggesting the importance of quick subsequent recognition. Amongst the other measures, self-reported memory ( How likely are you to remember the ad? ) and Happy emotion also demonstrate high correlations. Recall (12 th ), recognition (15 th ) and likeability (14 th ) achieved low correlation to sales. 1. Recognition of the 7 ads was generally high (from 41%-73%) with the correlation suggesting no apparent pattern between it and sales effectiveness. This is perhaps not surprising as recognition can be influenced by many factors, and has long ago proved unhelpful in determining sales effectiveness. Recall is only slightly more correlated to the sales index, but again is seen amongst many as unrelated to sales (Jones, 1986; Kuse, 1991; Lodish, et al., 1995). 15 th

6 2. Likeability: While likeability was important in the ARF s seminal study, it was poorly correlated with sales success here. 3. Self-reported emotions: Specifically a happy response to the advertising achieved the highest correlation as opposed to a generally positive or aroused response. Emotion appears important, but not as an end in itself. It appears that any emotion measure must capture the brand linked emotion, and not just emotion for its own sake. 4. Self-reported memory, attention, engagement and WOM: Interestingly, asking a respondent how likely they were to remember an ad demonstrated a higher correlation to sales (4 th ) than their actual recall of that ad (12 th ). Potentially a positive response to this question captures more of the processing of the execution than the actual recall itself. 5. Implicit memory: the speed with which a respondent recognizes the brand logo / packaging of an ad seen 2 days previously had a strong negative relationship with sales effectiveness (2 nd ). That is, respondents are quicker to recognise ads that are sales effective. 6. fmri: The early fmri results suggest that memory is important (i.e. engaging the hippocampus), and it has been the metrics used in the past that have been the issue (i.e. recall). Much has also been made of the importance of emotion in decision-making; and hence in advertising. These early results are consistent with the importance of emotion, but only when memory is engaged (i.e. emotion linked to the brand, rather than emotion for its own sake). 5. Summary and future research This study reports pilot results, and should be viewed as such. Despite this, the correlation between both the hippocampus and reaction time to sales effectiveness is exciting. The reaction time result is particularly interesting considering its potential use in multiple research platforms, particularly Computer Assisted Personal Interviewing (CAPI) and online. It could be scaleable, cost effective and deserves further research. The core strength of the CRVP was that it tested ads for the same brand (one was deemed a winner and one a loser ). Such a process removes some issues that our study has not. Key to this is the comparability of sales indexes between brands of different sizes. A correcting calculation can be applied, but the pairs of ads remove the need. In future phases we aim to collect pairs of ads for the same brand. Perhaps not surprisingly finding pairs of ads for the same brand with vastly different sales response is not an easy task. Continuing use of fmri carries with it a substantial cost (although if it confidently picked sales effective advertising, this cost may be covered by improved sales). Cost is however undeniably an issue, and therefore its application in commercial marketing may be limited in the short to medium term. Should the fmri results hold up upon further testing, it will be essential to determine suitable substitutes that may be more price competitive. Such options may include skin conductance, which we are currently testing, heart rate, Electromyography (EMG), Electroencephalography (EEG) in addition to implicit memory type techniques that observe behavioural responses rather than attitudinal.

7 References Abraham, M. M., & Lodish, L. M. (1990). Getting the Most Out of Advertising and Promotion. Harvard Business Review(May-June), Cramphorn, S. (2004). What Advertising Testing Might Have Been, If We Had Only Known. Journal of Advertising Research, 44(2), Damasio, A. R. (1994). Descartes' Error: Emotion, Reason, and the Human Brain. New York: Putnam. Faulkner, M., & Kennedy, R. (2008). A new tool for pre-testing direct mail. International Journal of Market Research, 50(4), Haley, R. I., & Baldinger, A. L. (1991). The ARF Copy Research Validity Project. Journal of Advertising Research, 31(2), Haley, R. I., & Baldinger, A. L. (2000). The ARF Copy Research Validity Project. Journal of Advertising Research, December/January, Jones, J. P. (1986). What's in a Name? Advertising and the Concept of Brands. Massachusetts: Lexington Books. Jones, J. P. (1995a). Advertising Exposure Effects Under a Microscope. ADMAP(February), Jones, J. P. (1995b). We have a breakthrough: single-source data is the key to proving advertising's short term effects. ADMAP(June), Kastenholz, J., Kerr, G., & Young, C. (2004). FOCUS AND FIT. Marketing Research, 16(1), Kuse, A. R. (1991). Measurement Tools for Ads that Sell. In Copy Research The New Evidence - Eighth Annual ARF Copy Research Workshop (pp ). New York: Advertising Research Foundation. Lodish, L. M., Abraham, M., Kalmenson, S., Livelsberger, J., Lubetkin, B., Richardson, B., & Stevens, M. E. (1995). How T.V. Advertising Works: A Meta-Analysis of 389 Real World Split Cable T.V. Advertising Experiments. Journal of Marketing Research, 32(May), McDonald, C. (2000). How single-source data research first developed. In J. P. Jones (Ed.), International Advertising - realities and myths (pp. Chapter 19): Sage Publications. Nayaradou, M. (2006). Advertising and economic growth. University of Paris, Dauphine. Ostlund, L. E., Clancy, K. J., & Sapra, R. (1980). Inertia in Copy Research. Journal of Advertising Research, 20(1), Phelps, E. A. (2004). Human emotion and memory: interactions of the amygdala and hippocampal complex. Current Opinion in Neurobiology, 14(2), Quartz, S., & Asp, A. (2005). Brain branding brands on the brain. In Annual Congress (pp. 1-13). Cannes: California Institute of Technology. Rossiter, J. R., & Bellman, S. (2005). Marketing Communications: Theory and Applications. Frenchs Forest: Pearson Education. Tellis, G. J. (2009). Generalizations about advertising effectiveness in markets. Journal of Advertising Research, 49(2), Watson, D., & Clark, L. A. (1999). The PANAS-X: manual for the positive affect and negative affect schedule - expanded form. University of Iowa, Iowa City.