The Value of Probing An in-depth study to demonstrate how interactive probing can bring life to open ends. This paper reviews the impact of interactive probing on word counts, prevalence of key qualitative themes, interview length and interview satisfaction. By: Andrea Richards, SVP
THE VALUE OF PROBING Abstract It s no secret that the value of open-ends has declined steadily as the industry moved to online research. Standard open-ends often yield generic, rational responses, and fail to provide the insight needed to help explain quantitative metrics. New methodologies, such as interactive probing software, have emerged to revive the value of open-ends. Interactive probing software offers a scalable solution to deliver probing to an entire quantitative sample, or to select respondents based upon key quantitative metrics. This paper explores the impact of incorporating interactive probing software by comparing three cells: Standard Likes and Dislikes open-ends, Probed Likes and Dislikes open-ends, and Probed Topic-Oriented open-ends. The following metrics are examined in this analysis: Prevalence of Key Themes Word Count Interview Length Survey Satisfaction Rating The findings of this analysis show that probing Topic-Oriented open-ends results in more frequent mention of emotional connections to the brand, as well as additional discussion on Shopping Channel, and the Impact of the Family s Preferences on brand decisions. Adding probing to standard open-ends results in additional detail surrounding key factors driving brand choice such as Taste, Health, Texture, and Price/Value. Word Count increases significantly from Standard to Probed, and from Probed to Topic-Oriented open-ends. Average Interview Length increases only slightly from Standard to Probed open-ends. Topic-Oriented open-ends add approximately 90 seconds to the average length of interview. Survey Satisfaction ratings are consistent across all three cells. Research Design Quester conducted interviews with 776 participants to explore snacking behaviors and understand the likes and dislikes associated with consumers favorite snacks. A Brand Exploratory interview was developed, incorporating a series of traditional quantitative metrics on purchase frequency, channel selection, and attribute ratings. In order to understand the impact of incorporating Quester probing, and Quester survey design, three cells were compared: Cell One: Standard Likes and Dislikes s, No Probing Cell Two: Standard Likes and Dislikes s, with Socrates Probing Cell Three: Quester Designed Topics, with Socrates Probing In Cell One and Cell Two, the questions were phrased as follows: Overall, what do you and your household like about [insert brand name]? What do you and your household dislike about [insert brand name], if anything? In Cell Three, the questions were written in topic form, and phrased as shown below: I want to make sure I have a good understanding of what your experience with this brand is like. So I would like you to tell me a story about your experience with this brand, one that is the best example of what it is like for you. Now I would like you to take a moment and think about your experience with this brand - if there is something that you could change about it to make it even better for you, what would that be...? Tell me in detail what specifically you would change or improve about this brand.
Study Results Interviews were analyzed by Quester s linguistic team. Key themes were identified by the analysts, and then quantified with Quester s proprietary analytics software, Aristotle. The full qualitative analysis is available for download on the Resources page of Quester s website. This paper will focus solely on the prevalence of the key themes, in an effort to illustrate the impact of the varying degrees of probing. The chart below illustrates the prevalence of the key themes across each cell. As shown, similar ideas are mentioned overall; however, the frequency with which those ideas are mentioned, as well as the detail with which those ideas are discussed, increases by incorporating probing. The most significant impact of the question phrasing ties to references to emotion. Consumers were more likely to offer unaided emotional connections to a brand when asked to share a story about their experience with it, rather than simply list their likes or dislikes. They were also more likely to discuss the impact of their family s preferences on snack choice, as well as selection of the channel shopped. The transcript below illustrates the flow of the conversation on an individual respondent basis. In this example, the idea of linking a brand back to one s childhood is revealed when using the story approach. I want to make sure I have a good understanding of what your experience with this brand is like. So I would like you to tell me a story about your experience with this brand, one that is the best example of what it is like for you. Please enter your response in the box below. Our online moderator will follow up with additional questions to make sure we fully understand your thoughts. I usually have great experiences with the Mars/Milky Way brand. The candy is usually gooey and soft like it should be, very rarely do I get a candy bar, that is stale. I love the candy because it reminds me of Halloween as a kid. You love the candy because it reminds you of Halloween as a kid, what does that mean to you? Well we always shared the treats that we got for Halloween between the kids in our family, often the chocolate was the first to disappear from the bowl and often I chose the milky way above all because I love the caramel
Probing Standard Open-Ends While probing standard open-ends did not yield the same level of emotional depth as the topic-oriented Story approach, it did add richness to the verbatims. Probing the Likes and Dislikes questions resulted in additional detail around Health, Price/Value, and Texture. The transcript below illustrates the additional detail collected to further define health benefits: Overall, what do you and your household like about this brand? Please be specific. Please enter your response in the box below. Our online moderator will follow up with additional questions to make sure we fully understand your thoughts. It is creamy, tastes good, healthy for you, and is reasonably priced. So, healthy, please tell me about that in even a little more detail. It is low in calories and fat, great for the digestive system. It also is a great source of calcium. Additional Interview Metrics Word Count: Probing open-ends results in significantly longer verbatim responses. Layering on the topic-oriented story approach results in an additional increase. A review of word counts, combining both open-ends in each cell, appears in the chart below: Word Count Story 57 Probed 38 Standard 9
Interview Length: Incorporating the Story approach added approximately 90 seconds to the average interview length. Interviews with Standard open-ends averaged 11.28 minutes. Interviews with Probed open-ends averaged 11.56 minutes. Interviews with the Story approach averaged 12.86 minutes. Interview Satisfaction: Incorporating the additional probing into the interview did not impact interview satisfaction scores. At the end of the interview, consumers were asked the following question: As surveys go, how would you rate this survey experience? Would you say you found it to be: 1) Extremely Enjoyable 2) Very Enjoyable 3) Somewhat Enjoyable 4) Not Very Enjoyable 5) Not At All Enjoyable Across the three cells, 75% and 78% of consumers rated the interview as Extremely or Very Enjoyable, as shown in the chart below: Interview Rating Not At All Enjoyable Not Very Enjoyable Somewhat Enjoyable Very Enjoyable Extremely Enjoyable 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% 45.0% 50.0% Extremely Very Somewhat Not Very Not At All Enjoyable Enjoyable Enjoyable Enjoyable Enjoyable Story 36.7% 39.0% 21.2% 3.1% 0.0% Probed 32.0% 46.7% 18.5% 2.3% 0.4% Standard 36.4% 41.9% 19.0% 1.9% 0.8%
In Summary Traditional open-ends are used in the hopes of gaining qualitative rationale for key quantitative metrics, yet often fail to reveal more than a few words, and a list of fragmented generalities. These findings demonstrate that adding a single layer of probing to these standard open-ends results in additional detail to aid in identifying secondary factors that drive brand loyalty. Adding this layer of probing helps to uncover the rationale behind issues such as taste, health, and variety, resulting in a more thorough understanding of the consumer mindset. Incorporating a topic-oriented approach alongside interactive probing reveals deeper insights not obtained using standard methodologies. In addition to providing further definition of general concepts, applying this topic-oriented approach leads to a greater understanding of the emotional connections to the brand. As stated by Brandt & Markowitz in their March, 2012 CASRO paper, Emotions are a key to success. They are what make the consumers sit up and pay attention. They make the idea sticky and retained by the consumer, and eventually, they drive behavior. Capitalizing on these emotional drivers increases the motivational capabilities of the messaging, and ultimately strengthens loyalty to the brand. About Quester Since 2001, Quester has leveraged our online software moderator, Socrates, to conduct in-depth qualitative one-on-one interviews, which allows for large sample sizes and further geographical reach versus traditional methods. For over 30 years Quester has been grounded in linguistic analysis, utilizing proprietary software and trained linguistic analysts to uncover the themes, ideas and nuances in consumer language and translate them into actionable insights. Quester s services can be deployed directly or through partners, within quantitative surveys, to enhance standard open-ended question.