Authors: Aaron Lee & Meredith Lind

Transcription

1 A Practical Look at Alternatives To Traditional Importance Measures Authors: Aaron Lee & Meredith Lind R ESEARCH CURRE N T S

2 Table of Contents Why Look for Alternatives to Traditional Importance Measures? So What Are Our Options? What is MaxDiff? Strengths of MaxDiff What is QSort? Strengths of QSort Comparing MaxDiff to Other Approaches Example of MaxDiff Results Example of MaxDiff Segmentation Comparing MaxDiff and 9-point Scale Mean Results Comparing MaxDiff and 9-point Scale Top 3 Box Comparing MaxDiff and 9-point Scale Ordering Comparing MaxDiff and QSort Comparing MaxDiff and QSort Top 5 Rankings Timing Considerations How Do They Compare? Summary of Pros/Cons of Each Approach In What Circumstances is MaxDiff Most Appropriate? About the Authors Contact Us

3 Why Look for Alternatives to Traditional Importance Measures? Importance Measures in Market Research: Measuring the importance of a set of attributes is a common goal of many market research studies, including purchase driver/buyer behavior studies, segmentation, customer satisfaction/ loyalty, brand image, etc. In particular, importance measures are often used to help marketers assess which product features or benefits to emphasize in messaging and communications, or product planners who need to decide which features must be included in a new product. Often these types of metrics will feed into driver modeling analyses as predictor variables, in order to help understand the impact that product feature has on the purchase decision (i.e., feature X has X amount of impact on the purchase decision, compared to features Y and Z). Driver modeling provides an important research output that helps companies make key marketing and product-related decisions, when faced with a variety of possible features or benefits available from which to choose. However, this analysis is only as effective as the data it is based on. Which leads to our key discussion topic at hand. Standard ratings have limitations: Most researchers are aware of the relatively poor job that stated importance ratings do of differentiating between options, items, categories, etc. Without any constraints or trade-offs to make, respondents tend to rate everything pretty high in importance, effectively truncating the scale and making it difficult to isolate the top drivers. When we find out that everything is important what do we do with that information and how do we use this to understand how to better develop and market products? 9-point Scale Mean Results Item C Item A Item G Item B Self-administered online surveys pose unique challenges: With most quantitative research shifting to web survey methodology, researchers are noticing more and more problematic data issues related to self-administration: Straight Lining: Speeding Inconsistent Answers: I m 24 and retired ; I don t own a car I get my oil changed every 6 months Inattention and respondent fatigue combined with professional gamers have led to careless attribute ratings further decreasing observed differentiation in scale ratings. YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 3

4 So What Are Our Options? We can take action to try to improve the quality of our rating scale data by eliminating straight liners and so forth. But are finding the % of our sample we d have to exclude is continually increasing contributing to increased costs for conducting market research. Or, we ask the questions in different ways that can: 1. Force respondents to differentiate among the various options 2. Use set-ups that require them to be more active, and discourages the snoozethrough radio button response pattern This presentation is an account of our experience testing a few alternatives to rating scales. What is MaxDiff? MaxDiff is a technique (available via Sawtooth Software) where respondents are essentially given a trade-off exercise where they must choose their most and least preferred choices in a list of items (usually between 3 to 5). The process is then repeated for a variety of different possible combinations. Least Preferred Item 1 Item 2 Item 3 Item 4 Item 5 Most Preferred MaxDiff represents an extension of Thurstone s Law of Comparative Judgment (Thurstone 1927). It assumes that respondents evaluate all possible pairs within the displayed subset and choose the pair that reflects the maximum difference in preference or importance (Louviere 1993). 1 1 Sawtooth Software Technical Paper Series: The MaxDiff/Web System Technical Paper, YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 4

5 Strengths of MaxDiff Humans are much better at judging items at extremes than in discriminating among items of middling importance or preference (Louviere 1993). 1 This makes it easier for respondents to answer, versus choosing from among points on a scale for a number of features/attributes that are judged to be about the same. The approach also avoids possible scale use biases that can be observed with rating scales, where we may find that some people systematically use the scale in a different way than others. Generates utility values at individual respondent level. Latent class and Hierarchical Bayes (HB) estimation methods extend analysis capabilities making it possible to estimate stable item scores from relatively sparse choice data. 1 With HB modeling, we can derive individual-level utility scores, even though respondents only evaluate a fraction of all possible subsets of items. With individual-level scores, we can apply common stat testing techniques. Additionally, we can submit these utility values to a cluster procedure to derive segments (using Latent Class). 1 Sawtooth Software Technical Paper Series: The MaxDiff/Web System Technical Paper, What is QSort? QSort is based on Q Methodology, which was originally developed by William Stephenson in the 1930s. 1 It is a subjective measure used to understand people s viewpoints on a set of items relative to one another. QSort can be implemented in several fashions the thoroughness of which depend on the goals of a study and the instrument time available for conducting the exercises. Note, what is done here does not involve sorting and ranking every item. A quick sorting of items by general importance Sort sub-groups until desired clarity achieved More Important Less Important More Important More Less The use of relative sorting, rather than asking subjects to rate their agreement with statements individually, is meant to capture the idea that people think about ideas in relation to other ideas, rather than in isolation. 1 YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 5

6 Strengths of QSort QSort s divide and conquer approach can be used with large item lists where asking respondents to individually rate every option would be very fatiguing. Depending on goals, QSort can be used to quickly identify the more interesting items from a large list which can then be followed up on with more detailed measures. The approach also avoids possible scale use biases that can be observed with rating scales, where we may find that some people systematically use the scale in a different way than others. Comparing MaxDiff to Other Approaches Case Study: Using MaxDiff vs other approaches to understand TV and computer viewing experiences To better understand how results can vary with different approaches, we decided to replicate a recent MaxDiff study we had done only this time use alternative rating scale and QSort methodologies. As a practical consideration, in addition to ratings and sorting, respondents also ranked their top five items in each approach: MaxDiff Rating Scale Ranking QSort Step 1 Choice of trade off exercise 9-point rating Rank top 5 of highest rated items Sort items into low, medium, high importance Top 5 Ranking Rank top 5 high importance items from step 1 MaxDiff originally done in full study with 1,800 respondents on software use and purchase motivations. Additional approaches tested in follow-up study of 300 respondents. YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 6

7 Example of MaxDiff Results Below are example results from a MaxDiff exercise. MaxDiff assesses the magnitude of difference between purchase drivers on a utility basis, with the total summing to 100% of available utility. It is important to remember that like other choice exercises, MaxDiff results are relative to the other items tested. Contribution to TV Viewing Pleasure Crisp/clear picture Uninterrupted/commercial free Free content Watch TV without distraction Watch what/when I want No buffering delays Large screen 7% 11% 11% 15% 14% 14% 23% An advantage of this approach is that we can conclude, for example, that crisp/clear picture makes 3x more contribution to TV viewing pleasure than large screen, and 11x more than getting online facts about show you re watching. Premium sound Surfing Web while watching TV Access from anywhere Interact w/others online while Own content Get info online about things see on Get online facts/history about Interact with TV program 4% 4% 3% 3% 2% 2% 2% 1% 10% 20% YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 7

8 Example of MaxDiff Segmentation MaxDiff can also provide effective segmentations with the differentiation achieved through a trade-off exercise often providing rich differences in segments. The segments below exhibit notably different patterns in TV viewing factor importance. Latent Class 3 Segment Solution Segment 1 33% of sample focused on factors 8, 1, Segment 2 29% of sample focused on factors 1, Segment 3 38% of sample focused on factors 2, 8, 4, 15, Factor #9 Factor #10 Factor #4 Factor #18 Factor #1 Factor #15 Factor #16 Factor #8 Factor #3 Factor #12 Factor #5 Factor #2 Factor #7 Factor #17 Factor #14 Factor #11 Factor #6 Factor #13 YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 8

9 Comparing MaxDiff and 9-point Scale Mean Results We first compare the 9-point scale mean results to those of the Max As illustrated below, the MaxDiff provides more observable trends than 9-point scale means. More tiers with clear shifts between each tier for MaxDiff A larger tier of top scores for mean scores Max Diff Results 9-point Scale Mean Results Crisp/clear picture Uninterrupted/commercial free Free content Watch TV without distraction Watch what/when I want No buffering delays Large screen 23% Tier 1 Crisp/clear picture % 14% 14% 11% 11% 7% Tier 2 Tier 3 Free content Uninterrupted/ No buffering delays Large screen Watch TV without Premium sound Tier 1 Premium sound Surfing Web while watching Access from anywhere Interact w/others online while Own content Get info online about things Get online facts/history about Interact with TV program 4% Watch what/when I want % Own content % Access from anywhere % Get info online about 4.75 Tier 4 2% 2% 2% More tiers and larger differences Surfing Web while Get online facts/history Interact w/ others online % Interact with TV program 3.85 Tier 2 Tier 3 YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 9

10 Comparing MaxDiff and 9-point Scale Top 3 Box We next compared the 9-point scale % top three box results to those of the MaxDiff. Top three box provides about the same differentiation as means; less than the MaxDiff. It is also difficult to see whether the top-three box middle tier is one large group or if there should be a break in the middle. Max Diff Results % Top 3 Box Scale Results Crisp/clear picture Uninterrupted/commercial free Free content Watch TV without distraction Watch what/when I want No buffering delays Large screen 23% Tier 1 Crisp/clear picture 79% 15% 14% 14% 11% Tier 2 Uninterrupted/commercial Free content No buffering delays Watch TV without 72% 71% 68% 62% 11% Tier 3 Large screen 60% 7% Watch what/when I want 57% Tier 2 Tier 1 Premium sound Surfing Web while watching TV Access from anywhere Interact w/others online while Own content 4% Premium sound 57% 4% 3% 3% Own content Access from anywhere Get info online about things 43% 42% 32% Tier 4 2% Get online facts/history 30% Tier 3 Get info online about things see on Get online facts/history about Interact with TV program 2% Surfing Web while 30% 2% Interact w/ others online 28% 1% Interact with TV program 22% Tier 4 YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 10

11 Comparing MaxDiff and 9-point Scale Ordering For the most part, the order of attributes is consistent between the two approaches; however, there are a few exceptions where the rank order varies by three or more positions. Rank MaxDiff Scale Crisp/clear picture 1 1 Uninterrupted/commercial free 2 3 Free content 3 2 Watch TV without distraction 4 6 Watch what/when I want 5 8 No buffering delays 6 4 Large screen 7 5 Premium sound 8 7 Surfing Web while watching TV 9 12 Access from anywhere Interact w/others online while watching Own content 12 9 Get info online about things seen on TV Get online facts/history about shows Interact with TV program YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 11

12 Comparing MaxDiff and QSort In our first QSort step, people grouped 18 items into two 9 item groups of more important and less important to their software purchase decision. While the tier differentiation is still not as good as MaxDiff, it is slightly better than what the rating scales yielded. Max Diff Results QSort % Contribute Greatly Crisp/clear picture Uninterrupted/commercial free Free content Watch TV without distraction Watch what/when I want No buffering delays Large screen 23% Tier 1 Crisp/clear picture 73% 15% 14% 14% 11% Tier 2 Free content No buffering delays Uninterrupted/commercial free Large screen 66% 57% 51% 49% 11% Tier 3 Watch what/when I want 48% 7% Watch TV without distraction 47% Tier 1 Tier 2 Premium sound 4% Premium sound 47% Surfing Web while watching TV Access from anywhere Interact w/others online while Own content Get info online about things see on Get online facts/history about Interact with TV program 4% 3% 3% Access from anywhere Own content Get online facts/history about 28% 24% 17% Tier 4 2% Get info online about things see 17% 2% Surfing Web while watching TV 15% 2% Interact with TV program 15% 1% Interact w/others online while 13% Tier 3 Tier 4 YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 12

13 Comparing MaxDiff and QSort Top 5 Rankings Comparing the QSort ranking with MaxDiff ordering, we see the top 5 ranking adds clarity. The QSort top 5 rank index is highly aligned with the MaxDiff results, only varying by more than 4 places on two occasions. Rank MaxDiff Scale Crisp/clear picture 1 1 Uninterrupted/commercial free 2 6 Free content 3 2 Watch TV without distraction 4 7 Watch what/when I want 5 4 No buffering delays 6 3 Large screen 7 5 Premium sound 8 8 Surfing Web while watching TV 9 10 Access from anywhere 10 9 Interact w/others online while watching Own content Get info online about things seen on TV Get online facts/history about shows Interact with TV program Scale rankings based on index calculation from top 5 rank results. YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 13

14 Timing Considerations While we have shown that the data from MaxDiff can be very powerful, we also need to consider how much room there is in the survey for gathering importance measures. Length of time to administer a 15 attribute model: 9-point scale rating: 1 to 2 minutes Limited QSort and top 5 ranking: 2 to 4 minutes MaxDiff: 4 to 6 minutes Clearly a practitioner needs to balance the need for highly differentiated data with the need to have room for other topics in the survey. How Do They Compare? We have seen consistently throughout our practical examination that the MaxDiff is the most differentiating, followed by rankings alone or combined with QSort, the QSort, and finally the scale ratings. To more scientifically compare performance, we indexed each approach to be on a 0 to 100 scale and analyzed their variances: Importance Measurement Approach Index score MaxDiff 14.9 QSort top 5 ranking point scale top 5 ranking 10.3 QSort point scale 1.1 In fact, MaxDiff yields 14x more differentiating power than scale ratings and roughly twice that of QSort. Top 5 rankings, however, do add notable improvements to variance. YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 14

15 Summary of Pros/Cons of Each Approach Traditional Rating/Ranking Pros: Provides both absolute and relative metrics Simple to program and execute/analyze Familiar to clients Relatively quick to administer Cons: Long list of attributes leads to inattention and less differentiated results on ratings; data quality may be compromised QSort Approach Pros: Easy for respondents to do with drag-and-drop Effectively breaks down the cognitive process of sorting through a long list of attributes improves data quality Avoids difference in scale use across respondents Relatively quick to administer Cons: Only provides relative measures, no absolute % s Limited in ability for more advanced analyses that can be run on either scale data or utility values, assuming a limited sort as conducted here (not a full set) Potential to over accentuate true differences by forcing assignment to groups MaxDiff Approach Pros: Provides relative order with magnitude of differences Results are clearly differentiated Provides utility values which can be analyzed for clusters Avoids difference in scale use across respondents Cons: Relatively long to administer Relatively difficult to program and analyze Burden on respondent is increased (length and tedium) need to limit the list of attributes May gloss over whether true differences exist by forcing respondents to choose between options YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 15

16 In What Circumstances is MaxDiff Most Appropriate? Use: We recommend using MaxDiff when your chief objective is to prioritize from amongst a list of possible choice drivers, and further understand the amount of impact each choice driver is likely to have on purchase decisions. Particularly valuable when using a list of attributes that have already been vetted and are known to be of value to respondents. Don t Use: If your primary objective is to understand in an absolute sense the percent of respondents that like a particular product or feature; for instance, if your client needs to provide a report card statistic or compare scores across studies. Use: If you have an interest in running advanced analyses on the data such as driver modeling or segmentation the results can be highly predictive and you can obtain better results than with scaled data (in cases where the scaled data fails to provide differentiation). Don t Use: If your list of attributes is too long (>20), the exercise becomes too laborious for respondents to provide good quality data. You d be better off using QSort. Don t Use: If your survey instrument is already pushing the limits of acceptability, including a relatively lengthy exercise is general not advisable. Be sure to include as close to the beginning of the survey as you can! YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 16

17 About the Authors Meredith Lind, Partner Meredith has over 17 years experience across a wide range of industries, with a focus on information technology over the past 10. Prior to YouGov Definitive Insights, she was a Principal at Momentum Market Intelligence and a Director at Gartner Custom Research and MORPACE International. She has been an MRA guest speaker. Meredith has a BA from The University of Michigan and earned an MBA with honors from Michigan State University. Aaron Lee, Partner Aaron has over 12 years experience with a heavy focus on information technology research, expert in both quantitative and qualitative approaches. Prior to YouGov Definitive Insights, he was a Director at Momentum Market Intelligence and a Senior Researcher at Gartner Custom Research. He has been an MRA guest speaker. Aaron holds a BA from the University of Oregon and an MBA from Willamette University. Contact Us Please contact us if you have any questions or interest in our research services: Meredith Lind meredith_lind@definitiveinsights.com phone: ext. 808 Aaron Lee aaron_lee@definitiveinsights.com phone: ext. 810 YouGov DefinitiveInsights A Practical Look at Alternatives To Traditional Importance Measures 17