Software for Typing MaxDiff Respondents Copyright Sawtooth Software, 2009 (3/16/09)

Transcription

1 Software for Typing MaxDiff Respondents Copyright Sawtooth Software, 2009 (3/16/09) Background: Market Segmentation is a pervasive concept within market research, which involves partitioning customers into homogeneous segments, with the expectation that different product offerings or marketing actions will be effective for each segment. The most common procedure for developing market segments is to administer a questionnaire to obtain information about customers needs and desires, and then to analyze the resulting information with some type of cluster or partitioning analysis. In recent years MaxDiff questionnaires have become increasingly popular for this purpose. MaxDiff presents items in sets (typically 12 to 18 sets) and asks respondents to choose the most and least desirable item in each set. Because respondents are forced to discriminate among items, MaxDiff responses usually contain more information than questionnaires using conventional rating scales. Importantly, they are also free from scale use bias. After the initial sample of respondents has been partitioned into segments, the need often arises to classify additional respondents into the same segments. The researcher could administer the same questionnaire to a new sample of respondents and repeat the partitioning analysis, but the resulting segments might be different from the earlier ones. Also, if the objective is just to classify respondents rather than to define segments, a shorter questionnaire (a typing tool ) might be used with less cost to the researcher and less inconvenience for the respondents. With conventional rating scales, discriminant analysis or tree-based methods can identify the subset of items to include in a typing questionnaire. However, developing typing questionnaires for MaxDiff is more complicated because it involves not only deciding which items to include, but also how to place those items within efficient MaxDiff sets. There is also the issue of how to classify new respondents (perhaps on-the-fly) based on MaxDiff answers, where the math is not so well-known as applying a discriminant function or simple if-then (tree-based) logic. For these reasons, many researchers have avoided using MaxDiff questionnaires for developing segmentation schemes if they expect that a typing tool eventually will be required. The good news is that if the researcher has used MaxDiff for the original questionnaire, it may be possible to construct a shorter questionnaire which is even more efficient than the original questionnaire at classifying respondents into segments. This is because MaxDiff presents items in sets. With, say, 50 items presented in sets of five, there are more than two million possible sets of items, and the most potentially useful sets may not have appeared in the original questionnaire. 1

2 The software described here solves this problem. Assume that a MaxDiff questionnaire has been administered to an initial sample of respondents, who have also been partitioned into segments of interest to the researcher. This software searches for an abbreviated MaxDiff questionnaire drawn from the original items, but potentially with combinations of items not represented within the original questionnaire, which is most efficient at classifying new respondents into the original segments. We also demonstrate the math for classifying respondents using the MaxDiff responses, which either can be done after data collection or on-the-fly using Sawtooth Software s SSI Web interviewing system or potentially another third-party application. Applying Bayesian Logic We use a logical framework similar to that of Latent Class analysis. Latent Class analysis assumes that there are a specified number of classes of respondents and that those in each class have identical needs and desires with respect to the subject matter being studied. Respondents do not belong unambiguously to any class, but rather each respondent has some probability of belonging to every class. Because MaxDiff analysis involves choices from sets of items, it is convenient to use logit analysis to estimate the underlying utility values thought to govern respondent answers. With the use of hierarchical Bayes (HB) analysis we can obtain a set of utilities indicating how each respondent values each item. Then, for any arbitrary grouping of items (not just those sets in the original questionnaire), we can estimate the probability with which each respondent would choose each item in that set as best, and as worst. Latent Class analysis uses a Bayesian model to predict each respondent s answers to a questionnaire. Even if we have not used Latent Class analysis to define segments, we can use the same model for classifying respondents into segments. We can average the utilities for respondents in each segment to produce a set of average utilities for the segment. 1 Using a logit model, we can calculate the likelihood that any pattern of questionnaire answers will occur, given that a respondent belongs to that segment. The relative sizes of the segments function as priors in our analysis. If we multiply the likelihoods (of the answers, for a respondent belonging to each segment) by the priors (relative segment sizes) we obtain relative posterior probabilities of the respondent belonging to each segment. We then classify the respondent into the segment with highest probability. We already know segment membership of each respondent in the initial sample. The computation described in the previous paragraph shows us the segment into which each respondent should be classified, based on predicted answers to any MaxDiff questionnaire involving those same items. The optimization problem then becomes one of finding the questionnaire which maximizes the number of correct classifications. 1 Averaging the individual utilities for members of a segment provides an approximation to the segment s utilities, but better estimates are obtained by computing aggregate logit estimates of the segment s utilities. The results reported in this paper are based on simple averages of HB utilities, but below we provide estimates of the improvement expected from using logit analysis instead of simple averages. 2

3 Our Search Strategy Suppose there are 50 items and we want to find the best questionnaire containing five sets, each consisting of five items. There are more than two million ways to choose just one set of 5 items. The number of possible questionnaires containing five such questions is greater than the number 4 followed by 30 zeroes. So it s clear that we can t just try all of the possibilities and pick the best one. We have adopted a simple search strategy, which turns out to be effective. We start by specifying the size of the desired questionnaire, in terms of the number of sets and the number of items per set. Suppose there are to be five sets of size five. We construct an arbitrary starting questionnaire by assigning items randomly to those 25 positions, with the restriction that no item can appear more than once in the same set. We then cycle through all 25 questionnaire positions, exploring the result of exchanging the item in that position with each item not in that set. Consider the first item in the first set. With a total of 50 items, there would be 45 items not already in that set, and we would try exchanging all 45 of those items with the first item, retaining the one that does the best job in combination with the other items already in the questionnaire. Then we repeat the process, finding the optimal item for the second position in the first set, etc., cycling through all 25 positions in the questionnaire. We repeat the process, continuing until there are no further improvements. This procedure does not always find the global optimum, because with large questionnaire sizes there are many possible questionnaires that are nearly equal in effectiveness. But the questionnaires that it does select are nearly optimal. The computation is accomplished in a few seconds, so it is reasonable to run it many times from different random starts and to accept the best solution. Computing Likelihoods We now consider how to compute the likelihood that members of a segment will make choices like those made by a particular respondent. Although MaxDiff questionnaires ask the respondent to choose both the best and the worst item in each set, we first consider just the choice of best. Given logit utilities for a segment, we can estimate the probability with which a member of that segment should choose each item as best. This is done by exponentiating the segment s utilities for each item in the set, and then rescaling the exponentiated values to have sum of unity. An example of this computation is provided in the table below, resulting in probabilities of.678 of a member of this segment choosing item A,.298 of choosing B, and.024 of choosing C. Probabilities of Selection of Each Item Average Exponen- Segment tiated Proba- 3

4 Item Utility Value bility A B C Sum We also have individual respondent utilities for each item, from the MaxDiff questionnaire that the respondent answered. With a similar computation using those utilities, we can estimate the probabilities that each respondent should choose item A, B, and C. We now need to combine the respondent s probabilities of choice with the segment s probabilities to estimate the likelihood that a member of this segment would choose as this respondent is predicted to choose. Suppose the predicted probabilities for this respondent s choice are.600 for A,.200 for B, and.200 for C. The probability of the segment and respondent both choosing A are.678 *.600. The probability of the segment and respondent both choosing B are.298 *.200, and the probability of both choosing C are.024 *.200. In those three cases the respondent chooses like the segment, and in all other cases the respondent chooses unlike the segment. Thus, the probability of this respondent making choices like those of this segment is the sum of joint probabilities:.678 * * *.200 =.471. Now consider choice of worst. By changing the signs of all utilities we can make similar computations for the probabilities of the segment and the respondent choosing each item as worst, and by a similar sum of joint probabilities we can estimate the likelihood that the respondent would make the same choice for worst as the segment. In the interest of simplicity we treat these choices of best and worst as independent, and therefore the likelihood of their both occurring is computed as the product of the two probabilities. 2 If the respondent has answered several item sets, then a similar likelihood can be computed for each set. If the answers for different sets are independent (an assumption market researchers usually make), then the likelihood of the entire pattern of answers is obtained by multiplying together the likelihoods for all the sets. This likelihood is the probability of the data (choices), given membership in a particular segment. We can make a similar computation under the assumption that the respondent belongs to each other segment. But what we really want is the probability of the 2 These two events are not really independent because once a respondent has picked A as best, item A is no longer available for choice of worst. These choices could be modeled as independent by deleting the item picked as best from the consideration set for the choice of worst. We have compared the results of doing that vs. simply ignoring the dependence among the two choices, and have found that the results are nearly identical. Therefore in this software we treat the two choices as independent of one another. 4

5 respondent belonging to each segment, given the data. Fortunately, Bayes theorem tells us how to do that. If we multiply the likelihoods of the answers, given membership in each segment, by the segment s relative sizes, the resulting products are proportional to probabilities of a respondent with those choices actually belonging to each segment. If we have those probabilities, we can assign each respondent to the segment to which he or she has highest probability of belonging. What to Optimize Let s restate our goal. We have a sample of respondents with known segment membership, who have responded to a MaxDiff questionnaire, and their answers have been processed into item utilities using HB (or a similar method that results in respondent-level, logit-scaled utilities). We want to construct a shorter questionnaire, based on a subset of the same items arranged in possibly different sets, that can be used to classify new respondents into the same segments. We plan to evaluate potential questionnaires by seeing how successful they are at classifying our current respondents into the proper segments. Our search strategy consists of starting with a randomly constructed questionnaire, and then systematically modifying it, trying to improve some measure of success. One measure of success is hit rate, the proportion of respondents who are classified into the correct segment. But hit rate is a poor choice as the objective of an optimization procedure. Hit rate isn t a smooth function. A respondent is either classified correctly or not. As a questionnaire is improved, an incorrectly classified respondent remains incorrect until some threshold is reached, at which point the respondent may become classified correctly. It would be better to have an objective function that is smooth rather than one that is step-shaped. Our procedure gives the user the choice of two objective functions. The one used most often is RLH ( root likelihood ), the geometric mean of respondents probabilities of being classified correctly. Recall that each respondent is assigned to the segment for which he or she has highest probability. If there were five segments and the respondent s highest probability were.201, we would not be at all confident of the assignment. If, on the other hand, the respondent s highest probability were.999, we would feel very confident. RLH is a smooth function, which measures the confidence we can have about the resulting classifications, and maximizing RLH also tends to maximize the hit rate. An alternate objective function is concerned with optimizing results for a particular segment (or segments). Suppose a researcher is particularly interested in minimizing the number of respondents erroneously classified into a particular segment. To do this, we consider the sum of those respondents probabilities of belonging to that segment who are truly members, and divide by the sum of all respondents probabilities of belonging to that segment. This is also a smooth function, and maximizing this objective should lead to fewer respondents being incorrectly classified into the segment of interest. 5

6 We turn now to a demonstration with real data of how effective this procedure is for finding shorter questionnaires which classify respondents effectively. An Example: A Political Segmentation In August 2008, we interviewed 800 respondents (US, 18+) using the E-Rewards panel. We collected the data for the purpose of illustrating the benefits of MaxDiff questioning and Cluster Ensemble Analysis for segmentation research, and we published an article in MRA s Alert Magazine featuring a six-group political segmentation. The MaxDiff questionnaire contrasted 25 statements describing various policy objectives. We use the same data here, but with a three-segment rather than a six-segment solution (for ease of illustration), and involving two demographic characteristics (gender and political party preference). To develop the three-group solution, we employed Cluster Ensemble Analysis, with some segmentation solutions in the ensemble based only on the 25 MaxDiff items, and some segmentations based solely on demographic characteristics. Thus, the consensus solution employed both aspects of the data, leading to segments that discriminated both on the attitudes and on the targetable characteristics. In addition to gender and party preference differences, the segments can be identified by the items on which members of each segment had highest utilities, expressed as deviations from the means for the entire sample. The items characterizing each segment most strongly are listed below, in order of their strength. (We offer segment labels for convenience, although we recognize that many readers will find them unsatisfying in some way.) Segment 1: Economic Populists (48% of sample; Male: 38%, Democratic 55%) Bring the troops home from Iraq Guarantee national health care and elder care program Reduce taxes for middle and lower income households Enact policies to solve housing/mortgage crisis Create a national jobs program Ensure the long-term health of Social Security Segment 2: Liberals (34% of sample; Male: 50%, Democratic 70%) Strengthen women s reproductive Right to choose Restrict carbon emissions to reduce global warming Restrict gun ownership Guarantee national health care and elder care program Increase worldwide humanitarian efforts Improve our relations / reputation with other countries Improve race relations Increase funding to help homeless / hungry 6

7 Segment 3: Conservatives (18% of sample; Male: 54%, Democratic 7%) Increase defense/military spending Increase spending in the war on terrorism Reduce illegal immigration Reduce illicit drug use Reduce our reliance on foreign oil imports Reduce corruption / Improve ethics in government Improve infrastructure such as roads and rails Reduce foreign trade All 800 respondents were used to search for a MaxDiff questionnaire which presented four sets of items, each containing four items, which would classify respondents optimally. This first computation did not use any demographic variables, and did not focus on any particular segments. A total of 100 replications were run from different random starting points. The average hit rate for those replications was.905, and the best was.929. The best replication is summarized in the table below, where the rows indicate true segment membership, and the columns indicate predicted segment membership. Attempting to Maximize Overall Hit Rate With Four Sets of Four Items Predicted Sum True Sum The values in the diagonal (in bold format) are hits and those off the diagonal are misses. The sum of diagonal elements is 743, for an overall hit rate of 743/800 =.929. The hit rate for any particular segment is equal to its diagonal element divided by its row sum. For example, Segment 3 has a hit rate of 262/269 =.974. From inspection of the attitude items characterizing each segment, it appears that segments 2 and 3 should be most different from one another, and that is confirmed by the zeros in those cells of the classification matrix above. Focusing on a Segment The classification into segments in the table above is good, and would be difficult to improve very much. However, suppose the researcher wants to identify members of segment 1, and wants as few people as possible to be identified incorrectly as belonging to segment 1. In the table above, 30 respondents who truly belong to segment 2 were 7

8 classified into segment 1, as were 7 respondents who truly belong to segment 3. The accuracy rate for segment 1 in this classification is 362 / 399 =.907. We can re-run the software using the alternative objective function in an attempt to minimize the off-diagonal elements in the first column. A new run with 100 replications was done with the alternative objective function, resulting in the table below: Attempting to Maximize Accuracy for Segment 1 With Four Sets of Four Items Predicted Sum True Sum Now there are only = 14 respondents misclassified into segment 1 rather than 37 as in the previous table, and the accuracy rate for segment 1 is 295 / 309 =.955 This has been accomplished at the cost of lowering the overall hit rate from.929 to 699 / 800 =.874. Using Auxiliary Variables Often the researcher has more information about respondents than just answers to MaxDiff questions. For example, in this political study we also recorded each respondent s gender and whether he or she was more aligned with Democrats or Republicans. Since this information is available at no cost, it makes sense to use it to improve our classification. This is done with a simple extension of the likelihood computation described above. Suppose that for a particular segment 40% of the respondents were male and 70% had said Democrat. We could use that information in the likelihood computation for a particular respondent by multiplying his or her likelihood by.4 if male or.6 if female, and by.7 if Democrat or.3 if Republican. This is an example of a naïve Bayes classifier, in which naïve refers to the assumption that all components of the likelihood computation are independent of one another. We expected the items in our questionnaire to be answered differently by Democrats and Republicans, so it would indeed be naïve to assume that we could also include those auxiliary variables in the likelihood computation without some risk of double-counting. However, the question of whether those variables can provide additional information can be answered empirically, as we shall see in the next section. Hit Rates for Different Conditions 8

9 Several questions come to mind that we may answer empirically: How well do the results hold up for a new sample of respondents? What is the value of including demographic variables? How would the hit rate vary with shorter or longer questionnaires? The following tables speak to these questions. For several questionnaire sizes, varying from 1 set of items up to 10 sets of items, we provide several hit rates. Each is based on 50 replications. For calibration we use all 800 respondents, and show both the average and the best hit rate for each set of 50 replications. For validation we choose approximately 400 respondents at random to search for the best questionnaire, and then compute the hit rate for the other half of the sample using that questionnaire. The sample is partitioned differently in each replication. For the Validation columns, likelihoods are computed differently than for the Calibration column. When designing a questionnaire, we want to make use of all available information, and for that reason we use the share of preference approach described earlier. However, the Validation column provides estimates of our success classifying new respondents on the basis of responses to the new abbreviated questionnaire and for whom we do not have HB estimates of utilities. For the Validation respondents, we simulate their answers to each question by adding Gumbel error to the point estimates of their utilities, which is similar to making a random draw from the posterior distribution of their utilities. Estimates of likelihood based on answers simulated this way are less precise, and as a result the Validation hit rates are lower. But we believe they should represent a good estimate of what would actually be obtained with a brief questionnaire in a new sample. We also show results for No Demos, and With Demos. The demographic variables were combined into a single four-level variable by crossing male/female with Democrat/Republican. Hit Rates for Questionnaires of Various Sizes Calibration Validation Avg Best Avg 1 Set of 4 Items No Demos Gender & Party Sets of 4 Items No Demos Gender & Party Sets of 4 Items No Demos Gender & Party Sets of 3 Items No Demos Gender & Party

10 4 Sets of 4 Items No Demos Gender & Party 5 Sets of 4 Items No Demos Gender & Party Sets of 4 Items No Demos Gender & Party Longer questionnaires always appear to help, although there are diminishing returns. Five sets of size four items are enough to achieve a hit rate more than 95% as high as that achieved by a questionnaire twice as long. To assess the value of adding the demographic variables it is most useful to look at the last column. The demographic variables appear to be somewhat harmful with very short questionnaires, somewhat useful with mid-length questionnaires, and of no value for longer questionnaires. We speculate that with very short questionnaires the demographic variables overpower the attitude variables, detracting from success, and that with the longest questionnaires classification is already so successful that they have little further to offer. For mid-length questionnaires with four or five choice tasks, it appears that the demographic variables are of some help. We included the data for four sets of three items to address the question of whether MaxDiff typing questionnaires could be used successfully in telephone interviewing, where smaller item sets are desirable. On average, four sets of three items work about as well as three sets of four items. Sample Size and Hit Rate We have used a second data set to explore the effects of different sample sizes. This data set had several thousand respondents who answered a MaxDiff questionnaire with 17 items. They were partitioned by Latent Class analysis into five segments. Here are calibration and validation hit rates, each averaged over at least 50 replications, for samples of various sizes: Average Hit Rates for Various Sample Sizes, with Five Segments, And Four Sets of Four Items Sample Calibration Validation Size Avg Avg

11 These hit rates are smaller than for the political segmentation. Classification is more difficult with more segments; because there are five rather than three segments, the expected hit rate due to chance alone is.200 rather than.333. Calibration hit rates decline slightly as sample size increases as would be expected, because with a smaller sample, the search algorithm can benefit from idiosyncratic aspects of the data. Indeed, with tiny sample sizes one might expect the calibration hit rate to approach unity. Validation hit rates appear insensitive to sample size. With many more replications they might be seen to rise slightly, but with only 50 replications the standard errors of these averages are large enough that we are unable to detect any meaningful trend. These hit rates are high enough to suggest that we need not be worried about failure to work with a new sample, so long as the typing questionnaire is developed using several hundred respondents. An Empirical Test with a Consumer Confidence Questionnaire After obtaining favorable results from re-analysis of existing data sets, we decided to conduct a study to see if we could verify those results with new data. This time, we used the context of consumer confidence (a deep recession had gripped the US and abroad throughout 2008, and was formally declared by economists in Q4 2008). The MaxDiff questionnaire included 30 items related to consumer confidence, such as unemployment rates, job security, interest rates, value of investments, real estate values, personal tax rates, affordability of food, etc., arranged in 18 choice tasks, each with 5 items. The data were collected using hotspex s Internet Panel. The sample consisted of about 700 Canadian respondents, age 18+. Our plan was to contact each respondent twice. The first wave of the study administered the full MaxDiff questionnaire. HB utilities from that questionnaire were used to partition respondents into five groups, each appearing to have a somewhat different mixture of concerns (we did not use any demographic variables in developing the segments). Those segments were used as a target to design an abbreviated MaxDiff questionnaire for classifying respondents. Here are results from analysis of data from the first wave: Split-Half Validation Hit Rates for First Wave of Consumer Confidence Questionnaire (No Demographics Involved) Number Validation 1 Validation 2 Of Tasks

12 The two columns in this table require explanation. Since the time that we conducted wave 1 of the study, our methodological thinking has advanced in two ways. We originally thought to simulate respondent choices in the questionnaire design phase by simply seeing which item had the highest or lowest utility (first choice rule). Choices simulated in that way produce less precise estimates than when likelihoods are estimated using shares of preference, as described above. We also originally planned to estimate segment utilities by simply averaging individual HB utilities of segment members, but we have since learned that there is considerable improvement to be had from using aggregate logit analysis to compute the segment probabilities of choice rather than simple averaging of HB utilities. The column headed Validation 1 describes split-half validity hit rates based on the simpler procedures we had planned to use at the outset. The column headed Validation 2 describes split-half validity hit rates based on improved methodology: likelihoods estimated by shares of preference, and segment utilities estimated by aggregate logit. We decided that the abbreviated questionnaire should contain six choice tasks, each with five items. Working from the results in the Validation 1 column (our best work at that time), we expected to be able to classify approximately 56% of the respondents correctly using only their six answers from that questionnaire. Respondents returned to complete wave 2 of the study from two to six days after the first wave and asked to answer the abbreviated questionnaire. Their segment membership was already known from the first wave. Our measure of success would be how accurately we could reclassify them based just on answers to the abbreviated questionnaire. We obtained high cooperation rates in both waves of the questionnaire, and after discarding a few respondents for too-short interview times, and a few more for inconsistent answers, we had a useable sample size of 556 respondents to the second wave. Reclassifying respondents correctly is not trivial, since respondents answer MaxDiff questions with some random degree of error, and there is even the possibility that respondents opinions may have shifted in the few days between questionnaire waves. Also, the reality is that segments aren t always cleanly delineated, and there are often substantial numbers of respondents in the valleys, distant from segment concentrations in the hills. For the second wave, the actual classification hit rates were as follows: Validation Hit Rates for Second Wave Condition Hit Rate No Demos.579 Income.579 Education.586 Age.594 All Three

13 All of these validity hit rates are better than our anticipated value of 56%. We speculate this is because we re-contacted the same respondents, rather than testing validity on a new sample, as we had in the split-half computations within the first wave. Even though demographics hadn t been used in developing the original segmentation scheme, the demographics appear to make a slight but consistent improvement. As evidenced by the split-half validity hit rates listed under Validation 2 in the previous table, we believe results would have been even better had we designed the abbreviated questionnaire using the more precise methods. Our best estimate is that we would have achieved hit rates closer to.64% than 56%. Reducing Rate of False-Positives We mentioned earlier that the researcher may be interested in creating a typing questionnaire that can accurately identify new respondents as belonging to a particular group. Perhaps the researcher needs to recruit new respondents to a focus group, and wants to ensure that these respondents indeed belong to the segment in question. We described a way that the search criterion for generating the typing questionnaire could be modified so that the classification rate into a particular segment (or segments) was maximized. However, there is yet another way for significantly boosting the classification rate and reducing the likelihood of misidentifying new respondents as belonging to a particular segment. Along with prediction into a group, the typing tool also reports the likelihood that the respondent belongs to that group 3. When we isolated the 231 (out of 556) respondents with at least 90% likelihood of belonging to the predicted segment, the actual hit rate (including demographic variables) increased from the base rate of 60% to 77%. Respondents with at least 95% likelihood (170 respondents) were classified with 81% accuracy. 3 The likelihoods are probably biased upwards because of the assumption that answers to different questions are independent of one another. Though likelihoods are useful for identifying well-classified respondents, they should not be interpreted as true probabilities. 13