A Model Based Approach for Achieving a Representative Sample. March 12, 2014 George Terhanian John Bremer Carol Haney

Transcription

1 A Model Based Approach for Achieving a Representative Sample March 12, 2014 George Terhanian John Bremer Carol Haney

2 The Idea of Employing a Model-Based Approach to Achieve a Representative Sample is not New We are investigating whether findings from huge samples of Internet respondents, coupled with sophisticated weighting processes, are as accurate as anything done on the telephone or door-to-door. Black and Terhanian (1998), Using the Internet for Election Forecasting, The Polling Report Page 2

3 Criticisms of Such Efforts are not New Either the willingness to discard the use of sampling frames as a means of selecting a sample and the feeble attempts at manipulating the resulting bias undermine the credibility of the survey process. Warren Mitofksy (1999). Pollsters.com. Public Perspective. Page 3

4 The Actual Critics have Included Individual Researchers as well as Entire Associations of Researchers SLOP = Self-Selected Opinion Polls Page 4

5 Evidence from the ARF s Foundation of Quality 1 Suggested that Such Concerns Can be Warranted In research carried out among seventeen different opt-in panels, the Advertising Research Foundation found wide variance, particularly on attitudinal and/or opinion questions (purchase intent, concept reaction, and the like) even after holding constant socio-demographic and other factors. Walker, Pettit, and Rubinson (2009). Page 5

6 Nevertheless, the Growth of Online Research has Remained Strong, Even Remarkable 7,000 6,000 5,000 4,000 Rest of World Europe US 3,000 2,000 1,000 0 '99 '00 '01 '02 '03 '04 '05 '06 '07 '08 '09 '10 '11 '12 '13E '14E Source: Inside Research Page 6

7 Partly for this Reason, AAPOR Changed its Tune in 2011 AAPOR needs to promote flexibility, not dogmatism, in determining which methods to use to achieve its objective of measuring, analyzing, and summarizing public opinion We have now established a new task force on non-probability research to broaden our reviews Nothing stands still, everything changes. Frank Newport (2011). Presidential Address. POQ. Page 7

8 And Two Years Later, the Task Force Concluded its Work The task force identified sample matching as the most promising non-probability approach for surveys after examining the conditions under which various survey designs that do not use probability samples have scientific merit. The task force did not propose explicit next steps, let alone a proposed agenda Nor did it mention the ARF s new large-scale effort to mount primary research FOQ 2 despite the potential for rich, relevant secondary analysis of the FOQ 2 data set to learn more about non-probability approaches Page 8

9 The ARF s FOQ 2 Explored Many Possible Sources of Bias 9 Respondent selection procedures (sampling) Post-hoc adjustment (weighting) procedures Procedures to improve within-survey behavior Identity verification Use of Multiple Sample sources (e.g., Panel 1, River 3) Survey participation frequency Respondent motivations and incentives Router sample optimization practices Mobile data collection Page 9

10 Additional Details on FOQ 2 A dual frame telephone survey, and an online survey, administered in parallel in January, The telephone survey included a sub-set of the items from the online survey. Target population of interest = the US general population, 18+ Seventeen online sample providers participated. The ARF provided expected counts for variables such as age, gender, and region to each provider in the interest of ensuring comparability among providers. A total of 1008 completed interviews via the phone survey, including about 300 via a cell-phone-only frame. 70,377 completed interviews online 57,811 from panels and 12,566 from non-panel sources. Page 10

11 FOQ 2 Topics Consumer product ownership Internet/Web orientation and usage Cognitive orientation Socio-emotional psychographics Consumer orientation Technology product ownership and usage Health and substance use Media exposure (TV, newspapers) Brand awareness and attitudes Political attitudes and behavior Health-related behaviors Demographics Page 11

12 (The Thinking Behind) Today s Presentation Researchers can (and should) use the ARF s FOQ 2 data set to test any number of hypotheses concerning non-probability sampling, and representativeness more generally. To that end, we rely on FOQ 2 data to develop an optimal model for obtaining a representative sample by that we mean one which minimizes bias and/or mean squared error across the different sample sources in the study. We accomplish this task by applying a multidirectional search algorithm (MSA) to the data from the various FOQ 2 (non telephone) sample sources We suggest possibilities for next steps We offer advise to researchers and insights professionals on how they might think about representativeness, and methods to achieve it. Page 12

13 What is Bias? A Practical Definition the difference between what survey respondents report and what we, or an omniscient observer, know to be true George Bohrnstedt (1983). Measurement. In: P.H. Rossi, J.D. Wright & A.B. Anderson (eds.) Handbook of Survey Research. Page 13

14 Measuring the Bias of a Single Question: An Example Q1. Do you smoke Cigarettes? Benchmark Panel 1 Abs Diff Every Day? Some Days Not at All? = Bias Page 14

15 Measuring the Bias of Multiple Questions Mean Sq Error= (Mean Abs Diff) 2 + Variance Bias Question Q1 8 Q2 9 Q3 10 Q4 11 Q5 12 Q6 4 9 = Mean Abs Diff (or Bias) 8 = Variance 89 = MSE Page 15

16 Reducing Bias: Two Ways (At Least) 1. Through Respondent Selection (Sampling) What variables should we account for when we select respondents for our survey? How might we use secondary information (e.g., Census information, market share information) to optimize our approach? 2. Through Post Hoc Adjustment (Weighting) What variables should we weight (up or down) to ensure that the characteristics of the sample match those of the target population? How might we use secondary information (e.g., Census information, market share information) to optimize our approach? Page 16

17 To Sample or To Weight to Reduce Bias? Debate Ongoing but Decision Should Be Informed by Evidence Hypothesis: When the interest lies in reducing bias. Age, Gender and Region, alone, are necessary but not sufficient The sampling scheme needs to include additional variables, and/or The weighting scheme need needs to include additional variables Page 17

18 What Additional Variables Might We Include? Many to choose from in the FOQ 2 data set: Primary demographics, such as race, ethnicity, education level, and income Secondary demographics, such as home ownership, # of adults in household, product ownership Behaviors, such as time spent online Attitudes, such as political ideology, sense of optimism, and privacy views Page 18

19 Method for Selecting these Variables: A Multidirectional Search Algorithm (MSA) MSA s minimize the average distance (bias) between the sample results (across all sources) and the chosen benchmarks (roughly 100, here) as they identify the optimal variables, and combinations of variables, to include in a model. For weighting, we add a penalty for each successive variable added into the scheme because of the effect on weighting efficiency. For sampling, we add no penalty. But we recognize, in general, that sampling plans with more variables tend to be harder to implement fully, all else equal Note: We restrict the # of variables in each scheme to no more than ten. Page 19

20 A Simple (Partially-Automated) 15 Step Process (1-9) 1. Determine the benchmarks. 2. Determine the convergence criterion. 3. Calculate the AGR bias for each sample source and each variable. Calculate the overall average. 4. Calculate the variance for each sample source, and calculate the overall average. 5. Calculate the average mean squared error. 6. Calculate the correlation between the usable survey variables and the bias. 7. Calculate impact. 8. Choose the variable to be added to the specification based on the highest level of impact. 9. Weight each sample source to the correct distributions for each variable in the specification. We used the SPSS Rake extension. Page 20

21 A Simple 15 Step Process (10-15) 10. Calculate the new average bias, variance, and mean squared error. 11. Determine the change in bias and mean squared error. Compare each to the convergence criterion. 12. If the change is larger than the convergence criterion, go back to step 6 and add another variable to the specification. 13. If the change is smaller than the convergence criterion, then start the process anew with a new initial variable being added to the specification. 14. The overall algorithm is complete when the initial variable produces metrics that, when compared to the AGR baseline, converge on the first iteration. 15. Sort through the results and identify the smallest bias and/or mean squared error. The optimal model is the set of variables that produce the minimum values. Page 21

22 Case 1. Sampling, No Weighting: Model Includes only Demographic Variables (or Standard Survey Questions) Optimal Demographic (Only) Models Overall Panel River AGR Bias - Baseline Case AGR, Time Spent Online, Race/Ethnicity, Education, Political Party, Housing Status, and Presence of a Landline Telephone Reduction in Bias, Model % 24.5% 20.8% AGR, Time Spent Online, Race/Ethnicity, Income, Housing Status, Number of Adults in the Household and Presence of a Landline Telephone Reduction in Bias, <odel % 22.4% 25.0% Note: Surprises include inclusion of time spent online, and the seeming interchangeability of education and income. Page 22

23 Summary of Key Sampling Variables across Two Optimal Demographic (or Standard Survey Question) Models 1. AGR, 2. Time Spent Online 3. Race/Ethnicity 4. Education 5. Income 6. Housing Status 7. Political Party 8. Housing Status 9. Presence of a Landline Telephone 10. # of Adults in the Household Page 23

24 Case 2. Sampling, No Weighting: All Variables Optimal All Variable Models Overall Panel River AGR Bias - Baseline Case AGR, Time Spent Online, Race/Ethnicity, Education, Q1260, Q2485B, Q2485C, Q2855r Reduction in Bias, Model % 36.7% 29.2% AGR, Time Spent Online, Race/Ethnicity, Income, q620a, Q1260, Q2485B, Q2855r Reduction in Bias, Model % 30.6% 33.3% Q620A = How often do you feel hopeful? Q1260 = How concerned are you about having records containing your personal information stolen over the Internet? Q2485B = How much do you agree with the following statement: It is best to treat those who disagree with you leniency and an open mind as they may be proven right? Q2485C = How much do you agree with the following statement: I am optimistic about my future? Q2855r1 = How many automobiles, vans or trucks of one ton capacity or less are kept Page 24 at home for use by members of your household?

25 Summary of Key Sampling Variables across Two Optimal All Variable Models. 1. AGR 2. Time Spent Online 3. Race/Ethnicity 4. Education 5. Income 6. Q620A = How often do you feel hopeful? 7. Q1260 = How concerned are you about having records containing your personal information stolen over the Internet? 8. Q2485B = How much do you agree with the following statement: It is best to treat those who disagree with you leniency and an open mind as they may be proven right? 9. Q2485C = How much do you agree with the following statement: I am optimistic about my future? 10. Q2855r1 = How many automobiles, vans or trucks of one ton capacity or less are kept at home for use by members of your household? Page 25

26 Some Questions not Answered Explicitly Here What is the incremental contribution toward bias reduction of each variable? How does complexity beyond AGR sampling or weighting affect efficiency? How much sampling versus how much weighting should be used? What effect might interaction terms have on bias and MSE? What other variables might we test? Are the models described here portable to new research? To different topics? To other countries? To tracking studies? How relevant, if at all, is the approach described here to telephone research? F2F research? Mobile research? Page 26

27 Recommendations (1) Depending on culture and philosophy, it might be necessary to pursue, or emphasize, one approach (e.g., weighting) rather than the other (e.g., sampling). Fortunately, MSA s are all about identifying variables that decrease bias, irrespective of whether they re used for sampling or weighting The decision on how to use the information from MSA s lies with the researcher. Getting the model right is crucial. Some researchers will add variables go the base (i.e., AGR) sampling scheme; some to the base weighting scheme, some to both. Behavioral variables such as time spent online and attitudinal variables can be particularly useful in reducing bias. Page 27

28 Recommendations (2) Figuring out how to implement a sampling or weighting scheme that includes 8-10 variables can be complex In both cases, efficiency can be a major challenge In general, the more complex the sampling quotas, the more difficult (expensive and time consuming) it is to fill those quotas. And the more complex the weighting scheme, the lower the effective sample size (which means it will be more difficult to determine significant results). Statistical matching methods for sampling, weighting or both have the added benefit of consolidating and summarizing information (e.g., the 8-10 variables), thereby increasing efficiency. We focus on them elsewhere. Page 28

29 Thank You! Page 29