Improving the Representativeness of Online Surveys

Transcription

1 Improving the Representativeness of Online Surveys Jeffrey Henning Researchscape International Event Sponsors

2

3

4

5 Total Survey Error Respondent Selection Issues Sampling Error Response Accuracy Issues Coverage Error Nonrespons e Error at Unit Nonresponse Error at Item Measurement Error due to Respondents Measurement Error due to Interviewers Survey Administration Issues Post-Survey Error Mode Effects Comparability Effects

6 Comparing Prices Niche Survey Topline Survey Probability Survey Mode Online Online Telephone Target > 5% incidence > 20% incidence Respondents Length 15 questions 25 questions Cost/response $5 $5 $20 General population 5 minutes Price $495 $1,995 $7,995

7 Agenda Probability sampling Probability online panels Open online panels Weighting Quota sampling Sample matching River sampling Intercept samples Practical ramifications

8 40.0% Average Absolute Errors 35.0% 30.0% 25.0% 20.0% Probability Non-probability 15.0% 10.0% 5.0% 0.0% 12.0% 3.3% 4.1% 4.7% 5.0% 5.3% 6.4% 6.4% 2.0% A B C D E F G H I Source: Yeager, Krosnick, et al, 2011

9 Accuracy = Value Probability Non-probability A B C D E F G H I Base = worst performing survey s average absolute error

10 Largest Absolute Errors 40.0% 35.0% 30.0% Probability Non-probability 35.5% 25.0% 20.0% 15.0% 10.0% 9.6% 11.7% 13.2% 13.7% 15.3% 15.6% 16.0% 18.0% 5.0% 0.0% A B E F H G D C I Source: Yeager, Krosnick, et al, 2011

11 Largest Absolute Errors 40.0% 35.0% 30.0% Probability Non-probability 35.5% 25.0% 20.0% 15.0% 10.0% 9.6% 11.7% 13.2% 13.7% 15.3% 15.6% 16.0% 18.0% 5.0% 0.0% A B E F H G D C I Source: Yeager, Krosnick, et al, 2011

12 Key Elements of Probability Sampling Coverage Known nonzero chance of selecting any member of the target population External selection Random selection of members of the population to participate in the survey

13 Robustness? Any method with a low response rate is not a random probability sample. We can t assume a known and non-zero chance of selection. This is true of telephone, so for most studies the gold standard is not a practical option, even if money were no object. Ray Poynter, director, Vision Critical, 2013 What about the vast majority of research that has 90% opt-out rates? Do we decide that those people weren t part of the population to begin with?...i m just having a hard time understanding the ongoing push to prove we are using probability samples. Annie Pettit, Research Now, 2010

14 Robustness? Any method with a low response rate is not a random probability sample. We can t assume a known and non-zero chance of selection. This is true of telephone, so for most studies the gold standard is not a practical option, even if money were no object. Ray Poynter, director, Vision Critical, 2013 What about the vast majority of research that has 90% optout rates? Do we decide that those people weren t part of the population to begin with?...i m just having a hard time understanding the ongoing push to prove we are using probability samples. Annie Pettit, Research Now, 2010 Response rates were positively associated with demographic representativeness, but only very weakly... In general population RDD telephone surveys, lower response rates do not notably reduce the quality of survey demographic estimates. Holbrook, Krosnick, Pfent, 2008

15 Probability sampling Agenda Probability online panels Open online panels Weighting Quota sampling Sample matching River sampling Intercept samples Practical ramifications

16 Probability Online Panels Build a large panel using Address Based Sampling Relentlessly invite candidates to join the panel Provide computers or tablets and Internet connectivity if needed Consistently perform as well as RDD Transitive property of probability sampling: a random sample of a random sample is highly accurate even though net response rates are low $900 per question from Knowledge Networks Perhaps the rise of smartphones will lead to new mobile probability panels that hit a lower price point

17 Probability Online Panels Build a large panel using Address Based Sampling Relentlessly invite candidates to join the panel Provide computers or tablets and Internet connectivity if needed Consistently perform as well as RDD Transitive property of probability sampling: a random sample of a random sample is highly accurate even though net response rates are low $900 per question from Knowledge Networks Perhaps the rise of smartphones will lead to new mobile probability panels that hit a lower price point

18 Impractical for Low Incidence Mothers of children 4 and under Families with chronically ill members Women who do yoga workouts Adventure racing enthusiasts Video game players Board and card game purchasers Purchasers of apps for smartphones and tablets E-book purchasers Purchasers of self-help books Golfers Small-business owners Middle managers

19 Bye, Bye, Probability But where randomized treatments are not possible... we must do the best we can with what is available to us. - Donald T. Campbell, social scientist, 1969

20 Agenda Probability sampling Probability online panels Open online panels Weighting Quota sampling Sample matching River sampling Intercept samples Practical ramifications

21 Open Online Panels Anyone can join the panel Panelists join for cash or prizes Many field surveys through web intercepts, collecting responses from non-panelists Inconsistent results A random sample of a convenience sample is still a convenience sample Random sampling does produce greater consistency for longitudinal studies

22 Examples of Online Panel Results 100% 90% 17% 80% 70% 60% 69% 50% 40% % 20% 10% 0% Panel U.S. Census

23 Bias of People Not on Internet

24 Agenda Probability sampling Probability online panels Open online panels Weighting Quota sampling Sample matching River sampling Intercept samples Practical ramifications

25 Weighting Post-stratification weighting viewed as a common solution to removing sampling bias from convenience samples Often misrepresented as a simple process of arithmetic

26 Cell Weighting 18 to 54 Men 79,184, responses 469K weight Women 79,017, responses 397K weight ,301, responses 2,420K weight 43,154, responses 2,539K weight

27 Rim Weighting / Raking Age Proprietary measure Gender Household income Region Education level Race/ethnicity

28 Weighting Implicit assumption is people we did survey in a particular demographic group are representative of the people we did not survey in that group Many researchers weight convenience samples... In the hope it does no harm In the belief it improves quality For the fact it redistributes demographics to match target population

29 Wait, Wait Waiting until the weighting stage to adjust is too late. The combination of coverage error and nonresponse in online panels generally creates a sample that is beyond fixing post hoc. We need to do more at the selection stage. - Reg Baker, former president and COO of Market Strategies, 2013

30 Agenda Probability sampling Probability online panels Open online panels Weighting Quota sampling Sample matching River sampling Intercept samples Practical ramifications

31 Quota Sampling Men 18 to 54 79,184, responses 595K weight Women 79,017, responses 599K weight ,301, responses 595K weight 43,154, responses 599K weight

32 Quota Sampling Divide the sample into cells and recruit to fill those cells Once 51% of respondents are women, stop accepting responses from women Each quota increases price: $1,000 for no quota $1,500 for 3 quota cells $2,000 for 12 quota cells Bad reputation among public opinion researchers, good reputation among corporate researchers

33 Quota Sampling Good reputation among corporate researchers Bad reputation among public opinion researchers

34 Quota Sampling

35 Agenda Probability sampling Probability online panels Open online panels Weighting Quota sampling Sample matching River sampling Intercept samples Practical ramifications

36 Sample Matching Rim weighting : cell weighting = Sample matching : quota sampling Imagine trying to fill 400 cells: 57-year old African American woman with associates degree in Newton Falls, OH 21-year old white male high school graduate in Worcester, MA Where do the cells come from?

37

38 YouGov Model of U.S. Population 2010 American Community Survey Age Gender Race Education Region Imputation from Registration and Voting Supplements Voter registration Imputation from Pew Religion in American Life Survey Religion Political interest Minor party identification Non-placement on an ideology scale

39 Sample Matching Finding 57-year old African American woman with associates degree in Newton Falls, OH Proximity function tests all members of panel, calculating distance from target (distance in age, gender, physical location, etc.) Invite 59-year old African American woman with GED in Warren, OH

40 Agenda Probability sampling Probability online panels Open online panels Weighting Quota sampling Sample matching River sampling Intercept samples Practical ramifications

41 River Sampling

42 Pros & Cons of Steady Panel Participation Practice Effects (Major) Regularly answering surveys may improve accuracy of responses Panel members may become more introspective and selfaware, improving their reporting Respondents answers to attitudinal questions improve with practice Panel Conditioning (Minor) Stimulus hypothesis that acting about future activity prompts that activity Past surveys makes panelists less like general population Panelist attrition nonrandomly affects panel representativeness Source: Chang & Krosnick, 2008

43 70% of NPD Panelists are Introverts vs. 50% in U.S. [Diligent panelists are] high on introversion, have a high need for cognition, enjoy thinking, and prefer complex to simple problems, and they like surveys they find surveys worthwhile. - Inna Burdein, direct of panel analytics for NPD, 2013

44 Agenda Probability sampling Probability online panels Open online panels Weighting Quota sampling Sample matching River sampling Intercept samples Practical ramifications

45 Intercept Surveys

46 Intercept Surveys

47 Key Elements of Probability Sampling? Coverage Known nonzero chance of selecting any member of the target population External selection Random selection of members of the population to participate in the survey

48 Agenda Probability sampling Probability online panels Open online panels Weighting Quota sampling Sample matching River sampling Intercept samples Practical ramifications

49 Mimicking Probability Sampling Probability panel Random selection of randomly recruited panelists Open panel Random selection of panelists Margin of error AAPOR is against reporting margin of error for nonprobability samples Coverage Known nonzero chance of selecting any member of the target population External selection Random selection of members of the population to participate in the survey Sample matching Random selection of members of the population to match in the panel Weighting Correcting for demographic underrepresentation

50 Recommendations When sourcing sample, ask for steps taken to minimize sampling bias When evaluating panels, ask how they select respondents for a given study Don t use weighting if sample was significantly demographically unbalanced Don t report sampling error but do consider reporting de factor error ranges

51 For Further Reading Free 125-page report from the American Association for Public Opinion Research:

52 Total Survey Error Respondent Selection Issues Sampling Error Response Accuracy Issues Coverage Error Nonrespons e Error at Unit Nonresponse Error at Item Measurement Error due to Respondents Measurement Error due to Interviewers Survey Administration Issues Post-Survey Error Mode Effects Comparability Effects

53 Thank you! Jeffrey Henning Researchscape International

54 The Sponsors for this Event Event Sponsors If you are interested in sponsoring a future NewMR event Michele.Poynter@TheFuturePlace.com

55 Q & A Jeffrey Henning Researchscape Ray Poynter The Future Place

56 Jeffrey Henning, PRC Researchscape International Up-to-date Research on the Changing Consumer Toll Free: +1 (888) x 701 jhenning@researchscape.com