Widening the Net. A comparison of online intercept and access panel sampling

Size: px

Start display at page:

Download "Widening the Net. A comparison of online intercept and access panel sampling"

Dominic Lawrence
6 years ago
Views:

1 Widening the Net A comparison of online intercept and access panel sampling Presented by: David G. Bakken PhD Senior Vice President and Chief Science Officer CASRO Panel Conference 4 th March 2011

2 Context for the Research In early 2010, Authentic Response developed a proprietary respondent Intercept capability to augment it s double opt-in Panel (MyView ) to expand the Company s access to respondents willing to take surveys. Before bringing this new capability online, Authentic Response, Inc. wished to gain a better understanding of the impact that Intercept sampling might have on survey results, especially in comparison with other sampling methods. Research was commissioned with David Bakken PhD to assess the impact of these different methods on survey results and ultimate decision-making.

3 The Industry Challenge Declining participation and response rates make it harder to acquire, convert and retain survey respondents. Increased demand for online research may strain the capacity of opt-in access panels. Some desirable consumer and B2B respondent pools show little interest in joining panels. 3

4 The Sampling Ideal

5 The Online Sampling Reality

6 The Decision Maker s Questions How wrong can my estimate be? What s the likelihood that I ll be wrong by that much? 6

7 Sources of Survey Error Failure to define the problem carefully and clearly Questionnaire-induced errors Non-response Coverage errors Sampling frame Non-representative spatial boundaries Non-representative temporal boundaries Estimation and statistical conclusion errors 7

8 Sampling is a Successive Filtering Process Total Population Addressable population units Qualified population units Willing population units Observed population units

9 A Variety of Sampling Models for MR Probability samples US Census General postal address surveys RDD Judgment/convenience samples Lists compiled for other reasons Volunteer samples/panels Mall intercepts 9

10 Objectives for This Research Compare results obtained from different online sampling methods: Traditional ed invitation to double opt-in panel sample Dynamic routing opt-in panel sample Consumer intercept volunteer sample 10

11 ed Opt-in Panel Sample Randomly selected members of the Authentic Response online consumer panel. These panelists received a traditional invitation with a link to a single specific survey. 1,015 completed surveys 11

12 Dynamic Routing Opt-in Panel Sample A separate random sample of Authentic Response consumer panel members These panelists received a link to the dynamic survey router Our test survey had the highest priority in the router queue 1,103 completed surveys 12

13 Online Consumer Intercept Sample Visitors to a variety of publisher sites exposed to general invitations to complete a survey Clicking through the link brought potential respondents to Authentic Response, where they were asked a few qualifying questions Qualified respondents were directed to the test survey 1,929 completed surveys 13

14 Sample Filtering and Quotas Replicate typical market research study Minimal inclusion criteria US residents Age 18 or older Quotas based only on sex and age U.S. Census targets Included to minimize sampling frame skews for age and gender 14

15 Sample Disposition Intercept Router Panel Completed Term/other Over quota Not eligible Visit Only

16 Frame Characteristics Intercept Router Panel % Males 51.3% 43.2% 46.2% Mean age 29.3 years 42.6 years 46.8 years Mean age ineligible Mean age over quota 13.8 years 30.1 years 17.0 years 29.5 years 44.8 years 54.7 years 16

17 Frame Characteristics Age Intercept Router Panel US Census % 20% 40% 60% 80% 100% 17

18 The Survey 28 question, 10- minute survey Designed to minimize measurement error Mostly factual questions with categorical responses Some questions from US Census as well as some American Community Survey demographic questions included for external comparison 18

19 Evaluating the Results Can we identify a respondent s sample source from his or her survey responses? How similar overall are the survey responses across the different samples? To what degree do any observed differences have managerial / decisionmaking significance? 19

20 Can We Identify a Respondent s Sample Source? Discriminant analysis 30 key survey measures as independent variables Sample type as dependent variable None of the key survey measures are predictive of sample source 85% of intercept cases correctly classified Most router and traditional panel cases also classified as intercept sample 20

21 How Similar Overall are the Responses? Correlation of Survey Responses Across Samples Intercept Router Panel Intercept -- Router Panel

22 % of Comparisons (n=10&) Are Differences Managerially Significant? 80% 70% 60% 50% 40% 30% Intercept vs. Panel Router vs. Panel Intercept vs. Router 20% 10% 0% <1% 1%-2% 2%-3% 3%-4% 4%-5% 5%-7% 7%-10% >10% Absolute Magnitude of Difference 22

23 Comparison to External Benchmarks 60% 50% 40% 30% 20% 10% Intercept Router Panel US Total 0% 4-year college degree Married African American Income over $100K

24 Blending Sample Results 30% 25% 20% 15% 10% 5% 0% Blu-ray DVD penetration Intercept Router Panel Simple weighted blending Re-sampling (bootstrap) analysis indicates that there is at least a 60% chance that the penetration for Blu-ray DVD is between 19% and 21% 24

25 Implications for On-line Sampling Blending intercept with opt-in panel respondents increases overall diversity in a sample. Combining intercept and opt-in sampling is one way to reduce coverage limitations of any single method or source. Sourcing matters the intercept net should be cast over a wide variety of publishers and other respondent sources. 25

ESOMAR 28 ANSWERS TO 28 QUESTIONS TO HELP BUYERS OF ONLINE SAMPLE

ESOMAR 28 ANSWERS TO 28 QUESTIONS TO HELP BUYERS OF ONLINE SAMPLE 2016 ESOMAR 28 COMPANY PROFILE 1. What experience does your company have in providing online samples for market research? Context: This