MODELING A PROBABILITY SAMPLE? AN EVALUATION OF SAMPLE MATCHING FOR AN INTERNET MEASUREMENT PANEL

Transcription

1 MODELING A PROBABILITY SAMPLE? AN EVALUATION OF SAMPLE MATCHING FOR AN INTERNET MEASUREMENT PANEL Lukasz Chmura, Douglas Rivers, Delia Bailey, Christine Pierce, Scott Bell 05/03/2013

2 SAMPLE MATCHING WHAT IS IT Statistical Method for Reducing Bias of Non-Probability Sampling Methodology Highlights Developed by Dr. Doug Rivers of YouGov & Stanford Consultant to Nielsen for Proof of Concept Uses convenience pool to construct a panel that best matches the characteristics of a probability sample Copyright 2013 The Nielsen Company. Confidential and proprietary. Potential Benefits Requirements Lower cost than probability methods, e.g. RDD Internet coverage is higher than landline coverage in many markets Bias correction beyond post-stratification weighting by controlling for characteristics beyond margins Large convenience pool with known demographics/characteristics Representative sample or population frame Tested Matching algorithm 2

3 SAMPLE MATCHING HOW IT WORKS Matching Properties of a Probability Sample Select Match Measure Copyright 2013 The Nielsen Company. Confidential and proprietary. Select probability sample from high quality population frame or survey Select convenience panelists that most closely match the characteristics of the probability panel. Measure the resulting matched sample from the convenience sample. 3

4 SAMPLES Nielsen Netview panel a panel of approximately 230,000 panelists whose work/home computer is metered; November 2011 data used TVPC Weighted a subset of the Nielsen Netview home panel selected using probability methods; weighted using demographic controls; unweighted sample size = 29,496 persons Megapanel Weighted a subset of the Nielsen Netview home panel selected using non-probability methods; weighted using an expanded set of demographic controls; unweighted sample size = 174,906 persons 2010 U.S. Census Bureau s American Community Survey (ACS), with Internet variables imputed using the October 2009 School Enrollment and Internet Use Survey, a supplement to that month's Current Population Survey (CPS), and the Social Side of the Internet Survey 2010 (Pew Research) Matched ACS - subset of Megapanel selected by matching to ACS 4

5 DICTIONARY Active persons that used an Internet-enabled computer during the specified period (November 2011) Channels websites focused on specific viewer interests such as travel or weather (Google Search would be one example of a Channel, Google Maps another) Brands - a collection of Channels with the same brand identity (for example, Google) Category - a collection of Channels with the same viewer interest (for example, Search Engines or Travel) Reach - the percentage of active panelists that visited the Brand/Channel at least once during the time period under consideration 5

6 SAMPLE MATCHING VARIABLES Variables used for stratification Active/Inactive Internet User Age (0-17, 18-29, 30-44, 45-64, 65+) Gender (M, F) Education (< HS, HS, Some College, College Grad, Post Grad) Race (White, non-white) DMU-time (Low, Medium, High) Additional variables used for matching: DMU Time in Deciles, U.S. Census Region Income (Under $25K, $25K-$50K, $50K-$75K, $75K-$100K, $100K- $150K, $150K+) Employment Status (Employed full-time, Employed part-time, Child under 18/Student, Unemployed, Retired, Homemaker) Ethnicity (Hispanic, non-hispanic) 6

7 PREDICTING UNIQUE VISITORS (000) FROM PROBABILITY WEIGHTED FOR TOP CHANNELS AND BRANDS There is a large correlation between the UV Estimates from Matched ACS and Megapanel Weighted and the UV Estimates from TVPC Weighted UV estimates from Matched ACS seem to be good predictors of UV estimates from TVPC Weighted for the top Channels and Brands Channels Brands Slope Intercept R 2 Slope Intercept R2 Matched ACS Megapanel Weighted 0.82* 2.95* * * significant at the 0.05 level (H o : Slope = 1, Intercept = 0) 7

8 8

9 UNIQUE VISITOR (UV) COMPARISONS Comparisons based on 500 Channels, 500 Brands, 99 Categories Relative Difference of Unique Visitors = X/Y 1 where X are estimates from Matched ACS and Megapanel Weighted and Y are estimates from TVPC Weighted Median of the Relative Differences Channels Brands Categories Matched ACS -7.2% -3.6% -2.7% Megapanel Weighted -13.6% -11.8% -16.8% 9

10 10

11 95% CONFIDENCE INTERVALS FOR REACH DIFFERENCES FOR TOP CHANNELS AND BRANDS TVPC Standard Errors calculated using the Jackknife method (20 replicates) = ( ) = 1 = G = number of replicates (20) R i = estimate for i th replicate R t = estimate for total sample Matched ACS Standard Errors calculated using the Bootstrap method (100 samples) NOTE: The Matched ACS Standard Errors only account for the variation due to the Sample Matching process. Total Standard Errors for Matched ACS cannot be calculated, as the matched sample is a subset of a convenience panel. Frequency Distribution of the Standard Errors for Matched ACS Channels Brands 0-0.2% 32% 40% % 29% 32% % 32% 26% % 7% 2% Median Reach 13.8% 9.9% 11

12 Points within the intervals are Channels/Brands where the differences in reach estimates can be explained by the sample matching process or the TVPC sampling error. In all, 39% of the top Channels and 34% of the top Brands are within the confidence intervals. 12

13 INTERNET USAGE AMONG ACTIVE PANELISTS Deciles: Decile 1 = lowest internet duration/page views; Decile 10 = highest internet duration/page views Compared to TVPC, Megapanel contained a large number of heavy online users (about twice as many active panelists in Decile 10) The sample matching process significantly reduced the overestimation of heavy online users 13

14 CONCLUSIONS The use of sample matching to select a subset of the nonprobability panel consistently resulted in smaller overall differences from the probability panel. The bias in the non-probability panel was not completely eliminated. The effectiveness of sample matching is limited by the availability of the matching variables. 14

15 CONTACT 15