Can Microtargeting Improve Survey Sampling?

Transcription

1 Can Microtargeting Improve Survey Sampling? An Assessment of Accuracy and Bias in Consumer File Marketing Data Josh Pasek University of Michigan NSF / Stanford Conference: Future of Survey Research Project in conjunction with S. Mo Jang, Curtiss Cobb, Charles DiSogra, & J. Michael Dennis

2 Surveys in the 21st Century Challenges and Opportunities

3 Surveys in the 21st Century Challenges and Opportunities Declining response rates Increasing costs Coverage challenges (for some modes)

4 Surveys in the 21st Century Challenges and Opportunities Declining response rates Increasing costs Coverage challenges (for some modes) Pew 2012 Also see: Curtin, Presser, & Singer, 2005

5 Surveys in the 21st Century Challenges and Opportunities Declining response rates Dual-frame costs (cf. Kennedy, 2007) Increasing costs Increasing refusal as a cost Coverage challenges (for some modes) (cf. Curtin, Presser, & Singer, 2005)

6 Surveys in the 21st Century Challenges and Opportunities Declining response rates Hispanic & Young Americans (Abraham, Maitland, Bianchi, 2006) Increasing costs Telephone Access Coverage challenges (for some modes) (Blumberg & Luke, 2011)

7 Surveys in the 21st Century Challenges and Opportunities Declining response rates Increasing costs Increasingly difficult to translate Coverage challenges (for some modes) from respondents to the population

8 Surveys in the 21st Century Challenges and Opportunities New forms of data New modes of data collection More sophisticated analytical tools

9 Surveys in the 21st Century Challenges and Opportunities New forms of data Social Media New modes of Mobile phone data data collection Tracking data & Paradata More sophisticated analytical tools Marketing data

10 Surveys in the 21st Century Challenges and Opportunities New forms of data New modes of Online surveys data collection Behavioral tracking More sophisticated analytical tools

11 Surveys in the 21st Century Challenges and Opportunities New forms of data Better weighting techniques New modes of Matching / propensity scores data collection Imputation More sophisticated analytical tools Machine learning

12 The big question Can the opportunities offset the challenges?

13 The big question Can the opportunities offset the challenges? Can we use new methods for data collection and analysis to help us understand the population?

14 The current exploration Consumer file marketing data

15 Why consumer file marketing data?

16 Why consumer file marketing data? Can be easily purchased

17 Why consumer file marketing data? Can be easily purchased Readily matched to addresses

18 Why consumer file marketing data? Can be easily purchased Readily matched to addresses Provides a rich source of data for all individuals in the sample (not just respondents)

19 Why consumer file marketing data? If the data are high quality:

20 Why consumer file marketing data? If the data are high quality: - Can enable efficient targeted sampling of hard-to-reach groups

21 Why consumer file marketing data? If the data are high quality: - Can enable efficient targeted sampling of hard-to-reach groups - Can provide information on systematic nonresponse

22 Why consumer file marketing data? If the data are high quality: - Can enable efficient targeted sampling of hard-to-reach groups - Can provide information on systematic nonresponse - Might allow corrections for nonresponse and sampling biases

23 Consumer file marketing data as a form of ancillary data Long history in statistics and survey methodology thinking about auxiliary sources of data that could translate between respondents and population (e.g. Deville, Sarndal, & Sautory, 1993; Holt & Smith, 1979; Jagers, Oden, & Trulsson)

24 Consumer file marketing data as a form of ancillary data Long history in statistics and survey methodology thinking about auxiliary sources of data that could translate between respondents and population (e.g. Deville, Sarndal, & Sautory, 1993; Holt & Smith, 1979; Jagers, Oden, & Trulsson) Emerging literature on using individual-level nonsurvey data to correct for errors due to nonresponse (e.g. Boehmke, 2003; Dixon & Tucker, 2010; Groves, 2006; Maitland, Casas-Cordero, & Kreuter, 2009; Kreuter & Olson, 2011; Little & Vartivarian, 2005; Peytchev, 2012; Smith, 2011)

25 Some key initial questions

26 Some key initial questions 1) What are we doing with the data? (supplement or source of inference)

27 Some key initial questions 1) What are we doing with the data? (supplement or source of inference) 2) How accurate are the data?

28 Some key initial questions 1) What are we doing with the data? (supplement or source of inference) 2) How accurate are the data? 3) How complete are the data?

29 Some key initial questions 1) What are we doing with the data? (supplement or source of inference) 2) How accurate are the data? 3) How complete are the data? 4) What model are we using to link the data with the world?

30 Some key initial questions 1) What are we doing with the data? (supplement or source of inference) 2) How accurate are the data? 3) How complete are the data? 4) What model are we using to link the data with the world? 5) How does the model perform for different types of inference?

31 Evaluating consumer file marketing data

32 Evaluating consumer file marketing data Sample

33 Evaluating consumer file marketing data Sample Ancillary Data

34 Evaluating consumer file marketing data Respondents Non-Respondents Sample Ancillary Data

35 The current project

36 The current project (1) Assess the correspondence of ancillary data and self-reports

37 The current project (1) Assess the correspondence of ancillary data and self-reports (2) Evaluate the nature of missingness in ancillary data

38 The current project (1) Assess the correspondence of ancillary data and self-reports (2) Evaluate the nature of missingness in ancillary data (3) Explore whether correctives using ancillary data (i.e. multiple imputations) could produce results that better reflect population parameters

39 Comparison data

40 Comparison data 25,000 households sampled by GfK from USPS Computerized Delivery Sequence File (>95% coverage)

41 Comparison data 25,000 households sampled by GfK from USPS Computerized Delivery Sequence File (>95% coverage) Address-Based Sample recruited via mail in January 2011, respondents were provided with Internet access

42 Comparison data 25,000 households sampled by GfK from USPS Computerized Delivery Sequence File (>95% coverage) Address-Based Sample recruited via mail in January 2011, respondents were provided with Internet access Self-report data from 4472 individuals in 2498 households recruited by GfK to KnowledgePanel AAPOR RR1=10.0%

43 Comparison data 25,000 households sampled by GfK from USPS Computerized Delivery Sequence File (>95% coverage) Address-Based Consumer Sample file data recruited from Marketing via mail in Systems January 2011, respondents Group merged were with provided all sampled with Internet households access 100% matched, data originally from InfoUSA, Experian, and Acxiom Self-report data from 4472 individuals in 2498 households recruited by GfK to KnowledgePanel AAPOR RR1=10.0%

44 Weights 4 sets of household weights: Pure household weight (1 / Rs in HH) Adult household weight (1 / Rs in HH over 18) Best ancillary match weight (1 / R(s) closest to Ancillary age in HH) Best ancillary match weight, full HH only (1 / R(s) closest to Ancillary age in HH for HHs with all respondents present)

45 Weights 4 sets of household weights: Pure household weight (1 / Rs in HH) Adult household weight (1 / Rs in HH over 18) Best ancillary match weight (1 / R(s) closest to Ancillary age in HH) Best ancillary match weight, full HH only (1 / R(s) closest to Ancillary age in HH for HHs with all respondents present)

46 Weights 2 sets of adjustment targets Respondents All Sampled Individuals

47 Weights 2 sets of adjustment targets Respondents Assessments of correspondence and missingness All Sampled Individuals

48 Weights 2 sets of adjustment targets Respondents Assessments of correspondence and missingness All Sampled Individuals Multiple imputations to match sampling frame

49 Weights Which weights we used did not matter for analyses We always used the most contextually appropriate weights for the data presented

50 Measures

51 Measures Home ownership Household income Household size } Household

52 Measures Home ownership Household income Household size Marital status Education Age } } Household Individual ( Head of Household )

53 The current project (1) Assess the correspondence of ancillary data and self-reported estimates (2) Evaluate the nature of missingness in ancillary data (3) Explore whether correctives using ancillary data could produce results that better reflect population parameters

54 (1) Assess the correspondence of ancillary data and self-reported estimates Basic strategy: Assess the proportion of matches between ancillary data and self-reported data for each variable among respondents

55 (1) Assess the correspondence of ancillary data and self-reported estimates Respondents Non-Respondents Sample Ancillary Data

56 Home Ownership Proportion of Households Ancillary Renter Ancillary Owner Renter Owner Self Report

57 Home Ownership Proportion of Households Ancillary Renter Ancillary Owner Renter Owner 88.9% Agreement Self Report

58 Household Income Proportion of Households Difference Between Self Report and Ancillary Estimates

59 Household Income Proportion of Households % Agreement Difference Between Self Report and Ancillary Estimates

60 Household Income Proportion of Households % Far Off Difference Between Self Report and Ancillary Estimates

61 Household Size Proportion of Households Difference Between Self Report and Ancillary Estimates

62 Household Size Proportion of Households % Agreement Difference Between Self Report and Ancillary Estimates

63 Household Size Proportion of Households % Far Off Difference Between Self Report and Ancillary Estimates

64 Marital Status Proportion of Households Ancillary Unmarried Ancillary Married Unmarried Married Self Report

65 Marital Status Proportion of Households Ancillary Unmarried Ancillary Married Unmarried Married 72.3% Agreement Self Report

66 Education Proportion of Households Difference Between Self Report and Ancillary Estimates

67 Education Proportion of Households % Agreement Difference Between Self Report and Ancillary Estimates

68 Education Proportion of Households % Far Off Difference Between Self Report and Ancillary Estimates

69 Age (Biased toward match) Proportion of Households to 2 1 Equal 1 2 to 5 5+ Differences Between Ancillary and Self Report Age in Years

70 Age (Biased toward match) Proportion of Households % Within 1 year 5+ 5 to 2 1 Equal 1 2 to 5 5+ Differences Between Ancillary and Self Report Age in Years

71 Age (Biased toward match) Proportion of Households % > 5 years 5+ 5 to 2 1 Equal 1 2 to 5 5+ Differences Between Ancillary and Self Report Age in Years

72 (1) Assess the correspondence of ancillary data and self-reported estimates Correspondence varies enormously across variables 23% - 89% Considerable discrepancies for all variables

73 The current project (1) Assess the correspondence of ancillary data and self-reported estimates (2) Evaluate the nature of missingness in ancillary data (3) Explore whether correctives using ancillary data could produce results that better reflect population parameters

74 (2) Evaluate the nature of missingness in ancillary data Basic strategy: See if missingness for ancillary measures differs by self-reports of the same variable See how well missingness can be predicted

75 (2) Evaluate the nature of missingness in ancillary data Respondents Non-Respondents Sample Ancillary Data

76 (2) Evaluate the nature of missingness in ancillary data Basic strategy: See if missingness for ancillary measures differs by self-reports of the same variable See how well missingness can be predicted

77 Missingness by Variable Missing Ancillary Data by Variable (N = 2277) Proportion Missing Ancillary Data (%) Home Ownership Household Income Household Size Marital Status Education Age Variable

78 Missingness by Respondent Distribution of Missing Ancillary Data Across Respondents (N = 2277) Proportion of Respondents (%) Number of Variables Missing

79 Home Ownership Distribution of Missing Home Ownership Data by Self Reported Home Ownership Status χ 2 (1, 2274) = 181.4, p<.001 Proportion Missing Ancillary Home Ownership Data (%) Own Rent Self Reported Home Ownership Status

80 Household Income Distribution of Missing Ancillary Income Data by Self Reported Income χ 2 (7, 1661) = 32.5, p<.001 Proportion Missing Ancillary Income Data (%) Self Reported Income Category in Thousands

81 Household Size Distribution of Missing Household Size Data by Self Reported Household Size χ 2 (4, 2274) = 15.15, p<.01 Proportion Missing Ancillary Household Size Data (%) Self Reported Household Size

82 Marital Status Distribution of Missing Marital Data by Self Reported Marital Status χ 2 (1, 2274) = 59.8, p<.001 Proportion Missing Ancillary Marital Data (%) Not Married Married Self Reported Marital Status

83 Education Distribution of Missing Education Data by Self Reported Education χ 2 (4, 1414) = 5.8, p=.21 Proportion Missing Ancillary Education Data (%) Less Than High School 19.2 High School Graduate 20.9 Some College 19.2 College Degree 27.3 Post Graduate Education Self Reported Education Category

84 Education Distribution of Missing Ancillary Age Data by Self Reported Age Proportion Missing Ancillary Age Data (%) χ 2 (6, 2277) = 188.2, p< Self Reported Age Category

85 (2) Evaluate the nature of missingness in ancillary data Basic strategy: See if missingness for ancillary measures differs by self-reports of the same variable See how well missingness can be predicted

86 Predictor Missing Ancillary Data Regressions predicting number of ancillary variables missing (range 0-6) using self-reported demographics

87 Predictor Home ownership Missing Ancillary Data Non-owners*** Regressions predicting number of ancillary variables missing (range 0-6) using self-reported demographics

88 Predictor Home ownership Income Missing Ancillary Data Non-owners*** n.s. Regressions predicting number of ancillary variables missing (range 0-6) using self-reported demographics

89 Predictor Home ownership Income Household size Missing Ancillary Data Non-owners*** n.s. Fewer persons*** Regressions predicting number of ancillary variables missing (range 0-6) using self-reported demographics

90 Predictor Home ownership Income Household size Marital status Missing Ancillary Data Non-owners*** n.s. Fewer persons*** Unmarried* Regressions predicting number of ancillary variables missing (range 0-6) using self-reported demographics

91 Predictor Home ownership Income Household size Marital status Education Missing Ancillary Data Non-owners*** n.s. Fewer persons*** Unmarried* n.s. Regressions predicting number of ancillary variables missing (range 0-6) using self-reported demographics

92 Predictor Home ownership Income Household size Marital status Education Age Missing Ancillary Data Non-owners*** n.s. Fewer persons*** Unmarried* n.s. Younger* Regressions predicting number of ancillary variables missing (range 0-6) using self-reported demographics

93 Predictor Home ownership Income Household size Marital status Education Age Race/ethnicity Missing Ancillary Data Non-owners*** n.s. Fewer persons*** Unmarried* n.s. Younger* n.s. Regressions predicting number of ancillary variables missing (range 0-6) using self-reported demographics

94 Predictor Home ownership Income Household size Marital status Education Age Race/ethnicity Missing Ancillary Data Non-owners*** n.s. Fewer persons*** Unmarried* n.s. Younger* n.s. R-squared 0.11 Regressions predicting number of ancillary variables missing (range 0-6) using self-reported demographics

95 Predictor Home ownership Income Missing Ancillary Data Non-owners*** n.s. Household size Fewer persons*** Missingness in ancillary data Marital status Education Unmarried* was not well accounted for n.s. Age Race/ethnicity Younger* n.s. R-squared 0.11 Regressions predicting number of ancillary variables missing (range 0-6) using self-reported demographics

96 (2) Evaluate the nature of missingness in ancillary data Missing ancillary data appears to be nonignorable and biased

97 The current project (1) Assess the correspondence of ancillary data and self-reported estimates (2) Evaluate the nature of missingness in ancillary data (3) Explore whether correctives using ancillary data could produce results that better reflect population parameters

98 (3) Explore whether correctives using ancillary data could produce results that better reflect population parameters Basic strategy: Impute the distribution of self-reports for all sampled individuals based on the ancillary data See if that represents a substantive improvement over the self-reports of respondents alone (cf. Peytchev 2012)

99 (3) Explore whether correctives using ancillary data could produce results that better reflect population parameters Respondents Non-Respondents Sample Ancillary Data

100 Analytical strategy - Impute self-reports for entire sample (not just respondents) - Compare imputed, raw self-report, and ancillary values to CPS

101 Measures Used in Imputations Home ownership Presence of telephone Number of persons in household Household income Marital status Education of head of household Age of head of household Number of children in household Hispanic status Region

102 The imputations 100 imputed datasets were created using MICE (multiple imputation via chained equations) Point estimates were generated for all imputed datasets as well as for raw self-reports, ancillary data, and CPS

103 Home ownership Home Ownership Estimates Weighted By Household Proportion MIs Raw GfK Estimate CPS Estimate Ancillary Estimate

104 Household Income Income Category Estimates Weighted By Household Proportion MIs Raw GfK Estimate CPS Estimate Ancillary Estimate Less than $15k $15k 25k $25k 35k $35k 50k $50k 75k $75k 100k $100k 150k More than $150k Household Income Category

105 Household size Household Size Estimates Weighted By Household Proportion MIs Raw GfK Estimate CPS Estimate Ancillary Estimate or more Persons in Household

106 Marital Status Marital Status Estimates Weighted By Individual Proportion MIs Raw GfK Estimate CPS Estimate Ancillary Estimate

107 Education Education Level Estimates Weighted By Individual Proportion MIs Raw GfK Estimate CPS Estimate Ancillary Estimate Less than HS HS Grad Some College College Grad Grad School Education Level

108 Age Age Category Estimates Weighted By Individual Proportion MIs Raw GfK Estimate CPS Estimate Ancillary Estimate and up Age Category

109 Differences from CPS Average Absolute Difference From CPS By Method And Variable Imputation Mean Raw GfK Estimate Ancillary Estimate Home Ownership Income Household Size Marital Status Education Age Average

110 Differences from CPS Data Household Individual Total Imputed mean 2.7% 3.3% 3.0% Raw self-report 4.0% 6.5% 5.3% Ancillary 7.5% 12.7% 10.1%

111 Differences from CPS Data Household Individual Total Imputed mean 2.7% 3.3% 3.0% Raw self-report 4.0% 6.5% 5.3% Ancillary 7.5% 12.7% 10.1% Imputations were better than raw selfreports, but not by an enormous amount

112 Distilling these results

113 Distilling these results Estimates from the raw self-report data (unweighted) were not very far off

114 Distilling these results Estimates from the raw self-report data (unweighted) were not very far off Imputations based on ancillary data eliminated a moderate portion of the error in the self-reports

115 Distilling these results Estimates from the raw self-report data (unweighted) were not very far off Imputations based on ancillary data eliminated a moderate portion of the error in the self-reports Ancillary data themselves do not seem particularly accurate

116 Across all analyses

117 Across all analyses Ancillary data estimates frequently differ from self-reports

118 Across all analyses Ancillary data estimates frequently differ from self-reports Missing ancillary data is systematic and appears to be non-ignorable

119 Across all analyses Ancillary data estimates frequently differ from self-reports Missing ancillary data is systematic and appears to be non-ignorable Standard Bayesian imputation algorithms do not fully correct biases

120 Using consumer file marketing data

121 Using consumer file marketing data Ancillary data may help identify members of hard-to-reach populations (possibly with bias)

122 Using consumer file marketing data Ancillary data may help identify members of hard-to-reach populations (possibly with bias) Ancillary data do not seem to be particularly efficient when correcting for non-response

123 Using consumer file marketing data Ancillary data may help identify members of hard-to-reach populations (possibly with bias) Ancillary data do not seem to be particularly efficient when correcting for non-response Unlikely that it would be possible to use these data to correct for a problematic sampling frame

124 What went wrong?

125 What went wrong? We can t know...

126 What went wrong? We can t know... The data are complete black boxes Proprietary

127 Moving forward from here Still lots of reasons to think that good ancillary data would substantively improve survey sampling But the demographic ancillary data used in this study were not sufficient for many purposes

128 Moving forward with current data

129 Moving forward with current data How do these results compare with traditional survey weighting techniques?

130 Moving forward with current data How do these results compare with traditional survey weighting techniques? Could a larger set of ancillary measures allow for better correctives?

131 Moving forward with current data How do these results compare with traditional survey weighting techniques? Could a larger set of ancillary measures allow for better correctives? Could linking other types of newly available data allow for better translations between respondents and society?

132 Ideally, we want data we can trust and evaluate

133 Ideally, we want data we can trust and evaluate The ancillary data need to be more transparent

134 Ideally, we want data we can trust and evaluate The ancillary data need to be more transparent The process of linking sources of data to oneanother needs to be more systematically addressed

135 Ideally, we want data we can trust and evaluate The ancillary data need to be more transparent The process of linking sources of data to oneanother needs to be more systematically addressed Is there an in-house option that could be used instead of purchasing data from corporations?

136 Ideally, we want data we can trust and evaluate The ancillary data need to be more transparent The process of linking sources of data to oneanother needs to be more systematically addressed Is there an in-house option that could be used instead of purchasing data from corporations? NSF can play a pivotal role in building such a dataset

137 Need to consider these questions with additional sources of data 1) What are we doing with the data? (supplement or source of inference) 2) How accurate are the data? 3) How complete are the data? 4) What model are we using to link the data with the world? 5) How does the model perform for different types of inference?

138 Can Microtargeting Improve Survey Sampling? An Assessment of Accuracy and Bias in Consumer File Marketing Data Josh Pasek University of Michigan NSF / Stanford Conference: Future of Survey Research Project in conjunction with S. Mo Jang, Curtiss Cobb, Charles DiSogra, & J. Michael Dennis