Measuring the Correlates of Intent to Participate and Participation in the Census and Trends in These Correlates:

Transcription

1 Measuring the Correlates of Intent to Participate and Participation in the Census and Trends in These Correlates: Comparisons of RDD Telephone and Non-probability Sample Internet Survey Data Josh Pasek Stanford & University of Michigan Jon A. Krosnick Stanford & U.S. Census Bureau

2 Monitoring Threats to Census Compliance U.S. Census Bureau commissioned surveys in late 2009 and early 2010 to determine factors that might cause people to complete or not complete the form

3 To Assess This... Examine what proportion of individuals hold particular beliefs about the Census Test whether those beliefs relate to intent to complete or completion of Census form Regularly examine prevalence of beliefs to assess emergent threats to Census completion

4 Two Synchronous Data Streams 2 data streams collected for Census Bureau RDD telephone (Gallup) and non-probability Internet (E-Rewards) 13 simultaneous weeks of data collection with > 900 interviews per data stream per week Very similar measures

5 Two Synchronous Data Streams Intent to complete Census (before and after forms mailed) Reported completion of Census (after forms mailed) Census will help/hurt respondent Locate illegal immigrants Trust confidentiality Time to fill out Importance of counting everyone Respondent's participation does not matter

6 Today's Question Would These Two Data Streams Lead To The Same Conclusions?

7 Three Comparisons Proportions Relations between variables Trends over time

8 Three Comparisons Proportions Who holds particular beliefs about Census? Relations between variables Trends over time

9 Three Comparisons Proportions Who holds particular beliefs about Census? Relations between variables Which beliefs relate to expected and actual completion? Trends over time

10 Three Comparisons Proportions Who holds particular beliefs about Census? Relations between variables Which beliefs relate to expected and actual completion? Trends over time Do the surveys indicate that purported predictors changed similarly over time?

11 Weighting Unweighted Weights provided by both houses Weighted identically using anesrake (Pasek, 2010)

12 Weighting Unweighted Weights provided by both houses Weighted identically using anesrake (Pasek, 2010) Both with and without matching on dates

13 Three Comparisons Proportions Relations between variables Trends over time

14 Three Comparisons Proportions Relations between variables Trends over time

15 Proportions Absolute difference between modal categories of non-demographic variables proportion in RDD - proportion in Internet Bootstrap to test significance Assessed within each week

16 Proportions Differences Between Data Streams Frequency of Differences Difference in Percentage Points

17 Proportions Differences Between Data Streams Frequency of Differences Week 10 Census can help you RDD: 43.4% Internet: 44.2% Difference: 0.8% Difference in Percentage Points

18 Proportions Differences Between Data Streams Frequency of Differences Week 3 Rs participation does not matter - agree RDD: 54.4% Internet: 40.5% Difference: 13.9% Difference in Percentage Points

19 Proportions Differences Between Data Streams Frequency of Differences Difference in Percentage Points Week 12 Important to Count Everyone - Agree RDD: 65.7% Internet: 32.2% Difference: 33.5%

20 Proportions Differences Between Data Streams Diff. Data Frequency of Differences > % > % > % > % Difference in Percentage Points > %

21 Three Comparisons Proportions Relations between variables Trends over time

22 Relations Between Variables Regressions predicting relevant outcomes Intent to complete Census Reported completion of Census Correlations between pairs of variables

23 Predicting Census Completion Variable RDD Internet Difference Numbers shown are from a logistic regression with all measures * p<.05 ** p<.01 *** p<.001

24 Predicting Census Completion Variable RDD Internet Difference Married.31*.58**.27 Same story, same magnitude Numbers shown are from a logistic regression with all predictors simultaneously * p<.05 ** p<.01 *** p<.001

25 Predicting Census Completion Variable RDD Internet Difference Married.31*.58**.27 Age *** -.68*.83* Same story, different magnitude Numbers shown are from a logistic regression with all predictors simultaneously * p<.05 ** p<.01 *** p<.001

26 Predicting Census Completion Variable RDD Internet Difference Married.31*.58**.27 Age *** -.68*.83* Don't have time to fill out - Disagree.88** Different story, same magnitude Numbers shown are from a logistic regression with all predictors simultaneously * p<.05 ** p<.01 *** p<.001

27 Predicting Census Completion Variable RDD Internet Difference Married.31*.58**.27 Age *** -.68*.83* Don't have time to fill out - Disagree Importance of counting everyone - Agree.88** * * Numbers shown are from a logistic regression with all measures Different story, different magnitude * p<.05 ** p<.01 *** p<.001

28 Predicting Census Completion Type Same story Same magnitude Same story Different magnitude Different story Same magnitude Different story Different magnitude Total (Number of Variables) One Predictor at a Time All Predictors 57% 31% % (23) 100% (16) Numbers shown are for variables that were significant in at least one data stream

29 Predicting Census Completion Type Same story Same magnitude Same story Different magnitude Different story Same magnitude Different story Different magnitude Total (Number of Variables) One Predictor at a Time All Predictors 57% 31% % (23) 100% (16) Numbers shown are for variables that were significant in at least one data stream

30 Predicting Intent to Complete Form Type Same story Same magnitude Same story Different magnitude Different story Same magnitude Different story Different magnitude Total (Number of Variables) One Predictor at a Time All Predictors 41% 48% % (41) 100% (25) Numbers shown are for variables that were significant in at least one data stream

31 Predicting Intent to Complete Form Type Same story Same magnitude Same story Different magnitude Different story Same magnitude Different story Different magnitude Total (Number of Variables) One Predictor at a Time All Predictors 41% 48% % (41) 100% (25) Numbers shown are for variables that were significant in at least one data stream

32 Three Comparisons Proportions Relations between variables Trends over time

33 Trends Over Time Correlations between variable categories over weeks Chi-squared tests comparing differences between data streams across weeks Among variables with significant variations over time in at least one data stream

34 Sometimes the trends match

35 Trends Over Time Percent Internet RDD Date

36 Trends Over Time Percent Internet RDD Date

37 Sometimes the trends don't match

38 Trends Over Time Percent Internet RDD Date

39 Trends Over Time Percent RDD Internet Date

40 And sometimes, the trends are opposites

41 Trends Over Time Percent Internet RDD Date

42 Trends Over Time Percent RDD Internet Date

43 Trends Over Time Frequency Correlations Between Data Streams Among variables with significant variations over time in at least one data stream

44 Differences Between Streams 100 vs. "Sampling Error" % significantly different 76% significantly different % significantly different Proportions Relations Trends Chance

45 Differences Between Streams 100 vs. "Sampling Error" % significantly different 30% significantly different 76% significantly different None of the weighting strategies changed these basic results Proportions Relations Trends Chance

46 Data from one of the top RDD firms and one of the most visible Internet firms produced very different results

47 Researchers need to choose, and that choice should depend on the validity of the results

48 But Which is More Valid? More accurate self-reports using Internet mode (Chang and Krosnick, 2009) Some theoretical reasons to prefer probability sampling (even with contemporary response rates) We cannot know based on these data alone

49 Measuring the Correlates of Intent to Participate in the Census and Trends in These Correlates: Comparisons of RDD Telephone and Non-probability Sample Internet Survey Data Josh Pasek Stanford & University of Michigan Jon A. Krosnick Stanford & U.S. Census Bureau

50

51

52 Not really the point... If the methods reach different conclusions, they can't both be correct

53 What Can We Make Of This? Mode effects Slight differences in question wording Neither survey is consistently within sampling error of accuracy benchmarks (though RDD is a little closer)

54 The results are not equivalent!

55 Benchmark Comparison Comparison of modal categories Variables not used in weighting or quotas Primary household language (English) Own or rent home (Own) Children in household (Yes) Absolute difference between sample mean and benchmark (weighted)

56 Benchmark Comparison Telephone Internet Frequency Percentage Point Difference Percentage Point Difference 11.4 Avg. Error Max Error 21.1

57 p<.001 difference Benchmark Comparison Telephone Internet Frequency Percentage Point Difference Percentage Point Difference 11.4 Avg. Error Max Error 21.1

58 Benchmark Conclusions Neither survey is consistently within sampling error RDD sample is somewhat more accurate than Internet sample