Use of a nonresponse follow-up survey to assess the validity of R-indicators as a measure of the risk of bias

Transcription

1 Use of a nonresponse follow-up survey to assess the validity of R-indicators as a measure of the risk of bias Caroline Vandenplas 1, Caroline Roberts 1 and Michèle Ernst Staehli 2 1 University of Lausanne, Switzerland 2 FORS, Swiss Foundation for Research in Social Sciences ESRA conference, Ljubljana, July,

2 Study Case: Swiss 2010 ESS data (round 5) 2

3 Two different sources of information about nonrespondents ESS 2010 Data Sample Register Nonresponse Survey R-indicator Maximal Absolute Bias Nonresponse Bias 3

4 ESS Switzerland: Response Rates Target: 70% Round 5 mean: 61% Different response rate enhancement techniques have been developed to improve the participation: monetary incentives, refusal conversion, extra contact attempts. 4

5 ESS5 Data Sample of individuals (n=2850) aged 15 and over, from the SFSO s register sampling frame (stratified by 7 NUTS regions) Automated matching to telephone numbers from a private database (AZ Direct): 61% with numbers Fieldwork by M.I.S. Trend SA October 2010 March 2011 Response rate 53.3% (n=1506) 5

6 Overview of fieldwork efforts Refusals Refusal conversion Sample Individuals Up to 5 face-to-face Nonresponse survey Noncontacts 6 to 10 face-to-face By telephone N=2850 N=1227 N=1293 N=1506 N=1421 N=2089 6

7 Response Rate, Contact Rate and Refusal Rate Response Rate Non-contact rate Refusal rate Up to 5 visits Extra visits Refusal Converts Telephone contacts NRFU 43.1% 45.4% 49.9% 52.8% 73.3% 14.6% 13.1% 13.1% 9.8% 5.9% 31.6% 32.5% 26.6% 28% 12.7% N % response rate -4.8% noncontact rate -3.6% refusal rate 7

8 Research questions Do we find evidence of bias from non-response on key survey variables? Is bias in key survey variables effectively reduced by additional fieldwork efforts? How effective is the R-indicator at predicting actual bias on key survey variables? 8

9 Nonresponse survey 9

10 Nonresponse survey Two months after the end of the main survey Consisting of a single sheet (double-sided) paper questionnaire with around 20 questions To sample members who remained un-contacted or who had refused to participate in the main survey Suffers from serious limitations: mode, context, timing,.. effects extreme nonrespondents Variables selected specifically to differentiate respondents from nonrespondents 10

11 Nonresponse bias After fieldwork, bias was detected on 6 variables: Having a fix line Having a fix line number registered Satisfy with democracy Extremely Happy Believe survey are useful Feel safe at night Meet people socially frequently suffered from some bias after 5 face-to-face visits but this bias was corrected by extra fieldwork efforts 11

12 Evolution of nonresponse bias 12

13 Evolution of nonresponse bias 13

14 Research questions Do we find evidence of bias from non-response on key survey variables? à Yes but only on 6 variables out of the nonresponse survey Is bias in key survey variables effectively reduced by additional fieldwork efforts? à Increased for Fixed line phone and Registered fixed number (directly due to fieldwork strategy), Satisfy with democracy à Decreased for Extremely Happy, Believe survey are useful, Feel safe at night, Meet people socially frequently 14

15 R-indicator and Maximal Absolute Bias 15

16 Building the R-indicator Available variables from the sampling frame and survey specific variables: sex, age *** (<30, 31-44, 45-64, 65+), marital status** (not married, married or legal partner), nationality*** (Swiss, border country, other), linguistic region (German, French, Italian), Urbanization ***(urban, rural) Whether telephone number was obtained from matching*** 16

17 Fieldwork effort & representativeness Response Rate Up to 5 visits Extra visits Refusal conversions Telephone contacts NRFU 43.1% 45.4% 49.9% 52.8% 73.3% R-indicator Confidence Interval Maximal Absolute Bias ( ) ( ) ( ) ( ) ( ) N (R-indicator based on logistic regression using frame & survey variables described earlier) 17

18 Fieldwork effort & representativness 18

19 ESS Fieldwork effort & representativity Problematic variables are Nationality, Telephone and Urbanization Maximal Absolute Bias goes down! 19

20 Representativity and nonresponse bias 20

21 Maximal Absolute Bias and Nonresponse Bias 21

22 Correlations between response propensities and key variables Pearson s r (for continuous variables), biserial and point-biserial correlations (depending on whether the dichotomy reflects a discrete or continuous relation between the response options) Convert the coefficients to z-scores to facilitate comparisons between them. 22

23 Variables affected by nonresponse bias Survey Variable n r z r p Fixed line telephone 1,506 r pb = *** Registered fixed line number 1,491 r pb = *** Extremely happy 1,506 r b = *** Satisfied with democracy Believes surveys are useful Feels safe walking after dark Meets people socially frequently 1,473 r b = ' 1,458 r b = ,505 r b = ** 1,506 r b = r = Pearson's r; r pb = point-biserial correlation; r b = biserial correlation. *** p <.001, ** p<.01, * p <.05, ' p<0.1 23

24 Representativity and Nonresponse Bias The detected nonresponse bias was always smaller that the MAB based on the R-indicator Most of the variables affected by nonresponse bias correlated with the (estimated) response propensities Believe survey are useful did not correlate but Meet people frequently socially did not display bias after fieldwork effort Correlation can also be low due to low variance in response propensities, which mean low nonresponse bias 24

25 Research questions Do we find evidence of bias from non-response on key survey variables? à The R-indicator being based on estimated response propensity is difficult to interpret Is bias in key survey variables effectively reduced by additional fieldwork efforts? à The representativity does not change much after additional fieldwork effort but partial indicators.. How effective is the R-indicator at predicting actual bias on key survey variables? à Correlates with most variables suffering from nonresponse bias and with 14 of the 20 key survey variables 25

26 Strengths and limitations of the two methods R-indicator: + No bias on the nonrespondents data + Permits to identify over- and underrepresented population subgroups (Nationality, Urbanization) - Socio-demographic variables, low correlation with key variables and response propensities(in ESS 2010, significant correlation with most of the variables affected by bias) Nonresponse survey: + Data on nonresponse designed to detect nonresponse bias - Bias due to extreme nonrespondents (R-indicator increases significantly when adding respondent to the NRFU, still overrepresentation of Swiss citizens) - Effects due to different survey protocols (different mode, timing, context of the item) 26

27 Conclusions Unique opportunities to have register data and NRFU to enriched the main survey data Some bias is detected but limited Fieldwork efforts do little to reduce bias (meet people frequently socially but also better representativity of age groups, divorced vs married people,..) Both methods agree Right interpretation of the correlations? 27

28 Thank you! 28