Case Study on mode effects in the German LFS

Transcription

1 Case Study on mode effects in the German LFS Final Workshop ESSnet on Data collection for Social Surveys using Multiple Modes Wiesbaden, 4-5 September 2014 Thomas Körner, Federal Statistical Office Germany

2 Acknowledgements This presentation is based on joint work with Anja Liersch, Katharina Marder-Puch and Gero Sdunek (Destatis) I am grateful for comments to Barry Schouten (CBS the Netherlands) and the entire ESSnet team The data used for the secondary analysis were collected in a project led by Sabine Köhne-Finster (formerly Destatis) together with colleagues from Destatis and the statistical offices of the Federal States Berlin-Brandenburg, Hesse, Mecklenburg-West Pomerania, Lower Saxony, North Rhine- Westphalia, Rhineland-Palatinate, Saxony, and Thuringia For details, see Körner/Liersch 2014

3 How to study mode effects? ESSnet recommends a combination of methods (Körner 2014) 1) Split sample experiments 2) Re-interview studies 3) Record-linkage approaches 4) Re-weighting approaches 5) Cognitive interviewing Case studies implemented in ESSnet Finland (combination of 1 and 4) presentation in this session Netherlands (combination of 1, 2, 4) Germany (combination of 1 and 4) plus cognitive testing in five countries (CAWI only)

4 The German case study Secondary analysis of split-sample experiment Q-MED/LFS carried out in 2009 and 2010 by Destatis and 8 State Statistical Offices (see Statistisches Bundesamt 2010) Random allocation of the respondents to four modes (CAPI, CATI, PAP, CAWI) Objectives of the secondary analysis to identify and quantify mode effects in the LFS, in particular regarding CAWI to analyse mode effects for LFS key variables (employment status, working time, questions with numerous response categories) to derive recommendations for the use of CAWI and multiple modes in the LFS

5 The study design Two step sample selection: Step 1: recruitment of households ready to participate in any mode (except CAWI) Step 2: Random allocation to data collection modes Allocation of online-ready households first Remaining households randomly allocated to other three modes Re-weighting of CAWI group to correct for structural differences Use of standard LFS questionnaire (CAWI excluding household context)

6 Limitations Random allocation blurred by (Very limited) unit nonresponse (Substantial) item nonresponse in PAP Selective allocation (and imperfect re-weighting) of CAWI subsample Restrictions for variables based on filter questions (e.g. working time) Differential share of proxy interviews (CAPI: 25%, CATI: 38%, PAP: 14%, CAWI: 9%) Limited sample size Sampling frame based on quota sample (former German HBS participants)

7 Selected Results

8 Employment status 70 Significance of differences (p-values) CAPI-CATI CAPI-CAWI in % CAPI-PAP < CATI-CAWI CATI-PAP <0.001 CAWI-PAP < CAPI CATI CAWI weighted PAP Employed Unemployed Outside labour force

9 Average weekly hours Hours worked (all employed) Significance of differences (t-test) Usu Actu CAPI-CATI CAPI-CAWI CAPI-PAP <0.05 < Hours usually worked CAPI CATI CAWI (weighted) PAP Hours actually worked CATI-CAWI <0.05 <0.1 CATI-PAP <0.01 CAWI-PAP <0.001 <0.001

10 Hours worked (full-time employees) 42 Significance of differences (t-test) 40 Usu Actu Average weekly hours CAPI-CATI CAPI-CAWI CAPI-PAP Hours usually worked CAPI CATI CAWI (weighted) PAP Hours actually worked CATI-CAWI CATI-PAP CAWI-PAP

11 Distribution of hours actually worked in % Significance of differences (p-values) CAPI-CATI CAPI-CAWI <0.01 CAPI-PAP < CATI-CAWI <0.01 CATI-PAP < CAPI CATI CAWI (weighted) PAP CAWI-PAP 0 up to Full-time employees only

12 Status in employment Significance of differences (p-values) 50 CAPI-CATI in % CAPI CATI CAWI (weighted) PAP Self-employed Employee Blue collar worker Public official Apprentice Other small job CAPI-CAWI < CAPI-PAP CATI-CAWI < CATI-PAP CAWI-PAP <0.01

13 Distribution higher educational attainment Significance of differences (p-values) 40 CAPI-CATI in % 30 CAPI-CAWI < CAPI-PAP 10 CATI-CAWI < CAPI CATI CAWI weighted PAP CATI-PAP <0.001 College of public administration University of applied sciences diploma CAWI-PAP University degree Doctorate

14 Conclusions There is evidence for mode-specific measurement effects Employment status: deviating estimates in PAP Hours actually worked: self-administered vs. interviewer administered modes Status in employment (and other variables with numerous response categories): self-administered vs. interviewer administered modes Mode effects are however not specific to CAWI, and no major impediment to use CAWI Careful questionnaire design needed, taking into account the sensitive elements of all modes Some caution due to limitations of the study

15 Similar findings from the Dutch case study Employment and unemployment estimates for CATI and CAWI differ significantly from those in CAPI Coverage, nonresponse and measurement effects all play a role (and were disentangled in the Dutch analysis) Coverage and nonresponse effects with respect to CAPI can be explained to a large extent by standard registry variables of Statistics Netherlands Measurement effects for both CATI and CAWI are not significantly different from zero at the 5% level For further details, see Schouten/van der Laan 2014

16 Recommendations Regarding the LFS Restrict use of self-administered PAP questionnaires Spend further effort on improving the measurement of questions with numerous response categories (problematic in CATI) Put more emphasis on mode sensitivity in designing and testing questions For further research Launch experimental studies on mode effects in employment variables Use sufficiently large samples to allow for an analysis of subpopulations Use integration of qualitative and quantitative designs to learn more about the sources of mode effects

17 Thank You. Thomas Körner