Treating Nonresponse in the Canadian National Longitudinal Survey of Children and Youth (NLSCY)

Size: px

Start display at page:

Download "Treating Nonresponse in the Canadian National Longitudinal Survey of Children and Youth (NLSCY)"

Felicity Woods
5 years ago
Views:

1 Treating Nonresponse in the Canadian National Longitudinal Survey of Children and Youth (NLSCY) An Evolution Over 6 Cycles Marcelle Tremblay ICCCS 2006, Oxford UK September 13, 2006

2 Talk Outline Treating nonresponse in NLSCY Original Cohort Review of methods: Cycles 1 through 5 Reweighting: : Cycle 6 Longer term investigation ideas Lessons learned Conclusion ICCCS 2006, Oxford UK 2

3 NLSCY Cumulative Response Rate 100% 80% 60% 86% 79% 77% 69% Actual 66% Expected 62% 58% 40% 20% 0% Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5 Cycle 6 ICCCS 2006, Oxford UK 3

4 NLSCY Treatment of Nonresponse Reweighting within RHGs (weighting classes) Group units that have a similar propensity to respond Reweight respondents by the inverse of the weighted response rate in the group Design-based variance can be estimated with Bootstrap ICCCS 2006, Oxford UK 4

5 Reweighting: : Cycles 1 through 5 ICCCS 2006, Oxford UK September 13, 2006

6 Review of Methods NLSCY 10-year review Findings: Segmentation modelling tends to produce many RHGs Inconsistent estimates across cycles Assumption that nonresponse is independent from one cycle to the next is incorrect ICCCS 2006, Oxford UK 6

7 Inconsistent Estimates Estimates of the number of children in the Atlantic Provinces who at cycle 1 lived in a lowincome household Estimate in thousands Cycle Current Approach Nested Groups ICCCS 2006, Oxford UK 7

8 In more technical terms Modelled nonresponse at each cycle separately using only previous cycle data overall adjustment obtained by multiplying the adjustment of each cycle (independence assumption) Nonresponse adjusted weight at cycle t w ( t) = w, i i d 1 ( t) φ ( t) i 1 φ ( t 1) i 1 φ ( t 2) i L ICCCS 2006, Oxford UK 8

9 In more technical terms (cont d) Nonresponse is not independent across cycles Response probability at cycles t and t+1 φ ( t, t + 1) = φ ( t) φ ( t 1 t) i i i + In other words, we have to nest RHGs over time Becomes impractical as the number of cycles grows (too many RHGs) Therefore, need a new approach for cycle 6 ICCCS 2006, Oxford UK 9

10 Reweighting: : Cycle 6 ICCCS 2006, Oxford UK September 13, 2006

11 The New Method Realized problems with the old approach at the start of the production window for cycle 6 Only had time to implement some changes Model cumulative nonresponse at cycle 6 with one logistic regression model leads to one nonresponse adjustment (no independence assumption) Nonresponse adjusted weight at cycle t wi ( t) = wi, d 1 ( t) φ ( t) i ICCCS 2006, Oxford UK 11

12 The New Method RHGs are constructed by grouping together units that have similar estimated response probabilities using the scoring method Scoring method derived from method in (Eltinge and Yansaneh 1997) Sort estimated probabilities and group into RHGs (e.g., deciles) Easy to control the number of RHGs ICCCS 2006, Oxford UK 12

13 Nonresponse Model Building Stepwise selection of significant variables Hosmer Lemeshow test used to assess model fit Variables: Sampling frame Approx. 70 useful for modelling 6 cycles of NLSCY data Approx variables per cycle ICCCS 2006, Oxford UK 13

14 Variables Available for Model Building To narrow down the variable list Automated program to run chi-square tests on all variables Sampling frame: 20 variables 6 Cycles of NLSCY Data Automated program doesn t t work Cycle x nonrespondents are missing all of their cycle x information missing data category drives chi-square, masking other relationships ICCCS 2006, Oxford UK 14

15 Using Survey Data in Nonresponse Process Modelling Cycle 1 information for 86% of our sample Cycle 1 respondents Timeline Which cycle 1 variables are related to their subsequent nonresponse? Automated chi-square program to identify key variables Impute key variables for cycle 1 nonrespondents Production deadlines limited our exploration for cycle 6 Key variable: cycle 1 cooperation Imputed for cycle 1 nonrespondents ICCCS 2006, Oxford UK 15

16 Cooperation Variable List of variables asked directly to the respondent Cooperation score = (n_quest_nr / n_quest_posed) * 100 Great predictor of future cycle total NR Will also be used in collection ICCCS 2006, Oxford UK 16

17 Total Response Rate vs. Cooperation Variable 100% 95% 90% 85% 80% 75% 70% 65% 60% 55% Cycle 2 Cycle 3 Cycle 4 Cycle 5 50% P_100- P_90 i.c. P_90- P_80 i.c. P_80- P_70 i.c. P_70- P_60 i.c. P_60- P_50 i.c. P_50- P_40 i.c. P_40- P_30 i.c. P_30- P_20 i.c. P_20- P_10 i.c. P_10-min i.c. Less cooperative More cooperative

18 The Cycle 6 Nonresponse Model Models all nonresponse from the start of the survey 10 sampling frame variables Highest level of education Province of residence Type of dwelling Class of worker Two Cooperation Variables Imputed for cycle 1 non-respondents 9 RHGs ICCCS 2006, Oxford UK 18

19 Longer Term Investigation Ideas ICCCS 2006, Oxford UK September 13, 2006

20 Developing Further the Logistic Model Incorporating More Longitudinal Data Interaction Terms Incorporating Paradata Further develop cooperation variable More collection information Modelling with Design Weights NLSCY design weights vary a lot among individuals Early investigations show design weights are having an impact Some ways to incorporate the design weights Weighted model An independent variable categorizing the weights as small, medium and large in size ICCCS 2006, Oxford UK 20

21 Remaining Challenge: Longitudinal Consistency Fixed a systematic problem in the estimator, but the new methodology doesn t t explicitly guarantee cycle x estimates to be the same, regardless of what weights are used. Research New Ideas Calibration Approach (Singh et al. 1995) No reweighting within RHGs Calibration to key estimates from previous cycles Mass Imputation Build RHGs Use RHGs as imputation classes and apply hot-deck imputation These approaches also have their limitations, need to investigate them further ICCCS 2006, Oxford UK 21

22 Lessons Learned Modular programming Design Weights Establish at the time of sampling NR modelling and adjustment Post-stratification stratification History file Importance of good documentation ICCCS 2006, Oxford UK 22

23 Conclusion Importance of revisiting and evaluating methods what makes sense at one cycle may prove inadequate a few cycles later Try to think, develop and apply strategies in a longitudinal frame of mind (as opposed to cross- sectional) Longitudinal modelling of nonresponse Do not hesitate to investigate new approaches when the classical ones do not seem to work as well as one would like Calibration and mass imputation approaches for nonresponse ICCCS 2006, Oxford UK 23

24 Contact Information Marcelle Tremblay / Statistique Marcelle.Tremblay@statcan.ca ICCCS 2006, Oxford UK 24

25 References Singh, et al. (1995), Longitudinal Survey Nonresponse Adjustment by Weight Calibration for Estimation of Gross Flows,, American Statistical Association, Proceedings of the Survey Research Methods Section Eltinge & Yansaneh (1997), Diagnostics for Formation of Nonresponse Adjustment Cells, With an Application to Income Nonresponse in the U.S. Consumer Expenditure Survey,, Survey Methodology, June, p ICCCS 2006, Oxford UK 25

Statistical Design and Estimation taking account of Quality

Statistical Design and Estimation taking account of Quality Chris Skinner NTTS, 23 February 2011 Quality Quality as accuracy Aim: quality improvement 2 Quality Context despite increasing threats to quality