Annamaria Bianchi 1 & Barry Schouten 2

Transcription

1 Annamaria Bianchi 1 & Barry Schouten 2 1 DSAEMQ - University of Bergamo, Italy 2 Statistics Netherlands & Utrecht University 15 June 2017

2 What are Adaptive Survey Designs? Ingredients Web mode as a possible design feature Application 2

3 ASDs assign different design features to different units in the sample, according to their characteristics. Objective: provide efficient trade-offs between quality and costs. Rationale: different units may prefer or react differently to different treatments. 3

4 Important distinction between ASDs and Responsive Survey Designs (RSDs): RSD posticipates on design decisions during DC. Experiments done as part of the main study. More flexible, stronger focus on quality-cost dashboards. Motivated by surveys with long DC period, where little known about the sample and effectiveness of treatments (Groves & Heeringa, 2006 JRSSA). ASD pre-specifies adaptations and interventions and uses paradata. Refers to short DC periods with limited possibility of intervention and strong prior knowledge. 4

5 Sample units are stratified into subgroups on the basis of a number of variables, available from sampling frame, administrative data, and paradata. Depending on the levels of availability of auxiliary variables, strata may be formed before or during DC (static/dynamic design).

6

7 A survey strategy is a list of survey design features. Survey mode one of most prominent. Web it has strong quality-cost differential: cheap but low response rates + different measurement properties than interviewer-administered modes. Other design features: length DC period, length questionnaire, invitation/advance letters, number/type of remainders, level/type incentives. We denote set of candidate strategies by S={, s 1,.,s M }

8 Choice of quality function depends on the errors one wants to account for. Focus of the literature mainly on non-response error. In contexts where web is a design feature measurement error is expected to play a role. Basic ingredient: response propensities g ( s) P( R 1 g,s) = = i

9 1) Response rate gg 2) R-indicator where 3) Coefficient of variation w gg( s g ) R 12S, 2 2 S w g g( s g ) gg S CV These quality indicators assume that measurement error is affected in the same way by all design features. survey mode affects both non-response and measurement error: web known to provide low RRs and to be prone to measurement error

10 Calinescu & Schouten (2016, SRM) introduce quality functions that take both non-response and measurement error into account. Basic idea: a) model underlying causes for differences in measurement error between design features response quality indicator b) Estimate their propensities to occur for different population subgroups response quality propensities

11 Costs to be included are those relative to the design features that are varied in the design. Cost functions may include overhead and variable costs components [Web: overhead costs may come from initial set-up costs, variable costs are those of mailed advance letters/remainders, incentives]. Denoting c g (s g ) unit cost incurred by allocating strategy s g to subgroup g: C n w c ( s ) gg g g g

12 Different methods proposed in the literature: 1) Structured trial-error approach: proxy measure of non-response monitored, when above threshold, additional effort made for under-represented groups. Feasible, pragmatic, allows quality-cost trade-off (Schouten & Shlomo, 2015 ISR) 2) Prioritize sample units with lowest propensities. Some guarantee that quality improved. Sensitive to accuracy of propensity estimates and harder to link strategies (Peychev et al., 2010, SRM; Wagner, 2013, SRM)

13 3) Set different stopping rules for different subgroups to decide whether continued efforts should be made to subgroups or not. No explicit optimization. Stopping rules set on the basis of quality functions, some guarantee that quality improved. (Lundquist & Särndal, 2013 JOS) 4) Mathematical optimization method. Optimal improvement of quality. Sensitive to inaccurate input parameters. Optimization can be complex. (Schouten et al., 2013 SM)

14 Maximize the response rate given constraints: max w ( s ) gg Cost-constraint (total available budget B) Cˆ n w c ˆ ( s ) B gg g g g g g g Representativeness-constraint (minimum level of representativeness α on AVs included in response propensity model) ˆR 1 2Sˆ Precision constraints nw ˆ ( s ) n, g G. g g g g

15 ASDs natural choice for surveys that employ Web DC ASDs for Web surveys most interesting in multi-mode surveys Web offers several design features that can be adapted: content of advance letter, type of invitation, number/type of remainders, incentives, design and lay-out of survey web portal, push to mobile devices, content of questionnaire Due to absence of interviewer, Web offers relatively few paradata when sample unit never logs in to the survey website most adaptation based on auxiliary data available at the outset of the survey 15

16 Dutch LFS data Web combined with F2F Adapted design features: optional use of incentives in Web and interviewer follow-up

17 Stratification: Age of sample person relevant and links to key survey variables + younger, middle-age, and elder tend to have different Web response rates In case Web applied, break-off may be sign of low motivation or low ability to do a survey + higher response rate in follow-ups. G = < 25, 25 45, 45 + {no break off, break offሽ Population distribution: Y&NB, Y&B, M&NB, M&B, E&NB, E&B = (26%, 7%, 30%, 3%, 31%, 2%ሻ

18 Strategies: s=(,, ), s=(,, F2F+), s=(web+,, F2F+), and s=(web-,, F2F+) not admissible.

19 Phase 1 action No Web- Web+ Phase Stratum Y&NB Y&B M&NB M&B E&NB E&B (0.01) 0.45 (0.01) 0.55 (0.01) (0.02) 0.25 (0.02) 0.10 (0.02) (0.02) (0.02) (0.02) (0.02) 0.60 (0.03) 0.35 (0.02) 0.70 (0.03) 0.30 (0.02) 0.80 (0.03) (0.02) 0.25 (0.03) 0.10 (0.02) 0.30 (0.03) 0.05 (0.02) 0.25 (0.03) (0.02) (0.02) (0.02) (0.02) 0.60 (0.03) 0.35 (0.02) 0.70 (0.03) 0.30 (0.02) 0.80 (0.03) (0.02) 0.25 (0.03) 0.10 (0.02) 0.30 (0.03) 0.05 (0.02) 0.25 (0.03) Y&B: without any action in Phase 1, unconditional response propensity in Phase 2 = 35%, conditional response propensity in Phase 3 is 20% Cumulative propensities lowest for s=(web-,, ), highest for s=(, F2F, F2F+) and s=(web+,f2f, F2F+)

20 Phase 1 action No Web- Web+ Phase Stratum Y&NB Y&B M&NB M&B E&NB E&B (1) 35 (1) 40 (1) 3 50 (1) 50 (1) 50 (1) (0.1) 2.5 (0.1) 2.5 (0.1) 2.5 (0.1) 2.5 (0.1) 2.5 (0.1) 2 35 (2) 40 (2) 30 (2) 50 (2) 35 (2) 55 (2) 3 40 (2) 55 (2) 45 (2) 55 (2) 35 (2) 50 (2) (0.1) 7.5 (0.1) 7.5 (0.1) 7.5 (0.1) 7.5 (0.1) 7.5 (0.1) 2 35 (2) 40 (2) 30 (2) 50 (2) 35 (2) 55 (2) 3 40 (2) 55 (2) 45 (2) 55 (2) 35 (2) 50 (2) Cumulative costs lowest for s=(web-,, ), highest for s=(, F2F, F2F+)

21 Optimization problem: gg g g g g g g max w ( s ) gg g g g Cˆ n w c ˆ ( s ) 360k Euro ˆR 1 2Sˆ 0.85 nw ˆ ( s ) 2000, g young,middle age,elder. N. of possible designs = 54,872 direct search

22 Around 36% of candidate designs has expected R- indicator above % of the designs has expected cost below threshold Only 8% out of all designs satisfies both constraints

23 Design with highest response rate and satisfying all constraints: Y&NB, Y&B, M&NB, M&B, E&B: Web- F2F E&NB: Web+ F2F RR = 50.2%, Costs per response = 59.5 Euros, R-Indicator = 0.913

24 R-indicator Optimal RRs when costs ranges between 40 and 90 Euros: Response rate Optimal RR increase from 40% to 60% R-indicator: no clear pattern, somewhat higher for higher cost levels

25 Sensitivity analysis: Draw response propensities from Beta distribution and costs per sample unit from normal. Recompute expected RR, costs, and R-indicator for optimal ASD. Some variation in RR, little variation in R-indicator. A lot of variation in cost cost larger than threshold in 46% of the samples.

26 We describe ASDs, highlighiting specific characteristics in case web is a design feature to be varied in the design We present a realistic (somewhat simplified) application to LFS data 26

27 Bianchi, A. & Schouten, B. (2017), Adaptive and responsive design, in: Handbook of Web Surveys. Calinescu, M. & Schouten, B. (2016), Adaptive survey designs for nonresponse and measurement error in multi-purpose surveys. Survey Research Methods, 10 (1), pp Groves, R.M., & Heeringa, S. (2006), Responsive design for household surveys: tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society Series A: Statistics in Society, 169 (Part 3), pp Lundquist, P. & Särndal, C.E. (2013), Aspects of responsive design for the Swedish Living Conditions Survey. Journal of Official Statistics, 29, pp Peytchev, A., Riley, S., Rosen, J., Murphy, J. & Lindblad, M. (2010), Reduction of Nonresponse Bias in Surveys through Case Prioritization. Survey Research Methods, 4(1), pp Schouten, B., Calinescu, M. & Luiten, A. (2013), Optimizing quality of response through adaptive survey designs. Survey Methodology, 39 (1), pp Schouten, B. & Shlomo, N. (2015), Selecting Adaptive Survey Design Strata with Partial R-Indicators. International Statistical Review. 27

28 Comments invited 28