Issues in Applying Adaptive Design to Establishment Surveys

Transcription

1 Issues in Applying Adaptive Design to Establishment Surveys Jaki S. McCarthy US Department of Agriculture National Agricultural Statistics Service The 6th International Workshop on Internet Survey and Survey Methodology Daejeon, Korea September 2014

2 Adaptive Design in Surveys In HH surveys, adaptive design can be used to strategically allocate data collection resources to reduce costs or improve data quality Contact prioritization Call scheduling Differential allocation of resources In establishment surveys, there are unique considerations

3 Key Differences Between Establishment and Household Surveys Some populations are small Populations often highly skewed some very large establishments may account for large proportion of estimate Often rich frame or auxiliary data are available

4 Establishment Samples Some units may be sampled with certainty These units may be unlike any others: small weights but large totals Cannot use other units to substitute or weight for them May be manually imputed for Large or unique establishments may be in many samples Much previous information may be known about them But we may also have to consider overall burden across multiple surveys

5 Establishment Survey Samples Estimates of interest are often population totals Small units (even with large weights) may not contribute much to estimates Change may be as important as absolute level Estimates within a survey may be relatively independent (e.g. the Crops/Stocks Survey produces estimates of corn, cotton, potatoes and tobacco; units reporting one may have little of others)

6 Adaptive Design in ESTABLISHMENT Surveys More PRE-planning is possible More information is known up front Can identify important units May have past response history Can predict likely nonrespondents Contact and data collection methods can be customized in advance

7 Questions to answer How do you decide which units to target? What are best data collection strategies for those establishments? What are best strategies for other subgroups? How to determine if efforts improve data quality (or lower costs)?

8 National Agricultural Statistics Service Quality Measures Response rates Coefficients of Variation Percent of Estimate from Respondents Key Statistics must be identified for the last two measures

9 Quality Measures Hog and Pig Survey Hog and Pig Survey N Response Rate (2012) United States 8, % N Response Rate (2013) 7, % Survey Estimates for Est. Coverage % (2012) Est. Coverage % (2013) CV (2012) All Hogs and Pigs Pig Crop CV (2013)

10 Quality Measures Agricultural Resource Management Survey Farm Production Expenditures N Response Rate (2011) United States 34, % N Response Rate (2012) 32, % Total Production Expenditures Livestock and Poultry Expenses Agricultural Chemicals Est. Coverage % (2011) Est. Coverage % (2012) CV (2011) CV (2012)

11 Nonstandard Data Collection for Increased Data Quality For any given survey identify subgroups of units for different data collection strategies Impact operations, size, demographics, establishment type, response propensity groups Design optimum data collection strategies for these groups

12 Special Handling for Impact Operations The largest operations may get special handling Review of known information about the unit (list frame, response history, ad hoc notes) Contact by senior staff Limited data collection customization (may be mode preference, allowance for nonstandard data formats, changes to field periods, etc.) Data collection prioritization (may be held out of some surveys to ensure participation in others) Historically, this is done informally and ad hoc

13 How to decide which unit has impact? We have also considered: Establishments likely to contribute larger % to key estimates Important for nonresponse weighting Maximizing data meeting publication standards

14 Which units should get priority? Impact operations may be handled on case by case basis in establishment survey adaptive design strategies Treated as a subgroup with n=1; handling for these is unique and individual Very reluctant to experiment on them due to their ongoing importance If included, they are treated very carefully

15 What are best data collection strategies for establishments? We can use HH data collection strategies (like call prioritization, changing mode, etc.) but also: Formal special handling program for impact operations Use of high status personnel for contacts Increased use of Internet reporting Pre-populating information from prior reports Accepting other data formats (e.g., spreadsheets, downloads) Managing data collection across surveys

16 An establishment example -- NASS s Agricultural Resource Management Survey Historically, all sample units assigned to interviewers in the same way, same procedures Models can predict likely nonrespondents Units can be ranked relative to NR weighting for impact High impact + likely NR cases recruited by field office director instead of regular interviewer Early work shows promise

17 Additional ARMS Survey Strategies Least expensive data collection strategies applied to easy subgroups Mailed questionnaires More expensive in person strategies assigned to other subgroups Interviewers visit in person and pick up form or interview respondent Testing new procedures now

18 How do you measure effects? Improvements in quality measures Nonresponse bias analyses Cost reductions What else?

19 Many important differences between HH and establishment surveys Adaptive design may increase data quality and optimize allocation of data collection resources But this may look VERY different in establishment than in household surveys