CHIS 2013 Sample Design and Survey Methodology TAC. August 30, 2012

Transcription

1 CHIS 2013 Sample Design and Survey Methodology TAC August 30,

2 Welcome Ninez Ponce, CHIS PI David Grant, CHIS Director A. A. Afifi, TAC Chair 2

3 Introductions (In order of Adobe Connect roster) 1) Name 2) Where you work 3) Expertise 3

4 Meeting Goals Overview of CHIS methodology past and present Discuss methodological topics most relevant to CHIS Sketch out design and experiment ideas as they arise 4 No burden to leave today with clear solutions to specific challenges work groups may follow

5 Meeting Themes 5 CHIS Field Schedule Methodological experiments and implementing methodological changes? Interview Length Shortening interview length; What role does questionnaire length play in overall survey cost and data quality? Measurement Improve measurement maintaining data series?

6 Costs and Cost/Error Tradeoffs Adapt CHIS methodology to reduce overall survey costs Proactive consideration (e.g., experiments and tests that will reduce costs long term) Some cost savings incur up front costs Most cost decisions are cost/quality trade offs Further adaptive/responsive design approach to data collection 6

7 The Interview Telephone survey on wide range of health issues (see questionnaire content handouts) Only mode for all data collection Landline RDD, Cell RDD, Surname List Pre notification letters with $2 incentive when address matched from records Westat interviewers conduct interviews from call centers and home 7

8 Sample designed to represent 8 Residents of CA Stratification by CA counties and county groups In 2011, 44 strata representing 58 counties Most with targets of n=600 ea. Smallest counties (17) in 3 grouped strata Key ethnic and racial subgroups Initial proportionate allocation accounts for this Vietnamese and Korean oversample since 2001 Am Indian/Alaskan Native in 2001 and current

9 Field Schedule Every two years since 2001 In past, data collected w/in approx. 9 month window forward in quarterly rolling sample starting in January of 2 year cycle T4 currently in field, finishing end of Dec. 9

10 TCE Building Health Communities 14 geographically defined communities in CA Chosen by TCE as part of 10 year health improvement effort CHIS data collection is baseline and follow up measurement (i.e., panel in development ) CHIS 2009 oversampled numbers in these areas Expansion of sample to insure 400 completes per site Participants re contacted in 2012 and asked about future participation (n 100 per site)

11 Status of 2013 Planning 11 SDSM TAC is one of 5 TACs Questionnaire being revised based on other TACs Room for experiments resulting from today Sample design Room for adjustments based on this meeting Increase/decrease in cell sample; alternative sampling methods for hard to reach populations (e.g., RDS)

12 CHIS Performance Over Time Royce Park 12

13 13 CHIS Response Rates Screener Interview Extended Interview Household Adult Child Adolescent Overall Household Adult Child Adolescent * Landline/List Sample (excludes cell phone sample), Weighted, AAPOR RR4

14 CHIS Response Rates Extended Child Response Rate (%) Extended Household Extended Adult Extended Teen Screener Interview Overal Household Overall Adult Overall Child Overall Teen * Landline/List Sample (excludes cell phone sample), Weighted, AAPOR RR4

15 Screener & Extended Response Rates Extended Child Response Rate (%) Extended Household Extended Adult Extended Teen 35.6 Screener Interview * Landline/List Sample (excludes cell phone sample), Weighted, AAPOR RR4

16 Overall Response Rates Response Rate (%) Screener Interview Overal Household Overall Adult Overall Child Overall Teen * Landline/List Sample (excludes cell phone sample), Weighted, AAPOR RR4

17 CHIS 2009 Screener Response Rate by Strata Overall State RR = San Francisco Santa Clara Solano San Mateo Los Angeles Ventura San Diego Riverside Contra Costa Lake El Dorado Fresno San Joaquin Sacramento Orange Nevada Alameda Imperial Santa Barbara Yolo San Bernardino Placer Kings Marin Sonoma Stanislaus Napa Mendocino Merced Madera Monterey Yuba Kern Santa Cruz Sutter San Benito Tulare Sierra Balance San Luis Obispo Shasta Butte Colusa, Glenn, Tehama Humboldt North Balance Response Rate (%) 17 CA County * Landline/List Sample (excludes cell phone sample), Weight, AAPOR RR4

18 CHIS 2009 Screener Response Rate by Strata Smallest Counties Overall State RR = San Francisco Santa Clara Solano San Mateo Los Angeles Ventura San Diego Riverside Contra Costa Lake El Dorado Fresno San Joaquin Sacramento Orange Nevada Alameda Imperial Santa Barbara Yolo San Bernardino Placer Kings Marin Sonoma Stanislaus Napa Mendocino Merced Madera Monterey Yuba Kern Santa Cruz Sutter San Benito Tulare Sierra Balance San Luis Obispo Shasta Butte Colusa, Glenn, Tehama Humboldt North Balance Response Rate (%) 18 CA County * Landline/List Sample (excludes cell phone sample), Weight, AAPOR RR4

19 CHIS 2009 Screener Response Rate by Strata Medium Counties Overall State RR = San Francisco Santa Clara Solano San Mateo Los Angeles Ventura San Diego Riverside Contra Costa Lake El Dorado Fresno San Joaquin Sacramento Orange Nevada Alameda Imperial Santa Barbara Yolo San Bernardino Placer Kings Marin Sonoma Stanislaus Napa Mendocino Merced Madera Monterey Yuba Kern Santa Cruz Sutter San Benito Tulare Sierra Balance San Luis Obispo Shasta Butte Colusa, Glenn, Tehama Humboldt North Balance Response Rate (%) 19 CA County * Landline/List Sample (excludes cell phone sample), Weight, AAPOR RR4

20 CHIS 2009 Screener Response Rate by Strata Largest Counties Overall State RR = San Francisco Santa Clara Solano San Mateo Los Angeles Ventura San Diego Riverside Contra Costa Lake El Dorado Fresno San Joaquin Sacramento Orange Nevada Alameda Imperial Santa Barbara Yolo San Bernardino Placer Kings Marin Sonoma Stanislaus Napa Mendocino Merced Madera Monterey Yuba Kern Santa Cruz Sutter San Benito Tulare Sierra Balance San Luis Obispo Shasta Butte Colusa, Glenn, Tehama Humboldt North Balance Response Rate (%) 20 CA County * Landline/List Sample (excludes cell phone sample), Weight, AAPOR RR4

21 Adult Response Rate by Sample Adult Characteristics Landline/List Cell phone Total Sex Male Female Age 18 to 30 years to 45 years to 65 years Over 65 years Type of household With child and/or Adolescent Without child or Adolescent Number of adults in household or more Adult was screener respondent Yes No

22 How do we compare to other surveys 22 RR s difficult to compare due to disposition definitions, RR calculations, and undocumented differences between surveys Comparable to CA BRFSS over the years 27.1% CA BRFSS RR v BRFSS has single adult interview, CHIS considers any one interview (Adult, Teen, Child) a complete

23 Address based Sampling Pilot Matt Jans 23

24 ABS Address based Sampling Pro: More complete coverage of HH population than telephone frames Con: Unless survey questionnaire is mail, requires switch to another mode for completion Challenge: How to obtain phone number, and eventually completed phone surveys, from sampled addresses 24

25 ABS Pilot Design 25 Implemented in 2 BHC sites Selected for lower survey saturation Spanish language prevalence Short Questionnaire (i.e., screener form) mailed to all addresses Requests adult response A few basic health and demography questions and Data expected end of year

26 Test of Messaging and Outreach 26 Random half in each site receive at 2 nd and 3 rd mailing TCE logo on the out going envelope Insert showing TCE/BHC support (1/3 page slip in bright color or with color logo) Seeking support of Program Managers (PMs) in the two communities Vouch for our survey with their members Distribute info via list, etc.

27 Designed to Compare Boyle Heights Compares to Usual CHIS Eligibility determined by usual CHIS rules and location within BHC area Random half of 2 nd and 3 rd mailing receive additional TCE logo on outgoing envelope and TCE insert. Yields 400 completed that fit usual CHIS protocol for comparison to CHIS data Merced Compares to Merced BHC, 2010 Eligibility determined by BHC age and parenting status criteria, and location within BHC area Random half of 2 nd and 3 rd mailing receive additional TCE logo on outgoing envelope and TCE insert. Yields 400 completed that fit BHC eligibility criteria for comparison to BHC oversample data 27

28 ABS Comments and Questions Experiences moving respondents from mail to another mode Dealing with DSF quality (e.g., non residential addresses, drop points) Other ABS ideas 28

29 Meeting Second Half TAC wide Discussion 29

30 Alternate Sampling Methods 30 Geographic Stratification Other approaches than county stratification ABS Best practices Challenges moving Rs from mail to phone (or other modes) Network sampling and related methods Synthetic estimates for costly areas

31 Multimode Data Collection We would expect higher response rates from other modes. What novel combinations are costcompetitive? What techniques work best for long surveys via the web? Mail? Combinations? T ACASI options: How long/how many questions? Nonresponse follow up for key estimates by 31 mail?

32 Matrix Sampling Random subgroups of respondents receive sections of the questionnaire; Missing data are imputed prior to release Models for best practices? Who has done this with public release data successfully? How large/small can sampled question sets be? How many units per question? How large should a common question core be, particularly as imputation core? 32

33 CHIS Questionnaire Duration (minutes) * Screener Interview Extended Interview Adult Child Adolescent Parental permission * Preliminary 33

34 CHIS Questionnaire Duration (minutes) Extended Adult 30.0 Minutes Extended Adolescent Extended Child * * Preliminary Parental permission Screener Interview

35 Parent and Teen Consent Focus group results Ask for teen participation more softly Allow parent to opt out of types/sets of questions Other reflections The order of selection and permission is important E.g., is Your child has been selected. May we interview them different from May we interview one of your children? Mary has been selected? Don t ask for teen names 35

36 Parent and Teen Consent Known methods for gaining parent consent? Novel incentives or incentive structures known to work with youth? Raffle for ipad or similar? Smaller, cheaper pre paid incentives? Multi phase incentives? E.g., parent given incentive for allowing teen (beyond original incentive); Teen given own incentive Message tailoring to avoid refusal at consent? To gain cooperation with teen? 36

37 Adolescent Rates Rates (%) Adol Int Rate Given Parent Consent ** (all interview types) Completion Rate** (all interview types) Permission Rate** Extended Adolescent RR* * Landline/List Sample (excludes cell phone sample), Weighted, AAPOR RR4 ** Unweighted rate 7.9 Overall Adolescent RR*

38 Interview Optimization Optimizing work assigned as well as follow up and call number/timing decisions Best call windows and time of day/week Behavior tailoring at first call and key points (e.g., consent) Do concepts of responsive/adaptive design apply to phone surveys? Statistical Process Control models for phone call management (in or out of survey research) 38

39 Prioritization of Projects What seems most tractable and interesting? Will follow up with quick survey to rate 39

40 In Closing Comments and summary available in a few weeks Possibility of forming working groups May send poll for interest Welcome calls, comments, and collaborations mjans@ucla.edu, npark@ucla.edu, dgrant@ucla.edu, nponce@ucla.edu 40 Thank You!