Applying the Tailored Design Method in a Randomized Control Trial Experiment Survey

Transcription

1 Agenda Applying the Tailored Design Method in a Randomized Control Trial Experiment Survey Benjamin Messer Research Into Action, Portland, OR PAPOR Annual Conference, San Francisco, CA December 14-15, PAPOR Annual Conference pg. 1

2 Disclaimer The information provided and views expressed in this presentation are those of Research Into Action and do not represent the California Time of Use Working Group PAPOR Annual Conference pg. 2

3 Agenda Background 01 Study Design 02 Response Rate Results 03 Nonresponse, Mode Effects, and Data Quality 04 Survey Costs 05 Conclusions PAPOR Annual Conference pg. 3

4 Background 2017 PAPOR Annual Conference pg. 4

5 RCTs are Increasingly Used to Inform Policies, but Can Be Complex and Involve High Stakes RCTs provide a level of scientific rigor that is difficult or impossible to achieve with quasi-experimental or nonexperimental studies Conducted with more frequency in policy areas: education*, health*, energy, criminal justice, tax policy, behavior change, etc. Assisted by methodological and technological advancements Costs, time/duration, and complexity remain challenges Getting the policy right raises stakes need high confidence in results 2017 PAPOR Annual Conference pg. 5

6 Tailored Design Method Provides a Framework There is not a one-size-fits-all approach to conduct surveys in a RCT study TDM provides a framework for customizing, or tailoring, various aspects of the survey to sampled members Focus on reducing errors Develop survey procedures that work together to encourage response from all sampled members Develop procedures based on positive social exchange and that consider elements such as survey sponsorship, the characteristics of the population and sample, content of survey questions and communications, etc PAPOR Annual Conference pg. 6

7 Study Design 2017 PAPOR Annual Conference pg. 7

8 California IOUs Tasked with RCT Electric Rate Study of Customers CA Legislature and Public Utilities Commission directed the three investor-owned utilities (IOUs) to switch to Time-Of-Use (TOU) electricity rates for residential customers IOUs: PG&E, SCE, and SDG&E (cover ~3/4s of CA) IOUs first had to conduct a pilot RCT study to determine if TOU rates work, and if the rates have harmful economic and/or health impacts on vulnerable segments of the population Compare customers on standard rate (Control) to customers on TOU rates (Treatment) Electricity prices vary on TOU rates based on the time of day (e.g. lower in morning and night, higher in afternoon and evening) vs. based on how much is used on the standard rate IOUs internally surveyed customers for Pilot recruitment, and provided bill credits for participation in the study Over 55,000 total participants 2017 PAPOR Annual Conference pg. 8

9 Complex RCT Study Design Climate Region Segment Control vs. Rate 1 Control vs. Rate 2 Control vs. Rate 3 Hot Moderate Cool Non-CARE/FERA CARE/FERA Below 100% FPG 100 to 200% FPG Seniors Non-CARE/FERA CARE/FERA Non-CARE/FERA CARE/FERA PG&E, SCE PG&E, SCE PG&E, SCE PG&E SCE None PG&E, SCE, SDG&E PG&E, SCE, SDG&E PG&E, SCE, SDG&E PG&E, SCE, SDG&E PG&E, SCE PG&E, SCE PG&E & SCE: 4 Rates, 9 segments, 3 regions 30 total cells SDG&E: 2 TOU Rates, 4 segments, 2 regions 12 total cells 2017 PAPOR Annual Conference pg. 9

10 Tailoring Survey Design to Study Design Offered web, mail, and phone modes, sequentially Designed modes to be as similar as possible Used user-friendly graphical design for web/mail modes Used client letterheads, logos, and other materials Included different framing/messaging in each contact Sent contacts and offered survey in five languages Addressed communications to account holder, but asked for person in household who makes decisions about energy usage/bills Randomized lists and randomly reversed order of scales to minimize primacy/recency effects Provided a bill credit incentive upon completion of survey Included point of contact at survey center and at client headquarters Unable to make web survey mobile-friendly 2017 PAPOR Annual Conference pg. 10

11 Survey Characteristics 11 research topics and 32 research questions Survey included 39 core questions with up to 93 items, and 14 non-core questions with up to 31 items Core questions included in all modes, non-core in web-only Required about 25 minutes PG&E SCE SDG&E Web Mail Phone Core Questions Core Questions Core Questions Web Non-Core Questions PG&E Questions Non-Core Questions SCE Questions Non-Core Questions SDG&E Questions 2017 PAPOR Annual Conference pg. 11

12 Survey Implementation If participant has Letter invite 2 invites Call Booklet If participant does NOT have Letter invite then reminder Booklet then reminder Call Fielded survey October to mid-december 2016 During the 2016 election Performed a pre-test with 25 respondents All nonrespondents received one phone call, then used phone to target low-response segments Shout-out to WSU SESRC 2017 PAPOR Annual Conference pg. 12

13 Response Rate Results 2017 PAPOR Annual Conference pg. 13

14 Overall Response Rates are High 44,558 total respondents Response Rates* 100% 90% 80% 81% 80% 87% 82% 70% 60% 50% 40% 30% 20% 10% 0% PG&E SCE SDG&E Total *Used AAPOR RR1 calculation 2017 PAPOR Annual Conference pg. 14

15 Response Rates are Consistent between Control and Treatment Groups PG&E SCE 100% 80% 83% 80% 81% 81% 100% 80% 82% 81% 79% 80% 60% 60% 40% 40% 20% 20% 0% Control Rate 1 Rate 2 Rate 3 0% Control Rate 1 Rate 2 Rate 3 100% 80% 60% 40% 20% 0% SDG&E 87% 87% 86% Control Rate 1 Rate 2 Within customer segments, largest differences in RRs between Control and Rate groups is 5 percentage points Average difference is 3 percentage points 2017 PAPOR Annual Conference pg. 15

16 Some Variation in Response Rates Across Customer Segments, but Sufficient Statistical Power All cells larger than 200 respondents, most larger than 350 Response Rates 100% 90% 80% 70% 88% 75% 68% 80% 83% 90% 73% 92% 74% 60% 50% 40% 30% 20% 10% 0% 2017 PAPOR Annual Conference pg. 16

17 Nonresponse, Mode Effects, and Data Quality 2017 PAPOR Annual Conference pg. 17

18 Nonresponse Analysis Used data from the IOUs participant recruitment survey to compare between 2016 customer survey respondents vs. nonrespondents: Income: six categories Language preference: English vs. non-english Household size: continuous # of seniors in household: continuous Conducted logistic regression for each region/segment/rate, using response to survey as dependent variable and p.05 significance level 2017 PAPOR Annual Conference pg. 18

19 Income, Language, and to lesser extent, HH Size and Seniors in HH are significant response predictors Lower income less likely to respond for majority of groups PG&E: 23 of 30 groups SCE: 15 of 30 groups SDG&E: 9 of 12 groups Non-English speakers were less likely to respond for some groups PG&E: 15 of 30 groups SCE: 10 of 30 groups SDG&E: 10 of 12 groups Larger households were less likely to respond for few groups PG&E: 4 of 30 groups SCE: 6 of 30 groups SDG&E: 4 of 12 groups Households with more seniors were less likely to respond for few groups PG&E: 5 of 30 groups SCE: 2 of 30 groups SDG&E: 0 of 12 groups Nonresponse did not vary across control and treatment groups 2017 PAPOR Annual Conference pg. 19

20 Minimal Mode Effects Web/mail respondents more likely to skip questions vs. phone respondents Phone respondents more likely to select Don t know vs. web/mail respondents Did not find primacy or recency effects on Likert scale or list-based questions Similar-as-possible design across modes, randomization of lists, and randomly reversing order of scales worked 2017 PAPOR Annual Conference pg. 20

21 Data Quality Was High About 2% of respondents were identified as satisficers Straightlined on questions with 8 or more items Selected all on questions with mutually exclusive answers Skipped more than 20% of items Satisficing more common among low-income and non- English speaking respondents 2017 PAPOR Annual Conference pg. 21

22 Survey Costs 2017 PAPOR Annual Conference pg. 22

23 Costs are High but Worth It for the Clients Total survey costs are ~$800,000, excluding incentives $800K / 44,558 respondents = ~$18/respondent Average incentive amount was $52 bill credit/respondent About 3% of respondents received $75 or $100 Including incentives, total costs/respondent = ~$ PAPOR Annual Conference pg. 23

24 Conclusions 2017 PAPOR Annual Conference pg. 24

25 TDM Is Useful for RCT Study Surveys TDM does not provide a step-by-step guide on how to design a survey but is a framework for making design decisions accounting for multiple factors that influence survey response We successfully used TDM to design a survey for a large-scale RCT High response rates, good data quality, and a fairly representative sample, at a cost acceptable to the clients Data was useful for making policy decisions The potential for economic and/or health hardship for CARE/FERA customers resulted in the CPUC excluding them from defaulting onto TOU rates Plan ahead (there are lots of decisions to make), try to be flexible, and closely monitor implementation 2017 PAPOR Annual Conference pg. 25

26 Questions? 2017 PAPOR Annual Conference pg. 26

27 Contact: Benjamin Messer, Ph.D.

28 Appendices 2017 PAPOR Annual Conference pg. 28