Puget Sound Energy s Home Energy Reports

Transcription

1 Puget Sound Energy s Home Energy Reports 2012 Impact Evaluation Prepared for Puget Sound Energy Prepared by KEMA, Inc. March 2013

2

3 Table of Contents 1. Executive Summary Program Background Evaluation Overview Key Findings Introduction Program Description Evaluation Objectives Overview of This Report Research Design and Data Collection Activities Experimental Design Data Sources and Disposition Billing Analysis Joint Savings Analysis Methodology Differences-in-Differences Joint Savings Analysis Results Program Savings Measured Program Savings Joint Savings Analysis Conclusions List of Appendices... A-1 A. Randomization Test... A-1 B. Impact Methodology... B-3 B.1 Difference-in-differences... B-3 C. Survey Instrument... C-4 KEMA, Inc. i

4 Table of Contents List of Exhibits Table 1-1: Summary of Savings for PSE HER Table 3-1. Billing Data Disposition Table 3-2. Survey Dispositions Table 5-1: Participating HER Households by Report Status and Mailing Frequency Table 5-2: HER Savings Per Household Based on Actual Consumption in Table 5-4: Annual Joint Rebate Savings Per Household for Electric and Gas, Current and Suspended Groups Table 5-5: Count of CFL Bulbs and Fixture Purchased Per Household in Table 5-6. Annual Joint Upstream Savings Per Household for Current and Suspended Treatment Groups KEMA, Inc. ii

5 1. Executive Summary 1.1 Program Background Puget Sound Energy (PSE) implemented the Home Energy Reports (HER) Program in The HER Program delivers customized information on energy consumption to participating households and compares the household s energy consumption to that of similar neighboring homes. In addition, the Report provides personalized tips on how to save energy based on the energy usage and house profile. The HER Program was designed to motivate households to reduce energy consumption through behavioral changes and participation on other PSE energy efficiency programs. The program was structured as a randomized controlled trial wherein the initial eligible population was randomly assigned to the treatment and control groups. Around 40,000 dual fuel, single family homes were randomly selected to receive the Report while 44,000 dual fuel, single family homes did not receive the report and were assigned as the control group. All households in the treatment group received the report either monthly or quarterly for two years. At the start of the third year of the HER Program, approximately 10,000 treatment group households were randomly selected to stop receiving the reports. This created a second treatment group designed to test the persistence of report-based savings after the cessation of reports. 1.2 Evaluation Overview The main goal of this impact evaluation was to develop HER Program savings estimate for year Specifically, the main objectives are as follows: 1. Measure the reduction in electric and natural gas consumption between the control group and the two HER treatment groups: a. The current treatment group that continues to receive reports in the fourth year. b. The suspended treatment group that received reports for the first two years of the program but has not received reports for the third and fourth years of the program. 2. Quantify the savings from HER-related increased uptake of other PSE programs which may be present in the measured consumption reduction: a. An increase in the number of participants and/or extent of participation in PSE rebate programs due to the home energy reports b. An increase due to home energy reports in the number of purchasers and/or number of purchased bulbs supported by PSE and NEEA upstream lighting programs KEMA, Inc. 1-1

6 3. To provide an estimate of 2012 HER savings which are free of double counted savings resulting from participation in PSE rebate and upstream lighting programs in previous HER years This evaluation used historical consumption data to measure the difference in consumption between the treatment and control groups. Savings estimates were also measured for the different treatment subgroups, such as the monthly and quarterly groups and the current and suspended groups. This evaluation also quantified the potential for double counting energy savings due to participation in other PSE rebate and upstream programs. DNV KEMA used the PSE program tracking data to quantify joint savings due to participation in other PSE rebate programs. To quantify joint savings from upstream programs where there is no tracking data, we used the participant survey. 1.3 Key Findings The primary goal of this evaluation is to develop the 2012 PSE HER Program savings estimates that are free of any joint savings due to participation in other PSE energy efficiency programs. The summary of results is presented in Table 1-1. Treatment Groups Table 1-1: Summary of Annual Savings for PSE HER 2012 HER Measured Savings (Per Household) Joint Savings (Per Household) Electric (kwh) Per Household Credited Savings All Households Current (+/- 47.9) ,330,705 Suspended (+/- 63.3) ,628,920 Total 6,959,625 Gas (therms) Current 12.7 (+/- 2.9) ,670 Suspended 8.7 (+/- 3.7) ,573 Total 272,243 Both current and suspended treatment groups generated statistically significant savings in The overall credited savings for electric and gas were 300 kwh and 11 therms per household, respectively. These savings constitute 2.8% and 1.3% of the household s average electric and gas consumption, respectively. KEMA, Inc. 1-2

7 These program savings exhibit two different kinds of persistence that remain open questions for programs of this type. Those current households that continued to receive reports through the fourth year generated savings at or above levels established in the first two years of the program. The suspended group households that were in their second year of not receiving reports still generated well over half of the savings of the current treatment group. The active HER Program continued to promote other PSE gas rebate programs causing a statistically significant increase in gas rebate program savings among the current treatment group. KEMA, Inc. 1-3

8 2. Introduction 2.1 Program Description In 2008, Puget Sound Energy (PSE) became the second utility in the U.S. to implement an innovative program designed to conserve energy. The program, referred to as the Home Energy Reports (HER) Program, utilizes a social marketing campaign, with normative messaging techniques, to encourage responsible energy behavior and choices. The campaign, administered by Opower, provides HER to households in PSE s combined gas and electric service territory. The program serves dual fuel, single family households. The HER program provides recipients with feedback on their household energy use by comparing the recipient s energy usage with that of neighboring homes, essentially using peer pressure to achieve energy savings. In addition, the reports provide tips regarding steps households can take to reduce energy consumption through behavioral changes and participation in other PSE energy efficiency programs. The program is structured as a randomized controlled trial experimental design to facilitate precise and unbiased estimates of average per household savings that are small on a percentage basis. This means that the initial eligible population was randomly assigned to treatment and control groups. In 2010, a subset (approximately 10,000) of the original HER treatment group were randomly selected to stop receiving the reports. This created a second treatment group designed to test the persistence of reportbased savings after the cessation of reports. The savings for this group is estimated separately in 2011 and Evaluation Objectives This report focuses on energy savings due to the PSE HER program for calendar year The specific objectives are as follows: 4. Measure the reduction in electric and natural gas consumption between the control group and the two HER treatment groups: a. The current treatment group that continues to receive reports in the fourth year. b. The suspended treatment group that received reports for the first two years of the program but has not received reports for the third and fourth years of the program. 5. Quantify the savings from HER-related increased uptake of other PSE programs which may be present in the measured consumption reduction: a. An increase in the number of participants and/or extent of participation in PSE rebate programs due to the home energy reports KEMA, Inc. 2-4

9 b. An increase due to home energy reports in the number of purchasers and/or number of purchased bulbs supported by PSE and NEEA upstream lighting programs 6. To provide an estimate of 2012 HER savings which are free of double counted savings resulting from participation in PSE rebate and upstream lighting programs in previous HER years 2.3 Overview of This Report This evaluation is organized as follows: Section 2 of the report presents the overall research design and data collection activities. Section 3 discusses the methodology used and Section 4 presents the results of the PSE HER program impact evaluation. KEMA, Inc. 2-5

10 3. Research Design and Data Collection Activities 3.1 Experimental Design Before the program launched, a group of 83,811 single family homes, located in PSE s combined gas and electric service territory, were selected to participate in the test and control group based on the following criteria: Dual Fuel (home uses both natural gas and electricity, which are both provided to the service address by Puget Sound Energy) Single family residential home Uses more than 80 MBtu of energy per year Home does not utilize a Solar PV system Address must be available with parcel data from the county assessor Has a bill history that starts on or before January 1, 2007 Home must have 100 similar sized homes (neighbors) within a two mile radius Home must have automatic daily meter reads After selection of participating households, 39,757 homes were randomly assigned to participate in the treatment group and the remaining homes were used to serve as a control group. Of the selected treatment homes, 25 percent were randomly selected to receive HER on a quarterly basis, while the remaining 75% percent homes received the report monthly. The random assignment of monthly and quarterly reports allows both PSE and Opower to test the effect of the frequency of receiving the report on energy savings. The program was implemented in October 2008 and for the first two program years (November 2008 October 2010) the 39,757 treatment group households received a report on a monthly or quarterly schedule per their assignment. Beginning November 2010, 9,674 treatment homes were randomly assigned to stop receiving the HER. This treatment group will be referred to as suspended treatment group while household who continued receiving report will be referred to as current treatment group. shows a diagram of the different HER groups used in this evaluation. KEMA, Inc. 3-6

11 Figure 3-1: HER Control and Treatment Groups HER Population Control Treatment Monthly Quarterly Suspended Suspended Current Current KEMA, Inc. 3-7

12 3.2 Data Sources and Disposition This impact evaluation used information collected from customer billing data, program tracking data and participant survey data. These data collection activities are described in the following subsections Billing Analysis PSE s Meter Data Warehouse provided daily billing data for each home included in the treatment and control groups from January 2007 to December PSE also provided weather data, and other sitespecific information such as move-out dates, zip code, house square footage, home value, number of occupancy, and other site characteristics. The pre-program period is from October 2007 to September This report focuses on program saving for calendar year Prior to analysis, KEMA examined billing for completeness and potential data issues such as duplicates, extreme values, missing observations and inconsistencies. Data preparation steps included: Duplicate reads When meters produced two identical reads in one day, one read was excluded from the analysis. When a meter produced two different reads in a day, both reads were excluded from the analysis. Negative reads were removed from the analysis. Extreme values, greater than 400kWh per day or 40 therms per day, were excluded from the analysis. Missing daily observations, caused by missed daily reads, were generally followed by a single read that covered the multiple missing days. Data imputation was employed by distributing energy consumption of that next non-missing meter read. Imputation was only done when the next non-missing read covered the missing period as indicated by start and end read dates. All households with less than 122 days of data during any of the four years (pre-period and 4 post-program) were removed from the final analysis dataset. Table 3-1 summarizes the program population, counts of households removed from the analysis and the final sample used in billing analysis. Households whose occupancy status changed during the analysis period and households without assigned control group were removed from the final HER population. KEMA, Inc. 3-8

13 Roughly 12% of the original treatment group was located in zip codes that did not have an assigned control group while 22% of the PSE sample population moved out during the analysis period. Table 3-1. Billing Data Disposition Population Control Treatment Total Original population 44,124 39,757 83,881 Not in customer/billing data Not randomly assigned 4,864 PSE sample population 44,089 34,854 78,943 Other Opower program 111 Inconsistent zip codes Other data issues Move-outs 9,765 7,816 Final Sample for ,693 26,590 60,283 Monthly - Current 12,703 Monthly - Suspended 6,348 Quarterly - Current 5,046 Quarterly - Suspended 2,493 Note: Some sites have multiple issues. A small percentage of households in the treatment group have opted to not receive the reports at some point since the Pilot began. The households are not removed from the analysis even though they are no longer receiving the treatment. This is referred to as testing the intent to treat and is necessary to get an unbiased estimate of the reports effect. Tests were performed on the final sample to confirm that the sample was balanced after the natural attrition of move outs and the other reductions in sample size. The tests compared differences in baseline usage and household characteristics between control and treatment groups. The tests confirmed that the final sample used in 2012 evaluation was balanced. Results from t-tests are provided in Appendix A. KEMA, Inc. 3-9

14 3.2.2 Joint Savings Analysis Program Tracking Data PSE provided tracking data for HER households participating in other PSE rebate programs during the HER period. The data included measure description, estimated date of install, number of units bought and claimed savings. The program tracking data was used to avoid double counting savings by adjusting HER savings for other savings related participation in PSE rebate program Participant Survey Data The DNV KEMA team utilized a Computer Aided Telephone Interview (CATI) survey to collect data used in the analysis of upstream lighting program. Table 3-2 provides counts of surveyed households and response rates. DNV KEMA formed a starting population by removing the participants from the year 3 HERs evaluation from the participating, suspended, and control households. This resulted in 16,571 households in the starting population. Table 3-2. Survey Dispositions Description Number Percent Starting Population 16,571 Never Called 2,711 Selected for Calling 13, % Known Not Eligible 2,801 20% Valid Sample 11,059 80% Full Completes 1,958 18% No eligible purchases 699 6% Refused 2,696 24% Not completed, eligible 2,768 25% Not completed, unknown eligibility 2,938 27% DNV KEMA stratified by household energy use and randomly selected 13,860 households to survey. One-fifth of these households were determined to be ineligible because of the following reasons: disconnected numbers, business numbers, fax or computer tones, cell phones, no such person at number, did not live at the address on record, or were not familiar with household lighting purchases in KEMA, Inc. 3-10

15 Of the valid sample, DNV KEMA completed 1,958 surveys with households that had at least one CFL or CFL fixture purchase in We also contacted 699 respondents who indicated they did not make any CFL or CFL fixture purchases in The survey screened out these 699 respondents, but tracked their purchase answers for use as zeroes in the upstream participation analyses. Taken together, we obtained a response rate of 24 percent, which is a typical rate for this type of survey. Calls took place from January 2 nd to 24 th, Respondents were called eight times over at least two weeks before being considered unreachable. DNV KEMA left voic messages after the third call that indicated what the call was about and provided a callback number. The surveys asked about CFL and CFL fixture purchases in 2012 and demographics. This survey was much shorter than the one used for the year 3 evaluation. For the questions common to both years, wording was kept as consistent as possible. KEMA, Inc. 3-11

16 4. Methodology This evaluation used daily household energy consumption data to estimate the reduction in energy consumption resulting from HER. This consumption reduction is the full measure of savings caused by mailing of reports and is referred to here as measured savings. Savings were estimated using a differencein-differences approach. Measured savings were compared for the following groups: Control vs Current and Suspended treatment groups, Current vs. Suspended treatment groups, and Monthly recipients vs. quarterly recipients. The HER program has a secondary objective of promoting other energy efficiency programs within PSE. If this promotion is successful, some portion of the measured savings will include part or all of the savings claimed by other programs. We refer to this as joint program savings because credit for these savings is shared by both HER program and other PSE programs. To account for joint savings, DNV KEMA utilized PSE tracking data and end-use load shape data to quantify the potential for double counting of energy savings with PSE rebate programs. Also, DNV KEMA used the household survey to address joint savings potential due to participation in upstream CFL and fixture programs, for which there is no tracking data. Joint savings are discussed in the subsequent sections and are ultimately removed from the 2012 savings estimate to avoid double counting. The measured savings with joint savings removed will be referred to as credited savings in this report Differences-in-Differences The difference-in-differences approach is a simple, robust approach to measuring program-related savings in a randomized experimental design framework. The approach compares mean energy consumption between the pre- and post-report periods for both the treatment and the control groups. 1 We explicitly avoid using the gross/net terminology here to avoid confusion with the more typical freeridership/spillover usage of those terms. It is important to note that because of the experimental design framework of the HER program, freeridership is not an issue. KEMA, Inc. 4-12

17 A simple pre-post comparison of treatment group consumption, without a control group, does not account for systemic effects (economic factors, fuel prices, etc) that impact all households consumption patterns during the measurement periods. It is possible that these systemic effects will increase or decrease consumption in the post-report period unrelated to the effects of the reports. This would bias the estimate of consumption reduction, a particular concern when expected reduction is relatively small. The control group, pre-post difference provides a robust estimate of the non-program, systemic effects on consumption that are observed in the post-report period. Because the control group was randomly assigned, their response to the systemic effects is representative of the treatment group response. The term difference-in-differences refers to the removal of the of the control group difference (systemic effects only) from the treatment group difference (program effects and systemic effects). A full discussion of the difference in difference approach can be found in Appendix B. 4.2 Joint Savings Analysis The goal of the joint savings 2 analysis is to quantify savings that are included in the measured savings but have already been credited to other PSE energy efficiency programs. Energy efficiency purchases that occur directly through a Puget Sound Energy rebate programs are tracked in PSE data systems. DNV KEMA analyzed PSE rebate program tracking data to identify possible increased uptake of other PSE energy efficiency programs by the treatment group. These programs include clothes washers, energy efficient heating systems, etc. In these program tracking data systems, rebate program participation and associated savings are tied directly to the customer within the HER program treatment and control groups. The experimental design framework makes it possible to accurately measure any increased activity in programs by the HER program group. For this analysis, DNV KEMA compiled data on all rebated installations, for both treatment and control groups. Savings were assigned on a daily basis starting with the installation date and carrying forward to the measure life. 3 Savings are apportioned across the days of the year based on measure level load shapes so that savings occur during the year approximately when they would be captured in the difference-indifferences calculations. For the 2012 rebate program joint savings calculation, the total accumulated savings of the control group in 2012 is removed from the total accumulated savings of the treatment 2 Sometimes referred to as uplift in other evaluations. 3 All measure lives are longer than the four years the HER Program has been in place. KEMA, Inc. 4-13

18 group in The difference is the effect of HER on rebate program activity. These are savings that would not occur if the HER Program was not operating. However, because the savings are already being claimed by the rebate programs that facilitate the participation, this difference will be removed from the overall measure consumption reduction caused by the HER Program. DNV KEMA uses a similar process to estimate joint savings associated with the upstream CFL bulb and fixture programs. DNV KEMA utilizes the survey data in place of the rebate program tracking data. The survey was conducted to gather information on the purchase and installation of CFLs for HER program treatment and control groups for calendar year The survey gathered store-specific information associated to respondent s CFL purchase (bulbs or fixtures). Data on participating retailers were used to calculate the number of purchased program CFL bulbs and fixtures. DNV KEMA calculated the difference in PSE sponsored CFLs between the treatment and control group households to determine the amount of savings associated to CFLs purchased in To expand these results across four years of the program, DNV KEMA assumed these bulbs were all installed on the first day of each program year (January 1 st ) and the joint savings carried forward on a load shape-weighted basis. The 2011 upstream purchase data were used as a proxy for purchases prior to 2011 before an upstream survey was conducted. The resulting estimates of joint savings for different time periods were then multiplied by the savings per CFL value of 24 kwh provided by PSE. Appendix C provides the survey instrument used to gather CFL installation for HER program KEMA, Inc. 4-14

19 5. Results Results of the impact evaluation are provided for calendar year These results will be used to support PSE savings claims for the 2012 HER Program. Table 5-1 provides the counts of households in each treatment category that were used in this evaluation. Table 5-1: Participating HER Households by Report Status and Mailing Frequency Treatment Group Monthly Quarterly Total Current 12,703 5,046 17,749 Suspended 6,348 2,493 8,841 Total 19,051 7,539 26,590 Section 5.1 provides the overall actual savings achieved in calendar year The results include average household and total savings for current and suspended treatment groups as well as total PSE HER savings. Sub-sections discuss each of the components of the overall savings the measured savings, the rebate program joint savings and the upstream joint savings Program Savings Table 5-2 provides the household- and program-level savings for the HER Program for calendar year These results are calculated separately for current and suspended treatment groups. The three components to estimating credited savings are the following: Measured savings is the average difference in consumption between HER treatment groups and the control group. It is calculated using a difference-in-differences approach that compares treatment and control group consumption in the pre- and post-report periods. Rebate program joint savings represents the increased activity in PSE rebate programs as a result of receiving, or having received, the report. This is the difference in PSE rebate program savings between the two PSE HER treatment groups (current and suspended) and the control group. Upstream program joint savings represents the increased use of PSE-supported CFL bulbs and fixtures as a result of receiving the Home Energy Report. This is the difference in PSE upstream KEMA, Inc. 5-15

20 program savings between the two PSE HER treatment groups (current and suspended) and the control group. Table 5-2: HER Savings Per Household Based on Actual Consumption in 2012 Treatment Groups HER Measured Savings (per household) Joint Savings (per household) PSE Rebate Program Upstream Program Credited Savings Per Household All Households Electric (kwh) Current (+/- 47.9) 0.2 (+/- 3.7) 5.5 (+/- 0.6) ,330,705 Suspended (+/- 63.3) 1.3 (+/- 4.9) 10.5 (+/- 0.6) ,628,920 Total 6,959,625 Gas (therms) Current 12.7 (+/- 2.9) 1.4 (+/- 0.6) n/a ,670 Suspended 8.7 (+/- 3.7) 0.7 (+/- 0.8) n/a ,573 Total 272,243 1 Values in parentheses are based on 90% confidence interval, two-tailed test. The +/- for upstream savings were based on 2012 survey results. To estimate credited savings per households, joint savings from rebate and upstream programs were subtracted from the measured savings derived from consumption analysis. Credited savings per household were expanded to the full population for the current and suspended groups using the counts in Table 5-1. The total program savings for electric and gas are composed of savings generated by current and suspended treatment groups. Table 5-3summarizes the HER program results with respect to average consumption. The current treatment group produced credited savings at 2.8 and 1.3 percent for electric and gas, respectively. The suspended treatment group incurred 39 percent less electric savings and 30 percent less gas savings when compared to current treatment group. Table 5-3: Credited Savings Per Household as a Percent of Consumption HER Treatment Electric (kwh) Gas (therms) Group Consumption* Savings Percent Consumption* Savings Percent Current Reports % % Suspended Reports 10, % % *Based on control group calendar year 2012 actual consumption KEMA, Inc. 5-16

21 5.1.1 Measured Program Savings Current vs Suspended Treatment Groups Figure 5-1 summarizes the calendar year 2012 measured savings for the current and suspended treatment groups. Savings for both current and suspended report groups were significantly different from zero based on a 90% confidence interval, two-tailed test. Figure 5-1: Average Annual Savings for Current and Suspended Treatment Groups The difference in electric savings between the two groups was statistically significant at the 95% level. However, the difference in gas savings between the suspended and current treatment groups was not statistically significant Monthly vs Quarterly Treatment Groups Figure 5-2 provides the 2012 program savings for monthly and quarterly recipients. The measured savings results for monthly- and quarterly-current treatment groups generally conform to expectation that is monthly recipients generate more savings than quarterly recipients. KEMA, Inc. 5-17

22 Figure 5-2: Average Annual Savings for Monthly vs Quarterly Current Recipients Results show that both monthly and quarterly groups generated electric savings but the difference in savings between the monthly and quarterly groups was not statistically significant. For the suspended treatment group, electric savings generated by quarterly recipients was higher than monthly recipients but the difference is not statistically significant. Gas savings were statistically significant for all groups. Gas savings for current-monthly treatment groups was found significantly higher than current-quarterly group but the difference in savings between monthly and quarterly recipients in the suspended group was not significant Joint Savings Analysis To understand the impact of the HER program on the uptake of other PSE programs, DNV KEMA conducted a joint savings analysis for rebate and upstream programs to avoid double counting of savings. The PSE rebate programs included purchases of energy efficient measures such as lighting, heating and cooling system, water heating systems, insulation and appliances. Table 5-4 presents the PSE rebate joint savings analysis for current and suspended treatment groups across HER post years. Electric savings from PSE rebate programs were not statistically significant for both current and suspended groups while gas savings were statistically significant for both HER treatment groups. In 2012, rebate program joint savings were 0.20 kwh and 1.36 therms for the current group and 1.27 kwh and 0.74 therms for the suspended group. KEMA, Inc. 5-18

23 Fuel Electric (kwh) Gas (Therms) Table 5-4: Annual Joint Rebate Savings Per Household for Electric and Gas, Current and Suspended Groups Year HER Groups Joint Rebate Savings per household Control Current Suspended Current Suspended ** 0.27** ** 0.75** ** 0.95** ** 0.74 **Indicates statistically significant at 95% confidence level Table 5-5 provides the number of CFL bulbs and fixture purchases for the control, current treatment and suspended treatment groups in The difference in savings resulting from participation in the CFL program between the continued group and the control group was just a fraction of a single light bulb. The number of CFL fixtures purchased was smaller compared to CFL bulbs but none of the differences were statistically different from zero. Table 5-5: Count of CFL Bulbs and Fixture Purchased Per Household in 2012 CFL Purchased HER Groups Joint Rebate Savings per household 1 Control Current Suspended Current Suspended CFL Bulbs CFL Fixtures Total CFL Purchased Not statistically significant at 90% level PSE upstream savings were calculated by assigning 24 kwh savings per purchased bulb. Table 5-6 provides the estimates of the annual joint savings from CFL bulbs and fixture purchase across all post years. The 2012 estimate included savings generated from CFLs purchased by HER participants in prior years as measured in The estimate of upstream joint savings was higher for the suspended group, though, consistent with the continued treatment group findings, none of the results for the suspended group were statistically different than zero. KEMA, Inc. 5-19

24 Table 5-6. Annual Joint Upstream Savings Per Household for Current and Suspended Treatment Groups 4 Year Joint Upstream Savings (kwh) Per Household Current Suspended 2009* *Includes last two months of 2008 The joint savings analysis was used to provide an estimate of credited savings for PSE HER. Overall, around 5.7 kwh and 1.4 therms savings per household were shared between HER and other PSE programs for current treatment group. For the suspended group, HER and other PSE program share 11.8 kwh and 0.7 therms savings per household. These joint savings were deducted from the HER measured savings to avoid double counting of savings with other PSE programs. The HER savings for 2012 was based on savings with these joint program savings netted out. The joint savings analysis for upstream programs has been limited to CFLs. Expanding the analysis to include LED light bulbs is recommended to accurately quantify double counted savings due to participation in PSE upstream programs 6. Conclusions The purpose of this evaluation was to measure the impact of the PSE HER program for calendar year The PSE HER program generated statistically significant reduction in consumption. For calendar year 2012, PSE HER program generated credited savings of 300 kwh and 11 therms per household. These savings constitute 2.8% and 1.3% of the household s average electric and gas consumption, respectively. The Overall HER Program results are a combination of savings generated by an ongoing, current treatment group and a suspended treatment group that only received reports for the first two years of the 4 The 2011 joint upstream savings estimate has been revised from the third year report values. Developing a year three estimate of joint upstream savings required backcasting upstream joint savings for years one and two from the year three results. The updated results use 2011 current report receiving households as the basis for 2009 and 2010 estimates for both current and suspended groups. In 2009 and 2010, all treatment households received reports and should have been represented by the households that continued to receive reports in KEMA, Inc. 6-20

25 program. The current treatment group savings are slightly higher than those reported for the same group in the year three program evaluation. The suspended group, despite not receiving reports for the last two years, produced only 39 percent less electric savings and 30 percent less gas savings when compared to current treatment group. These program savings exhibit two different kinds of persistence that remain open questions for programs of this type. Those current households that continued to receive reports through the fourth year generated savings at or above levels established in the first two years of the program. The suspended group households that were in their second year of not receiving reports still generated well over half of the savings of the current group. The active HER Program continued to effectively promote other PSE gas rebate programs causing a statistically significant increase in gas rebate program savings among the current treatment group. As of the fourth year, the suspended group no longer shows the increase in gas rebate activity that was present in the previous three years. There has never been any evidence of increased activity in electric rebate programs or upstream program. All of these savings are removed from the final credited savings estimate, but it is important to remember the additional value of the HER program in promoting the other gas rebate programs. The joint savings analysis for upstream programs has been limited to CFLs. Expanding the analysis to include LED light bulbs is recommended to accurately quantify double counted savings due to participation in PSE upstream programs. KEMA, Inc. 6-21

26 List of Appendices A. Randomization Test Table A.1: Testing for differences in electric and gas consumption between treatment and control Fuel Monthly Treatment Control Control - Treatment Count Mean Std Error Count Mean Std Error Difference Pr > t Electric Oct ,590 1, ,693 1, (1.032) Nov , , (0.567) Dec , , Jan , , Feb , , Mar , , Apr , , May , , Jun , , Jul , , (0.218) Aug , , (0.166) Sep ,590 1, ,693 1, Gas Oct , , (0.009) Nov , , (0.139) Dec , , Jan , , (0.051) Feb , , (0.059) Mar , , Apr , , May , , Jun , , Jul , , Aug , , Sep , , KEMA, Inc. A-1

27 Table A.1: Testing for differences in individual characteristics between treatment and control Characteristics Treatment Control Control - Treatment Count Mean Std Error Count Mean Std Error Difference Pr > t Age 26, , Bathrooms 26, , (0.000) Bedrooms 26, , (0.005) Fireplace 26, , (0.001) House value 26, ,537 1,041 33, , No. of occupancy 23, , House size 26,590 2, ,693 2, (2.915) KEMA, Inc. A-2

28 B. Impact Methodology B.1 Difference-in-differences The difference-in-differences approach is the most direct and simple way of leveraging the experimental design of the HER program. The approach compares the difference in the average consumption of the treatment group between pre- and post-report period with the same difference for the control group. The treatment group pre-post difference captures all changes between the two periods including those related to receiving the reports. The control group captures all changes with the exception of those related to the report, because the control group did not receive the reports. The random selection of the treatment and control groups ensures that, on average, the control group will appropriately reflect the non-report related changes experienced by treatment and control group alike between the pre-and post-report periods. Removing the non-report differences, as represented by the control group difference, from the treatment difference produces an estimate of the report s isolated effect on consumption. The average energy consumption is calculated for both treatment and comparison group in both pre- and post-report periods. The difference-in-differences estimate is then produced with the following equation. where = Pre-post difference in annual consumption for household i; = Intercept T = Treatment indicator (value of 1 if treatment and 0 otherwise) β = Treatment effect or savings estimate ε = error term The difference-in-differences approach can be applied on a monthly or seasonal basis. As long as time periods are balanced in the pre- and post-report periods, the savings estimate will be consistent for that time period. KEMA, Inc. B-3

29 C. Survey Instrument Puget Sound Energy Home Energy Report Savings Double Counting CATI Survey I INTRODUCTION I0 May I please speak with <CONTACT NAME>? [IF CONTACT NAME IS AVAILABLE, READ I1] [IF CONTACT NAME IS NOT AVAILABLE ARRANGE FOR CALLBACK] I1 Hello, my name is from on behalf of Puget Sound Energy. PSE is conducting a survey about how households in their service area use energy. They are interested in learning more about their customers purchases of energy using equipment. This is NOT a sales call and the information that you provide will be kept strictly confidential. [IF NECESSARY: Puget Sound Energy will use your input to improve the programs they offer to residential customers.] CELL1 First, have you received this call on a wireless phone or on a landline phone? 1 WIRELESS GOTO CELL2 2 LANDLINE GOTO I2 CALLBACK CALLBACK CELL2 Are you driving a vehicle or using any equipment or machinery that requires your attention? [INTERVIEWER: IF RESPONDENT SAYS YES, READ] Due to safety reasons we will need to call you back at a more convenient time. Thank you very much. 1 YES CALLBACK 2 NO CALLBACK CALLBACK KEMA, Inc. C-4

30 I2 Do you or anyone else in your household work for a gas or electric utility, including Puget Sound Energy? 1 Yes SPECIFY: THANK & TERMINATE 2 No PS1 I am calling about <ADDRESS>. Do you live at this address? 1 Yes GOTO I3 2 No Thank and Terminate Thank and Terminate Thank and Terminate I3 Are you knowledgeable about the energy related purchases made for this household in the past year? 1 Yes GOTO C1 2 No [ASK: May I speak to someone who is? SCHEDULE INTERVIEW IF PERSON NOT AVAILABLE OR ARRANGE FOR CALLBACK] THANK & TERMINATE INTERVIEW THANK & TERMINATE INTERVIEW C CFL PURCHASE(S) C1 I m going to ask you about two kinds of equipment your household may have purchased in They are: Compact Fluorescent Bulbs and Compact Fluorescent fixtures. Compact Fluorescent fixtures are special lighting fixtures that use pin-based bulbs that plug in to the fixture. These fixtures often have an Energy Star label. We ll talk about these later. First, I m going to ask you some questions about Compact Fluorescent bulbs. [NOTE TO CALLER: SPEAK OUT THE LETTERS OF C-F-Ls see- eff- els ] C2 Compact Fluorescent bulbs are small fluorescent bulbs that typically screw into regular light sockets. They often have a twisty shape. Sometimes people call them CFLs. Have you heard of these? KEMA, Inc. C-5

31 1 Yes 2 No SKIP TO X1 THANK AND TERMINATE. DO NOT COUNT AS COMPLETE First time = Repeat C2. Second time = SKIP TO X1 C3 I ll refer to Compact Fluorescent bulbs as CFLs for the rest of this call. Did you or anyone in your household purchase any CFLs in 2012? 1 Yes 2 No SKIP TO X1 SKIP TO X1 SKIP TO X1 C4 Approximately, how many compact fluorescent bulbs did your household purchase in 2012? If you purchased any multi-packs, please tell me the total number of BULBS you purchased. For example, a pack with three bulbs would count as three. [IF NECESSARY: Your best estimate is fine.] 1 One SKIP TO C8 SET TOT_LAMPS=1 2 More than one [SPECIFY] SKIP TO C5 SET TOT_LAMPS=ANSWER SKIP TO C15 ASK C4b C4b Could you tell me a range? For example, did you purchase between X and Y bulbs? C4b [RECORD LOW END OF RANGE] C4c [RECORD HIGH END OF RANGE] SKIP TO C15 SKIP TO C15 C4d I have some later questions that will reference the number of bulbs you purchased in For those questions, I will use the midpoint of this range, which is [(C4b + C4c)/2, round up to a whole number]. [SET TOT_LAMPS = (C4b + C4c)/2, round up to a whole number] C5 Did you purchase all the CFLs on the same shopping trip? KEMA, Inc. C-6

32 1 Yes SKIP TO C8 2 No SKIP TO C12 SKIP TO C12 C6 REMOVED C7 On how many different trips did you purchase CFLs in 2012? [IF NECESSARY SAY Your best estimate is fine] 1 [RECORD # OF TRIPS] SKIP TO C12 SKIP TO C12 BEFORE STARTING STORE MODULE, SET SUM_LAMPS = 0 STORE MODULE C8a to C11 REPEAT UP TO 6 TIMES] [IF (TOT_LAMPS = 1) OR (C5 =1) Program so that if only one store. Ask STORE MODULE only one time] IF (TOT_LAMPS = 1) OR (C5 =1), SKIP TO C8] C8a [READ ONLY FIRST TIME THROUGH MODULE I m going to ask about up to six of your CFL purchases in I ll ask you some questions about the first purchase, then repeat some questions for the later purchases. C8b [READ ONLY SECOND TIME THROUGH MODULE] Now let s go through those questions for your 2 nd CFL purchase in C8c [READ ONLY THIRD TIME THROUGH MODULE] Now let s go through those questions for your 3rd purchase. C8d [READ ONLY THIRD TIME THROUGH MODULE] Now let s go through those questions for your 4th purchase. C8e [READ ONLY THIRD TIME THROUGH MODULE] Now let s go through those questions for your 5th purchase. KEMA, Inc. C-7

33 C8f [READ ONLY THIRD TIME THROUGH MODULE] Now let s go through those questions for your 6th purchase. C8 At what store did you buy the CFLs? [DO NOT READ] [ACCEPT ONLY ONE RESPONSE] 1 ACE HARDWARE 2 ARIRANG ORIENTAL MARKET 3 BARTELL DRUGS 4 BEST BUY 5 CARNICERIA LA CHIQUITA 6 COSTCO 7 DO IT BEST HARDWARE CENTER 8 DOLLAR PLUS 9 DOLLAR TREE 10 FOSS' GROCERY 11 FRED MEYER 12 FRY'S ELECTRONICS 13 GOODWILL 14 GROCERY OUTLET 15 HADLOCK BUILDING SUPPLY 16 HAGGEN 17 HARDWARE SALES 18 LA TEJANA MEXICAN STORE 19 LAKE SAWYER GROCERY 20 LOWE'S 21 LUMBERMENS 22 MAPLE VALLEY MARKET 23 MCLENDON HARDWARE 24 MERCADITO DEL VALLE 25 OAK HARBOR MARKETPLACE 26 OLYMPIA LIGHTING CENTER 27 ONLY A DOLLAR PLUS 28 PORT ORCHARD MARKETPLACE 29 PUGET PANTRY 30 RITE AID 31 SEBO'S DO IT CENTER 32 THE MARKETS 33 THE STAR STORE, INC. KEMA, Inc. C-8

34 34 TRUE VALUE HARDWARE 35 VALLEY HARVEST MARKET / VALLEY HARVEST II INTERNATIONAL MARKET 36 VILLAGE LIGHTING 37 WALGREENS 38 WALMART 39 WALT'S LYNWOOD CENTER MARKET 40 WESTSIDE BUILDING SUPPLY DO IT CENTER 41 WINCO 42 HOME DEPOT 95 OTHER (SPECIFY) SKIP TO C11 SKIP TO C11 C9 In what city or town was this store located? [DO NOT READ] [ACCEPT ONLY ONE RESPONSE] 1 ANACORTES 2 AUBURN 3 BAINBRIDGE ISLAND 4 BELLEVUE 5 BELLINGHAM 6 BLACK DIAMOND 7 BLAINE 8 BONNEY LAKE 9 BOTHELL 10 BREMERTON 11 BURIEN 12 BURLINGTON 13 CLINTON 14 COVINGTON 15 DES MOINES 16 EDGEWOOD 17 ELLENSBURG 18 ENUMCLAW 19 EVERSON 20 FEDERAL WAY 21 FERNDALE 22 FREELAND 23 GRAHAM KEMA, Inc. C-9

35 24 ISSAQUAH 25 KENMORE 26 KENT 27 KINGSTON 28 KIRKLAND 29 LACEY 30 LANGLEY 31 LYNDEN 32 MAPLE VALLEY 33 MERCER ISLAND 34 MOUNT VERNON 35 NEWCASTLE 36 NORTH BEND 37 OAK HARBOR 38 OLYMPIA 39 PORT HADLOCK 40 PORT ORCHARD 41 PORT TOWNSEND 42 POULSBO 43 PUYALLUP 44 REDMOND 45 RENTON 46 SAMMAMISH 47 SEDRO WOOLLEY 48 SILVERDALE 49 SUMNER 50 TUKWILA 51 TUMWATER 52 WOODINVILLE 53 YELM 95 OTHER (SPECIFY) [C10 DELETED] [IF TOT_LAMPS=1, AUTO POPULATE C11=1 AND SKIP TO C12] [IF C5=1 (ONE TRIP) AUTO POPULATE C11 = TOT_LAMPS AND SKIP TO C12] KEMA, Inc. C-10

36 C11 Of the [TOT_LAMPS SUM_LAMPS] bulbs left to discuss, how many did you purchase at that time? 1 [RECORD #] END OF LOOP RULES: SUM_LAMPS = SUM_LAMPS + C11 IF SUM_LAMPS >= TOT_LAMPS, BREAK LOOP AND GOTO C12 IF (TOT_LAMPS = 1) OR (C5 =1), BREAK LOOP AND GO TO C12 GOTO C8b FOR 2 nd PURCHASE, GOTO C8c FOR 3 rd PURCHASE, GOTO C8d FOR 4 th PURCHASE, GOTO C8e FOR 5 th PURCHASE, GOTO C8f FOR 6 th PURCHASE, or BREAK LOOP AND GOTO C12 IF NO MORE PURCHASES TO ASK ABOUT C12 How many of the [TOT_LAMPS] bulbs that you purchased in 2012 are currently installed in or around your home? 1 [RECORD #] [IF C12 <= TOT_LAMPS (DID NOT INSTALL MORE THAN PURCHASED) SKIP TO C13] C12b To confirm, you purchased [TOT_LAMPS] and installed [C12]. Is that correct? [INSTRUCTION TO CALLER: IF YOU GET TO C12B A SECOND TIME FOR THIS PURCHASE, CHOOSE DON T KNOW AND MOVE ON TO AVOID AN INFINITE LOOP] 1 Yes GOTO C12c 2 No GO BACK TO C12 GOTO C13 KEMA, Inc. C-11

37 GOTO C13 C12c How were you able to install more bulbs than you purchased? [DO NOT READ. SELECT ALL THAT APPLY] 1 Had some in storage 2 Got some from someone else 3 Something else (SPECIFY) 4 Did not install more than purchased GO BACK TO C12 C13 What type of bulb did the majority of these CFLs replace? Was it... [IF TOT_LAMPS=1 USE ALTERNATE WORDING: What type of bulb did the CFL replace? Was it ] [READ 1-5. ACCEPT ONE ANSWER] 1 Other CFLs, 2 Regular/incandescent bulbs, 3 Halogen bulbs, 4 A mix of CFL and other bulbs, or 94 They did not replace other bulbs? 95 SOMETHING ELSE [IF C12 >= TOT_LAMPS (DID NOT INSTALL LESS THAN PURCHASED) SKIP TO C15] C14 What did you do with the bulbs you did NOT install. Did you..? [READ 1-4. ACCEPT MULTIPLE ANSWERS] 1 store them in your home, 2 give them away, 3 return them to the store, or 95 do something else with them? (SPECIFY: ) 94 I INSTALLED THEM ALL KEMA, Inc. C-12

38 C15 What, if anything, influenced your household to purchase the CFLs? Anything else? [DO NOT READ] [ACCEPT MULTIPLE REPLIES] 1 SAVING MONEY 2. SAVING ENERGY [ALSO TO BE GREEN OR ENVIRONMENTALLY CONCIOUS] 3. ANY REBATE 4 LETTER OR BILL INSERT FROM PSE 5 PSE WEBSITE, PSE ADVERTISING 6 SALESPERSON 7 MANUFACTURER ADVERTISING, NEWS, OR OTHER MEDIA 8 HOME ENERGY REPORT 9 LAST LONGER 10 PRICE OF BULB (ON SALE/WAS FREE) 11 ONLY TYPE AVAILABLE/PHASING OUT OF INCANDESCENT LIGHT BULBS 12 LIGHT QUALITY 13 WANTED TO TRY THEM/TRY SOMETHING NEW/SEE HOW THEY WORK 94 NOTHING/NO INFLUENCE 95 OTHER (PLEASE SPECIFY) C16 Next, I d like you to think about your lighting purchases over the past several years and how they might have changed. About what percent of your household light bulb purchases were CFLs in each of the following years? Your best guess is fine. C16a. 2008? C16b. 2009? C16c. 2010? C16d. 2011? C16e. 2012? [REPEAT AS NECESSARY: About what percent of your household light bulb purchases were CFLs in <<YEAR>>] KEMA, Inc. C-13