Vectren Indiana Natural Gas DSM Program Impact Evaluation

Transcription

1 Vectren Indiana Natural Gas DSM Program Impact Evaluation Final Report Prepared for Vectren Madison, Wisconsin, August 18, 2009 Experience you can trust.

2 Copyright 2009, KEMA, Inc. The information contained in this document is the exclusive, confidential and proprietary property of KEMA, Inc. and is protected under the trade secret and copyright laws of the U.S. and other international laws, treaties and conventions. No part of this work may be disclosed to any third party or used, reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording, or by any information storage or retrieval system, without first receiving the express written permission of KEMA, Inc. Except as otherwise noted, all trademarks appearing herein are proprietary to KEMA, Inc. Experience you can trust.

3 Table of Contents 1. Executive Summary Overview and Discussion of Prescriptive Results Evaluation Overview Prescriptive Program Program Background Evaluation Methodology Prescriptive Results Spillover Effect of Fuel Prices Conclusions and Recommendations Custom Program Results Conclusions and Recommendations Introduction Prescriptive Program Custom Program Organization of Report Prescriptive Program Program Background Data Collection Sample Design Sample Disposition Data Collection Prescriptive Evaluation Methodology Demand-Supply Decision Influence Analysis Participating Supplier Sales Change Analysis Market-Level Sales Change Analyses Discussion of Methods Prescriptive Impact Evaluation Results Demand-Supply Decision Influence Analysis Participating Supplier Sales Change Analysis Market-Level Sales Change Analysis Comparison of Results and Final Savings Vectren i August 18, 2009

4 Table of Contents 3.5. Prescriptive Program Process Results Supplier Survey End-User Survey Prescriptive Program Conclusions and Recommendations Recruiting New Suppliers More Training for Suppliers Offer Incentives Directly to Suppliers More Marketing to Nonresidential Customers More Marketing to High-Use Residential Customers Additional Research Custom Program Program Background Overview of Approach End-User Data Collection Analysis Methodology Results Other Data Conclusions and Recommendations APPENDIX A: Sample Frames, Tracking Systems, and Documentation Review... A-1 APPENDIX B: Supplier and End-User Demographics... B-1 APPENDIX C: Programmable Thermostats and Water Heaters... C-1 APPENDIX D: Residential New Construction Builders... D-1 APPENDIX E: Decision Influence Attribution Analysis Methodology Details... E-1 APPENDIX F: Ratio Estimation Details...F-1 APPENDIX G: Survey Instruments... G-1 List of Exhibits: Figure 1-1 Monthly Natural Gas Prices in Vectren North Service Territory Figure 3-1 Demand-Supply Decision Influence Analysis Narrow Overall Attribution Determination Figure 3-2 Simple Lifetime Net Savings Figure 3-3 Influence Designation for Participating End-Users Vectren ii August 18, 2009

5 Table of Contents Figure 3-4 Decision Tree to Determine Supplier Influence Figure 3-5 Program Attribution by Influence Category Figure 3-6 Participating Supplier Sales Data Used for Method C Figure 3-7 Participating Supplier Sales Data Used for Method B Figure 3-8 End-user Satisfaction with Contractor Figure 3-9 End-user Satisfaction with Equipment Figure 3-10 End-user Satisfaction with Rebate Amount Figure 3-11 End-user Satisfaction with Paperwork Figure 3-12 End-user Satisfaction with Staff Interactions Figure 3-13 End-user Satisfaction with Energy Audit Figure 3-14 End-user Satisfaction with Program as a Whole Figure 3-15 Monthly Natural Gas Prices in Vectren North Service Territory Figure 4-1 Relationship Between Tracking and Verified Gross Savings Figure 4-2 Relationship Between Gross and Net Savings Table 1-1 Prescriptive Program Evaluation Results Table 1-2 Comparison of Methods and Results Table 1-3 Attribution Results from the Demand-Supply Decision Influence Methods A and B..1-8 Table 1-4 Participating Suppliers Changes in High Efficiency Recommendations and in Supplier-Influenced Sales Table 1-5 Prescriptive Program Net Savings Table 1-6 Gross Energy Savings Estimates Table 1-7 Life Cycle Tracking and Verified Gross Savings Estimates Table 1-8 Overall Results for Year 1 of the Custom Program Table 3-1 Contractor Sample Design Table 3-2 Contractor Sample Design with Breakouts by Equipment Installed Table 3-3 Distributor Sample Design Table 3-4 Prescriptive Program End-User Characterization Table 3-5 Distribution of End-User Sample Target Completes Table 3-6 End-User Sample Design with Breakouts for Equipment Installed Table 3-7 Contractor Sample Disposition Table 3-8 Distributor Sample Disposition Table 3-9 Prescriptive End-User Sample Disposition Table 3-10 Summary of Survey Instruments Vectren iii August 18, 2009

6 Table of Contents Table 3-11 Comparison of Methods Table 3-12 E7a Influence Group Assignment Table 3-13 E6 and E6a Influence Assignment Table 3-14 E7b and E7c Influence Assignment Table 3-15 E7d and E7e Influence Assignment Table 3-16 Demand-Supply Decision Influence Analysis Results Table 3-17 Participating Suppliers Changes in High Efficiency Recommendations and in Supplier-Influenced Sales Table 3-18 Frequency Suppliers Use Rebate as Sales Tool Table 3-19 Effectiveness of Program to Encourage Suppliers to Sell High-Efficiency Furnaces 3-42 Table 3-20 Program Effectiveness by Supplier Strata Table 3-21 Reasons for Program Ineffectiveness Table 3-22 Results of E7a: Contractor Role in Equipment Selection Table 3-23 Results of E6a: Boiler/Furnace Features End-users Wanted Table 3-24 Results of E7d: Features Emphasized by Contractors Table 3-25 Results of E7e: Most Important Features Table 3-26 End-User Attribution for Vectren Only and Both Influence Categories Table 3-27 Percent of Customers Aware of Rebates Table 3-28 When End-user Became Aware of Rebate Table 3-29 Information Sources About Equipment Table 3-30 With Whom Respondent Discussed Equipment Options Table 3-31 Sources of Information about Program Table 3-32 Reasons for Installing Equipment Table 3-33 Benefits Received from Program Table 3-34 Reasons Customers Interested in High-efficiency Furnaces Table 3-35 Other Influences On Customers Interest in High-efficiency Furnaces Table 3-36 Effect of Recent Economic Uncertainties on Customers Interest in High-efficiency Furnaces Table 3-37 Effects on Willingness to Install New Equipment Table 3-38 Effects on Willingness to Install High-efficiency Equipment Table 3-41 Change in High Efficiency Sales among Participating Suppliers Table 3-42 Comparison of Methods and Results Vectren iv August 18, 2009

7 Table of Contents Table 3-43 Prescriptive Program Net Savings Table 3-44 Percent of Residential Furnace Sales Table 3-45 Supplier Reports of Retrofit Sales Table 3-46 Frequency Supplier Filled out Rebate Form for Customers Table 3-47 Frequency Suppliers Compared 80% to 90% AFUE Furnaces Table 3-48 Average Price Difference Between 80% and 90% AFUE Furnaces Table 3-49 Suppliers Satisfaction with Rebate Table 3-50 Reason Suppliers Unsatisfied with Rebate Levels Table 3-51 Suppliers Satisfaction with Paperwork Table 3-52 Reasons Suppliers Unsatisfied with Paperwork Table 3-53 Suppliers Satisfaction with Marketing Effort Table 3-54 Reasons Suppliers Unsatisfied with Program Marketing Efforts Table 3-55 Supplier s Satisfaction with the Program Staff Table 3-56 Reasons Suppliers Unsatisfied with Interactions with Program Staff Table 3-57 Suppliers Satisfied with the Program as a Whole Table 3-58 Reason Suppliers Unsatisfied with Program as a Whole Table 3-59 Supplier s Indication of What Aspects Worked Well Table 3-60 Supplier Suggestions for Program Improvement Table 3-61 Type of Project Table 3-62 Changed Thermostat Settings Since Installing New Boiler/Furnace Table 3-63 Changes to Thermostat Setting Table 3-64 Reasons for Changing Thermostat Settings Table 3-65 Improvements Made Due to Energy Audit Table 4-1 Gross Energy Savings Estimates Table 4-2 Life Cycle Tracking and Verified Gross Savings Estimates Table 4-3 Overall Results for Year 1 of the Custom Program Table B-1 Commercial/Builder End-User Type of Organization... B-5 Table B-2 Commercial/Builder End-User Number of Locations... B-7 Table B-3 Commercial/Builder End-User Headquarters... B-7 Table B-4 Residential End-User Home Square Footage... B-8 Table B-5 Residential End-User Home Age... B-8 Table B-6 Residential End-User Weeks/Year at Secondary Address... B-9 Table B-7 Residential End-User Number of Residents... B-9 Vectren v August 18, 2009

8 Table of Contents Table B-8 Residential End-User Highest Level of Education Attained... B-10 Table B-9 Residential End-User Total Pre-tax Household Income... B-10 Table C-1 Percent of Energy Star Thermostats with New Furnaces... C-3 Table C-2 Number of Separate Programmable Thermostats Installed per Year... C-5 Table C-3 Percent of Energy Star Separate Programmable Thermostats... C-5 Table C-4 Frequency Suppliers use Rebate on Energy Star Thermostats... C-7 Table C-5 Reasons for Not Using Thermostat Rebate More Often... C-8 Table C-6 Percent of Programmable Thermostat Sales That Would be Energy Star Rated Without Rebates... C-10 Table C-7 Supplier Sells Storage Water Heaters... C-10 Table C-8 Number of Storage Water Heater Units Installed by Suppliers in a Typical Year.. C-11 Table C-9 Percent of Water Heaters Sold That Were High Efficiency... C-11 Table C-10 Frequency Supplier Used Rebate on High Efficiency Water Heater... C-13 Table C-11 Reasons for Not Using Storage Water Heater Rebates More Often... C-14 Table C-12 Percent of High Efficiency Water Heaters That Would be Sold without Rebate.. C-15 Table C-13 Percent of Suppliers that Sell Tankless Water Heaters... C-16 Table C-14 Number of Tankless Units Installed by Supplier in a Typical Year... C-17 Table C-15 Percentage of High-efficiency Tankless Water Heaters Sold... C-18 Table C-16 Percent of Tankless Water Heater Sales Purchased with a Vectren Rebate... C-19 Table C-17 Percent of High Efficiency Tankless Water Heaters... C-20 Table C-18 When End-users Became Aware of Water Heater Rebate... C-22 Table C-19 Where End-users Heard about Water Heater Rebates... C-22 Table D-1 Percentage of Rebated Builder Projects that were Spec Homes... D-1 Table D-2 Percentage of Rebated Builder Projects that were Energy Star Homes... D-2 Table D-3 Builders who Pass on Extra Cost to Home Buyers... D-2 Table D-4 How Many Home-buyers Ask about Energy Efficiency... D-3 Table E-1 Efficiency Attribution Assignments... E-8 Table E-2 Efficiency Attribution Assignments... E-10 Vectren vi August 18, 2009

9 1. Executive Summary This report is the final deliverable of KEMA s evaluation of the Vectren Natural Gas DSM programs in Indiana. The principal objective of this evaluation was to study the performance and results of Phase 2 Programs and estimate net impacts. Specifically, KEMA reviewed and evaluated participating Prescriptive Program end users that completed projects between December 1, 2007 and November 30, 2008 and participating Custom Program end users that completed projects between the start of the program and March 31, Overview and Discussion of Prescriptive Results The prescriptive program was evaluated using five methods, the Demand-Supply Decision Influence Analysis narrow and broad methods and three sales change analyses, one for participating suppliers and two at the market level. The results of the evaluation for the five methods is found in Table 1-1. The methods used to determine the results are explained later in this report. KEMA recommends that the Participating Supplier Sales Change analysis attribution of 48 percent be used to determine program net savings. Table 1-1 Prescriptive Program Evaluation Results Analysis Method Program Attribution Demand-Supply Decision-Influence - Narrow 22% Demand-Supply Decision-Influence - Broad 45% Participating Supplier Sales Change 48% Market-Level Sales Change: Vectren-Adjusted 46% Market-Level Sales Change: Statewide 42% Suppliers reported that the energy efficient equipment sales share was already near 50 percent in 2006, a pre-program year. This value was confirmed with confidential market data provided by a leading manufacturer for the state of Indiana. Therefore, if the majority of customers who qualify for a rebate take it, then the free ridership rate would be approximately 50 percent even if the energy efficiency equipment sales share rose to 100 percent. Therefore, given the state of the market at the time the program started, the recommended attribution estimate is a strong result. All of the analyses produce attribution results that may be lower than expected given other recent studies and program attribution assumptions. Vectren 1-1 August 18, 2009

10 A technical potential study was conducted by Forefront Economics Inc. and H. Gil Peach and Associates LLC for the northern part of Vectren s territory in which assumed that furnace measures would achieve a net-to-gross ratio of 90 percent. We did not find a basis for this assumption in the report. The Vectren Market Research Department produced an internal report in 2008 titled Conservation Study: Effectiveness of Indiana Conservation Programs and Customer Attitudes Regarding Conservation which suggests a net-to-gross ratio of 76 percent. 2 This study is not specific to furnaces and boilers, but approximately one-half of the participants interviewed for the study installed one of the two measures. The Database for Energy Efficient Resources (DEER) in California changed its net-to-gross assumption for high efficiency furnaces from 80 percent to 60 percent during the last round of review. All of these sources would suggest a net-to-gross ratio higher than those found in KEMA s study. Vectren s internal Conservation Study uses a 50 percent free rider rate based on the fraction of customers who said the rebate was important or very important to their purchase decision. That report also estimates a spillover increment of 7 percent for participants and 9 percent for nonparticipants, and a market transformation effect of 37 percent, which combine to an overall netto-gross ratio of 76 percent. Based on our analysis, these overall spillover and market transformation estimates appear to be high for furnaces and boilers. The conservation study also indicates that 78 percent of customers say they would have bought the same equipment without the rebate. This result is reasonably consistent with our finding of 16 percent pure free riders for furnaces. Determining attribution in the presence of multiple influences is always challenging. Our surveys asked participating end users about the influence of several factors on their willingness to install high-efficiency equipment. These factors included the cost of fuel, environmental 1 Vectren DSM Action Plan: Final Report. Prepared by Forefront Economics, Inc. and H. Gil Peach and Associates, LLC. December 9, Conservation Study: Effectiveness of Indiana Conservation Programs and Customer Attitudes Regarding Conservation. Conducted by the Vectren Market Research Department. Finalized November Vectren 1-2 August 18, 2009

11 concerns, the severity of winters, and the economic downturn, as well as Vectren s endorsement, the rebate, and their contractor. For each of these factors, 75 to 80 percent of respondents said that the factor had increased or greatly increased their willingness to install the qualifying equipment. Clearly it would not make sense to say that each of these factors deserves 75 to 80 percent of the credit for the rebated units. The key question for crediting a purchase to the program is whether that purchase would have been made in the absence of the program. Admittedly this is a challenging question for an end user to answer. Our end-user surveys attempt to make it easier by using a series of framing questions to return the respondent to the thought process at the time the decision was made. The confidential manufacturer s data used for the two Market-Level Sales Change Analyses and our result both indicate a pre-program market share for eligible equipment around 45 percent. A net-to-gross ratio as high as 90 percent is highly unlikely at that starting point. Rather, a relatively high free ridership level would be expected. For example, if all qualifying units took rebates and Vectren was successful in raising the overall market share to 90 percent, the free rider rate would be 45 percent divided by 90 percent, or 50 percent. If the overall market share increased only to 60 percent and all qualifying units took rebates the free rider rate would be 45 percent divided by 60 percent, or 75 percent. Thus, the planning assumption of 90 percent netto-gross or even the recent DEER estimate of 60 percent seems too high for this market. On the other hand, our finding of net-to-gross ratios close to 50 percent indicates that the program has been as successful as possible among participating suppliers. The two market analyses also indicate that the nonparticipating suppliers are selling qualifying units in a total quantity comparable to the number sold within the program. These non-program units are part of the existing baseline activity, and are not program attributable. The question remains why these suppliers are not working with the program to provide rebates to their customers, and whether their efficient sales shares could be increased if these suppliers were brought into the program Evaluation Overview Prescriptive Program KEMA s evaluation of the prescriptive program emphasized the furnace and boiler measures which account for 79 percent of program gross annual savings. In the Prescriptive program, the gross savings estimates that result from changing furnace or boiler efficiency are well understood; therefore, KEMA was not asked to analyze the gross savings estimates for this Vectren 1-3 August 18, 2009

12 program. Instead, we concentrated our efforts on the effect of the program on the adoption of high efficiency equipment, or program attribution. KEMA also looked at how effectively the program operates from the perspective of participating suppliers and end users. We addressed program satisfaction, take-back effects, and other factors that could affect the ultimate attribution. Custom Program The Custom program is small and not as established as the Prescriptive program. The Custom program was started on April 1, 2008, a full year after the start of the Prescriptive program. KEMA addressed net savings by performing a detailed review of a sample of projects. We performed an engineering review of the savings calculations provided by the program to verify the gross estimates. We also analyzed survey data to determine program attribution. The combined gross savings adjustment and attribution adjustment produce total net savings. As with the prescriptive program, KEMA looked at how effectively the program operates from the perspective of participating end users. We asked questions to determine program satisfaction, take-back effects, and why the project was installed Prescriptive Program Program Background The Prescriptive program was started by Vectren in The program was created to help customers get relief from rapidly rising gas costs. In the Phase 2 year, 3 the program awarded over 14,000 rebates to approximately 11,250 participants. The Prescriptive Program pays rebates to end users for installing high efficiency equipment. The program was designed to deliver services primarily through market providers such as HVAC suppliers or distributors. Program staff identify and visit market providers and deliver program materials, rebate forms, and training if requested. Program staff also educate market providers on the benefits of taking additional time with customers, up-selling, and encouraging 3 The Phase 2 program began December 1, 2007 and ran through November 30, Vectren 1-4 August 18, 2009

13 energy efficiency. Staff gave providers tools to assist in these efforts. Typical measures installed include high efficiency furnaces, boilers, water heaters, and programmable thermostats. The Prescriptive Program provides set rebate amounts for qualifying equipment and a select number of measures. Standard, prescribed energy savings have been developed and are applied to each piece of equipment installed in the tracking database. The rebate amount and energy savings are directly tied to the piece of equipment purchased, not the actual energy savings resulting from an individual installation. Both general service and residential customers of Vectren Energy Delivery are eligible for participation in the Prescriptive program. For large, long-lived investments like furnaces and boilers, end users don t pay a lot of attention to information about efficiency until they re starting to think about replacing their unit. The supplier-driven program addresses these people; therefore, it is more effective at reaching the target market than a program that is only promoted through direct marketing to end users. The Vectren program incorporates both elements through the supplier-driven program design and a series of direct marketing campaigns. In 2007 and 2008, Vectren sponsored a number of advertisements to encourage end users to consider energy efficiency and promote the program. The campaigns were implemented in a number of media, including newspaper, radio, television, and bill inserts Evaluation Methodology As mentioned above, our evaluation of the Prescriptive Program concentrated on program attribution. Determining attribution means determining the cause of end users decisions, or how decisions would have been different without the program. This determination always has challenges and uncertainties. The challenges and uncertainties are greater in two contexts: Substantial market change that might otherwise be affecting customer decisions Upstream or supplier-driven programs, where the influence of the program on customer decisions is indirect. Both situations apply to the Phase 2 Prescriptive Program. To address these challenges, the evaluation adopted multiple approaches. Each has strengths and weaknesses and each sheds some light on the program s effects. The first method is the Demand-Supply Decision Influence Analysis. This approach assesses the influences of the program and the supplier on the end-user s decision, and the influence of Vectren 1-5 August 18, 2009

14 the program on the supplier s promotional efforts. The analysis used data from surveys that were delivered to participating suppliers and end users to gauge the effect of the program on participant behavior. With this overall method, we used the available data 2 different ways to assess the influence of the program on suppliers. The first method took a narrow interpretation, resulting in relatively low attribution estimates. The second took a broader interpretation, resulting in attribution estimates that are more in line with the other methods. A second general approach was a sales share change analysis for participating suppliers. This analysis estimates the total change in high efficiency sales among participating suppliers. Estimating the change attributable to the program requires additional assumptions, but provides another perspective on the program s effect on the market. We call this method the Participating Supplier Sales Change analysis. Our third approach was a market-level sales share analysis, using confidential statewide market share data. This analysis assumes that other energy efficiency programs in Indiana have negligible effect on participating suppliers energy efficient sales shares. While we are not able to share the details of this analysis, the results confirm the findings from our supplier surveys and associated methods. We call this method the Market-Level Sales Change: Vectren- Adjusted analysis. Finally, we conducted a second market-level sales share analysis, again using confidential statewide market share data. This analysis assumes that the 2006 energy efficient sales share represents the non-program baseline for the entire state, not just the Vectren territory. We call this method the Market-Level Sales Change: Statewide analysis Prescriptive Results For the prescriptive evaluation, KEMA completed surveys with 70 contractors, 16 distributors, 30 Commercial end users, and 471 Residential end users. The Demand-Supply Decision Influence Analysis used data from all of these surveys. The Participating Supplier Change Analysis used results from the supplier surveys only. The two Market-Level Sales Change Analyses used data from the supplier surveys together with the confidential statewide sales data. The analysis results are summarized in Table 1-2. Vectren 1-6 August 18, 2009

15 Table 1-2 Comparison of Methods and Results Method A B Description Demand-Supply Decision Influence: Narrow Demand-Supply Decision Influence: Broad Demand-Side Influence Survey demand-side influence classification demand-side influence classification Supply-Side Influence Survey change in recommendations change in sales Adjustment for Participating Supplier Non-Rebated Qualifying Units included included Net-To- Gross Key Assumptions Ratio Program influence on suppliers is captured by change in recommendations 2006 EE recommendations rate share represents 2008 nonprogram recommendations baseline 22% Negligible program influence on non-participating suppliers or customers 2006 EE sales share represents 2008 non-program baseline Negligible program influence on non-participating suppliers 45% C Participating Supplier Sales Change not used change in sales included 2006 EE sales share represents 2008 non-program baseline Negligible program influence on non-participating suppliers 48% D Market-Level Sales Change: Vectren- Adjusted not used change in sales not needed Non-Vectren change in EE sales share represents 2008 non-program baseline for Vectren Negligible program influence on non-participating suppliers Negligible effect of other programs on Vectren participating suppliers' energy efficient sales shares 46% E Market-Level Sales Change: Statewide not used change in sales not needed 2006 energy efficient sales share represents non-program baseline for the state 42% With the exception of the narrowly interpreted Demand-Supply Decision Influence (Method A), all the methods give similar results, around 46 percent attribution. We believe the lower estimate from Method A is unrealistically low for several reasons: Method A measures the influence of the program on suppliers in terms of the change in the rate at which the supplier recommends energy-efficient equipment. However, the influence of the program on sales is broader than simply whether or not the equipment is recommended. The broad interpretation (Method B) captures these effects more completely by using instead the changes in sales for the portion of sales that were supplier influenced. Participating suppliers indicated substantial increases in energy efficient sales shares after the start of the program. The 47 percent net-to-gross ratio for Method C corresponds to a roughly 80 percent increase in energy efficient sales share within the Vectren territory. Participating suppliers also indicated that nearly all customers who were eligible to receive a Vectren rebate applied for one. Unless the sales increase is either in error or is attributable to something other than Vectren, program attribution associated with participating customers must be at least big enough to account for this increase. The Vectren-adjusted statewide market data analysis (Method D) provides a pre-program market share very close to that determined from our participating supplier data. This result supports the initial similarity of the participating suppliers to the general market, Vectren 1-7 August 18, 2009

16 and also supports the validity of our survey-based method for estimating sales shares, Method C. The statewide market data analysis (Method E) provides a similar result to that of the Vectren-adjusted version of this analysis, but slightly lower. It makes sense that the Vectren program could be somewhat more effective in its second year than the statewide average. The other utility programs were in their first year during More detailed results for the narrow and broad interpretations of program influence on suppliers are shown in Table 1-3. The table shows that only 16 percent of end users were pure free riders, end users that were not influenced to buy energy efficient units by the supplier nor directly by Vectren. The program receives zero attribution credit for these end users. Another 2 percent were influenced by Vectren but not by the supplier. The program attribution for these end users, based on additional end-user survey questions, was a little over 50 percent. Table 1-3 Attribution Results from the Demand-Supply Decision Influence Methods A and B Weighted Fraction of Program Attribution Narrow Supply- Broad Supply- Side Influence Side Influence Influence Category Sample Count Participants Supplier Only % 19% 51% Vectren Only 13 2% 55% 55% Both Supplier and Vectren 57 10% 60% 51% Neither Influence 90 16% 0% 0% Overall w/o Adjustment % 22% 43% Overall Including Adjustment 22% 45% The majority of end users were influenced by their suppliers; some of these were also influenced by Vectren. The extent of influence for supplier-influenced end users was roughly twice as high using the broad, sales-based interpretation as using the narrow, recommendations-based interpretation. Supplier Attribution Results from Demand-Supply Decision Influence Surveys The supplier survey analysis produced a supplier attribution estimate that reflects the effect of the program on supplier business practices. In terms of the overall program results, the supplier attribution is also the attribution of the Supplier Only influence category end users. Vectren 1-8 August 18, 2009

17 For the narrow interpretation of program influence, KEMA determined the supplier program attribution by examining the change in the percent of high efficiency equipment recommendations from 2006 to For the broad interpretation, we determined the sales share of supplier-influenced energy efficient sales for each year. Supplier-influenced sales are total efficient sales less the portion associated with pure free riders or, in 2008, with direct program influence. Table 1-4 shows the percentage of recommendations and supplierinfluenced sales in 2006 and 2008 and the average change over that period. Also shown are the changes in recommendations and sales as fractions of 2006 and 2008 levels. Table 1-4 Participating Suppliers Changes in High Efficiency Recommendations and in Supplier-Influenced Sales 2006 % 2008 % Average Change Change as % of 2006 Level Change as % of 2008 Level Method Sector Used For A: DSDI-N Residential recommendations (n=54) 79% 97% 18% 23% 19% Attribution for Non-residential recommendations (n=29) 71% 95% 25% 35% 25% Supplier Influence B: DSDI-B Energy efficiency sales, both sectors 44% 78% 35% 80% 51% Attribution for Supplier Influence C Overall supplier-influenced sales, both sectors 43% 78% 35% 81% 46% Total Attribution D Overall market sales, both sectors U/A* U/A* U/A* U/A* 46% Total Attribution E Overall market sales, both sectors U/A* U/A* U/A* U/A* 42% Total Attribution *Data is confidential and therefore cannot be reported. The table shows an increase in energy efficiency recommendations of 22 percent for residential customers and 35 percent for nonresidential customers using Method A. The table also shows an 81 percent increase in energy efficiency sales using Method B and an increase of 106 percent using Method C. These are indications of program influence. However, the portion of 2008 recommendations attributable to the program is the change as a percent of 2008 levels, not as a percent of the starting level. Since most participating suppliers started out with relatively high rates of recommending efficient equipment, there was relatively little room for improvement in this dimension. There was substantially more room for improvement in successful sales to customers not already inclined to buy high efficiency. This rate more than doubled from 2006 to Put another way, 51 percent of participating suppliers energy efficient sales to supplier-influenced end users are attributable to the program Spillover Spillover takes two forms: 1) influence of the program on efficient equipment sales outside the program, and 2) adoption of additional efficiency measures outside any program as a result of Vectren 1-9 August 18, 2009

18 program participation. We accounted for incremental furnace/boiler sales outside the program based on participating suppliers reports of the extent of rebate use by qualifying equipment, as well as the tracking data indicating likely qualifying units that could not be rebated. The two market sales data analyses indicate little additional increase in efficient sales beyond what is accounted for by the in-program increases and these adjustments. More definitive information on the first category of spillover would require surveys of nonparticipating suppliers and end users. Our surveys did not probe for the data to determine the second type of spillover. While some end users reported using lower setpoints because they were able to be comfortable at a lower temperature, we do not know that this is specifically because a high-efficiency unit was installed rather than simply because a new unit of any efficiency was installed Effect of Fuel Prices One reason this program was launched was to help provide customers relief from rising fuel prices. Vectren s retail prices rose substantially from 2000 up to 2006 but then flattened off. Vectren provided KEMA with a file listing all of the monthly gas charges from February 2000 to July Figure 1-1 shows a plot of the results. Vectren 1-10 August 18, 2009

19 Figure 1-1 Monthly Natural Gas Prices in Vectren North Service Territory $ per therm Dec-99 Apr-01 Sep-02 Jan-04 May-05 Oct-06 Feb-08 Jul-09 The figure shows that gas prices have been rising since 2000 with a large increase in Gas prices remained at 60 cents or lower prior to the beginning of 2005 then rose to twice the previous maximum in 2005 and until they dropped slightly in 2006 and leveled at approximately 150 percent of the previous maximum. Gas prices were relatively constant from 2006 through Thus, a relatively high preprogram rate of recommending and selling energy efficient equipment is not surprising. Further changes after 2006 would not be expected related to fuel prices. Thus, 2006 rates serve as reasonable estimates of the baseline of what 2008 would have looked like without the program. The confidential sales data is also consistent with the assumption of little change outside the program area Conclusions and Recommendations KEMA recommends that the attribution found using Method C (48%) be used to determine net program savings. Table 1-5 shows the net savings using the Method C attribution rate. In the Vectren 1-11 August 18, 2009

20 table, the attribution is applied to the furnace and boiler savings only because those measures were the focus of this evaluation. The savings from the other measure are reported but are not verified. Table 1-5 Prescriptive Program Net Savings Lifetime Gross Savings Attribution Rate Lifetime Net Savings Measure Group Furnace/Boiler 13,071,850 48% 6,274,488 Other 3,455,009 N/A 3,455,009 Prescriptive Overall 16,526,859 N/A 9,729,497 KEMA recommends that the Vectren Conservation Connection program be continued. To that end, we have assembled a number of recommendations regarding the prescriptive program based on the impact and process survey results. We also recommend that the program adopt 48 percent as an estimate of the net-to-gross ratio for furnaces and boilers going forward. Recruiting New Suppliers A key area where the Program could improve its energy savings claims and program attribution ratings is through the recruitment of new HVAC contractors. This will have a number of benefits. First it will allow the Program to reach end users that it is not reaching now. Residential customers often are loyal to their incumbent HVAC contractor and if their furnace breaks down they are likely to first call the phone number on the sticker that s attached to their furnace. If that contractor is not already participating in the Vectren Program, then these customers are unlikely to be made aware of the rebates available for higher-efficiency equipment. The Program will also likely get higher program attribution ratings from the new HVAC contractors since the Program will likely be changing their business practices. As noted, the reported average incremental cost of a 90% AFUE vs. an 80% AFUE model is over $600. Therefore the nonparticipating contractors are likely to be currently promoting the 90%+ AFUE furnaces to a lesser degree than the participating contractors since they do not have a Vectren rebate to help close that cost gap. In contrast, the survey evidence from the participating contractors shows that the vast majority (95-97%) claim to be recommending high-efficiency equipment to their customers. As these contractors make recommending high efficiency equipment part of their standard business practices, they are less likely to give the Program credit for these practices, especially over time as memory fades and/or new staff are hired that are unaware of the Program s influences. Vectren 1-12 August 18, 2009

21 There are a number of ways that the Conservation Connection Program could try to recruit more new HVAC contractors. Do more marketing to end users: While relying on contractors as the primary means of recruiting end users can reduce Program marketing costs, it has the potential, as discussed above, to shut out a substantial share of the end-user population who are loyal to their nonparticipating contractors. Marketing to end users directly about the benefits of energyefficient equipment and the availability of the Vectren rebates can encourage customers to ask their incumbent contractors about these benefits and opportunities. In addition to bringing new customers into the Program fold, this will also encourage new contractors to join the Program. Do more marketing to new suppliers with financial inducements for new suppliers: The Program should do more direct mail marketing to HVAC contractors in Indiana, putting particular focus on those who are currently not participating in the Program. Although Vectren or the Conservation Connection Program implementers may already have a list of HVAC contractors in Indiana, it has been our experience that such lists are often incomplete or outdated. For example, we have found that many smaller HVAC contractors do not appear on utility trade ally lists. This is because they do not respond to surveys by industry trade data compilers like Dun & Bradstreet or are not involved with trade associations another frequent source of such trade ally lists. Yet it is important to recruit these smaller HVAC contractors since they likely account for a large percentage of the non-participating contractors (larger companies are usually savvier about exploiting utility rebate opportunities). We have found that yellow page listings and state contractor licensing lists are goods way to locate some of these smaller HVAC contractors. In addition to developing an attractive direct mail piece to recruit these new contractors, the Program might also consider giving a financial incentive for contractors who have never participated in the Program before and are able to convince a certain number of their customers to buy the rebated energy-efficient equipment. More Training for Suppliers Although percent of the participating suppliers reported promoting the high-efficiency equipment, they also reported that it only accounted for percent of their sales (depending on whether it was residential or nonresidential sales). This and other evidence (e.g., the low incidence of contractors reporting the discussion of life-cycle costs with their customers) suggests that there is some room for improvement in terms of how contractors try to sell the high-efficiency equipment. The Wisconsin Focus on Energy program has offered training Vectren 1-13 August 18, 2009

22 sessions on energy-efficiency salesmanship for many years and has reported very positive feedback from contractors who attended these sessions. To encourage greater participation at these salesmanship training sessions, one idea would be to combine it with the Conservation Connection Program s annual kickoff meeting where it meets with suppliers to discuss the Program s latest rebate offering and program requirements. This would increase the potential audience and reduce the burden, especially for the smaller HVAC contractors, of having to travel to two separate Program events. Other ways to recruit contractors to the trainings would include highlighting these events in the supplier direct mail campaigns mentioned above as well as such traditionally-popular inducements such as free lunches or breakfasts. Since many suppliers may not have the time to travel to a training event, creating a webinar on energy-efficiency salesmanship would allow the Program to reach such suppliers. Vectren could further facilitate an emphasis on life-cycle costs and return-on-investment (ROI) by providing the suppliers with worksheets that show the ROI of high-efficiency equipment at current gas prices, if gas prices increase by 10 percent, and if gas prices increase by 20 percent. These worksheets could carry the official Vectren logo to increase their legitimacy. Vectren could also consider supplying laminated worksheets to participating suppliers. Offer Incentives Directly to Suppliers Some suppliers suggested offering an incentive directly to contractors (SPIFFs). They cited two reasons for doing so. First, contractors would be more motivated to sell the higher-efficiency units if there was something in it for them directly. Second, some of the contractors said that compensation for the time it takes them to fill out the rebate paperwork would be nice. We suggest that Vectren could provide a small SPIFF to suppliers. A small SPIFF could compensate the suppliers for the time it takes to fill out the paperwork and provide a small incentive for them to sell the high-efficiency units. More Marketing to Nonresidential Customers This report presents some evidence that suggest that the Program could be doing more to recruit nonresidential end users. HVAC contractors and distributors reported that the proportion of their nonresidential furnace sales that were high efficiency were much lower (34%) than their residential sales (59%). Another reason for the Program to focus more on these nonresidential customers was that these customers had higher program attribution ratings than the Vectren 1-14 August 18, 2009

23 nonresidential customers. A direct mailing to commercial property managers would be one way to recruit more nonresidential customers to the Program. More Marketing to High-Use Residential Customers Our results also indicate that most of the end users purchase new HVAC equipment because their old equipment stopped working, rather than upgrading functional equipment. A direct marketing effort may be able to increase the number of end users who upgrade functioning equipment. This would increase program attribution by giving the program more credit for acceleration of replacements. One group to focus this direct marketing on is residential end users with older homes and/or high gas bills. These end users are the most likely to benefit from upgrading their HVAC equipment, and the direct marketing effort could also emphasize lifetime savings/roi. To further encourage end users with older equipment to upgrade, Vectren could offer a bonus rebate or bounty for anyone who replaces equipment of a certain age (e.g.: 15+ years old) or lower efficiency equipment (e.g.: 70% AFUE equipment). Additional Research This study conducted surveys only with participating suppliers and customers. The work described here could be enhanced by surveying nonparticipating suppliers and customers in addition. Surveys with nonparticipating suppliers could assess: The rate of recommendations and sales shares of energy-efficient equipment by nonparticipating suppliers within and outside of Vectren territory. Program awareness Reasons for not participating. Characteristics of participating versus nonparticipating suppliers. Likelihood of increasing sales with further program support. The recommendations rate and sales share data could provide further confirmation of or adjustment to the analysis described here. Similarly, surveys with nonparticipating customers who acquired new furnaces or boilers could assess: Program awareness Reasons for not participating. Vectren 1-15 August 18, 2009

24 Current attitudes toward energy efficiency Characteristics of participating versus nonparticipating customers Likelihood of increasing energy efficient purchases with further program support Custom Program The Custom Program impact analysis included a review of the program s estimate of gross savings as well as a net savings analysis. The primary purpose of the evaluation was to produce an estimate of the net savings achieved by the program. The custom program impact evaluation methodology used data from two sources to determine program attribution: the end-user survey and the engineering review. KEMA attempted to deliver end-user surveys to the entire program population. The engineering review was an analysis of the gross savings estimates claimed by the program. A KEMA engineer used data from the program documentation, the end-user contact, and secondary sources to determine whether the program s gross savings estimate was reasonable. If the estimate was deemed not reasonable, a verified gross savings amount was determined for the project. The engineering reviews were conducted for every project with a completed enduser survey and produced a gross savings adjustment factor for the program. The end-user survey data was analyzed to verify measure installation and to produce an estimate of the influence that the program had on end users decision to install high efficiency equipment. The survey asked for the program s effect on the timing, efficiency, and quantity of equipment installed and the results were built into an end-user attribution. End users were also asked to represent the degree of influence provided by their supplier. The supplier survey was delivered for a particular project only if the end user was primarily influenced by his or her supplier. Unlike the prescriptive program supplier survey, the custom supplier survey asked for the program s effect on the timing, efficiency, and quantity installed for that particular project, not for market information or business practices in general. The data was analyzed to produce an estimate of the influence that the program had on the supplier s decision to sell high efficiency equipment for that project. As with the prescriptive program, the attribution results of the supplier and end-user surveys were combined into one overall program attribution. For supplier-influenced projects, the enduser attribution was compared to the supplier attribution and the maximum value was taken as Vectren 1-16 August 18, 2009

25 the program attribution. For all other projects, the end-user attribution was taken as the program attribution. The gross savings adjustment factor and attribution results were weighted according to the sample design and ratio estimation was used to apply the sample results to the population, resulting in total savings attributable to the custom program. This is the first year of implementation for the prescriptive program and only six projects were completed. The program is still in its infancy; therefore, the results of this evaluation should be assessed as case studies and not used to anticipate future program performance Results The Custom Program completed six projects with five end users. Unlike the prescriptive program, the custom program has only been in effect for one year. The program year started on April 1, 2008 and ended on March 1, KEMA attempted to complete surveys with all five of the custom program end users representing six projects. We were able to complete surveys with three end users representing four projects. Our analysis of the custom program began with a review of the program s estimate of gross savings. A KEMA engineer reviewed the gross savings estimates from four of the six projects installed during the first year of the program. Table 1-6 shows the tracking estimate and evaluation-verified estimate of gross savings for all four projects. Table 1-6 Gross Energy Savings Estimates Tracking Gross Savings Verified Gross Savings Project Participant A.1 7,611 5,269 Participant A.2 5, Participant B 3,352 6,627 Participant C 1, The primary goal of the evaluation was to produce a life cycle estimate of savings attributable to the program. Therefore, KEMA applied measure lives to our estimates of verified energy savings to produce the verified gross life cycle savings for Program Year 1 based on the reviews we completed. Table 1-7 shows the annual tracking and verified gross savings, the Vectren 1-17 August 18, 2009