2014 Demand-Side Management Evaluation Final Report

Transcription

1 2014 Demand-Side Management Evaluation Final Report June 30, 2015 Indianapolis Power and Light One Monument Circle Indianapolis, IN 46204

2 This page left blank.

3 Prepared by: Cadmus With Subcontractor Opinion Dynamics

4 This page left blank.

5 Table of Contents Introduction... 1 Methodology Overview... 3 Commercial and Industrial Summary... 6 Residential Summary... 6 Peer Comparison Report Program... 7 Multifamily Direct Install Program... 7 Appliance Recycling Program... 8 Home Energy Inspector Program... 8 Commercial and Industrial Programs... 9 Program Budgets, Goals, and Scorecard... 9 Methodology Overview... 9 Audit and Engineering Review... 9 Sampling, Review, and Analysis... 9 Evaluated Ex Post Gross Impacts Ex Post Net Savings Evaluation Findings Audit Engineering Review Gross Impacts Net Impacts Conclusions and Recommendations Peer Comparison Report Program Program Description Program Goals, Budgets, and Scorecard EM&V Methodology Equivalency Analysis Billing Analysis Impact Analysis Overall Program Savings Paper-only vs. Paper/ Models i

6 Channeling Analysis Process Analysis Engagement with Report Neighborhood Comparison Energy Saving Attitudes and Energy Tips Energy Efficiency Actions Taken Customer Satisfaction Demographics Program Insights and Recommendations Multifamily Direct Install Program Program Description Program Design Changes Program Budget, Goals, and Scorecards EM&V Methodology Impact Analysis Verified Volume Ex Post Gross Savings Ex Post Net Savings Process Analysis Program Participation Knowledge Increase Customer Satisfaction Program Influence Property Characteristics Program Insights and Recommendations Appliance Recycling Program Program Description Planned PY2015 Program Design Changes Appliance Efficiency Standards Program Implementation Program Budget, Goals, and Scorecards ii

7 EM&V Methodology Impact Analysis Verified Measure Volume Appliance Characteristic Factors and Adjustments Part-time Use Adjustment Factors Ex Post Per-Measure Savings Algorithms Gross Energy and Demand Savings Program Insights and Recommendations Home Energy Inspector Program Program Description Program Budget, Goals, and Scorecards EM&V Methodology Impact Analysis Ex Ante Deemed Savings Ex Post per-measure Savings Ex Post Gross Savings Ex Post Net Estimates Program Insights and Recommendations CoolCents Program Residential and C&I Program Budgets, Goals, and Scorecards Appendix A: Self-Report Net-to-Gross Evaluation Methodology Appendix B: Peer Comparison Report Modeled Estimates Appendix C: Peer Comparison Report Fixed Effects Model Output for Wave Appendix D: Peer Comparison Report Fixed Effects Model Output for Wave 1 (Paper vs. Paper/ ) Appendix E: Peer Comparison Report Fixed Effects Model Output for Wave Appendix F: Peer Comparison Report Fixed Effects Model Output for Wave 2 (Paper vs. Paper/ ) Appendix G: Peer Comparison Report Fixed Effects Model Output for Wave Appendix H: Peer Comparison Report Fixed Effects Model Output for Wave 3 (Paper vs. Paper/ ) iii

8 Appendix I: Multifamily Direct Install Program Assumptions and Algorithms Appendix J: Appliance Recycling Program Measures, Assumptions, and Algorithms Appendix K: Home Energy Inspector Program Measures, Assumptions, and Algorithms iv

9 Introduction This evaluation report presents impacts achieved by the Indianapolis Power and Light (IPL) Demand Side Management Portfolio in PY Table 1 provides a list of programs evaluated in PY2014 as well as a brief description of each. Table 2, Table 3, and Table 4 present goals, energy savings, and demand savings, respectively, that the portfolio achieved in PY2014. Program C&I DSM Programs Business Energy Incentive CoolCents- Commercial* Residential DSM Programs Peer Comparison Multifamily Direct Install Appliance Recycling Home Energy Inspector CoolCents Residential* Table 1. IPL DSM Program Offerings Brief Program Description This program offers customers custom and prescriptive incentives on a wide range of energy efficient equipment including lighting, HVAC, and appliances. The commercial CoolCents program helps IPL reduce peak load by curtailing air conditioning units during peak periods in the summer. Participating customers receive a credit on their bill for participating in the program. The program provides information to customers so they can reduce their energy consumption through behavioral changes as well as increased customer engagement The main goal of the program is to direct install energy efficiency measures such as CFLs, lowflow showerheads, and kitchen and bathroom aerators at multifamily premises The program collects and recycles functioning refrigerators, freezers and room air conditioners to remove less-efficient appliances from the power grid. In PY2014 the program also offered a light-bulb kit with three CFLs to customers who were present at the time of the appliance removal. The program offers free energy efficiency measures to customers after completion of an online energy audit The objective of the program is to reduce peak load by curtailing air conditioning during peak usage periods during summer months, between May and September. This program targets residential single-family, multifamily and mobile home premises. Monthly summer bill credit provided based on number of controlled HVAC units. *The CoolCents program evaluations will be delivered under separate cover, in summer IPL also offered DSM programs such as Ecohouse, Residential New Construction and Residential renewable incentive programs. However, these programs were not included in the contracted scope of work for the PY2014 electric evaluation. 1

10 Program Commercial & Industrial Table Program Savings Goals PY2014 Energy Savings Goal (kwh) PY2014 Demand Savings Goal (kw) Business Energy Incentive- Combined 34,505,911 5,785 Residential Peer Comparison 29,045,262 - Appliance Recycling 7,490, Multifamily Direct Install 2,272, Home Energy Inspector 1,779, Residential- TOTAL 40,587,551 1,480 Portfolio Total 75,093,462 7,265 Portfolio savings values follow below. The detailed program sections of the report that follow present findings and observations regarding these values. Table Portfolio Energy Savings Summary Energy Savings (kwh) Program Ex Ante Audited Verified Ex Post Gross Ex Post Net Commercial and Industrial Business Energy Incentive Custom 24,802,245 26,457,689 26,457,689 24,799,765 16,591,043 Business Energy Incentive Prescriptive 13,755,234 13,108,870 13,108,870 13,449,495 13,786,618 Business Energy Incentive Total 38,557,479 39,566,559 39,566,559 38,249,260 30,377,661 Residential Peer Comparison Multifamily Direct Install Appliance Recycling Home Energy Inspector 29,061,000 29,061,000 (a) 29,061,000 (a) 28,497,349 (b) 28,497,349 5,857,744 5,857,744 5,470,271 5,300,381 5,300,381 2,101,658 2,103,037 1,991,883 2,092,791 1,643, , , , , ,631 2

11 Program Residential Total Energy Savings (kwh) Ex Ante Audited Verified Ex Post Gross Ex Post Net 37,878,341 37,879,718 37,276,432 36,611,284 36,080,044 Portfolio- Total 76,435,820 77,446,277 76,842,991 74,860,544 66,457,705 (a) The Evaluation Team could not confirm the ex ante deemed savings value for the Peer Comparison program and, as such, the ex ante energy savings are carried forward for the audited and the verified energy savings. (b) The Peer Comparison program evaluation establishes ex post net savings only. The ex post values are net of attrition. Program Commercial and Industrial Business Energy Incentive Custom Business Energy Incentive Prescriptive Business Energy Incentive Total Residential Table Portfolio Demand Savings Summary Demand Savings (kw) Ex Ante Audited Verified Ex Post Gross Ex Post Net 3, a 3, , , ,339 3, a 3, , , ,756 6, , , , ,095 Peer Comparison - - (b) - (b) 6,814 (c) 6,814 Multifamily Direct Install Appliance Recycling Home Energy Inspector Residential Total 1,132 1,149 1,063 7,858 7,772 Portfolio-Total 7,946 7,769 7,703 14,043 12,867 a The tracking database served as a source; the utility Scorecard exhibited minor variance from the tracking data (+/- 10 kw). b The Evaluation Team could not confirm the ex ante deemed savings value for the Peer Comparison program and, as such, the ex ante energy savings are carried forward for the audited and the verified energy savings. c The Peer Comparison program evaluation establishes ex post net savings only. The ex post values are net of attrition. Values rounded for reporting purposes Methodology Overview This evaluation primarily sought to perform an impact analysis for the IPL DSM portfolio, with the report covering the PY2014 program year. The Evaluation Team used the relative consistency among the programs and implementation methods (as well as continuity among Evaluation Team personnel) to 3

12 leverage parameters and findings from previous years evaluations. Where possible (or required), the team used information such as in-service rates (ISRs), net-to-gross (NTG) ratios, and similar parameters to inform this year s evaluation findings. An impact analysis seeks to accurately quantify demand and energy savings from an efficiency measure or a collection (program/portfolio) of efficiency measures. These analyses employ statistical and engineering-based analysis techniques. Following a progression, analysis results for each savings type correspond to a specific step in the evaluation process. Quantified impacts correspond to the following steps: Step 1: Audited savings Step 2: Verified savings Step 3: Evaluated ex post gross savings impact Step 4: Evaluated ex post net savings impact Figure 1 provides a visualization of the EM&V process that the team has taken for estimating savings from the Portfolio. Figure 1. Demand-Side Management Impact Evaluation Steps Table 5 provides definitions and purposes for each evaluation step. 4

13 Table 5. Evaluation, Measurement, and Verification Impact Definitions Category Definition Purpose Ex Ante Savings Reported savings values in IPL s Scorecard. Goal setting Audited Savings Verified Savings Checks for tracking system accuracy; program savings based on adjusted program tracking data. The following approach provided this value: Review of the program tracking databases Review of a sample of hardcopy program applications to verify consistency with data recorded in the program tracking databases Confirms measures have been installed and currently operate. This step uses a random sample of installations selected for detailed analysis. Typical methods for collecting necessary data include telephone surveys and site visits. For installed Custom measures, engineering assumptions may be reviewed to provide a statistically representative sample of projects. This step may be adjusted to address issues such as the following: Measures rebated but never installed Measures not meeting program qualifications Measures installed but later removed Measures improperly installed This step further adjusts audited savings as follows: Check ex ante-deemed savings estimates and calculations to ensure the implementer/utility applied previously agreed-upon values appropriately/correctly Adjust program tracking data as necessary to correct identified errors or omissions Recalculate program savings based on adjusted program tracking data Intermediate step only Confirmation of program reach and persistence of installed measures Ex Post Gross Savings Calculated using engineering analysis, building simulation modeling, billing analysis, metering analysis, or other accepted methods. Adjustments may include changes to baseline assumptions, adjustments for weather, adjustments to occupancy levels, and adjustments to decreased or increased production levels. Indication of actual savings achieved by program Ex Post Net Savings Determined by adjusting ex post gross savings estimates to account for a variety of circumstances, including savings-weighted, 1 freerider, 2 and spillover effects. 3 The NTG ratio for the two types of net savings estimates derived from the following formula: Participant Net Savings: Annual Net-to-Gross Ratio = (1- freerider adjustment + participant spillover adjustment) Program design improvements, planning future programs, costeffectiveness analysis, calculations of lost revenues and shared savings 5

14 Category Definition Purpose 1 Freerider, spillover, and market effects adjustments to the NTG ratio are weighted to reflect the savings levels associated with those effects, compared to the savings levels directly achieved from installed measures. Savings are weighted so adjustments to net savings are proportionate to savings levels associated with the actions taken. Thus, small savings actions result in small adjustments and large savings actions result in larger adjustments, depending on the occurrence level. 2 Freeriders would have taken exactly the same action (or made the same behavior change) installing a measure (or changing a behavior) with exactly the same energy efficiency result at the same time as they did in taking the program-incented action. Partial freeriders would have taken exactly the same action, but the program expedited that change; or they would have taken similar action, but not at the same efficiency level as the program-incented action; or they would have made the same behavior change, but at a later time than the program-encouraged behavior change. 3 Savings produced due to the program s influence on participants energy use, through technology purchases and usage changes, or through behavioral changes induced or significantly influenced by the program or the portfolio, but not directly incented by the program. Commercial and Industrial Summary In PY2014, the Business Energy Incentive program exceeded its gross energy and demand savings targets, while spending fewer funds than budgeted. IPL delivers the Business Energy Incentive as a single program with a Custom and Prescriptive track. The Custom track delivered approximately double the savings (24.8 million kwh) of the Prescriptive track (13.4 million kwh); however, Custom freeridership was 33.1% (with no spillover savings); this can be attributed to one of the eight respondents being estimated as 100% freerider, and three of the eight respondents being estimated as 50% freeriders. The full freerider respondent stated that they had already intended to purchase the measure before learning about the program. While the Prescriptive track had lower savings compared to the Custom track, program freeridership was very low (4.9%); the program also had spillover savings, resulting in an overall Net-to-Gross ratio of 103% (i.e., ex post net savings were higher than ex post gross savings). Residential Summary The IPL DSM programs showed strong growth in PY2014 when compared to the previous year. Overall, the Residential Core Plus portfolio increased energy savings by 13% more than PY2014. All programs expanded in terms of the number of participants, number of measures installed (or recycled), and in trade ally/builder reach. The residential portfolio contributed 36,611,284 kwh in ex post gross energy savings and achieved 7,859 kw in ex post gross demand savings (a 5% increase over the PY2013 ex post gross demand savings of 7,481 kw, excluding the CoolCents programs). Despite this growth, IPL s overall portfolio is also in transition. It absorbed some of the statewide programs (as of January 2015). The program also reviewed the viability of ongoing programs, resulting in the closure of three residential programs including: 6

15 The Residential New Construction program as it has historically incented few homes, and the demand for all electric homes has drastically curtailed with low prevailing natural gas market prices. HVAC High Efficiency program (in 2013) because the program was not deemed cost-effective. The Residential Renewables program due to low historic participation rates The major upsurge in energy savings are due to the large increase in the reach of the Peer Comparison Report program that more than doubled by reaching 210,961 homes in PY2014. The IPL Peer Comparison program contributed 78% of ex post gross energy savings. The PY2014 residential evaluation focused on impact results for all programs. In addition, detailed results from participant surveys with Peer Comparison and Multifamily Direct Install (MFDI) program participants are included in this report. As per the focus on conducting participant surveys every other year, no surveys were fielded for the Appliance Recycling and Home Energy Inspector (HEI) programs. Highlights of each evaluated residential DSM program follow. Peer Comparison Report Program The Peer Comparison Report program is the largest contributor of savings to the IPL residential Core Plus programs portfolio, achieving an ex post net adjusted energy savings of 28,497,349 kwh, with a realization rate of The program achieved an ex post net adjusted peak coincident demand savings of 6,814 kw. The Evaluation Team could not establish a realization rate as the program did not report demand savings. The program sent quarterly Home Energy Reports (HER) to 210,961 households in the IPL service territory (after attrition). The program more than doubled in PY2014 with the addition of over 120,000 households to the treatment group. IPL reports that the program now reaches upwards of 54% of its total residential population. As the program expands, it now reaches lower energy consuming households, including multifamily and mobile properties. In order to manage program expenditures, IPL has reduced the number of mailed reports to earlier participants (Wave 1) to twice a year, maintaining a quarterly schedule for the other waves. As with previous evaluations, there does not appear to be an increase in savings in households that also receive reports compared to those that receive both report types. To test effectiveness of reports, IPL may consider setting up a treatment group that only receives HER reports via . Multifamily Direct Install Program The Multifamily Direct Install (MFDI) program is the second largest contributor to IPL s Residential Core Plus portfolio, contributing 5,300,381 kwh to the program with a realization rate of The program contributed 593 kw in gross ex post demand savings with a realization rate of The program introduced new measures in PY2014 such as programmable thermostats and LED night lights. Thermostats represent 0.36% of program savings. 7

16 Property managers who responded to the participant survey have indicated satisfaction with the program overall and that they would participate in the program again. The leading drivers of satisfaction were the time it took to schedule and complete the installations, the professionalism of the implementer, and the CFLs that were installed. Some property managers have suggested that the program may be enhanced by sharing more program information with the properties tenants, adding measures installed through the program such as LED bulbs and fixtures and power strips, and increasing the number of CFLs and programmable thermostats installed. Appliance Recycling Program The Appliance Recycling program is the third largest contributor to IPL s Residential Core Plus portfolio, contributing 2,092,791 kwh to the program with a realization rate of 1.0. The program contributed 370 kw in gross ex post demand savings with a realization rate of The Appliance Recycling Program recycled 2,389 appliances and reportedly had customers install 1,775 CFL bulbs (the program distributed 4,425 bulbs, a 126% increase compared to PY2013), establishing a record performance for the program. While there were no major program changes in PY2014, IPL plans to increase the incentive payment beginning in 2015 to $40 per appliance (refrigerators and freezers; Room ACs incentives remain unchanged at $20/unit). Home Energy Inspector Program The Home Energy Inspector (HEI) program contributes 720,763 kwh to the program with a realization rate of The program contributed 81 kw in gross ex post demand savings with a realization rate of The program distributed 2,370 energy kits during PY2014, a slight increase over the 2,355 kits distributed in the previous year. IPL changed kit suppliers during the year, and included LED night lights in addition to standing CFL bulbs and water conservation measures. The program continued operating in partnership with Citizen s Gas where IPL claims savings only from lighting measures distributed to homes with gas heating. 8

17 Commercial and Industrial Programs This section presents impact evaluation results for the Commercial and Industrial programs. The Evaluation Team conducted a series of interviews with program staff to learn of program changes, but did not conduct process evaluation activities as part of this assessment. For Custom projects, prospective applicants must submit a pre-application, documenting the intended project and projected savings. After customers have been approved (based on the pre-application), they must submit a final application following the retrofit; this documents any changes from the original pre-application. Program Budgets, Goals, and Scorecard Table 6 displays program budgets and expenditures for PY2014. Commercial and Industrial Table 6. Commercial and Industrial Budgets and Expenditures Program Program Budget Program Expenditures Percentage of Total Business Energy Incentive Combined $3,798,417 $3,758,301 96% Methodology Overview Audit and Engineering Review The Evaluation Team collected IPL and implementer program savings claims and measured tracking data, reviewing and comparing the data at an aggregate level to ensure alignment among sources. Such activity often detects systematic tracking or documentation issues, such as inconsistent program year definitions, midyear implementation changes, or differing extract and report dates. The team then collected available program and project documentation (including application files, calculation workbooks, implementer methodology work papers, and other pertinent information) and reviewed the files for the following features: Alignment with program tracking data; Agreement with the Indiana Technical Reference Manual (TRM), as appropriate; Completeness of information; Reasonableness of assumptions; and Appropriate algorithms and data sourcing. Sampling, Review, and Analysis The Evaluation Team selected representative samples for each program offering. These samples served as the focus for evaluation analysis, with analysis findings then extrapolated to the larger program population. 9

18 Each sample was random and weighted (stratified or probability proportional to savings), with claimed energy savings at the measure level selected for the weighting factor. Sample targets were set at 80% confidence and 10% precision for each offering (90/10 one-tailed), presuming a 0.5 coefficient of variation for the total population. This target approach yielded 90% confidence and 10% precision when applied at the program level. The Evaluation Team then performed engineering desk reviews for each sampled measure. The team employed the following review and analysis techniques: Recreating claimed savings values using appropriate algorithms and deemed parameters, where applicable; Comparing relevant secondary source values, such as TRMs, energy codes, case studies, and other publicly available research; Interviewing facility personnel via phone or to understand operating practices, baseline conditions, and other pertinent details; and Researching key data affecting measures, including manufacturer specification sheets and performance data. Finally, the Evaluation Team coordinated and cross-checked findings to determine spillover savings values. Evaluated Ex Post Gross Impacts The Evaluation Team determined evaluated ex post gross impacts from synthesizing all prior measurement and verifications activities, all pertinent data, and other applicable secondary sources (e.g., TRMs, energy codes). These values represented the best estimate of true demand and energy savings from the associated measures or programs. The team generated these savings values for the sample populations and then calculated adjustment factors (i.e., ex post savings/ex ante savings) or realization rates (i.e., verified savings/ex ante savings). The team applied the resulting factors to the gross population savings values, thus projecting evaluation findings for the total program. Ex Post Net Savings The Evaluation Team implemented a NTG methodology to evaluate the Business Energy Incentive Program in This NTG methodology consisted of two components freeridership and spillover. In 2014, the Evaluation Team implemented two surveys one for each program track. As tracking data for the Business Energy Incentive Program s Custom option were unavailable at the measure level, the team modified the survey to capture decision-making processes at the project level as a whole. The following formula provided final NTG ratios for the program: NTG Ratio = 1 Freeridership + Spillover 10

19 To determine freeridership, the Evaluation Team drew upon a previously developed approach that ascertained freeridership using patterns or responses to a series of 13 simple questions. The questions which allowed yes, no, and don t know responses asked whether participants would have installed the same equipment in the program s absence, at the same time, at the same amount, and at the same efficiency level. By assigning freerider scores to question response patterns, confidence and precision estimates could be calculated on score distributions. 2 The Evaluation Team calculated participant spillover by estimating the following: Savings attributable to additional measures installed; and Whether respondents credited IPL with influencing those decisions. The team counted measures as spillover if customers did not request an incentive, even though the measure qualified for a program incentive. Appendix A provides a detailed explanation of the team s NTG methodology, including descriptions of freeridership and spillover evaluation methodologies and full-text versions of NTG survey questions asked of participants. Evaluation Findings Findings follow for the Business Energy Incentive program, including Custom and Prescriptive offerings. Audit Program tracking data and program Scorecard values generally aligned very well, albeit with some exceptions. The final iterations of the Scorecard and tracking data were issued in March/April Updates to these data sources included late, year-end projects and measures. The Evaluation Team found a minor discrepancy (+/-10 kw) between claimed demand savings listed on the Scorecard and totals presented in the final program tracking database: in this case, program tracking data had been used as the claimed savings values of record. This audit process detected several notable disagreements between the tracking data and spot-checked program files. The database incorrectly tracked a pair of prescriptive HVAC measures. Quantities and savings values were significantly inflated from the actual values presented in the project file, although reasonable reported incentive values were tracked. This likely resulted from an input error during data entry of the measure data as the units total tonnage matched the claimed quantity, and the two sets of values were presented side-by-side on the application form. Similarly, the measure count tracked for a sampled lighting controls measure did not match the count from the supporting files. 2 This approach follows methods outlined in: Schiller, Steven et al. National Action Plan for Energy Efficiency. Model Energy Efficiency Program Impact Evaluation Guide Available online: The IPL Oversight Board reviewed and approved the evaluation team s NTG questions and response pattern scoring approach. 11

20 Figure 2 and Figure 3 illustrate program population savings details by measure type. This measure type was characterized as Lighting, Non-Lighting or Both Lighting & Non-Lighting, and was determined by the Evaluation Team during the audit process, based on tracking data information. Figure 2. Custom Population, Ex Ante Savings by Measure Type (kwh) Figure 3. Prescriptive Population, Ex Ante Savings by Measure Type (kwh) Engineering Review The Prescriptive and Custom offerings exhibited excellent evaluated savings performance, relative to claimed savings values. Notably, however, the evaluation process found many projects with inaccurate savings claims. The Evaluation Team found many measures with evaluated savings lower than claims, 12

21 but the many projects yielding values higher than the original claimed values counterbalanced these measures. Energy savings claims generally yielded higher realization rates than demand savings claims. The complexity of peak demand periods and coincidence factors likely drove this outcome. The Evaluation Team applied adjustments relating to demand savings for a number of measures examined; these adjustments included most lighting measure savings calculations evaluated to accurately incorporate coincidence factors and waste heat factors. For a number of Prescriptive measures, the exact application of the TRM methodology remained unclear. Lighting control measures particularly seemed to use deemed savings values that did not correlate to the TRM methodology or available measure data. Though the TRM methodology may have been used for these calculations, the application was very likely inappropriate. Conversely, nearly all lighting projects used appropriate wattage values that reflected the true power demand of fixtures (including ballast factors) and not just nominal lamp wattages. For Custom measures, a number of deviations were detected during this review process resulting in adjustments to savings. However, no clear patterns of deviations were detected. The Evaluation Team found most project documents complete and well organized. The review process found several incomplete files, although the implementation contractor was prompt in providing additional information and clarification. Overall, despite a few inconsistencies, the implementer s tracking and documentation system appeared effective. Gross Impacts A summary of impact evaluation activities follows, organized by program. In general, a relatively large share of program savings could be evaluated directly, in part due to a weighted sampling scheme. Program Commercial and Industrial Business Energy Incentive Custom Business Energy Incentive Prescriptive Gross Population Unit Count Table 7. Impact Evaluation Activities Gross Population Measure Count Evaluation Sample Measure Count Target Sample Measure Count Evaluation Sample Energy Savings Share of Program % 48, % 1 Represents the proportion of ex ante energy savings from the sampled measure population, relative to the gross program population (sample population energy savings/gross population energy savings). Summaries follow for each program s savings values, as does a chart of impact parameters resulting from the evaluation process and analysis. Additionally, a breakdown follows of evaluated adjustment 13

22 factors (ex ante savings divided by ex post savings), illustrating some issues and inconsistencies with claimed savings. In general, though, very good agreement occurred between claimed savings values and evaluated findings. Program Commercial and Industrial Business Energy Incentive Custom Business Energy Incentive Prescriptive Applied ISR 1 Table 8. Impact Evaluation Parameters Demand Realization Rate 2 1 Source: Participant Surveys 2 Source: Review and Surveys = (Verified/Ex Ante) 3 Source: Engineering Analysis = (Ex Post/Ex Ante) 4 Source: NTG Analysis = (1 Freeridership) + Spillover Energy Realization Rate 2 Demand Adjustment Factor 3 Energy Adjustment Factor 3 NTG Ratio 4 100% 99% 107% 97% 100% 66.9% 100% 95% 95% 84% 98% 102.5% Table 9. Business Energy Incentive Program Custom Offering Savings Summary Savings Type Goal 1 Ex Ante 2 Audited 3 Verified 4 Ex Post 5 Net 6 Custom Offering Peak-Coincident Demand Savings Not specified 3,601 3,583 3,583 3,497 2,339 (kw) Energy Savings (kwh) Not specified 24,802,245 26,457,689 26,457,689 24,799,765 16,591,043 1 The Scorecard only provides goals at a program level (i.e., Custom and Prescriptive combined). 2 Source: Scorecard and program tracking. 3 Source: audit. 4 Source: review and surveys. 5 Source: engineering analysis. 6 Source: NTG analysis. Table 10. Business Energy Incentive Program Prescriptive Offering Savings Summary Savings Type Goal 1 Ex Ante 2 Audited 3 Verified 4 Ex Post 5 Net 6 Prescriptive Offering Peak-Coincident Demand Savings (kw) Energy Savings (kwh) Not specified Not specified 3,212 3,038 3,038 2,689 2,756 13,755,234 13,108,870 13,108,870 13,449,495 13,785,732 1 The Scorecard only provides goals at a program level (i.e., Custom and Prescriptive combined). 14

23 2 Source: Scorecard and program tracking. 3 Source: audit. 4 Source: review and participant surveys. 5 Source: engineering analysis. 6 Source: NTG analysis. Measure Type Table 11. Savings Breakdown by Measure Type Evaluation Sample Measure Count Evaluation Sample Unit Count Demand Adjustment Factor Energy Adjustment Factor Prescriptive Offering Air Compressor with VSD % 100% ENERGY STAR LED 15 8, % 128% Exterior Lighting % 92% F32 T8 HB Lighting % 100% High Performance T8 Lighting % 198% Low Watt T8 Lighting w/ CEE Ballast % 137% Occupancy Sensors % 63% Other Efficient Lighting Technologies % 42% Parking Garage Lighting % 92% Rooftop and Unitary AC % 8% Custom Offering Medium Project - $5K-$25K % 99% Small Project - $1-5K % 120% Net Impacts After conducting 49 Prescriptive participant phone surveys, the Evaluation Team converted participant responses into individual freeridership scores. The team converted scores using the Excel-based matrix approach described in Appendix A and then determined the freerider score by translating a survey respondent s responses into a matrix value, and applying a rules-based calculation to obtain the final score. The team calculated freeridership scores, confidence intervals, and precision estimates based on distributions of scores within the matrix. The following tables and figures present all combinations of responses received for the program, along with the score assigned to each combination, revealing participants responses did not reflect every potential combination. Responses tended to group around subsets of common patterns. Table 12 shows freeridership calculation results for the program, indicating an overall 23% freeridership score. 15

24 Table 12. Freeridership Results Program Option Responses (n) Freeridership Score Total Commercial and Industrial 63* 23.2% * Customers installing more than one measure were asked the freeridership battery of questions for a maximum of two measures, while custom participants were asked about the project as a whole, resulting in 63 unique responses. Table 13 shows the percentage of total gross evaluated kwh energy savings, total number of survey responses and savings weighted freeridership estimates by participant type. Fewer survey completes were collected for C&I Custom Program compared to the C&I Prescriptive Program due to a small population from which to draw a sample and because the C&I Custom Program participants were difficult to reach during the phone survey process. The C&I Prescriptive Program freeridership estimate of 5% is less than the 12% freeridership estimated in PY2013. C&I Custom Program participant surveys were not completed in PY2013. Program Category Table 13. Freeridership Results by Program Evaluated Gross Percentage of Total Gross Number of Responses* Freeridership C&I Prescriptive Energy Savings (kwh) 13,449, % %** C&I Custom Energy Savings (kwh) 24,799, % %** C&I Overall Energy Savings (kwh) 38,249, % %*** *Customers installing more than one measure were asked the freeridership battery of questions for a maximum of two measures, while custom participants were asked about the project as a whole, resulting in 63 unique responses. **Weighted by survey sample ex post gross program kwh savings. ***Weighted by population ex post gross program kwh savings. Table 14 shows the unique C&I Prescriptive Program response combinations resulting from the participant survey, freeridership scores assigned to each combination, and the number of responses for each combination. An x indicates a skipped question (due to the participant s response to a previous question). A few common patterns appeared regarding the respondents answers to freeridership questions: Twenty four respondents would not have purchased any equipment at all without the incentives. As these respondents indicated they would not have purchased the measures at all without the incentive, they could be considered 0% freeriders. Three respondents who reported purchasing or installing rebated measures before learning about the program were estimated as 100% true freeriders. Eight respondents were estimated as 0% freeriders because they reported they would have installed less efficient equipment in absence of the program incentive. 16

25 Table 14. Frequency of Prescriptive Participant Freeridership Scoring Combinations 1. Installed same measure without incentive? 2. Already planning to purchase? 3 Already planning to purchase? 4. In capital budget? 5. Installed same quantity? 6. Installed same efficiency? 7. Installed at the same time? 8. Increased ROI? 9. Confirm, would have installed same measure? [Ask if question 9 is No] 10. Installed same efficiency? [Ask if question 9 is No and if measure QTY > 1] 11. Installed same quantity? [Ask if question 9 is No] 12. Installed at the same time? [Ask if question 9 is No] 13. Increased ROI? Freeridership Score Response Frequency Yes Yes x x x x x x x x x x x 100% 3 Yes No Yes Yes Yes Yes Partly No x x x x x 25% 1 Yes No Yes Yes Yes Partly Yes No x x x x x 25% 1 Yes No Yes Yes Yes No x x x x x x x 0% 2 Yes No Yes No Yes Yes Yes No x x x x x 12.5% 1 Yes No Yes No Yes No x x x x x x x 0% 1 Yes No Yes No No Yes Partly No x x x x x 0% 2 Yes No Partly x Yes Partly Yes No x x x x x 12.5% 1 Yes No No x Yes Yes Partly Yes x x x x x 25% 1 Yes No No x Yes Yes Yes No x x x x x 12.5% 1 Yes No No x Yes Yes Partly No x x x x x 0% 1 Yes No No x Yes Partly Yes No x x x x x 0% 2 Yes No No x Partly Yes No x x x x x x 0% 1 Yes No No x No Yes No x x x x x x 0% 1 Yes No No x No No x x x x x x x 0% 1 Partly No Yes No Yes No x x x x x x x 0% 1 Partly No Partly x Yes Yes Partly No x x x x x 0% 1 Partly No Partly x Yes Partly Partly No x x x x x 0% 1 Partly No No x Yes Yes Partly No x x x x x 0% 1 Partly No No x Partly Yes Partly No x x x x x 0% 1 Partly No No x Partly Partly Partly No x x x x x 0% 2 Partly No No x No No x x x x x x x 0% 1 No x x x x x x x Yes No x x x 0% 3 No x x x x x x x No x x x x 0% 24 17

26 Table 15 shows the unique C&I Custom Program response combinations resulting from the participant survey, freeridership scores assigned to each combination, and the number of responses for each combination. An x indicates a skipped question (due to the participant s response to a previous question). A few common patterns appeared regarding the respondents answers to freeridership questions: Three respondents would not have purchased the measure at all without the incentives. As these respondents indicated they would not have purchased the measures without the incentive, they could be considered 0% freeriders. One respondents was estimated as a 0% freerider because they reported they would have installed less efficient equipment in absence of the program incentive. Three respondents were estimated as 50% freeriders, as they planned to purchase the measures before hearing about the incentive, planned the purchase in their most recent capital budget, and confirmed they would have purchased the same efficiency and quantity of the measures at the same time without the incentive, but the program incentive increased the return on investment for the measure thereby allowing the project to receive implementation approval. One respondent who reported purchasing or installing rebated measures before learning about the program was estimated as a 100% true freerider. Table 15. Frequency of Custom Participant Freeridership Scoring Combinations 1. Installed same measure without incentive? 2. Already planning to purchase? 3 Already planning to purchase? 4. In capital budget? 5. Installed same quantity? 6. Installed same efficiency? 7. Installed at the same time? 8. Increased ROI? 9. Confirm, would have installed same measure? [Ask if question 9 is No] 10. Installed same efficiency? [Ask if question 9 is No and if measure QTY > 1] 11. Installed same quantity? [Ask if question 9 is No] 12. Installed at the same time? Yes Yes x x x x x x x x x x x 100% 1 698,516 [Ask if question 9 is No] 13. Increased ROI? Freeridership Score Response Frequency Total Ex Post Gross kwh Savings Yes No Yes Yes Yes Yes Yes No x x x x x 50% 3 575,853 Yes No No x No No x x x x x x x 0% 1 1,223,627 No x x x x x x x No x x x x 0% 3 477,730 18

27 Individual freeridership scores can be analyzed by examining their distribution, as shown in Figure 4 and Figure 5 for the C&I Prescriptive and Custom Programs, respectively. Figure 4. Distribution of Prescriptive Freeridership Scoring Combinations Approximately 84% of C&I Prescriptive respondents did not indicate freeridership; almost 10% of respondents indicated low levels of freeridership (12.5% to 25%), while 5% of respondents were estimated as full freeriders. Figure 5. Distribution of Custom Freeridership Scoring Combinations Four of the eight C&I Custom respondents did not indicate freeridership; three of the eight respondents indicated moderate freeridership (50%), one respondent was estimated as a full freerider. As detailed in Appendix A, the team estimated spillover measure savings using specific information from the evaluation about participants and through the use of the Indiana TRM as a baseline reference. The team estimated the program s spillover percentage by dividing the sum of additional spillover savings 19

28 (as reported by survey respondents) by the total incentivized gross savings achieved by all survey respondents from the relevant program track (i.e., Custom or Prescriptive). Program Track Table 16. Surveyed Participant Savings Analysis Spillover Savings (kwh) Surveyed Program Participant Savings (kwh) Spillover Percentage Prescriptive 166,050 2,253, % Custom 0 2,975, % Overall, surveyed C&I Prescriptive Program participants highly influenced by the program installed three efficient lighting controls, 1,344 LEDs, and 12 CFLs that did not receive funding from IPL. Table 17 displays the number of participants installing these additional measures considered spillover and the total KWh savings resulting from these installations. No C&I Custom Program participants surveyed attributed influence to their program participation on additional energy-efficient purchases. Table 17. Spillover Measures, Quantity, and Savings Spillover Measure Participants Project Energy Savings (kwh) Efficient Lighting Controls 1 2,443 LEDs 5 162,460 CFLs 1 1,147 Total 1 166,050 Table 18 summarizes the C&I Program s freeridership, spillover, and NTG percentages. Program Option Responses (n) Table 18. NTG Ratio for C&I Participants Freeridership Percentage Spillover Percentage NTG Ratio NTG Absolute Precision C&I Prescriptive %* 7.4% 102.5%* ±7.8% C&I Custom %* 0.0% 66.9%** ±41.9% Overall %*** 2.6%*** 79.4%*** ±37.1% *Weighted by survey sample ex post gross program kwh savings ** Weighted by survey sample ex post gross program kwh savings. Estimate does not meet the 10% absolute precision threshold at the 90% confidence interval. ***Weighted by program category ex post gross population program kwh savings. Conclusions and Recommendations Conclusion: The Custom and Prescriptive offerings produced relatively accurate claimed savings estimates. Despite some definite inaccuracies and issues, the relative agreement between ex ante (claimed) savings values and ex post (evaluated) savings is a positive outcome. This conclusion especially holds true for a program that offers Custom measures with many varying factors (e.g., weather, production, occupancy). 20

29 Recommendation IPL and its implementers should continue to maintain and improve upon these high standards for measure tracking, savings estimation, and documentation. Conclusion: Project and measure tracking methods generally proved effective and transparent. The implementation contractor generally applied an effective and understandable organization and documentation scheme. Project files were demarcated with cover pages to organize and label materials appropriately. Where provided, measure savings calculations were presented in a tabular (spreadsheet) format; so stakeholders could understand assumptions and algorithms via direct examination. This year s final utility Scorecard generally was accurate, but it did not track Prescriptive offerings separately from Custom offerings, and exhibited a minor deviation from totals presented in the final program tracking data. Recommendation Currently, all data sources present generic labels for savings values such as kw Saved. IPL and its implementers should consider specifying peak-coincident demand savings in project and tracking materials and in the utility Scorecard to eliminate ambiguity among stakeholders. Additionally, the implementer did not use tracking numbers for Prescriptive measures; amending this policy could provide added clarity regarding savings attribution and avoid variances between claimed savings data sources. Conclusion: The claimed savings calculation methodologies for some measures were unclear and likely inappropriate.the implementation contractor probably leveraged the TRM and previous evaluation findings to generate claimed savings values, particularly for Prescriptive measures. Custom projects generally employed a standardized workbook, used to collect pertinent data. Prescriptive measures lacked documentation beyond project applications. Recommendation IPL and its implementer should consider publishing methodology work papers that detail the calculation methodology and assumptions used for a given measure. Such work-papers often serve as a bridge between programs and TRMs. These documents also can contribute to a feedback loop with evaluation findings informing updates to the applicable TRM. 21

30 Program Description Peer Comparison Report Program IPL funds the Peer Comparison Report program to meet the following primary objectives: Obtain measurable and cost-effective energy savings Increase program participation in select IPL Energy Efficiency, Renewables and Demand Response programs The program uses behavioral science-based marketing to provide customized energy consumption information to selected IPL residential households. Treated households receive information on their energy consumption as compared to their peers, with the theory that such comparisons will prompt measurable energy savings among treated households. Every quarter, treated households are mailed a hardcopy of a Home Energy Report (HER) that contains customized suggestions for reducing energy consumption. Households are randomly chosen using a randomized control trial (RCT) design. In an RCT, a target population is randomly assigned to treatment and control groups. Due to the randomization process, treatment and control group customers are theoretically equivalent and, therefore, any difference between the treatment and control groups energy use after receiving treatment is considered attributable to the treatment intervention. The IPL Peer Comparison program was launched in March The program initially targeted highusage households that fell within the top 60% of energy usage in the residential sector. Given that the program met its program goals during its first year and was deemed to be cost-effective, IPL added an expanded group of customers in March PY2013. However, overall usage for this expanded group of customers is almost half that of the original customers (implication of this are discussed in the impact results section). In PY2014, the program added 183,993 treatment households to meet program goals established in the 2014 IURC filing (this reduced to 143,520 after attrition). By extending the program to households not traditionally in the high-energy users population, the program now reaches approximately 54% of its overall residential customer population (implication of this are discussed in the impact results section). The Evaluation Team defines treatment households by waves as defined by OPower, the program implementer, generally based on when they first started receiving HER reports. Table 19 shows that in PY2014, while 273,778 IPL households received HER reports, this number was reduced to 210,961 due to the number of accounts that became inactive, given that as customers move, their account at the premise becomes inactive. 22