Public Service Company of Oklahoma 2017 Energy Efficiency & Demand Response Programs: Annual Report

Transcription

1 Public Service Company of Oklahoma 2017 Energy Efficiency & Demand Response Programs: Annual Report Prepared for: Oklahoma Corporation Commission In accordance with annual reporting requirements: Title 165: Oklahoma Corporation Commission Chapter 35. Electric Utility Rules Subchapter 41. Demand Programs 165: Reporting June 22, 2018

2 Table of Contents 1 EXECUTIVE SUMMARY PROGRAM OFFERINGS SUMMARY OF ENERGY IMPACTS SUMMARY OF PEAK DEMAND IMPACTS SUMMARY OF PORTFOLIO BENEFIT-COST RATIOS CUMULATIVE PORTFOLIO PERFORMANCE SUMMARY OF PROCESS EVALUATION FINDINGS INTRODUCTION REDUCED EMISSIONS AND WATER CONSUMPTION MILESTONES ACHIEVED IN MARKET TRANSFORMATION PROGRAMS ANNUAL UTILITY GROWTH METRICS AND PORTFOLIO RATIOS HIGH-VOLUME ELECTRICITY USER OPT OUT FUEL SWITCHING IMPACTS PROGRAM IMPLEMENTATION & STRATEGIC ALLIANCES TRAINING AND CUSTOMER OUTREACH ENERGY EFFICIENCY PROGRAMS HIGH PERFORMANCE BUSINESS PROGRAM (HPB) HOME WEATHERIZATION PROGRAM ENERGY SAVING PRODUCTS PROGRAM HIGH PERFORMANCE HOMES PROGRAM EDUCATION PROGRAM BEHAVIORAL MODIFICATION DEMAND RESPONSE PROGRAMS PEAK PERFORMERS OVERVIEW APPENDIX A. GLOSSARY APPENDIX B. PORTFOLIO COST-EFFECTIVENESS B.1 COST EFFECTIVENESS SUMMARY B.2 ENERGY EFFICIENCY PROGRAMS B.3 DEMAND RESPONSE PROGRAMS B.4 AVOIDED COSTS APPENDIX C. IDENTIFICATION OF PROGRAM IMPLEMENTERS APPENDIX D. TRAINING AND CUSTOMER OUTREACH i

3 APPENDIX E. MARKETING SYNOPSIS APPENDIX F. OKDSD, AR, & IL TRM DEEMED SAVINGS AND ALGORITHMS F.1 ENERGY EFFICIENCY PROGRAMS APPENDIX G. M&V SAFETY APPENDIX H. LIGHTING DISCOUNTS PRICE RESPONSE MODEL DETAILS H.1 INTRODUCTION H.2 DATA SOURCES AND PROCESSING ii

4 1 Executive Summary This report presents an evaluation of the performance of the energy efficiency and demand response programs, also known as the Demand Portfolio, offered by Public Service Company of Oklahoma (PSO) in PSO is submitting this report to fulfill the requirements outlined in Title 165: Oklahoma Corporation Commission Chapter 35. Electric Utility Rules Subchapter 41. Demand Programs 165: On July 1, 2015, PSO submitted a comprehensive portfolio of energy efficiency and demand response programs (Portfolio Filing) to the Oklahoma Corporation Commission (OCC) for Program Years This portfolio was approved by the OCC in Cause No. PUD by Order No on December 1, The focus of this report is participation during the second program year (PY2017) of the implementation cycle, spanning from January 1, 2017 to December 31, Program Offerings PSO offered customers seven programs: four residential, two commercial/industrial, and one cross-sector program in Program names, PY2017 start dates, and targeted customer sectors are shown in Table 1-1. Table 1-1: Program Start Dates Program Sector Start Date Energy Efficiency Programs High Performance Business Commercial & Industrial January 01, 2017 Home Weatherization Low-Income Residential January 01, 2017 Energy Saving Products Residential December 01, High Performance Homes Residential January 01, 2017 Education Residential & Commercial January 01, 2017 Behavioral Modification Residential January 01, 2017 Demand Response Programs Business Demand Response Commercial & Industrial January 01, All the programs represent program participation from January 1, 2017 December 31, 2017 except the ESP program. The reported savings for LED retail discounts span the time period of December 1, 2016 November 30, This month offset allows for reconciliation of retail sales data and manufacturer/retailer invoices. Executive Summary 1

5 For the purposes of this report, projected, reported, and verified impacts are defined as follows: Projected Impacts refer to the energy savings (kwh) and peak demand reduction (kw) estimates submitted to the OCC as part of PSO s portfolio filing approved on December 1, Reported Impacts refer to energy savings (kwh) and peak demand (kw) reduction estimates based on actual customer participation in PY2017 before program evaluation activities. Verified Impacts refer to energy savings (kwh) and peak demand (kw) reduction estimates for PY2017 developed through independent program evaluation, measurement, and verification (EM&V). Verified impacts reflect actual program participation (as opposed to projected participation) and adjust for any findings from ADM s independent evaluation; which includes a detailed review of program materials, interviews with program participants, and, in some cases, detailed on-site data collection. A glossary of these and other energy efficiency and demand response evaluation specific terms is provided in Appendix A. All impacts presented in this report represent energy savings or peak demand reduction at-the-meter except for Section 1.4 and Appendix B, where impacts at-the-generator are adjusted using an estimated line loss factor of for calculating program cost effectiveness. Program impacts including projected, reported, and verified annual energy savings and peak demand reduction during PY2017 are summarized in the following sections. 1.2 Summary of Energy Impacts At the portfolio level, reported energy savings in PY2017 were 125,410 MWh. Total gross verified energy savings were 130,447 MWh. This realization rate for gross energy savings was 104%. The verified gross annual energy savings is 114% of the projected gross annual energy savings of 114,440 MWh. The Net-to-Gross (NTG) ratio indicates the percentage of gross savings directly attributable to program influences. The portfolio-level NTG ratio was estimated as 80%, resulting in a net annual energy savings of 104,661 MWh. The net annual energy is 113% of the projected net annual energy of 91,950 MWh. Table 1-2 summarizes the energy impacts of PSO s energy efficiency and demand response programs during PY Order number for Cause No. PUD Executive Summary 2

6 Table 1-2: Summary of Gross Energy Impacts PY Program Gross Annual Energy Savings (MWh) Gross Projected Reported Verified Realization Rate Energy Efficiency Programs Net Impacts (MWh) Projected Net Net NTG Energy Energy Ratio Savings Savings High Performance Business 52,644 57,385 58, % 42, ,642 Home Weatherization 3,865 5,204 4,930 95% 3, ,930 Energy Saving Products 31,233 44,517 51, % 21, ,569 High Performance Homes 7,493 10,355 7,832 76% 6, ,800 Education 4,949 7,949 7,906 99% 3, ,178 Behavioral Modification 14, , Conservation Voltage Reduction Energy Efficiency Totals 114, , , % 91, ,119 Demand Response Programs Business Demand Response Demand Response Totals Portfolio Totals 114, , , % 91, , Summary of Peak Demand Impacts At the portfolio level, reported peak demand reduction in PY2017 was MW. Total gross verified peak demand reduction was MW. The realization rate for peak demand reduction was 80%. The verified gross peak demand reduction is 97% of the projected gross peak demand reduction of 73.8 MW. The portfolio-level NTG ratio for peak demand reduction was estimated as 93%, resulting in a net peak demand savings of MWh. The net peak demand savings is 95% of the projected net peak demand savings of MW. Table 1-3 summarizes the peak demand impacts of PSO s energy efficiency and demand response programs during PY Rounding may affect totals and net-to-gross ratio multiplication/division in table. Executive Summary 3

7 Table 1-3: Summary of Demand Impacts PY Program Gross Peak Demand Reduction (MW) Gross Projected Reported Verified Realization Rate Energy Efficiency Programs Net Impacts (MW) Projected Net Peak Net Peak NTG Demand Demand Ratio Reduction Reduction High Performance Business % Home Weatherization % Energy Saving Products % High Performance Homes % Education % Behavioral Modification Conservation Voltage Reduction Energy Efficiency Totals % Demand Response Programs Business Demand Response % Demand Response Totals % Portfolio Totals % Summary of Portfolio Benefit-Cost Ratios ADM calculated the cost-effectiveness of PSO s 2017 programs based on reported total spending, verified net energy savings, and verified net demand reduction for each of the energy efficiency and demand response programs. Additional inputs to the cost effectiveness tests included estimates of natural gas savings, line-loss adjustments, emissions reductions, measure lives, discount rates, participant costs, and avoided costs. All program spending inputs were provided by PSO as shown in Appendix B. The total portfolio spend was $31,549,637. The methods used to calculate cost-effectiveness were informed by the California Standard Practice Manual. 5 4 Rounding may affect totals and net-to-gross ratio multiplication/division in table. 5 California Standard Practice Manual: Economic Analysis of Demand Side Management Programs, October Available at: _Electricity_and_Natural_Gas/CPUC_STANDARD_PRACTICE_MANUAL.pdf. Executive Summary 4

8 The specific tests used to evaluate cost-effectiveness for the Oklahoma Corporate Commission are the Utility Cost Test and the Total Resource Cost Test. The benefit-cost ratios for those tests as well as the Rate Payer Impact Test, the Societal cost test, and the participant cost test are presented in Table 1-4. Detailed cost-effectiveness assumptions and findings are presented in Appendix B. Table 1-4: Benefit-Cost Ratios Program Utility Cost Test Total Resource Cost Test Ratepayer Impact Measure Societal Cost Test Participant Cost Test Energy Efficiency Programs High Performance Business Home Weatherization Energy Saving Products High Performance Homes Education Behavioral Modification Conservation Voltage Reduction Energy Efficiency Total Demand Response Programs Business Demand Response Demand Response Total Portfolio Total Another way to view portfolio performance for 2017 is on a levelized dollar per kwh savings or dollar per peak kw reduction basis. Energy efficiency programs are designed to reduce energy usage while providing the same or improved service to the end-user in an economically efficient way, regardless of whether energy usage occurs during peak or non-peak periods. Energy savings occur for the lifetime of the energy efficiency measures installed. As such, program performance was assessed on a levelized dollar per kwh basis for energy efficiency programs. Levelized cost in $/kwh is calculated as shown in the formula below: Levelized Cost (in $/kwh) = C x Capital Recovery Factor / D Capital Recovery Factor = [A*(1+A)^(B)]/[(1+A)^(B)-1] Executive Summary 5

9 Where: A = Discount rate (5%, 7.32%) B = Estimated measure life in years 6 C = Total program costs D = Annual kwh savings Table 1-5 shows how PSO s portfolio of energy efficiency programs performed on a levelized cost basis for PY2017. The verified net lifetime energy savings in Table 1-5 include a line loss adjustment factor of Program Year Table 1-5: Levelized $/kwh for Energy Efficiency Programs Total Costs Verified Net Lifetime Energy Savings (kwh) 5% Levelized $/kwh - 5% Verified Net Lifetime Energy Savings (kwh) 7.32% Levelized $/kwh % 2017 Residential 7 $18,152,406 $479,635,542 $ $415,893,125 $ Commercial 8 $10,283,729 $519,023,188 $ $454,301,640 $ EE Programs $28,436,135 $998,658,730 $ $870,194,764 $ A study conducted by the American Council for an Energy Efficient Economy (ACEEE) 9 in 2014 compared levelized $/kwh for electric energy efficiency programs in 20 different states between 2009 and Levelized costs estimated as part of this study ranged from $0.016 to $0.048 with an average of $0.028 (2011$). Compared to the ACEEE analysis, the levelized $/kwh for PSO s energy efficiency programs are more favorable than 9 of the 20 states when using a discount rate of 5% and are more favorable than 6 of the 20 states when using a discount rate of 7.32% 11. Demand response programs are designed to encourage customers to change their normal consumption patterns during periods when prices are high or system reliability is potentially constrained. These programs encourage load reduction during a short period of time, usually a limited number of days during the summer. As such, demand response program performance was assessed on a peak kw reduction per dollar basis. Table 1-6 shows how PSO s portfolio of demand response programs performed on a $/kw reduction 6 Calculated as described in Appendix B. 7 Residential Programs include: Home Weatherization, High Performance Homes, Energy Savings Products, and Education Residential and School Kits. 8 Commercial Programs include: High Performance Business and Education Non-Residential Kits. 9 Molina, Maggie, The Best Value for America s Energy Dollar: A National Review of the Cost of Utility Energy Efficiency Programs. ACEEE, March This study used a discount rate of 5% to calculated levelized $/kwh 11 Weighted Average Cost of Capital. Provided by PSO in the Portfolio Plan. Executive Summary 6

10 basis for PY2017. The verified net peak demand reduction in Table 1-6 includes a line loss adjustment factor of Table 1-6: $/kw for Demand Response Programs Program Year Total Costs Verified Net Peak Demand Reduction from DR (kw) $/kw 2017 $2,594,726 48,534 $ Cumulative Portfolio Performance PY2017 was the second program year for the Demand Side Management (DSM) portfolio. Portfolio-level energy and demand impacts for PY2017 are shown in Table 1-7. Portfolio-level energy and demand impacts from are shown in Table 1-8. Table 1-7: 2017 Portfolio Performance Verified Energy and Peak Demand Impacts Program Year Verified Gross Annual Energy Savings (GWh) Verified Net Annual Energy Savings (GWh) Energy Efficiency Programs Verified Gross Peak Demand Reduction (MW) Verified Net Peak Demand Reduction (MW) Cumulative EE Totals Demand Response Programs Cumulative DR Totals Cumulative Portfolio Totals Executive Summary 7

11 Table 1-8: Cumulative Portfolio Performance Verified Energy and Peak Demand Impacts Program Year Verified Gross Annual Energy Savings (GWh) Verified Net Annual Energy Savings (GWh) Energy Efficiency Programs Verified Gross Peak Demand Reduction (MW) Cumulative EE Totals Demand Response Programs Cumulative DR Totals Summary of Process Evaluation Findings During the first quarter of 2018, ADM completed their process evaluation for PY2017. Program participants, service providers, and program staff were largely satisfied with PY2017 portfolio offerings. Key process related findings and recommendations are summarized below: High Performance Business -Custom and Prescriptive While the design of the Custom and Standard Component remained largely the same in 2017 as in 2016, staff continued to make improvements to program delivery. Most notably was a mid-year introduction of a simplified application process. Under the new process, customers submit a single page application and are then contacted by an account manager who collects additional information on the project. Additionally, the online intake tool is now fully operational. Staff noted that this tool works well for customers submitting smaller projects and eases the application process by using dynamic forms that narrow the information needed based on information provided in earlier portions of the application. Staff is in the process of implementing two new program offerings to increase uptake of non-lighting measures: a new commissioning and energy management service, and offering targeted efficiency improvements in oil pumping operations. Executive Summary 8

12 Small Business Energy Solutions (SBES) Staff noted that there was a high level of interest in the SBES Program in Staff noted that a key driver of program activity was the addition of the TLED lamps. One challenge noted was the development of refrigeration projects. The barriers to refrigeration projects identified were: (1) not having a local representative to sell refrigeration projects, and (2) the additional education required to encourage customers to adopt refrigeration measures as compared to what is needed for lighting measures. Staff is continuing to work with the refrigeration contractor to get a local salesperson as well as scoping out other firms that can deliver refrigeration measures. 92% of customers surveyed reported that they were somewhat or very satisfied with the program overall. Service providers demonstrated an understanding of guidelines for installing the TLED lamps added in Service providers stated that they typically recommend Type B over Type A lamps because these provide the best value and fit customers needs. Most lighting service providers reported that they also promote refrigeration and other equipment incentives offered by PSO. Service providers viewed the support provided by and communication with program very favorably. Service providers indicated that SBES compares favorably to other small business programs they are familiar with and Audit Direct Install (ADI) online proposal tool continues to be viewed as a useful administration tool. All lighting service providers rated their satisfaction as either a 4 or 5 on a 5-point Likert scale (with 5 meaning very satisfied ). Home Weatherization Overall, the program design and operations for the Home Weatherization Program remained unchanged. Two noteworthy developments resulted in a potential expansion of the program s reach: First, funds contributed by Oklahoma Natural Gas (ONG), who have partnered with PSO to implement the program, increased in PY2017 compared to PY2016 to approximately 18% of the program. The additional funds allowed the program to serve more customers. Second, the program reached an agreement with the Ki Bois Community Action Partnership to complete weatherization services for 10 homes in While no homes were weatherized through Ki Bois, staff intends to continue this arrangement in Executive Summary 9

13 A direct mail campaign followed by telephone outreach continued to be the primary recruitment method. Surveyed customers indicated that they learned about the program primarily through the direct mail and telephone campaign, as well as by word-of-mouth. The program was also promoted in a PSO newsletter. Program satisfaction remained high with more than 90% of customers reporting that they were satisfied with the information provided about the program, the quality of the work performed, the energy savings realized, and the overall experience with the service. Key Process Findings for Energy Saving Products PSO continues to add additional measures to the retail program. The incentives for ENERGY STAR heat pump water heaters, ENERGY STAR dryers, and APS are consistent with those used in other jurisdictions. o PSO offers $500 for ENERGY STAR heat pump water heaters. Incentives offered by three utilities in Illinois and Missouri ranged from $300 - $500. o PSO offers $50 for ENERGY STAR dryers, which is the same amount offered through an Illinois utility rebate program. o PSO offers $12 for APS. Two utilities in Illinois offered incentives of $10 and $13. Additionally, the planned sales targets are reasonable given the achieved market penetration in other jurisdictions. o PSO rebated.11 ENERGY STAR heat pump water heaters per 1,000 customers. The values found for three other utilities ranged from.05 to.36. o PSO rebated.32 ENERGY STAR dryers per 1,000 customers. An Illinois utility achieved.74 units per 1,000 customers. o PSO plans on rebating 3.87 APS per 1,000 customers, a value which is somewhat higher than the 2.05 units per 1,000 customers rebated by an Illinois utility. Ninety-five percent of customers who received rebates for appliances were satisfied with the program overall and the application process, and ninety percent were satisfied with the wait time to receive the rebate. Executive Summary 10

14 Performance Homes - New Homes, Multiple Upgrades, and Single Upgrades Components The program exceeded its energy savings goals for Changes to the air sealing incentive structure resulted in an overall decrease in the average incentive payment for this measure. Staff noted that this change, as well decreased activity from a contractor who had relatively high acquisition costs for duct sealing, resulted in greater program participation with the available incentive budget than initially projected. Compared to PY2016, the average incentive payment made to homes in the Multiple Upgrades program component decreased while average reported kwh savings increased (from 3,105 kwh to 3,963 kwh). Additionally, the average number of measure types installed per home increased from 3.5 to 3.7. The program offered the New Green Appraiser training in 2017, and ICF staff indicated that this change resulted in an increase in the number of green appraisers in Oklahoma from 1 to 18. Two program improvements for the New Homes program component were identified and were in the process of being implemented. One change is to reduce the bonus for ENERGY STAR homes from $1,000 to $800. Second, the staff has identified a means to streamline the processing of REM/Rate files through bulk processing. This change should allow program staff to process builder submissions more frequently than once a month (as is currently done), allowing for faster payment and an improved experience with the program for builders. Most participants surveyed reported being somewhat or very satisfied (96%) with the Multiple Upgrades program component overall, with 2% indicating they were neither satisfied nor dissatisfied, and 1% indicating they were dissatisfied. Similarly, most participant surveyed reported being somewhat or very satisfied (95%) with the Single Upgrades component overall. Energy Education The School Kits component continued to operate smoothly and did not have any challenges meeting its goals. Staff noted that program participation is currently limited by available budget and interest in the component exceeds what can be supported by the available funding. The Residential Kits Program achieved its distribution goals without difficulty. The Non-residential Kits component did not meet its annual goal of 3,000 kits distributed. Based on the program participation in PY2016 and PY2017, PSO decided to reduce the goal to 1,000 kits, which PSO was able to meet in Executive Summary 11

15 Twenty-percent of respondents reported that they were aware of the PSO discounts and incentives and learned of them from information provided in the kit. Participants who received a residential kit were satisfied with the program overall (94%) and the time it took to get the kit (87%). Behavioral Modification The program design and logic incorporate the strategies most commonly used to influence customer behavior in home energy report programs; namely, using social norms, information on historical energy consumption, and tips on how to reduce energy consumption. Future planned components of the program will make use of additional behavioral strategies (e.g., increasing self-efficacy for reducing energy use, using rewards and points, etc.). A key goal of the program is to drive customers to use the My Energy Advisor Portal, through which customers can get more detailed information on their homes energy use, access additional energy saving tips and view information on PSO program information, and in the future, track what actions they have taken to save energy and monitor their progress. Metrics on use of My Energy Advisor show a clear link between the delivery of HERs and increased use of My Energy Advisor. Because of the time required to set up key program infrastructure, the program first sent home energy reports in October. As such, the reports were sent after the cooling season, the time of year when behavioral programs tend to have their greatest impacts on electricity usage. Continued implementation of the program should see greater savings in future program years. Although the sample size was limited (47 respondents), 15% of customers that completed the participant survey reported that they had completed their My Energy Advisor Profile and 36% stated that they had logged on to My Energy Advisor to view the information on their home s energy use provided. While 74% of customers indicated that they have not logged into the My Energy Advisor portal, 25% to 33% of customers surveyed indicated being unable to log onto My Energy Advisor, indicating greater interest in using My Energy Advisor overall. Most respondents reported that they were somewhat or very satisfied with the number of s received (62%) and the information on the homes energy use (67%). Approximately 25% of surveyed participants indicated that they were neither satisfied nor dissatisfied. Few respondents (12% or less) indicated dissatisfaction with either aspect of the program. Executive Summary 12

16 Business Demand Response Overall the Business Demand Response program remained consistent in Staff continued to make incremental improvements to the notification and results delivery process, which improved customers experience with the program. Targeted outreach efforts in Lawton were successful in increasing the number of participating sites from this region. All new participants in the program were satisfied with the program overall and none reported dissatisfaction with the program. Most participants reported being very likely to participate again in PY2018 (75%). The number of events called were consistent with 55 percent of respondents expectations, 25 percent reported that there were fewer events than expected. Executive Summary 13

17 2 Introduction This report presents an evaluation of the performance of the energy efficiency and demand response programs offered by Public Service Company of Oklahoma (PSO) in PSO is submitting this report to fulfill the requirements outlined in Title 165: Oklahoma Corporation Commission Chapter 35. Electric Utility Rules Subchapter 41. Demand Programs 165: PSO contracted with ADM Associates, Inc. (ADM) to perform comprehensive program evaluation, measurement, and verification (EM&V) for PY2017. ADM s evaluation findings for each 2017 energy efficiency program are provided in Chapter 3 of this report while evaluation findings for the demand response program are provided in Chapter 4. Table 2-1 summarizes program-level participation, program contribution to portfolio-level savings, and number measures offered. Table 2-1: Program Level Participation Program % of Portfolio Savings (Ex Ante) Participants Measures Offered High Performance Business 45.76% 1, Home Weatherization 4.15% 2,239 6 Energy Saving Products 35.50% - 1,469,793 High Performance Homes 8.26% 4, Education 6.34% 26,212 7 Behavioral Modification 0.00% 92,939 1 Cumulative EE Totals % 127,387 1,469,838 Business Demand Response 0.00% Cumulative DR Totals 0.00% Cumulative Portfolio Totals % 127,608 1,469, Reduced Emissions and Water Consumption Reduced emissions occur because of energy savings achieved through PSO s Demand Portfolio displacing marginal fossil fuel based electric generation. The EPA s Emissions and Generation Resource Integrated Database (egrid) is a comprehensive source of emissions data related to the electric power sector in the U.S. The technical support Introduction 14

18 document for egrid, based on 2016 data, was released in February of Included in the database are estimates of non-baseload emission rates for various greenhouse gasses in different sub regions of the country. The PSO service territory falls into egrid sub region SPP South (SPSO). Table 2-2 below lists the most recent egrid non-baseload output emission rates for SPSO. Table 2-2: egrid GHG Annual Output Emission Rates egrid Sub region Annual Non-baseload Output Emission Rates Carbon dioxide Methane Nitrous oxide (CO2) (CH4) (N2O) (lb/mwh) (lb/gwh) (lb/gwh) SPP South (SPSO) 1, Using the egrid emission rates and lifetime energy savings for measures installed through the PSO Demand Portfolio in 2017 results in the estimated emissions reductions listed in Table 2-3. Table 2-3: Emission Reduction Estimates Lifetime Energy Savings 13 Carbon dioxide reduction Methane reduction Nitrous oxide reduction (Net at Generator) (CO2) (CH4) (N2O) (MWh) (tonnes) (tonnes) (tonnes) 1,406,447 1,060, Reductions in water consumption at participant homes/facilities resulting from PSO s 2017 portfolio of programs were not tracked. Many of the energy efficiency measures commonly associated with water savings in the residential sector (faucet aerators, lowflow shower heads, efficient clothes washers and dishwashers, etc.) were limited in the portfolio design because of the high prevalence of natural gas water heating in the PSO service territory. The High Performance Business program does offer incentives for measures that have water saving potential for C&I customers (e.g., variable frequency drives on pump motors). The effects on water consumption for these measures were not quantified for PY2017. There are also water savings associated with reduced energy generation attributable to PSO s energy efficiency and demand response programs. PSO s generation fuel mix is made up of coal (~14%), natural gas (~17%), purchased power (~48%) and wind (~21%) 14. Most of the purchased power comes from natural gas plants Lifetime energy savings listed are based on measure lives from the OK Deemed Savings Documents, AR TRM, PA TRM, or IL TRM, annual net energy savings estimated through EM&V of the 2017 portfolio, and a line-loss adjustment factor of Introduction 15

19 All non-wind generation fuel sources are used in thermoelectric power plants which boil water to create steam, which in turn drives turbines. After the steam passes through a turbine, it is cooled so that it condenses, and the water can be reused. The process of cooling the steam accounts for almost all water use in most thermoelectric power plants, as the steam itself circulates in a closed system. A portion of the water used for this cooling process is lost to evaporation. The specifics regarding how much water is consumed in the process depend largely on the technologies used in each power plant (once-through water cooling, recirculating water cooling, dry-cooling). A 2003 report by the National Renewable Energy Laboratory (NREL) 15 provides estimates of water consumption per kwh of energy consumed for all U.S. states. The estimate in Oklahoma is 0.51 Gallons per kwh consumed. Using the NREL water consumption estimates and lifetime energy savings for measures installed through the PSO Demand Portfolio in 2017 results in the lifetime water savings estimates listed in Table 2-4. Lifetime Energy Savings (Net at Generator) (MWh) Table 2-4: Water Savings Estimates, Thermoelectric Generation Overall Generation Percentage Thermoelectric Water Consumption per kwh Consumed (Gallons/MWh) Lifetime Water Savings (Gallons) 1,406,447 78% ,484, Milestones Achieved in Market Transformation Programs While all seven of PSO s energy efficiency programs are designed primarily as energy efficiency resource acquisition programs, there are some market transformation characteristics, briefly summarized below. Energy Saving Products (ESP) Program: The main component of the ESP program in 2017 was retail markdowns of certain LED light bulbs. The goal of the markdowns is to increase sales to customers who would have otherwise purchased less efficient options in the absence of the price discount. These programs have long been considered to have market transformation effects in terms of retailer stocking decisions and manufacturer shipment decisions. During the 2018 program year PSO will collaborate with ADM on a shelving study to quantify the impact of market transformation, specifically for lighting. High Performance Homes New Homes: PSO s New Homes component of the High Performance Homes program, a 2017 ENERGY STAR Partner of the Year Award winning program, continues to expand its influence in the market for new homes. PSO was recognized by ENERGY STAR in 2012, 2013, 2015, 2016, and 2017, receiving Sustained Excellence Awards each year. The program provided educational trainings for both builders and raters that influenced energy efficiency offerings in new homes. 15 Source: Introduction 16

20 Service Provider Recruitment and Training: PSO s High Performance Business and High Performance Homes programs include service provider training opportunities that focus on increasing awareness and knowledge of building science approaches to energy efficiency. This aspect of the programs has potential market transformation effects beyond the energy savings induced through the program. For example, a seminar is in the planning phases for February 2018 to provide energy efficiency training to commercial and industrial customers. The seminar will inform participants on energy trends as well as the latest in energy efficiency technologies. For a complete list of service provider training events refer to Appendix D. Service provider participation continues to grow for the High Performance Business program. 2.3 Annual Utility Growth Metrics and Portfolio Ratios The Oklahoma Title 165: reporting rules provide guidance for providing context on the utility load growth and the Demand Portfolio relative to load and revenue. Table 2-5 shows weather-normalized annual growth rates for PSO s total utility energy sales, distribution, and peak demand as well as four-year compound growth rates. Year Table 2-5: Utility Growth Net Sales (GWh) Sales Growth Energy at Generator (GWh) Energy Growth Peak Demand (MW) Demand Growth , % 19, % 4, % , % 19, % 4, % , % 19, % 4, % Compound Growth Rate 1.09% % % - Table 2-6 shows weather-normalized annual growth rates and four-year compound growth rates for utility energy sales by customer class. Table 2-6: Weather Normalized Retail Meter Sales Residential Commercial Industrial Other Retail Total Retail FERC Total System Year GWh %Chg GWh %Chg GWh %Chg GWh %Chg GWh %Chg GWh %Chg GWh %Chg , % 5, % 5, % 1, % 18, % % 18, % , % 5, % 5, % 1, % 18, % % 18, % , % 5, % 5, % 1, % 18, % % 18, % Compound Growth Rate % % % % % % - Introduction 17

21 Table 2-7 provides a comparison of program costs to operating revenue. Table 2-7: Demand Portfolio Funding 2017 Demand Portfolio Program Cost ($M) $ Operating Revenues ($M) $1,444 Program Cost as % of Utility Operating Revenue 2.2% Table 2-8 provides a comparison of energy savings to total utility energy sales. Table 2-8: Demand Portfolio Energy Savings 2017 Demand Portfolio Energy Savings (GWh) Metered Energy Sales (GWh) 19,053 Savings as % of Utility Sales 0.7% 2.4 High-Volume Electricity User Opt Out The Oklahoma Title 165: rules allow for High-Volume Electricity Users to opt out of some or all energy efficiency or demand response programs by submitting a notice of such decision to the director of the Public Utility Division and to the electric utility. A High- Volume Electricity User is defined as any single customer that consumes more than 15 million kwh of electricity per year, regardless of the number of meters or service locations. Table 2-9 provides a summary of high volume customers who opted out of energy efficiency programs. Table 2-9: High Volume Electricity User Opt Out Energy Efficiency Metric Opt-out eligible 2017 Chose to opt-out -EE Number of accounts 4,564 4, Electric Sales (GWh) 6,601 6,300 Aggregate load as a percentage of total sales 34.6% 33.1% Introduction 18

22 Table 2-10 provides a summary of high volume customers who opted out of demand response programs. Table 2-10: High Volume Electricity User Opt Out Demand Response Metric Opt-out eligible 2017 Chose to opt-out -DR Number of accounts 4,564 3, Electric Sales (GWh) 6,601 6,067 Aggregate load as a percentage of total sales 34.6% 31.8% 2.5 Fuel Switching Impacts PSO did not provide incentives for installation of electric heating or electric water heating to replace natural gas fueled equipment during PY2017. A review of the program tracking data found three instances in which natural gas equipment were replaced with electric equipment that was rebated through a PSO program. Two projects in the High Performance Homes Single Upgrades Program went from gas to heat pumps and one project in the High Performance Homes Multiple Upgrades Program went from 80 AFUE to a mini-split system. 2.6 Program Implementation & Strategic Alliances PSO had seven full-time employees dedicated to the implementation of energy efficiency and demand response programs in Additionally, PSO entered contracts with several energy services companies (ESCOs) and contractors to aid in program implementation. A complete list of implementation contractors, including contact name, title, business address, phone number, address, and program associations, is provided in Appendix C. ICF International (ICF) was contracted to implement the High Performance Business and High Performance Homes programs. CLEAResult was contracted to implement the Energy Saving Products program. The Home Weatherization program was largely implemented by Titan ES, LLC, with some program participation also coming through Rebuilding Together Tulsa, a volunteer organization working to preserve and revitalize low-income homes and communities. PSO contracted with Resource Action Programs to provide energy efficiency kits distributed through the Education program. Finally, the Business Demand Response program was implemented in-house by PSO, with database support provided by AEG. Additional customer engagement materials and services for the entire portfolio of programs were provided by VI Marketing and Branding. Examples of marketing materials used during 2017 to promote PSO s energy efficiency and demand response programs are provided in Appendix E. Introduction 19

23 For most programs in the 2017 portfolio, service providers were recruited to participate by submitting rebate applications on behalf of customers implementing qualifying energy efficiency measures. PSO s website contains lists of registered service providers and the associated products/services they provide. 2.7 Training and Customer Outreach PSO regularly conducts various service provider training and customer outreach events, which are summarized in Appendix D. During 2017, PSO s energy efficiency and demand response programs sponsored: 76 in-store residential lighting promotional events 47 other customer outreach and service provider training events, including: Portfolio overview presentations Program specific service provider training One-on-one presentations with potential participants Trade show and event booths promoting the portfolio Introduction 20

24 3 Energy Efficiency Programs PSO s energy efficiency portfolio in 2017 consisted of seven programs: two commercial/industrial and five residential (a small component of the Education program distributed kits to small businesses). Program-level energy savings is summarized in Table 3-1. Table 3-1: Annual Energy Savings Energy Efficiency Programs Program Gross Peak Annual Energy Savings (MWh) Projected Reported Verified Energy Efficiency Programs Gross Realization Rate Net Impacts NTG Ratio Net Annual Energy Savings (MWh) High Performance Business 52,644 57,385 58, % ,642 Home Weatherization 3,865 5,204 4,930 95% ,930 Energy Saving Products 31,233 44,517 51, % ,569 High Performance Homes 7,493 10,355 7,832 76% ,800 Education 4,949 7,949 7,906 99% ,178 Behavioral Modification Energy Efficiency Totals 100, , , % ,119 Program-level peak demand reduction is summarized in Table 3-2. Table 3-2: Peak Demand Reduction Energy Efficiency Programs Gross Peak Demand Reduction (MW) Net Impacts Net Peak Gross Program NTG Demand Projected Reported Verified Realization Ratio Reduction Rate (MW) Energy Efficiency Programs High Performance Business % Home Weatherization % Energy Saving Products % High Performance Homes % Education % Behavioral Modification Energy Efficiency Totals % Demand Response Programs 21

25 The remainder of this section provides evaluation findings for each of the PY2017 PSO energy efficiency programs including program performance metrics, evaluation methodologies, energy and demand impacts, and process evaluation findings. 3.1 High Performance Business Program (HPB) Program Overview PSO s High Performance Business (HPB) program provided rebates for a total of 1,437 projects completed by 1,210 unique participant premises in The program seeks to generate energy and demand savings for small and large commercial and industrial customers, schools, and municipalities by incentivizing high efficiency electric end-use products including, but not limited to, lighting, HVAC, and Variable Frequency Drives (VFDs) for motors. Starting in 2016, the program was expanded to include a Small Business Energy Solutions (SBES) rebate program. To participate in the SBES program, businesses must use 220,000 kwh or less annually and use a PSO approved service provider. PSO s customers can participate by self-sponsoring or working through a program service provider to leverage technical expertise. Reported expenditures for the HPB program in PY2017 was less than budgeted (97% of budget). Table 3-3 summarizes projected, ex ante, and ex post demand impacts as well as other program performance metrics. Energy Efficiency Programs 22

26 Table 3-3: Performance Metrics High-Performance Business Program Metric PY2017 Number of Customers 1,210 Budgeted Expenditures $10,549,799 Actual Expenditures $10,176,406 Energy Impacts (kwh) Projected Energy Savings 52,644,002 Ex Ante Energy Savings 57,385,066 Gross Ex Post Energy Savings 58,211,117 Net Ex Post Energy Savings 52,642,377 Peak Demand Impacts (kw) Projected Peak Demand Savings 8,752 Ex Ante Peak Demand Savings 10,126 Gross Ex Post Peak Demand Savings 10,314 Net Ex Post Peak Demand Savings 9,233 Benefit / Cost Ratios Total Resource Cost Test Ratio 2.14 Utility Cost Test Ratio 4.24 Table 3-4 summarizes the achieved sample size for the various data collection activities for the HPB program evaluation. Table 3-4: Sample Sizes for Data Collection Efforts High Performance Business Data Collection Activity Achieved Sample Size Custom/Prescriptive SBES On-Site M&V visits Customer Decision Maker Survey Service Provider Survey 4 6 In-depth Interviews with Program Staff - 2 Energy Efficiency Programs 23

27 3.1.2 High Performance Business Program (HPB) - Custom and Prescriptive For custom and prescriptive projects, ADM found a 101% realization rate for gross energy savings and a 98% realization rate for gross peak demand reduction. Impact Evaluation Overview PSO s HPB prescriptive and custom projects provided rebates for a total of 1,064 projects completed by 864 unique premises in Three projects did not have energy savings. Lighting system retrofit projects continued to be the main source of program savings (66% of ex ante annual energy savings). Additionally, savings attributable to all lighting projects (new construction lighting, custom lighting, and lighting system retrofit) contribute 81% of ex ante kwh savings for prescriptive and custom projects. Custom non-lighting applications account for 16% of ex ante savings, while the remaining 3% of ex ante savings are attributable to prescriptive HVAC measures. Overall, the number of rebated projects in HPB increased from 1002 in PY2016 to 1064 in PY2017. Ex ante energy savings increased from 47,878 MWh to 49,092 MWh. Ex post gross energy savings likewise increased from 48,506 MWh (PY2016) to 49,689 MWh (PY2017). The gross energy savings realization rate for PY2017 was 101%, while the gross peak demand reduction realization rate was 98%. The estimated annual energy savings net-to-gross (NTG) ratio changed from 95% in PY2016 to 89% in PY2017. The estimated peak demand NTG ratio changed from 95% in PY2016 to 87% for PY2017. Table 3-5 provides a summary Custom and Prescriptive project savings in the HPB program. Energy Efficiency Programs 24

28 Table 3-5: Performance Metrics High Performance Business Program Prescriptive and Custom Metric PY2017 Number of Customers 864 Energy Impacts (kwh) Ex Ante Energy Savings 49,092,291 Gross Ex Post Energy Savings 49,357,803 Net Ex Post Energy Savings 44,061,144 Peak Demand Impacts (kw) Ex Ante Peak Demand Savings 7,966 Gross Ex Post Peak Demand Savings 7,803 Net Ex Post Peak Demand Savings 6,796 Benefit / Cost Ratios Process Evaluation Overview Total Resource Cost Test Ratio 2.21 Utility Cost Test Ratio 3.99 The PY2017 process evaluation consisted of participant surveys and program staff interviews. The objective of the participant survey was to assess the source of program awareness, factors that influenced project decision making, experience with the application process or energy consultant, and program satisfaction. A total of 68 customer decision makers responded to the participant survey. Participation in the HPB program was somewhat consistent throughout the year. Figure 3-1 displays the accrual of ex ante energy savings throughout Energy Efficiency Programs 25

29 Figure 3-1: Accrual of Ex Ante kwh Savings during the Program Year Table 3-6 summarizes the program savings by measure type as listed in the data provided by the implementation contractor. Lighting (Retrofit Lighting, New Construction Lighting, and Lighting) accounts for approximately 81% of savings. Energy Efficiency Programs 26

30 Table 3-6: Summary of Program Activity by Measure Type Measure Type Incentive Number Percent of Total Ex Ante kwh Dollars of Ex Ante Rebate Savings per kwh Projects Savings Dollars Saved Retrofit Lighting ,981,549 63% $1,746,535 $0.06 New Construction Lighting 92 7,399,755 15% $601,707 $0.08 Expansion 7 1,627,398 3% $172,194 $0.11 Custom 7 1,556,152 3% $114,847 $0.07 Lighting 81 1,293,997 3% $57,525 $0.04 New Construction 6 1,290,632 3% $139,357 $0.11 Unitary HVAC & VFDs 39 1,153,071 2% $315,168 $0.27 Refrigeration & Kitchen Equipment ,957 2% $74,046 $0.10 Equipment Replacement ,135 2% $50,198 $0.07 Plug Load and Controls 2 755,880 2% $39,065 $0.05 Unitary HVAC & VFDs, Refrigeration & Kitchen 2 486,785 1% $92,048 $0.19 Equipment Other 3 432,245 1% $28,319 $0.07 Unitary HVAC ,637 1% $223,650 $0.81 High Efficiency Equipment and Controls ,792 <1% $10,788 $0.07 HVAC VFD 2 35,956 <1% $1,125 $0.03 Renovation Project 2 33,313 <1% $4,326 $0.13 Building Envelope 2 31,460 <1% $2,236 $0.07 Unitary HVAC & VFDs, New Construction Lighting 1 19,163 <1% $6,500 $0.34 Lighting, Custom 1 14,415 <1% $1,195 $0.08 Total 1,064 49,092, % $3,680,828 $0.07 Table 3-7 summarizes the share of ex ante savings by district. Table 3-7: District Share of Reported kwh Savings Region Ex Ante kwh Percent of Ex Ante Savings Savings Eastern District 8,340,793 17% Tulsa District 30,872,946 63% Tulsa Northern District 3,980,622 8% Western District 5,897,930 12% Energy Efficiency Programs 27

31 Figure 3-2 provides a heat map of the number of projects located in a specific area of the state grouped by zip code. Description of Program Figure 3-2: Distribution of Custom and Prescriptive Projects PSO s High Performance Business program seeks to generate energy savings for custom and prescriptive projects by promoting high efficiency electric end-use products including, but not limited to, lighting, HVAC, process improvements, and variable frequency drives (VFD s). The program allows PSO s customers to participate by either self-sponsoring or by working through a third-party service provider to leverage technical expertise. The program seeks to combine the distribution of financial incentives with access to technical expertise to maximize program penetration across the range of potential C&I customers. Additionally, the program aims to accomplish the following: Increase customer awareness and knowledge of applicable energy saving measures and their benefits, Increase the market share of commercial grade high efficiency technologies sold through market channels, And increase the installation rate of high efficiency technologies in C&I facilities by businesses that would not have done so in absence of the program. Prescriptive rebate amounts are provided to participating customers for some measures including certain types of lighting, lighting controls, hotel & kitchen equipment, and HVAC equipment. Custom projects that do not fall into prescriptive measure categories are Energy Efficiency Programs 28

32 rebated on a per kwh and kw impact basis. To assist customers with identifying energysaving capital improvements, the program offers capital improvement consultation. Methodology This chapter provides a brief overview of the data collection activities, gross and net impact calculation methodologies, and process evaluation activities that ADM employed in the evaluation of the HPB program. Data Collection Data for analysis was collected through review of program materials, on-site inspections, end-use metering, and interviews with participating customers and service providers. PSO uses Sightline in conjunction with an SQL Server Reporting Services (SSRS) system as their central tracking and reporting system. Based on program tracking data provided by PSO through SSRS, a sample was developed for on-site data collection. On-site visits were used to collect data for gross impact calculations, verify measure installation, and determine measure operating parameters. Facility staff members were interviewed to determine the operating hours of the installed systems and provide any additional operational characteristics relevant to calculating energy savings. For a subset of sampled projects, lighting equipment, HVAC equipment, or motors/variable Frequency Drives (VFDs) were monitored to obtain accurate operational profiles. Appendix G: M&V Safety provides a description of ADM s safety standards and procedures. In addition to on-site data collection, customer surveys provided self-reported data for the NTG analysis and process evaluation. A total of 68 customer decision makers completed the survey. In-depth interviews with four PSO and implementation staff members were conducted to provide additional perspectives for process evaluation. Table 3-8 shows the achieved sample sizes for the different types of data collection utilized for this study. Table 3-8: Sample Sizes for Data Collection Efforts Data Collection Activity Achieved Sample Size On-Site M&V visits 73 Customer Decision Maker Survey 68 In-depth Interviews with Program Staff 4 Energy Efficiency Programs 29

33 Sampling Plan ADM created a stratified sample based on the amount of energy savings and type of measure installed in a given project. For this approach, ADM utilized algorithms in an R programming language package based on the Bethel-Chromy algorithm. 16 Realization rates (the ratio of ex post to ex ante savings) for projects sampled in each stratum are only extrapolated to other projects within that stratum. Verification of sample precision, by means of each stratum s contribution to variance, is then performed on the ex post extrapolated annual energy savings (kwh) for the program. Sample sizes were designed to meet 10% precision at the 90% confidence level at the program level. Separate samples were drawn for custom and prescriptive projects and SBES projects. Table 3-9 shows the sample design that was used for custom and prescriptive projects. Stratum classifications were based on verified measure installations. The 73 projects that were sampled for on-site measurement and verification account for approximately 29% of ex ante program kwh savings Energy Efficiency Programs 30

34 Table 3-9: Sample Design for the High-Performance Business Program Prescriptive and Custom Stratum Name Ex Ante kwh Savings Strata Boundaries (kwh) Population of Projects Design Sample Size Custom 1 641,116 <33, Custom 2 609,509 33,452-96, Custom 3 2,149,811 96, , Custom 4 958, , , Custom 5 3,576,649 > 350, HVAC 1 317,930 < 16, HVAC 2 217,477 16,795-59, HVAC 3 331,420 59, , HVAC 4 599,836 > 120, NC Lighting 1 1,138,005 < 53, NC Lighting 2 1,869,717 53, , NC Lighting 3 2,676, , , NC Lighting 4 1,715,745 > 324, Retrofit Lighting 1 7,798,483 < 42, Retrofit Lighting 2 10,561,318 42, , Retrofit Lighting 3 5,555, , , Retrofit Lighting 4 4,605, , , Retrofit Lighting 5 3,768,669 > 987, Total 49,092,291-1, Energy Efficiency Programs 31

35 Impact Evaluation Methodology The evaluation of gross energy savings and peak demand reduction from projects rebated through the HPB program can be broken down into the following steps: The program tracking database was reviewed to determine the scope of the program and to ensure there were no duplicate project entries. The tracking database was used to define a discrete set of rebated projects that made up the PY2017 program population. A sample of projects was then drawn from the population established in the tracking system review. A detailed desk review was conducted for each project sampled for on-site verification and data collection. The desk review process includes a thorough examination of all project materials including invoices, equipment cut sheets, preand post-inspection reports, and estimated savings calculators. This review process informs ADM s fieldwork by identifying potential uncertainties, missing data, and sites where monitoring equipment is needed to verify key inputs to the ex ante savings calculations. After reviewing project materials, on-site verification and data collection visits are scheduled for each sampled project. The visits are used to collect data for savings calculations, verify measure installation, and determine measure operating parameters. The data collected during the on-site verification visits is used to revise savings calculations as necessary. For example, if the ex ante savings calculations relied on operating hours for a given measured that were found to be inaccurate based on the on-site verification and data collection, changes are made to more accurately reflect actual operating conditions. After determining the ex post savings impacts for each sampled project, results are extrapolated to the program population using project-specific sampling weights. This allows for the estimation of program level gross ex post energy (kwh) savings with a given amount of sampling precision and confidence. Net-to-Gross Estimation Net savings analysis determines what portion of gross savings is the direct result of program influence. Net savings may be less than gross savings because of free ridership impacts. Free-riders for a program are defined as those participants that would have installed the same energy efficiency measures without the program. Conversely, net savings may be greater than gross savings due to energy saving spillovers attributable to 17 Three projects with no savings reported were excluded from the sampling plan. Energy Efficiency Programs 32

36 the program. Participants or non-participants may implement additional energy efficiency measures due to the influence of the program without receiving program incentives for implemented measures. Information collected from a sample of program participants through a customer decision maker survey is used for NTG analysis. These responses are reviewed to assess the likelihood that participants are free riders. Several criteria are used for determining what portion of a customer s savings for a project should be attributed to free ridership. The criteria are organized into the following three factors: Plans and intentions of the premise to install a measure without support from the program; Influence that the program had on the decision to install a measure; and A premise s previous experience with a measure installed under the program. For each of these factors, rules are applied to develop binary variables indicating whether a participant shows free ridership behavior. The first required step is to determine if a participant stated that his or her intention was to install an energy efficiency measure without the help of the program inducements. Two binary variables were constructed to account for customer plans and intentions: one, based on a more restrictive set of criteria that may describe a high likelihood of free ridership, and a second, based on a less restrictive set of criteria that may describe a relatively lower likelihood of free ridership. The first, more restrictive criteria indicating customer plans and intentions that likely signify free ridership are as follows: The respondent answers, yes to the following two questions: Did you have plans to install the [Equipment/Measure] before participating in the program? and Would you have gone ahead with this planned installation of the measure even if you had not participated in the program? The respondent answers, definitely would have installed to the following question: If the rebates from the program had not been available, how likely is it that you would have installed [Equipment/Measure] anyway? The respondent answers, did not affect the timing of purchase and installation to the following questions: We would like to know whether the availability of information and rebates through the program affected the timing of your purchase and installation of [Equipment/Measure]. Did you purchase and install [Equipment/Measure] earlier than you otherwise would have without the program? The respondent answers, no, the program did not affect the level of efficiency that we chose for equipment in response to the following question: Did you choose Energy Efficiency Programs 33

37 equipment that was more energy efficient than you would have chosen had you not participated in the program? The respondent answers, no, the program did not affect timing of project in response to the following question: Did you install the [Equipment/Measure] earlier than you otherwise would have because of the information and rebates from PSO s program? The second, less restrictive criteria indicating customer plans and intentions that likely signify free ridership are as follows: The respondent answers yes to the following two questions: Did you have plans to install the [Equipment/Measure] before participating in the program? and Would you have gone ahead with this planned installation of the measure even if you had not participated in the program? Either the respondent answers, would have installed or probably would have installed to the following question: If the rebates from the program had not been available, how likely is it that you would have installed [Equipment/Measure] anyway? Either the respondent answers, did not affect timing of purchase and installation to the following question: Did you purchase and install [Equipment/Measure] earlier than you otherwise would have without the program? or the respondent indicated that while program information and rebates did affect the timing of equipment purchase and installation, in the absence of the program they would have purchased and installed the equipment within the next two years. The respondent indicated that no, the program did not affect level of efficiency that we chose for equipment. The third factor is determining if a customer reported that a recommendation from a program representative, or experience with the program, was influential in the decision to install a piece of equipment or measure. This criterion indicates that the program s influence may lower the likelihood of free ridership when either of the following conditions is true: The respondent answers, very important to the following question: How important was previous experience with PSO energy efficiency programs in making your decision to install the [Equipment/Measure] at your facility? The fourth factor is determining if a participant in the program indicated that he or she has previously installed an energy efficiency measure like one that they installed under the program without an energy efficiency program incentive during the last three years. A participant indicating that he or she had installed a similar measure is considered to have a higher likelihood of free ridership, due to the potential influence of the prior experience. Energy Efficiency Programs 34

38 The criteria indicating that previous experience may signify a higher likelihood of free ridership are as follows: The respondent answers yes to the following question: Before participating in the program, had you installed any equipment or measure like [Equipment/Measure] at your facility? The respondent answers yes, purchased energy efficient equipment but did not apply for financial incentive. to the following question: Has your organization purchased any energy efficient equipment in the last three years for which you did not apply for a rebate through the program? The four sets of rules just described are used to construct four different indicator variables that address free ridership behavior. For each customer, a free ridership value is assigned based on the combination of variables. With the four indicator variables, there are 12 applicable combinations for assigning free ridership scores for each respondent, depending on the combination of answers to the questions creating the indicator variables. Table 3-13 shows these values. NTG ratios are determined on a program level basis, for both kwh and kw, so the ratio is applied to the final program level gross ex post savings values. The ratio may differ for kwh and kw depending on customers measure level energy savings and peak demand reduction. The customer decision maker survey also includes a series of questions used to analyze whether there are potential spillover effects associated with non-rebated purchases by program participants. 18 Specifically, survey respondents are asked: We would like to know if you have installed any additional energy efficient equipment because of your experience with the program that you DID NOT receive an incentive for. Since participating in the program, has your organization installed any ADDITIONAL energy efficiency measures at this facility or at your other facilities within PSO's service territory that did NOT receive incentives through PSO's program? Customers who indicate yes are identified as potential spillover candidates. Potential spillover candidates are additionally asked to identify the type of additional equipment installed and provide information about the equipment for use in estimating energy savings. For each type of equipment that respondents report installing, respondents are asked the following two questions, which are used to assess if any savings resulting from the additional equipment installed are attributable to the program. 18 The spillover analysis is limited to participant spillover. Non-participant spillover effects may exist for the program, but they are not estimated and therefore assumed to be zero. Energy Efficiency Programs 35

39 [SP1] How important was your experience with the program in your decision to install this [Equipment/Measure]? [Rated on a scale where 0 meant not at all important and 10 meant very important] [SP2] If you had NOT participated in the program, how likely is it that your organization would still have installed this [Equipment/Measure]? [Rated on a scale where 0 meant not at all likely and 10 meant very likely] A spillover score was developed based on these responses as follows: Spillover Score = Average (SP1, 10-SP2) The energy savings of equipment installations associated with a spillover score of greater than five are attributed to the program. Process Evaluation The process evaluation is designed to research and document the program delivery mechanisms and collective experiences of program participants, partners and staff. The process evaluation is designed to answer the following research questions: What changes, if any, were made to the program design or implementation procedures? How did new participants learn of the program? What factors motivated their decision to participate? Were program participants satisfied with their experience? What were the key successes and challenges during PY2017? To address these questions, ADM s process evaluation activities include surveys to program participants and in-depth interviews with program staff. Table 3-10 provides a summary of data collection activities for the process evaluation. Table 3-10: HPB Custom and Prescriptive Program Research Questions Data Collection Activity Program Staff Interviews Process Evaluation Research Objectives Assess program staff perspectives regarding program operations, strengths, or barriers to success. Participant Surveys Assess source of program awareness, factors that influenced project decision making, experience with the application process or energy consultant, and program satisfaction. Energy Efficiency Programs 36

40 Impact Evaluation Findings The ex post gross kwh savings for the PY2017 HPB Prescriptive and Custom projects are summarized, by sampling stratum, in Table Table 3-11: Ex Ante and Ex Post Gross kwh Savings by Sampling Stratum Prescriptive and Custom Stratum Ex Ante kwh Savings Ex Post Gross kwh Savings Gross kwh Realization Rate Custom 1 641,116 1,013, % Custom 2 609, ,747 92% Custom 3 2,149,811 2,263, % Custom 4 958, ,114 74% Custom 5 3,576,649 3,164,300 88% HVAC 1 317, , % HVAC 2 217, , % HVAC 3 331, ,516 67% HVAC 4 599, ,486 38% NC Lighting 1 1,138,005 1,442, % NC Lighting 2 1,869,717 2,456, % NC Lighting 3 2,676,288 3,293, % NC Lighting 4 1,715,745 1,824, % Retrofit Lighting 1 7,798,483 9,009, % Retrofit Lighting 2 10,561,318 9,722,365 92% Retrofit Lighting 3 5,555,768 5,087,731 92% Retrofit Lighting 4 4,605,723 4,140,637 90% Retrofit Lighting 5 3,768,669 3,269,302 87% Total 49,092,291 49,357, % Energy Efficiency Programs 37

41 The achieved sample design results in ex post gross kwh estimates with ±9.1% relative precision at the 90% confidence interval. 19 Overall, ex post gross energy savings were relatively close to the original ex ante values at the program level (101% realization rate). There was, however, a wide range of kwh realization rates at the sample project level. The following sections discuss specific measure types from the PY2017 sample. Lighting Projects Lighting projects were included in two strata categories; retrofit (Retrofit Lighting 1-5), and new construction (NC Light1-4) lighting. The five retrofit lighting strata had a combined realization rate of 89%, with strata realization rates varying from 87% to 116%. The combined new construction strata had a realization rate of 116%, with strata realization rates varying from 106% to 131%. Retrofit Lighting Projects Differences between ex ante and ex post energy savings can be explained by differences in reported and verified hours of use (HOU). ADM used lighting schedules from detailed interviews with facility staff as well as deemed hours of use when applicable. Lighting settings from Energy Management Systems (EMS), timers, and photocells were used, where appropriate, based on fieldwork findings. When an accurate HOU was not available, or the HOU varied, deemed values from the Arkansas TRM v6.1 were used. For PY2017, ADM used deemed HOU for school applications in which monitoring was not possible over the course of all operating conditions; summer, breaks, holidays, etc. In addition to differences in the HOU, differences in savings was attributable to: Baseline wattage differences, Savings differ in provided documentation compared to SSRS report, No adjustment for EISA 2007 for baseline conditions, Additional or different controls found on-site, And heating and cooling interactive effects. During PY2017, the implementation contractor added notes for some sites regarding HOU. ADM found these notes helpful and would encourage further detail provided to compare evaluation findings. ADM has compiled a historical list of projects to review the comparison of HOU by facility type. Findings indicate that some facility types have greater variability in the total annual HOU than others. These include offices, public assembly, religious facilities, and schools. It would be beneficial if more detail could be provided in the project notes when using interview-based hours of use for these facility types. 19 That is, we are 90% confident that the true verified gross savings are between 53,837,450 and 44,878,156 kwh based on the uncertainty introduced by sampling. Energy Efficiency Programs 38

42 New Construction Lighting Projects Energy savings analyses for new construction lighting projects require a lighting power density (LPD) approach to determine the proper baseline condition. The allowable LPD baseline condition is based on allowable building codes. Building codes for Oklahoma date back to IECC ADM found higher energy savings in all but one new construction lighting project. The difference in savings can be attributed to the following reasons: A difference in verified square footage, A difference in annual hours of use, Incorrect space type LPD, And a difference in fixture quantities. Some projects had lighting fixtures from different area types combined into a single line item in the ex ante calculator. For example, exterior fixtures were sometimes included with interior fixtures. This led to an incorrect LPD being applied. HVAC Projects Heating, Ventilation, and Air Conditioning (HVAC) projects represented the strata with the largest variability in the sample. Energy savings for HVAC measures can be highly variable due to differences in equipment efficiency, capacity, etc. The most common measure that fell into the ADM sample is rooftop AC units. However, some of the other measures include split system AC, economizers, dual enthalpy controls (for economizers), and VFDs. For most projects installed prior to June 30 th, 2017, discrepancies between ex ante and ex post savings could be primarily attributed to the implementation team and evaluation team using differing EFLH values. From June 30 th onward, the EFLH used to calculate savings was consistent between both the implementation team and evaluation team. Despite this realignment, ex ante and ex post savings continued to deviate for the remainder of the program year. Savings calculations were not retained as part of the project documentation. Therefore, ADM was not able to pinpoint whether or not the discrepancy was attributable to a difference in calculation inputs. Custom & Equipment Projects The variance in realization rates for custom and equipment projects (noted as C in Table 3-9 and Table 3-11) vary by measure and savings algorithm implemented. Custom projects in the ADM PY2017 sample included HVAC systems and controls, lighting, air compressors, kitchen equipment, and plug load equipment. These measure types were grouped together in the sample due to the nature of the measure, the number of projects, and the annual energy savings (kwh). Projects were also placed into this category that had multiple measures. Energy Efficiency Programs 39

43 The reasoning for the various realization rates include: A difference in savings algorithms. For several projects, ex ante savings are based on manufacturers calculation tools which are often proprietary and not available. In these instances, ADM used custom calculations, equest, or deemed values. The deemed value instances included HVAC measures and air dryers. The air dryer ex post kwh was based on the Massachusetts TRM as the measure is not included in the Arkansas TRM. Collected power consumption data was not available for the ex ante savings but was available for the ex post savings calculation. This was the result of power monitoring by ADM or provided trend data from the customer. A difference in quantity or specification of the installed new construction equipment. For all custom and equipment strata in the ADM sample, the gross energy realization rate was 91%. As custom projects often require the acquisition of power monitoring or trend data, for the ex post analysis, this realization rate is reasonable. Gross Coincident Peak Demand Reduction (kw) The ex post gross kw reduction for the PY2017 High Performance Business program is summarized by sampling stratum in Table The peak demand reduction realization rate for prescriptive and custom projects is 98%, suggesting good consistency between ex ante and ex post savings. Energy Efficiency Programs 40

44 Table 3-12: Ex Ante and Ex Post Gross kw Reduction by Sampling Stratum Stratum Ex Ante Peak kw Reduction Ex Post Gross Peak kw Reduction Ex Post Gross kw Realization Rate Custom % Custom % Custom % Custom % Custom 5 1, % HVAC % HVAC % HVAC % HVAC % NC Lighting % NC Lighting % NC Lighting % NC Lighting % Retrofit Lighting 1 1,347 1,305 97% Retrofit Lighting 2 1,597 1, % Retrofit Lighting % Retrofit Lighting % Retrofit Lighting % Total 7,966 7,803 98% The achieved sample design resulted in ex post gross kw estimates with ±18.2% relative precision at the 90% confidence interval. 20 Peak kw reduction was highly variable from project to project, resulting in a relative precision higher than 10%. Despite this, the Energy Efficiency Programs 41

45 relative precision of the sample in PY2017 is roughly the same as PY2016, the realization rate is close to 100%, and the confidence interval remains significantly different than 0. Differences between ex ante and ex post demand reduction could be attributed to: 1) calculation error in ex ante demand reduction, 2) use of stipulated coincidence factors (CF) that did not align well with actual equipment schedules, 3) varying peak demand reduction from the defined peak period, or 4) differences in the definition of peak demand savings. 21,22 For lighting projects, the ADM ex post lighting calculators generate an hourly curve (8760 hours) for 2017 to determine the average kw value across the peak demand period for each lighting schedule. Custom calculations and energy simulations provide similar results. For other prescriptive measures, the ADM calculators used the deemed coincidence factors provided in the AR TRM v6.1. Net-to-Gross Estimation The data used to assign free ridership scores were collected through a survey of customer decision makers for projects rebated through the HPB program during PY2017. Free ridership was estimated using the methodology described in Custom and Prescriptive Net-to-Gross Estimation section. Table 3-13 shows percentages of total gross ex post savings associated with different combinations of free ridership indicator variable values for the custom and prescriptive incentive component. 20 That is, we are 90% confident that the ex post gross peak demand reduction is between 6,384 and 9,222 kw based on the uncertainty introduced by sampling. 21 ADM noted that the implementation team defined peak demand savings as the maximum kw savings across the peak demand period, while ADM defined the peak demand savings as the average kw savings across the peak demand period (as recommended in the UMP). Despite this difference, the peak demand reduction realization rate was still close to 100%. 22 Uniform Methods Project: Methods for Determining Energy Efficiency Savings for Specific Measures. National Renewable Energy Laboratory (NREL), Energy Efficiency Programs 42

46 Table 3-13: Estimated Free-ridership for PY2017 HPB Program kwh Custom and Prescriptive Had Plans and Intentions to Install Measure without C&I Program? (Definition 1) Had Plans and Intentions to Install Measure without C&I Program? (Definition 2) C&I Program had influence on Decision to Install Measure? Had Previous Experience with Measure? Free Ridership Score Percentage of Total Gross kwh Savings Y Y Y Y 100% 1% Y Y N N 100% 7% Y Y N Y 100% 0% Y Y Y N 67% 2% N Y N Y 67% 0% N Y N N 33% 1% N Y Y N 0% 0% N Y Y Y 33% 2% N N N Y 33% 1% N N N N 0% 26% N N Y N 0% 25% N N Y Y 0% 5% Required program to implement measures 0% 31% Total 10.8% 100.0% Overall, the estimated percentage of program free ridership is 10.8%. This level of free ridership is higher than in 2016 (6%). This increase is driven by participants who claimed that they had to plans to install the measures without the HPB program and that the program did not have an influence on which energy efficient equipment they installed. Customer decision maker survey responses were also analyzed to estimate participant spillover effects. Only one of the respondents reported installing efficient equipment that met the attribution criterion and for which energy savings could be estimated. The final net-to-gross ratio (NTGR) for the program is calculated as 1 free-ridership + participant spillover. This results in a NTGR of 89.3% for kwh savings and 87.1% for peak demand reductions. Table 3-14 shows the amount of savings and peak demand reduction impacted by free ridership and spillover. These values were calculated based on the percent of free-ridership and spillover found in the sample of survey respondents. Energy Efficiency Programs 43

47 Table 3-14: Free-Ridership and Spillover for Custom and Prescriptive Savings Free Ridership Spillover kwh 5,346,622 49,963 kw 1, The gross and net ex post energy savings and peak demand reduction of the HPB program during PY2017 are summarized by program in Table Table 3-15: Summary of Ex Post Gross and Net Impacts Program Ex Post Gross kwh Savings Ex Post Net kwh Savings Net-to-Gross Ratio Ex Post Gross kw Reduction Ex Post Net kw Reduction High Performance Business - Custom and Prescriptive 49,357,803 44,061, % - kwh 87.1% - kw 7,803 6,796 Process Evaluation Findings A total of 68 customer decision makers completed a survey about their experience with the High-Performance Business Program in The building types associated with the responses are shown in Table 3-16 are based on the building type listed in the program tracking data. The common building type identifier among survey respondents was other (18%). Other common building types were schools (15%), retail (13%), and industrial buildings (12%). Energy Efficiency Programs 44

48 Table 3-16: Survey Respondent Organization Types Response Percent (n = 68) Other 18% School 15% Retail 13% Industrial 12% Office 10% Religious Facility 10% Warehouse 10% Lodging 4% Education: College and Vocational 1% Grocery 1% Health Facility 1% Manufacturing 1% University 1% Overall Satisfaction Eighty-eight percent of participants were satisfied or very satisfied with the program overall, while 3% were neither satisfied nor dis-satisfied as shown in Figure 3-3. Additionally, most respondents indicated satisfaction with the various elements of the program. No single aspect of the program experience stood out as something participants were particularly satisfied or dissatisfied with. Energy Efficiency Programs 45

49 Figure 3-3: Program Satisfaction Respondents were given an opportunity to provide additional comments about their experience with the HPB program. Most comments made by the respondents indicated satisfaction with the program. Some examples of participant comment responses are: Continuation of this program would be a great benefit to my organization. We have been very happy with PSO and the incentive program Nothing short of a great program. Extremely helpful staff. They really make the experience smooth. Probably would have not spent the time to participate in the program if it wasn't for the help with the paperwork from ICF. They were amazing. The PSO representative was very kind and helpful - checking with me regularly on my project High Performance Business Program (HPB) - SBES This section reports the findings from the Small Business Energy Solutions (SBES) projects within the HPB program. ADM s verified ex post annual energy savings estimates for SBES resulted in a 107% realization rate for gross energy savings and a 116% realization rate for gross peak demand reduction. Energy Efficiency Programs 46

50 Impact Evaluation Overview PSO s HPB SBES projects provided rebates for a total of 372 projects completed by 346 unique premises in Lighting system retrofit applications continued to be the main source of program impacts, representing 93% of ex ante annual energy (kwh) savings. Refrigeration contractor applications accounted for 7% of reported savings. Overall, the number of rebated projects in HPB SBES increased from 186 in PY2016 to 372 in PY2017. The ex ante energy savings increased from 4,054 MWh (PY2016) to 8,293 MWh (PY2017). The ex post gross energy savings likewise increased from 4,423 MWh (PY2016) to 8,853 MWh (PY2017). The program reported coincident peak demand reductions of 2,160 kw while verified savings were 2,511 kw, an increase from PY2016 (1,091 kw and 1,074 kw, respectively). The evaluation resulted in a gross energy savings realization rate of 107% for PY2017 and a gross coincident peak demand reduction realization rate of 116%. The estimated annual energy savings NTG ratio changed from 99% in PY2016 to 97% in PY2017. The estimated peak demand NTG ratio changed from 99% in PY2016 to 98% for PY2017. Table 3-17 provides projected, ex ante, and ex post energy and demand impacts, as well as other program performance metrics for the SBES projects in the HPB program. Energy Efficiency Programs 47

51 Table 3-17: Performance Metrics High Performance Business Program Small Business Energy Solutions Metric PY2017 Number of Customers 346 Energy Impacts (kwh) Ex Ante Energy Savings 8,292,775 Gross Ex Post Energy Savings 8,853,314 Net Ex Post Energy Savings 8,581,233 Peak Demand Impacts (kw) Ex Ante Peak Demand Savings 2,160 Gross Ex Post Peak Demand Savings 2,511 Net Ex Post Peak Demand Savings 2,437 Benefit / Cost Ratios Process Evaluation Overview Total Resource Cost Test Ratio 1.90 Utility Cost Test Ratio 5.53 The PY2017 process evaluation included participant surveys and interviews with participating contractors as well as program staff. The objectives of the participant survey are to assess the source of program awareness, factors that influenced project decision making, experience with the application process or energy consultant, and program satisfaction. A total of 49 customer decision makers responded to the participant survey. Participation in the SBES portion of the HPB program was consistent throughout the year. Figure 3-4 displays the accrual of ex ante energy savings throughout Energy Efficiency Programs 48

52 Figure 3-4: Accrual of Reported kwh Savings during the Program Year Table 3-18 summarizes program activity by service provider. The total number of completed projects (based on unique project numbers) doubled from 2016 and the number of completed refrigeration projects increased from 5 to 34. Table 3-18 summarizes the average number of days between when the contractor proposal was completed and when the project was committed to, and the days between the commitment and when installations were completed. As shown, the period of between proposals and installations was close to three months on average. Once the project was committed to, it was completed in short order, approximately five days on average. There was little variation in the time between the commitment and the project completion date across the contractors. Energy Efficiency Programs 49

53 Contractor Table 3-18: Summary of Project Timelines Number of Projects Days Between Proposal and Commit Date Days Between Commit Date and Completion Lighting Lighting Lighting Lighting Lighting Lighting Refrigeration Refrigeration Refrigeration All Contractors 371* 99 5 *Note that there was one record that did not have an audit date that was omitted from these averages. The total population of projects is 372. Program Activity by Location Table 3-19 displays the share of SBES savings by district. Because of the assignment of territories and budgets to service providers, two-thirds of program savings resulted from projects completed outside the Tulsa district. Table 3-19: District Share of Reported kwh Savings Region Reported kwh Savings Percent of Reported Savings Eastern District 2,648,669 32% Tulsa District 2,666,064 32% Tulsa Northern District 814,120 10% Western District 2,163,922 26% Energy Efficiency Programs 50

54 Figure 3-5 provides a heat map of the location of SBES projects across the service territory. Figure 3-5: Distribution of Small Business Energy Solutions Projects Description of Program PSO s High Performance Business program seeks to generate energy savings for small commercial and industrial customers by promoting high-efficiency electric end-use products including lighting and refrigeration measures. The program seeks to combine provision of financial inducements with access to technical expertise to maximize program penetration across the range of potential small business customers. The program has the following additional goals: Increase customer awareness and knowledge of applicable energy saving measures and their benefits Increase the market share of commercial grade high-efficiency technologies sold through market channels Increase the installation rate of high-efficiency technologies in small businesses by customers that would not have done so absent the program Direct install rebates are available to customers that qualify for the SBES portion of the program. To qualify for the program, businesses must use 220,000 kwh or less annually and use a PSO approved service provider. Customers may request an exemption of these requirements. Exemptions are granted on a case-by-case basis, determined by how a customer fits within the program goals. Offerings in this portion of the program are limited Energy Efficiency Programs 51

55 to lighting retrofits and refrigeration measures such as motor replacement, fan controls, anti-sweat heaters, etc. Methodology This chapter provides a brief overview of the data collection activities, gross and net impact calculation methodologies, and process evaluation activities that ADM employed in the evaluation of the HPB program. Data Collection Data for the analysis were collected through review of program materials, on-site inspections, end-use metering (for one project), and interviews with participating customers and service providers. A sample was developed for on-site data collection based on data obtained via SSRS. Participating contractors used an online proposal tool called Audit Direct Install (ADI) software. Within ADI, space-by-space inventories are created for each project. The implementation team can generate reports directly from ADI which contain sufficient information to conduct on-site verification visits. On-site visits were used to collect data for gross impact calculations, to verify measure installation, and to determine measure operating parameters. Facility staff members were interviewed to determine the operating hours of the installed systems and provide any additional operational characteristics relevant to calculating energy savings. For one sampled project, lighting controls were monitored to obtain accurate operational profiles. In addition to the on-site data collection effort, customer surveys provided self-report data for the net-to-gross analysis and process evaluation. A total of 49 customer decision makers who completed SBES incentive projects completed the survey. In-depth interviews with two PSO and implementation staff members were conducted to provide additional perspectives for the process evaluation. Table 3-20 shows the achieved sample sizes for the different types of data collection employed for this study. Table 3-20: Sample Sizes for Data Collection Efforts SBES Data Collection Activity Achieved Sample Size On-Site M&V visits 23 Customer Decision Maker Survey 49 Trade Ally Interviews 6 In-depth Interviews with Program Staff 2 Energy Efficiency Programs 52

56 Sampling Plan As with the Custom and Prescriptive projects, ADM created a stratified sample based on the amount of energy savings and type of measure installed in a given project. Sample sizes were designed to meet 10% precision at the 90% confidence level at the program level. Table 3-21 below shows the sample design that was used for SBES projects. Stratum classifications were based on verified measure installations. The 23 projects that were sampled for on-site measurement and verification account for approximately 15% of reported program kwh savings. Table 3-21: Sample Design for the High-Performance Business Program Small Business Stratum Name Reported kwh Savings Strata Boundaries (kwh) Population of Projects Design Sample Size Lighting 1 1,010,220 <12, Lighting 2 1,630,129 12,006 25, Lighting 3 1,870,519 25,904 46, Lighting 4 1,189,844 46,606 78, Lighting 5 1,128,243 78, , Lighting 6 901,405 >151, Refrigeration 1 162,215 <14, Refrigeration 2 205,016 14,251 25, Refrigeration 3 195,183 >25, Total 8,292, Impact Evaluation Methodology The evaluation of gross ex post energy savings and peak demand reduction from projects rebated through the High Performance Business program can be broken down into the following steps: The program tracking database was reviewed to determine the scope of the program and to ensure there were no duplicate project entries. The tracking database was used to define a discrete set of rebated projects that made up the PY2017 program population. A sample of projects was then drawn from the population established in the tracking system review. Energy Efficiency Programs 53

57 A detailed desk review was conducted for each project sampled for on-site verification and data collection. The desk review process included a thorough examination of all project materials including invoices, equipment cut sheets, preand post-inspection reports, and estimated savings calculators. This review process informed ADM s fieldwork by identifying potential uncertainties, missing data, and sites where monitoring equipment was needed to verify key inputs to the reported savings calculations. Additionally, the review process involved assessing the reasonableness of deemed savings values and calculation input assumptions. After reviewing the project materials, on-site verification and data collection visits were scheduled for each sampled project. The visits were used to collect data for savings calculations, to verify measure installation, and to determine measure operating parameters. The data collected during the on-site verification visits was used to revise savings calculations as necessary. For example, if the reported savings calculations relied on certain measure operating hours that were determined inaccurate based on the facilities actual schedule, changes were made to more accurately reflect actual operating conditions. After determining the ex post savings impacts for each sampled project, results were extrapolated to the program population using project-specific sampling weights. This allows for the estimation of program level gross ex post energy (kwh) savings with a given amount of sampling precision and confidence. For the HPB program SBES projects, the sample was designed to ensure ±10% or better relative precision at the 90% confidence level for kwh reductions. Net-to-Gross Estimation The purpose of net savings analysis is to determine what portion of gross savings achieved by PSO customers is the direct result of program influence. Information collected from a sample of program participants through a customer decision maker survey was used for the net-to-gross analysis. These responses were reviewed to assess the likelihood that participants were free riders. Several criteria were used for determining what portion of a customer s savings for a project should be attributed to free ridership. The criteria are organized into the following three factors: Plans and intentions of the firm to install a measure even without support from the program; Influence that the program had on the decision to install a measure; and A firm s previous experience with a measure installed under the program. Energy Efficiency Programs 54

58 For each of these factors, rules were applied to develop binary variables indicating whether a participant showed free ridership behavior. The first required step was to determine if a participant stated that his or her intention was to install an energy efficiency measure without the help of the program inducements. Two binary variables were constructed to account for customer plans and intentions: one, based on a more restrictive set of criteria that may describe a high likelihood of free ridership, and a second, based on a less restrictive set of criteria that may describe a relatively lower likelihood of free ridership. The first, more restrictive criteria indicating customer plans and intentions that likely signify free ridership are as follows: The respondent answered, yes to the following two questions: Did you have plans to install the [Equipment/Measure] before participating in the program? and Would you have gone ahead with this planned installation of the measure even if you had not participated in the program? The respondent answered, definitely would have installed to the following question: If the rebates from the program had not been available, how likely is it that you would have installed [Equipment/Measure] anyway? The respondent answered, did not affect the timing of purchase and installation to the following question: We would like to know whether the availability of information and rebates through the program affected the timing of your purchase and installation of [Equipment/Measure]. Did you purchase and install [Equipment/Measure] earlier than you otherwise would have without the program? The respondent answered, no, the program did not affect the level of efficiency that we chose for equipment in response to the following question: Did you choose equipment that was more energy efficient than you would have chosen had you not participated in the program? The respondent answered, No, the program did not affect timing of project in response to the following question: Did you install the [Equipment/Measure] earlier than you otherwise would have because of the information and rebates from PSO s program? The second, less restrictive criteria indicating customer plans and intentions that likely signify free ridership are as follows: The respondent answered yes to the following two questions: Did you have plans to install the [Equipment/Measure] before participating in the program? and Would you have gone ahead with this planned installation of the measure even if you had not participated in the program? Energy Efficiency Programs 55

59 Either the respondent answered, would have installed or probably would have installed to the following question: If the rebates from the program had not been available, how likely is it that you would have installed [Equipment/Measure] anyway? Either the respondent answered, did not affect timing of purchase and installation to the following question: Did you purchase and install [Equipment/Measure] earlier than you otherwise would have without the program? or the respondent indicated that while program information and rebates did affect the timing of equipment purchase and installation, in the absence of the program they would have purchased and installed the equipment within the next two years. The respondent indicated that no, the program did not affect level of efficiency that we chose for equipment. The second factor is determining if a customer reported that a recommendation from a program representative or experience with the program was influential in the decision to install a piece of equipment or measure. This criterion indicates that the program s influence may lower the likelihood of free ridership when either of the following conditions is true: The respondent answered, very important to the following question: How important was previous experience with PSO energy efficiency programs in making your decision to install the [Equipment/Measure] at your facility? The respondent either answered probably would not have or definitely would not have installed the equipment if it was recommended by a program staff member or the energy consultant they worked with (for SBES participants). The third factor is determining if a participant in the program indicated that he or she had previously installed an energy efficiency measure like one that they installed under the program without an energy efficiency program incentive during the last three years. A participant indicating that he or she had installed a similar measure is considered to have a higher likelihood of free ridership, due to the potential influence of the prior experience. The criteria indicating that previous experience may signify a higher likelihood of free ridership are as follows: The respondent answered yes to the following question: Before participating in the program, had you installed any equipment or measure like [Equipment/Measure] at your facility? The respondent answered yes, purchased energy efficient equipment but did not apply for financial incentive. to the following question: Has your organization purchased any energy efficient equipment in the last three years for which you did not apply for a rebate through the program? Energy Efficiency Programs 56

60 The four sets of rules just described were used to construct four different indicator variables that address free ridership behavior. For each customer, a free ridership value was assigned based on the combination of variables. With the four indicator variables, there were 12 applicable combinations for assigning free ridership scores for each respondent, depending on the combination of answers to the questions creating the indicator variables. Table 3-13 in the subsequent chapter of this report shows the scoring breakdown. The customer decision maker survey also included a series of questions used to analyze whether there were potential spillover effects associated with non-rebated purchases by program participants. 23 Specifically, survey respondents were asked: We would like to know if you have installed any additional energy efficient equipment because of your experience with the program that you DID NOT receive an incentive for. Since participating in the program has your organization installed any ADDITIONAL energy efficiency measures at this facility or at your other facilities within PSO's service territory that did NOT receive incentives through PSO's program? Customers who indicated yes were identified as potential spillover candidates. Potential spillover candidates were additionally asked to identify the type of additional equipment installed and provide information about the equipment for use in estimating energy savings. For each type of equipment that respondents reported installing, respondents were asked the following two questions used to assess if any savings resulting from the additional equipment installed were attributable to the program. [SP1] How important was your experience with the program in your decision to install this [Equipment/Measure]? [Rated on a scale where 0 meant not at all important and 10 meant very important] [SP2] If you had NOT participated in the program, how likely is it that your organization would still have installed this [Equipment/Measure]? [Rated on a scale where 0 meant not at all likely and 10 meant very likely] A spillover score was developed based on these responses as follows: Spillover Score = Average (SP1, 10-SP2) The energy savings of equipment installations associated with a spillover score of greater than five were attributed to the program. 23 The spillover analysis is limited to participant spillover. Non-participant spillover effects may exist for the program, but they are not estimated and therefore assumed to be zero. Energy Efficiency Programs 57

61 Process Evaluation The process evaluation is designed to research, and document, the program delivery mechanisms and collective experiences of program participants, partners and staff. The process evaluation was designed to answer the following research questions: What changes, if any, were made to the program design or implementation procedures? How did new participants learn of the program? What factors motivated their decision to participate? Were program participants satisfied with their experience? What were the key successes and challenges during PY2017? Are small business service providers satisfied with the program? Are referrals effectively shared amongst them? How does PSO s small business program compare to other small business programs that they work with? Are service providers aware of any barriers to participation? To address these questions, ADM s process evaluation activities included surveys program participants, in-depth interviews with service providers, and in-depth interviews with program staff. Table 3-22: Process Evaluation Data Collection Activities Summary Data Collection Activity Program Staff Interviews Process Evaluation Research Objectives Assess program staff perspectives regarding program operations, strengths, or barriers to success. Participant Surveys SBES Service Provider Interviews Assess source of program awareness, factors that influenced project decision making, experience with the application process or energy consultant, and program satisfaction. Assess barriers to participation, gaps in measures offered, satisfaction with program processes. Impact Evaluation Findings The ex post gross kwh savings for the PY2017 HPB SBES projects are summarized by sampling stratum in Table Energy Efficiency Programs 58

62 Table 3-23: Ex Ante and Ex Post Gross kwh Savings by Sampling Stratum SBES Stratum Ex Ante kwh Savings Ex Post Gross kwh Savings Gross kwh Realization Rate Lighting 1 1,010,220 1,138, % Lighting 2 1,630,129 1,592,194 98% Lighting 3 1,870,519 1,980, % Lighting 4 1,189,844 1,271, % Lighting 5 1,128,243 1,368, % Lighting 6 901,405 1,052, % Refrigeration 1 162, ,539 67% Refrigeration 2 205, ,183 83% Refrigeration 3 195, ,335 87% Total 8,292,775 8,853, % The achieved sample design resulted in ex post gross kwh estimates with ±8.4% relative precision at the 90% confidence interval. 24 Overall, ex post gross energy savings were relatively close to the original reported values at the program level (107% realization rate). There was, however, a wide range of kwh realization rates at the sample project level. The following sections discuss specific measure types from the PY2017 sample. Lighting Contractor Projects For the lighting projects in the program, the realization rate for annual kwh savings is 109%. The wattage table used in the ex ante calculations is not an exact match for the Arkansas TRM v6.1, but the discrepancies are negligible. For all sampled sites, ADM verified measure counts to be accurate. There were minor discrepancies between claimed HOU and verified HOU, but these discrepancies were not biased in any direction. Interactive effects were not included in ex ante calculations (1.09 in most cases) when applicable. The application of interactive effects served as the primary driver of the additional verified savings. 24 That is, we are 90% confident that the true verified gross savings are between 8,990,031 and 7,595,519 kwh based on the uncertainty introduced by sampling. Energy Efficiency Programs 59

63 Refrigeration Projects ADM relies on the savings algorithms from the Arkansas TRM v6.1 for refrigeration measures. The ex ante calculations appear to use various sources for savings algorithms. This leads to variations in savings by measure, most notably electronically commutated (EC) motors and evaporator fan controls. When specifications cannot be verified during the on-site inspections, deemed values from the TRM are used. In general, this has resulted in greater ex post savings for EC motors, and lower ex post savings for evaporator fan controls. These two measures represent 49% of ex ante savings. Gross Coincident Peak Demand Reduction (kw) The ex post gross kw reduction for the PY2017 HPB SBES program is summarized by sampling stratum in Table Overall, the ex post gross kw is equal to 116% of the reported reduction for the program. Table 3-24: Ex Ante and Ex Post Gross kw Reduction by Sampling Stratum Stratum Ex Ante Peak kw Reduction Ex Post Gross Peak kw Reduction Ex Post Gross kw Realization Rate Lighting % Lighting % Lighting % Lighting % Lighting % Lighting % Refrigeration % Refrigeration % Refrigeration % Total 2,160 2, % Energy Efficiency Programs 60

64 The achieved sample design resulted in ex post gross kw estimates with ±21.7% relative precision at the 90% confidence interval. 25 The high level of uncertainty associated with peak kw reductions is due to the significant amount of variance from project to project. Much of the difference between ex ante and ex post demand reduction, as in past program years, is explained by either 1) calculation error in ex ante demand reduction, 2) use of stipulated coincidence factors (CF) that did not align well with actual equipment schedules. For lighting projects, the ADM ex post lighting calculators generate an hourly curve (8760 hours) for 2017 to determine the average kw value across the peak demand period for each lighting schedule. Net-to-Gross Estimation The data used to assign free ridership scores were collected through a survey of customer decision makers for projects rebated through the HPB program during PY2017. Free ridership was estimated using the methodology described in Small Business Energy Solutions: Net-to-Gross Estimation section. Table 3-25 shows percentages of total gross ex post savings associated with different combinations of free ridership indicator variable values for the SBES incentive component. 25 That is, we are 90% confident that the ex post gross peak demand reduction is between 2,629 and 1,692 kw based on the uncertainty introduced by sampling. Energy Efficiency Programs 61

65 Had Plans and Intentions to Install Measure without C&I Program? (Definition 1) Table 3-25: Estimated Free-ridership for PY2017 HPB Program SBES Had Plans and Intentions to Install Measure without C&I Program? (Definition 2) C&I Program had influence on Decision to Install Measure? Had Previous Experience with Measure? Free Ridership Score Percentage of Total Gross kwh Savings Y Y Y Y 100% 0% Y Y N N 100% 1% Y Y N Y 100% 0% Y Y Y N 67% 0% N Y N Y 67% 0% N Y N N 33% 0% N Y Y N 0% 0% N Y Y Y 33% 0% N N N Y 33% 7% N N N N 0% 33% N N Y N 0% 1% N N Y Y 0% 0% Required program to implement measures 0% 58% Total 3.2% 100% Overall, the estimated percentage of program free ridership is 3.2%. This level of free ridership is higher than in 2016 (2016 free ridership was 0.5%). This score indicates that a very few customers stated that they would have installed the equipment in absence of the program. Customer decision maker survey responses were also analyzed to estimate participant spillover effects. Overall, one of the respondents reported installing efficient equipment that met the attribution criterion and for which energy savings could be estimated. The final net-to-gross ratio for the program is calculated as 1 free-ridership + participant spillover. This results in an NTGR of 97.0% for kwh savings and 96.9% for peak demand reductions. The gross and net ex post energy savings and peak demand reduction of the HPB program during PY2017 are summarized by program in Table Table 3-26: Summary of Ex Post Gross and Net Impacts Program Ex Post Gross kwh Savings Ex Post Net kwh Savings Net-to-Gross Ratio Ex Post Gross kw Reduction Ex Post Net kw Reduction High Performance Business - SBES 8,853,314 8,581, % - kwh 96.9% - kw 2,511 2,437 Energy Efficiency Programs 62

66 Process Evaluation Findings An online survey was administered to program participants during January and February A total of 49 customer decision makers completed a survey about their experience with the Small Business Energy Solutions Program in Table 3-27 summarizes the types of business and building types represented among survey respondents. Table 3-27: Survey Respondent Business and Building Types Response Percent (n = 49) Retail: Excluding Enclosed Mall 29% Office 14% Auto Dealership/Repair Shop 10% Religious Facility 8% Government 6% Manufacturing 6% Education: K-12 w/ Summer Session 4% Service (Excluding Food) 4% Warehouse: Non-refrigerated 4% Education: K-12 w/o Summer Session 2% Financial Institution 2% Gas Station 2% Gymnasium 2% Other 2% Police Station/Firehouse 2% Workshops 2% Overall Satisfaction Of the survey respondents, six reported that they had interactions with PSO program staff during the completion their project. These respondents indicated that staff was very knowledgeable, and they were very satisfied with the thoroughness and timeliness of the response received. Energy Efficiency Programs 63

67 Overall, participant satisfaction with the program was high. Ninety-two percent of the respondents reported that they were somewhat or very satisfied with the program. In total, 5 of the 49 respondents (10%) reported some dissatisfaction with an aspect of the program. One respondent elaborated on their reasons for dissatisfaction and noted that the LED light strips have failed Conclusions Figure 3-6: Program Satisfaction The following summarizes the key findings of the process evaluation of the custom and standard components of the High-Performance Business Program: While the design of the Custom and Prescriptive component remained largely the same in 2017 as in 2016, staff continued to make improvements to the program delivery. Most notably among these was the introduction of a simplified application process at midyear. Under the new process, customers submit a single page application and are then subsequently contacted by an ICF account manager who Energy Efficiency Programs 64

68 collects additional information on the project from the customer. Additionally, the online intake tool is now fully operational. Staff noted that this tool works well for customers submitting smaller projects and eases the application process using dynamic forms that narrow the information needed based on information provided in earlier portions of the application. Both PSO and ICF staff indicated that they would prefer to make quicker progress towards reaching the energy savings goal during the year rather than being dependent on many projects completing in December. Staff is in the process of implementing two new program offerings to increase uptake of non-lighting measures. These are a new commissioning and energy management service and an offering targeting efficiency improvements in oil pumping operations. Most participants were satisfied or very satisfied with the program overall. Additionally, most respondents indicated satisfaction with the various elements of the program. No single aspect of the program experience stood out as something participants were particularly satisfied or dissatisfied with. Review of the program tracking data found that the measure type categories used were at times not descriptive of the type of efficiency measure implemented and that multiple category names were used to describe the same type of project. The following summarizes the key findings of the process evaluation of the SBES component of the High Performance Business Program. Staff noted that there was a high level of interest in the program in A key driver of program activity was the addition of the TLED lamps. Staff reported that there was a lot of interest in this measures during the year. One challenge noted was the development of refrigeration projects. The barriers to refrigeration projects identified were (1) not having a local representative to sell refrigeration projects and (2) the additional education required to encourage customers to adopt refrigeration measures as compared to what is needed for lighting measures. Staff is continuing to work with the refrigeration contractor to get a local salesperson as well as scoping out other firms that can deliver refrigeration measures. Ninety-two percent of the respondents reported that they were somewhat or very satisfied with the program overall. Service providers demonstrated an understanding of guidelines for installing the TLED lamps added in Service providers stated that they most typically recommend Type B over Type A lamps because these provide the best value and fit to the customer s needs. Energy Efficiency Programs 65

69 Most of the lighting service providers reported that they promote refrigeration and other equipment incentives offered by PSO. Service providers suggested that case studies and co-branded materials would enhance their marketing and outreach efforts. Support provided by, and communications with, program staff were viewed very favorably by all service providers. Service providers indicated that SBES compares favorably to other small business programs they are familiar with and ADI continues to be viewed as a useful administration tool. All lighting service providers rated their satisfaction as either a four or a five using a scale where one meant very dissatisfied and five meant very satisfied. The PY2017 evaluation of the HPB Custom and Prescriptive projects has resulted in a gross savings realization rate of 101% compared to the ex ante annual energy savings. An NTG score was calculated based on survey information to inform ADM of the amount of spillover and free-ridership in the program, which resulted in a decrease in gross savings by 10.7%. The PY2017 evaluation of the HPB SBES has resulted in a gross savings realization rate of 107% compared to the ex ante annual energy savings. A NTG score was calculated based on survey information to inform ADM of the amount of spillover and free-ridership in the program, which resulted in a decrease in gross savings by 3%. The energy savings and peak demand reduction results are shown in Table 3-28 and Table These tables include the summary for the entire HPB program. Table 3-28: Summary of Program Level Energy Impacts Program Custom and Prescriptive Small Business Energy Solutions Ex Ante Gross kwh Savings Ex Post Gross kwh Savings Realization Rate Net- to- Gross Ratio Ex Post Net kwh Savings 49,092,291 49,357, % 89% 44,061,144 8,292,775 8,853, % 97% 8,581,233 Total 57,385,066 58,211, % 90% 52,642,377 Energy Efficiency Programs 66

70 Table 3-29: Summary of Program Level Demand Impacts Program Custom and Prescriptive Small Business Energy Solutions Ex Ante Gross kw Savings Ex Post Gross kw Savings Realization Rate Net- to- Gross Ratio Ex Post Net kw Savings 7,966 7,803 98% 87% 6,796 2,160 2, % 97% 2,437 Total 10,126 10, % 90% 9,233 Updates from PY2016 Upon completion of the PY2017 evaluation, several updates were implemented to have a positive influence on the impact evaluation results. The updates for High Performance Business Custom and Prescriptive include: Updating the EFLH for HVAC units that the implementation team is using to be consistent with ADM s evaluation Increased vetting of refrigeration gasket contractors and energy savings methodologies. The updated methodology includes the determination of linear feet of gasket that needs to be replaced for energy savings; as opposed to applying energy savings to the entire gasket replaced. The addition of project notes provided by the implementation team to provide more detailed information in terms of annual operating hours. The updates for High Performance Business Custom and Prescriptive include: Incentives were added for TLED retrofits to have a positive influence on program flexibility and customer satisfaction. Planned Program Changes The following changes are planned or have been implemented for the 2018 program year. Staff is working with Trane to offer a commissioning and energy management service. This effort is primarily targeting municipalities, universities, schools, and hospitals. A pilot sub-program targeting efficiency in the oil industry was launched in early Staff contracted with an organization to work with oil pumping facilities to identify efficiency improvements. There are no major changes planned for SBES at this time. Energy Efficiency Programs 67

71 3.2 Home Weatherization Program Program Overview PSO s Home Weatherization program seeks to generate energy and demand savings for limited income residential customers through the direct installation of weatherization measures. The Weatherization Program provided no-cost energy efficiency improvements to PSO customers with household incomes of $45,000 or less a year. PSO partnered with two organizations to deliver the efficiency improvements, Titan ES and RTT. Titan ES provided energy audits, customer education, and installation of efficient equipment. Titan ES delivered services to single family homes that are either leased or owner occupied. RTT is a Tulsa based non-profit organization that provides a variety of home improvement services for low-income homeowners. The services provided by RTT include program sponsored energy efficiency improvements as well as other repairs such as roof repairs. The program also partnered with Oklahoma National Gas (ONG) to co-fund gas and electric improvements to a small number of homes that receive energy services from both utilities. Incentive costs are split 50/50 between PSO and ONG for these joint-funded projects. Through the Home Weatherization program, participants received diagnostic energy assessments, weatherization measures such as air sealing, insulation, and duct sealing, and direct installation of up to eight LED light bulbs in higher use areas. The project must have a savings-to-investment ratio (SIR) of 1.0 or higher, although individual measures may fall below this ratio. Table 3-30 shows measures installed through the program. Energy Efficiency Programs 68

72 Table 3-30: Summary of Measures Implemented Measure Number of Projects % Ex Ante kwh Savings Air Infiltration 1,842 61% Attic insulation 1,942 38% LED 1,565 1% Room AC 1 <1% Water Heater Jacket 133 <1% Water Heater Pipe Insulation 99 <1% PSO s Home Weatherization program serviced 2,239 households during the 2017 program year. Participants saved an average of 2,324 kwh and installed and average of 2.5 measure types through the weatherization program. Actual program spending in 2017 was nearly identical to projected spending, PY2017 performance metrics are summarized in Table Energy Efficiency Programs 69

73 Table 3-31: Performance Metrics Weatherization Metric PY2017 Number of Customers 2,239 Budgeted Expenditures $3,923,754 Actual Expenditures $3,808,927 Energy Impacts (kwh) Projected Energy Savings 3,865,198 Reported Energy Savings 5,204,471 Gross Verified Energy Savings 4,929,881 Net Verified Energy Savings 4,929,881 Peak Demand Impacts (kw) Projected Peak Demand Savings 1,225 Reported Peak Demand Savings 1,584 Gross Verified Peak Demand Savings 1,526 Net Verified Peak Demand Savings 1,526 Benefit / Cost Ratios Total Resource Cost Test Ratio 2.47 Utility Cost Test Ratio EM&V Methodologies This section provides a brief overview of the data collection activities, gross and net impact calculation methodologies, and process evaluation activities that ADM employed in the evaluation of the Home Weatherization program. A process evaluation memo was provided to PSO after the completion of the 2017 program year. Data Collection Several primary and secondary data sources were used for the evaluation. Tracking data and supporting documentation for the program was obtained from Sightline database and SQL Server Reporting Services (SSRS). This tracking data was used as the basis for quantifying participation and assessing program impacts. Additional data was collected through the phone surveys and onsite verification visits. Energy Efficiency Programs 70

74 The telephone survey included questions to verify: Whether the participant indeed had their home serviced through the program, The measures claimed to be installed in the tracking database matched participant responses, and The heating, cooling, and water heating systems for respondents matched records in the tracking database. During the site visits, ADM field staff verified that the claimed energy efficiency measures were installed. and recorded key inputs to savings calculations such as R-value of installed insulation and quantity of LED bulbs replaced. Data collected through these activities was used to develop measure level verification rates, which were then used to adjust the deemed savings estimates where necessary. One hundred and thirty-five participants were surveyed by telephone. A total of 49 homes were visited independently to verify all measures were installed to program standards. ADM completed an interview with the PSO program manager in November of 2017 to inform the process evaluation memo. Table 3-32 below summarizes the data collection activities and sample sizes. Table 3-32: Sample Size Data Collection Activity Achieved Sample Size On-site Verification Visits 49 Customer Survey 135 In-Depth Interviews with Program Staff 1 Sampling Plan For the calculation of sample size for survey completes, a coefficient of variation of 0.5 was assumed. 26 With this assumption, a minimum sample size of 68 participants was required, as shown in the following formula: Equation 3-1: Minimum Sample Size Formula for 90 Percent Confidence Level 26 The coefficient of variation, cv(y), is a measure of variation for the variable to be estimated. Its value depends on the mean and standard deviation of the distribution of values for the variable (i.e., cv(y) = sd(y)/mean(y)). Energy Efficiency Programs 71

75 Where: n 0 = minimum sample size Z = Z-statistic value (1.645 for the 90% confidence level) CV = Coefficient of Variation (assumed to be 0.5) RP = Relative Precision (0.10) ADM conducted phone surveys with 135 participants across the service territory. The additional survey completes were obtained to increase the chance of participation in all areas the program impacted. The sample for in-home inspections was designed to achieve ±10% relative precision or better at the 90% confidence interval. Reported home savings values were placed in one of four strata and the number of in-home inspections needed per stratum was calculated. Table 3-33 below shows the achieved sample design. Table 3-33: Sample Design Home Weatherization Stratum Reported kwh Strata Boundaries (kwh) Population Size CV Sample Size 1 108,694 < ,940, , ,389,704 2,250 8, ,255 > 8, Total 5,204, Gross Impact Methodologies The methodology used to calculate energy (kwh) and demand impacts (kw) consisted of: Reviewing a census of program tracking data The tracking data was reviewed for a census of homes and measures. ADM verified there were not any duplicate project data entry errors. Verifying measure installation ADM calculated installation rates (ISR) by measure for a sample of program participants utilizing data from phone and onsite verification visits. Reviewing ex ante savings estimates for each measure Energy Efficiency Programs 72

76 ADM reviewed ex ante savings calculations for all measures to provide an explanation of any savings discrepancies. Calculating ex post verified savings utilizing: Oklahoma Deemed Savings Document, (OKDSD) Arkansas Technical Reference Manual v6 (AK TRM) A brief description of each measure calculation methodology is described in the sections below. Appendix F includes the measure level algorithms and deemed savings values utilized for the ex post and kwh and kw savings calculations. Infiltration Reduction: ADM performed a benchmarking analysis for the infiltration reduction measure. The analysis compares the deemed savings values in the OKDSD and AR TRM to savings calculations from similar weather zones across the county. The kwh savings calculated from the OKDSD were found to be an outlier, while the AR TRM produced values that were in the expected savings range. The AR TRM was utilized to calculate energy and demand impacts of infiltration reduction measures. Savings are calculated by multiplying the air infiltration reduction (CFM), with the energy savings factor corresponding to the climate zone and HVAC type. The air infiltration reduction estimate in CFM is obtained through blower door testing performed by the program contractor for each home serviced. Only homes with electric cooling systems are eligible for the measure (central AC or room AC). Duct Sealing: This measure involves sealing leaks in ducts of the distribution system of homes with either central AC or a ducted heating system. ADM utilized the Oklahoma Deemed Savings Document in conjunction with the duct leakage reduction results in order to calculated measure savings. The duct leaked reduction estimate in CFM is obtained through duct blaster testing performed by the program contractor for each home serviced. Ceiling Insulation: Deemed savings values were calculated for each weather zone in accordance to the OKDSD. Deemed savings listed based on the R-value of the baseline insulation. Savings are calculated by multiplying the corresponding savings value by the square footage insulated. Retrofit insulation must meet a minimum R-value of R-38 in order to be considered for savings. Pipe Insulation and Water Heater Jackets: Installation of pipe insulation and/or water heater jackets required electric water heating in serviced homes. As such, the number of recipients was significantly smaller than other program measures. The deemed savings for water heater jackets installed on electric water heaters are sourced from the Oklahoma Deemed Savings Document. For water heater jackets, a review of the tracking system showed that conservative assumptions were used to inform the use of the deemed savings. Savings values corresponding to 2 thick jackets on Energy Efficiency Programs 73

77 40 gallon tanks were used for all sites. The deemed savings for this measure depend on 1) insulation thickness and 2) water heater tank size. Water heater pipe insulation involves insulating of all hot and cold vertical lengths of pipe, plus the initial length of horizontal hot and cold water pipe, up to three feet from the transition, or until wall penetration, whichever is less. The Oklahoma Deemed Savings Document specifies deemed values for energy and demand impacts of water heater pipe insulation measures, the deemed values can be found in Appendix F. The verified savings for water heater pipe insulation resulted in a realization rate equal to 100%, for both energy savings (kwh) and peak demand reduction (kw). LED Light Bulbs: This measure provides savings for replacing an inefficient lamp with an Omni-directional LED in residential applications. The replacement must be Energy Star qualified. The Oklahoma Deemed Savings Document specifies the algorithms for use in calculating energy and demand impacts of ENERGY STAR LEDs. ADM utilized these algorithms with a modification to the hours of use per year (960 HOU per year). The modification of the hours of use was sourced from a benchmarking study performed in Net-to-Gross Estimation The Home Weatherization program specifically targets customers with limited income, providing all services at no cost to the customer. It is likely that participating customers would not have funded the installed energy efficiency measures on their own. As a result, ADM assumed an NTG ratio of 100% Process Evaluation Activities The process evaluation component was designed to answer the following research questions: How do participants hear about the Program? Are there any changes in how participants learn about the program as compared to the prior year? Were the program participants satisfied with their experience? What were the levels of satisfaction with the work performed, the scheduling/application process, and other aspects of program participation? What are the perceived energy and non-energy benefits associated with the Program? Were there any changes to program design or implementation in 2017? How did these changes affect participation or outcomes? Are participation/savings goals still being reached? Are there changes that should be considered in future years? 27 ADM HOU Memo, Energy Efficiency Programs 74

78 Were there any significant challenges or new obstacles during the 2017 program year? Looking forward, what, if any, are key barriers and drivers to Program success within PSO s market? As many of the above research questions were addressed during prior evaluation years, each aspect of the process evaluation was assessed in the context of previous evaluation results. To address these questions, ADM s process evaluation activities included participant surveys and an interview with the PSO Program manager. ADM provided a portfolio level process evaluation memo to PSO in the first quarter of Impact Evaluation Findings Ex post and ex ante peak demand by measure are shown in Table The savings estimates result in a gross annual kwh savings realization rate of 95% and a peak kw reduction realization rate of 96%. The reported savings were calculated utilizing the Oklahoma Deemed Savings Document and the Arkansas TRM V5. Energy Efficiency Programs 75

79 Table 3-34: Reported and Verified kwh and Peak kw Measure Reported Energy Savings (kwh) Reported Peak Demand Savings (kw) Verified Gross Energy Savings (kwh) Verified Gross Peak Demand Savings (kw) kwh Realization Rate 28 kw Realization Rate Attic Insulation 1,877, ,879, % 98% Duct Sealing 1,615, ,743, % 113% Air Infiltration 1,418, ,016, % 70% Water Heater Jacket 9, , % 100% Water Heater Pipe Insulation 4, , % 100% LED Light Bulbs 278, , % 106% Total 5,204,471 1,584 4,929,881 1,526 95% 96% Table 3-35 shows the installation rates by measure type for the phone and field verification survey effort. Table 3-35: Home Weatherization In-Service Rates Measure Verified/Claimed Phone Survey On-site Verification ISR Attic Insulation Duct Sealing Air Infiltration Water Heater Jacket Pipe Insulation LEDs Verified Claimed Verified Claimed Verified Claimed Verified 4 2 Claimed 4 2 Verified 4 1 Claimed 4 1 Verified Claimed % 100% 100% 100% 100% 96% 28 SSRS Database was not updated to reflect changes to kwh and kw savings that were brought about by a change in methodology during the portfolio cycle. The evaluation, implementation, and utility decided to instead capture the changes in the realization rates. Energy Efficiency Programs 76

80 Infiltration Reduction: Three of the survey respondents with infiltration reduction measure indicated that they had not received the installation. ADM utilized data from the SIGHTLINE web portal and verified the measure was complete in all three residences. This measure can be difficult for a home owner to see, for example if the plumbing fixture penetration under a kitchen sink were done a customer may not notice the measure, or if the windows were sealed using a clear caulking with a thin bead. Additionally, ADM s on-site verification work found evidence of air sealing in all 44 visited homes. Based on these findings, an ISR of 100% was applied. ADM calculated the deemed savings values for each home and determined realization rates of 72% for energy savings (kwh) and 70% for peak demand reduction (kw). The difference between the ex post and ex ante savings estimate was the utilization of the energy savings factors from the AR TRM for the ex post savings calculation and the energy savings factors in the OK DSD for the ex ante calculation. Duct Sealing: A total of 77 survey respondents represented homes where duct sealing was reported to have occurred. Two of these respondents indicated that they had not had duct sealing measures installed. ADM logged on to SIGHTLINE web portal and verified the measure was done on both respondents homes. This measure can be difficult for a home owner to see, as the duct system is typically in the attic covered by insulation and if the furnace plenum was done in a furnace closet a home owner may not open that closet very often and notice the sealed duct. The on-site verification visits included 36 homes where duct sealing savings had been claimed. ADM s on-site verification work found evidence of duct sealing in all the visited homes where savings had been claimed. An ISR of 100% was applied for duct sealing. ADM calculated the deemed savings values for each home and determined realization rates of 108% for kwh and 113% for peak demand reduction (kw). The difference between the ex post and ex ante saving the difference in the baseline SEER value applied algorithm. The ex ante calculation applied the default baseline SEER value from the OKSDS. The ex post calculation utilized 11.5 as the baseline SEER value. The lower SEER value was applied because the measure is being implemented income qualified homes which tend to be serviced by older air conditioning units. The 11.5 SEER value is the average of U.S. DOE minimum allowed SEER for air conditioners form (10 SEER) and after January 23, 2006 (13 SEER). Energy Efficiency Programs 77

81 Ceiling Insulation All survey respondents verified insulation was installed. Similarly, evidence of additional ceiling insulation was verified during all on site visits. During the on-site verification visits, the pre-existing and new insulation levels along with square feet installed were measured. Approximate square footage measurements closely matched reported values. As a result, an ISR of 100% was applied for attic insulation. The verified kwh savings realization rate was 100% and peak demand kw savings realization rate was 98%. Pipe Insulation and Water Heater Jackets Installation of pipe insulation and/or water heater jackets required electric water heating in serviced homes. As such, the number of recipients was significantly smaller than other program measures. ADM completed 8 verification surveys and 3 on-site inspections for customers that had one or both measures installed in their homes. All 8 survey respondents indicated that the measures were installed. On-site verifications provided evidence of pipe insulation and water heater jacket installation, with electric water heating confirmed. In all sampled cases, the measure was verified as installed resulting in an ISR 100%. For water heater jackets, a review of the tracking system showed that conservative assumptions were used to inform the use of the deemed savings. Savings values corresponding to 2 thick jackets on 40 gallon tanks were used for all sites. The deemed saving for this measure depend on 1) insulation thickness and 2) water heater tank size. The deemed savings for water heater jackets installed on electric water heaters can be found in Appendix F and are sourced from the Oklahoma Deemed Savings Document. The verified savings for water heater jackets resulted in a realization rate equal to 100%, for both energy savings (kwh) and peak demand reduction (kw). Water heater pipe insulation involves insulating of all hot and cold vertical lengths of pipe, plus the initial length of horizontal hot and cold water pipe, up to three feet from the transition, or until wall penetration, whichever is less. The Oklahoma Deemed Savings Document specifies deemed values for energy and demand impacts of water heater pipe insulation measures, the deemed values can be found in Appendix F. The verified savings for water heater pipe insulation resulted in a realization rate equal to 100%, for both energy savings (kwh) and peak demand reduction (kw). LED Light Bulbs: Ninety-three customers who received LEDs through the weatherization program were surveyed, for a total 532 LEDs. Twenty of the 532 LEDs were reported by survey respondents to have either failed or were not installed. Energy Efficiency Programs 78

82 The on-site verification visits included 30 homes where LED installation was reported, representing a total of 191 LEDs. One hundred and eighty-four of these LEDs, were verified to be in service. Based on these findings, an ISR of 96% was applied to the ex post energy saving calculation. LED bulb calculations resulted in realization rates of 99% kwh and 106% for peak demand reduction. The difference in ex post and ex ante savings was attributed to the difference in hours of use and ISRs applied. The ex ante calculation used an annual HOU of 1023 and an ISR of 91%, while the ex post calculation used an annual HOU of 961 and as found 2017 ISR of 96% Conclusions Overall the program design and operations remained unchanged. Two noteworthy developments resulted in a potential expansion of the program s reach. First, the partnership with ONG continued and the funds ONG contributed in 2017 increased from The additional funds allowed the program to serve more customers through the program. Second, the program reached an agreement with the Ki Bois Community Action Partnership to complete weatherization services for 10 homes in While no homes were weatherized through Ki Bois, staff intends to continue this arrangement in A direct mail campaign followed by telephone outreach continued to be the principal means by which the program recruited customers. The program was also promoted in a PSO newsletter. Survey responses suggest that customers continued to primarily learn about the program through the direct mail and telephone recruitment campaign, as well as by word of mouth. Staff noted a longer-term concern with market saturation, that is, at some point, the program will have exhausted the supply of eligible customers willing to participate. Program satisfaction remained high with more than 90% of customers reporting that they were satisfied with the information provided about the program, the quality of the work performed, the energy savings realized, and the overall experience with the service. Planned Program Changes There are no planned program changes for the Home Weatherization Program. Energy Efficiency Programs 79

83 3.3 Energy Saving Products Program Program Overview PSO s Energy Saving Products (ESP) program seeks to generate energy and demand savings for residential customers through the promotion of LEDs, room air purifiers, advanced power strips, clothes dryers, bathroom ventilation fans, water dispensers and heat pump water heaters. The purpose of this program is to provide PSO residential customers inducements for purchasing products that meet high efficiency standards. The ESP upstream program consists of retail price discounts for qualifying LED light bulbs, room air purifiers and advanced power strips in The upstream program uses a price mark down mechanism where participating retailers advertise and offer discounted pricing for program sponsored products. The retailers/manufacturers are then reimbursed by PSO for the difference between the discounted price and the normal retail price. This program component also included distribution of free LEDs in partnership with food banks and local food pantries within the PSO service territory during PY2017. Discounted LED bulbs, including the free LEDs distributed through local food pantries, make up the vast majority (~99.0%) of the reported energy savings for the PY2017 ESP program. The downstream program provides mail-in rebates from PSO for qualifying clothes dryers, bathroom ventilation fans, water dispensers and heat pump water heaters in The actual number of participants in the ESP lighting component of the program is unknown, as light bulb, advanced power strips and room air purifier purchaser information is not tracked by participating retailers. In total, 375,846 packages of LEDs (65 packages of CFLs were sold in December 2016 but were part of the program in 2017). A total 1,467,303 individual bulbs were discounted through participating retailers or distributed in partnership with local food pantries. The total number of advanced power strips, room air purifiers, clothes dryers, bathroom ventilating fans, water dispensers and heat pump water heaters sold in the ESP program was 2,490. A total of 1,469,793 bulbs, advanced power strips, room air purifiers, clothes dryers, bathroom ventilating fans, water dispensers and heat pump water heaters were discounted through participating retailers or distributed in partnership with local food pantries. Table 3-36 provides a summary of program metrics for the 2017 program year. Program costs were $3,887,827, while reported kwh savings exceeded program projections. Gross verified energy savings developed through ADM s impact evaluation were slightly higher than reported savings, representing a gross realization rate of 115%. Verified peak demand reduction represents 116% of reported value. Energy Efficiency Programs 80

84 Table 3-36: Performance Metrics Energy Saving Products Program Metric PY2017 Number of Products 1,469,793 Budgeted Expenditures $3,687,883 Actual Expenditures $3,887,827 Energy Impacts (kwh) Projected Energy Savings 31,233,280 Reported Energy Savings 44,516,965 Gross Verified Energy Savings 51,024,824 Net Verified Energy Savings 32,569,214 Peak Demand Impacts (kw) Projected Peak Demand Savings 3,615 Reported Peak Demand Savings 7,297 Gross Verified Peak Demand Savings 8,491 Net Verified Peak Demand Savings 5,398 Benefit / Cost Ratios Total Resource Cost Test Ratio 7.37 Utility Cost Test Ratio 5.48 The remainder of this section details the EM&V methodologies and findings for the Energy Saving Products (ESP) program EM&V Methodologies The following section details the methodologies that ADM used to verify retail sales, estimate energy and peak demand impacts, and assess the performance for the Energy Saving Products program. Data Collection Several primary and secondary data sources were used for the evaluation. Tracking data and supporting documentation for the program was obtained from the Energy Federation, Inc. (EFI) database. This tracking data was used as the basis for quantifying participation Energy Efficiency Programs 81

85 and assessing program impacts. Supplemental tracking data was provided by CLEAResult which included the following information for each combination of retailer, model number, and discount level for upstream lighting: Package sales per week (program sales only) Original retail price Manufacturer/Retailer sponsored discounts (if any) PSO sponsored discounts Retail price, including all discounts Number of bulbs per package Rated wattage Rated lumens Rated lifetime in hours Additional documentation including retailer agreements, retailer/manufacturer invoices, promotional event documentation and general program materials were reviewed as part of the evaluation. Primary data collection activities included a general population telephone survey, interviews with program staff members and interviews with retailers. For the general population telephone survey, a newly acquired sample was collected from September - November The final sampling size for each primary data collection activity is presented in Table 3-37 below. Table 3-37: ESP Data Collection Activities Data Collection Activities N LED RDD Survey 152 Downstream Rebate Participant Surveys 14 Program Staff Interviews 1 There were two survey efforts conducted: A Random Digit Dialing (RDD) technique and a Downstream Rebate Participant Survey. For the RDD survey, residential customers within Oklahoma were contacted and interviewed about recent LED purchases. The RDD survey was organized by zip codes, targeting locations within specific areas of Oklahoma. Because customer contact information is not tracked for the LED markdowns, the RDD methodology provided a cost-effective way of reaching many potential program participants or representative consumers. In addition, both landlines and cellular phones were targeted in a breakdown of 50:50. Interviewers used screening questions to determine whether respondents were (a) a PSO or other electric utility Energy Efficiency Programs 82

86 customer, (b) recently purchased light bulbs and (c) that the respondent had a general understanding of different light bulb technologies. 29 In total, the RDD survey was completed by 152 PSO customers. Of the RDD respondents, 125 felt they could correctly identify different light bulb technologies. One-hundred and fifty-two (152) of these respondents had purchased light bulbs during the eight months prior to the survey. All 152 of the respondents indicated that they had purchased LEDs. For the downstream rebate participant survey, customers that purchased clothes dryers, water dispensers or bathroom ventilation fans though the PSO ESP program were contacted through an survey (only one customer purchased a heat pump water heater and it was after the survey collection period). Screening questions were asked to understand customer program awareness, participation process and customer satisfaction. In total, the downstream rebate participant survey was completed by 14 PSO customers that purchased rebated items in the downstream ESP program. These results have been consolidated in a separate memo, 2017 Process Evaluation Memo. To inform the process evaluation, ADM conducted in-depth interviews with program staff at PSO and implementation contractor CLEAResult. These interviews provide insight into various aspects of the program and its organization. The interviews focused on changes to the program that occurred during Interview respondents also discussed aspects of the program operations and results that they considered to be successful as well as challenges faced. These results have been consolidated in a separate memo, 2017 Process Evaluation Memo. Gross Impact Estimation Methodology: Upstream Program Lighting Only Reported energy and peak demand impacts for the program were calculated using deemed per-unit impacts from the Oklahoma Deemed Savings Documents. For LEDs, the deemed savings algorithms came from the 2013 updated Deemed Savings Documents, which reflect baseline bulb wattage changes resulting from the Energy Independence and Security Act of 2007 (EISA). ADM s evaluation consisted of (1) verifying the quantity of program eligible measures that were discounted in-store, (2) reviewing the assumptions and inputs associated with the deemed savings values (3) verifying that the deemed per-unit impacts were applied appropriately and (4) making appropriate adjustments for in-service rates, leakage, and cross sector sales. 29 Customers were asked if they felt they could correctly identify a typical incandescent bulb, a CFL, and an LED if all three were place in front of them. Energy Efficiency Programs 83

87 Verification For LED markdowns, ADM reviewed the program tracking database consisting of retailer transaction data. Important fields included: item description, number and type of package sold, bulbs per package, bulb lumens, bulb wattage, program and original retail pricing, retail location, and transaction period. This tracking data was compared to participating retailer/manufacturer invoices to verify the quantity of units sold and discounted through the program. The retailer/manufacturer invoices submitted to the program rebate processing center are based on actual sales transaction data from each retailer. Manufacturer invoices were also reviewed for the bulbs distributed through local food pantries. Calculation of Gross Annual kwh Savings Gross annual energy savings for discounted LEDs were calculated using the algorithm from the Oklahoma Deemed Savings Documents, except for the Hours of Use (HOU). For program year 2017, ADM had to determine the most appropriate method for calculating energy savings for residential lighting and the ESP program. Ideally, a metering study conducted in PSO territory would provide the best estimate for HOU, but the results of a metering study would not be available until In the absence of meter data, it is difficult to discredit or uniquely support the 2.8 HOU from the 2005 KEMA study in California or the 2.17 HOU from the 2014 Cadmus study in Arkansas. As an alternative, ADM determined the best approach was to review well-regarded and recent metering studies and calculate an unweighted average across these studies to reduce any possibility of bias. ADM calculated an unweighted average of 2.63 hours using the recent metering studies reported in The Uniform Methods Project (UMP) combined with four additional studies (2014 Cadmus study in Arkansas, the 2005 KEMA study in California, the 2014 GDS/Nexant metering study in Pennsylvania, and 2015 ADM LED study in Nevada) to calculate a robust HOU for use in PSO and OG&E territories for the 2017 program year. The savings algorithm is described in Appendix F. In-Service Rate Adjustments The cost-effectiveness testing for the program requires calculating lifetime energy savings for purchased LEDs. Less efficient incandescent and EISA compliant halogen bulbs typically have rated lifetimes considerably lower than LEDs. Additionally, calculating lifetime energy savings requires an estimate of when the newly purchased bulbs are installed. The Deemed Savings Documents stipulate an In-Service Rate (ISR) of 97%, but this reflects the percentage of bulbs estimated to be installed eventually. Previous studies have found that immediate or first-year installation rates are generally lower, as some bulbs are shelved for later use. Energy Efficiency Programs 84

88 To estimate a second-year ISR, ADM asked RDD survey respondents to estimate the number of purchased light bulbs they plan to install within one week and within one year. It was then assumed that the full ISR of 97% is achieved within three years. 30 The secondyear ISR is assumed to be the average of the first-year ISR and the full ISR, reflecting an assumed linear rate of installation. The ISR only affects first and second year savings as well as the discounting of energy and demand impacts for cost-effectiveness testing purposes. Annual savings estimates are unaffected. Leakage Adjustments Leakage refers to cross-territory sales that occur when program discounted bulbs are installed outside of PSO s service territory. When this occurs, the energy and demand impacts from the discounted bulbs are not being realized within the territory that paid for and claimed the savings. An estimate of leakage was calculated in PY2015 at 3.6% and this was used for PY2017. Cross Sector Sales Adjustments ADM used estimated annual hours of use (HOU) of (as described in Calculation of Gross Annual kwh Savings). This reflects an average daily HOU of 2.63 times days per year. While this is within the range of HOU estimates from previous studies 31 of residential lighting use, it likely underestimates HOU for bulbs that are installed in nonresidential buildings. The higher annual HOU for bulbs in non-residential savings implies a shorter expected useful life for the bulbs (in years). The time period in which the savings occur affects the applicable baseline wattage and discount factor for cost-effectiveness savings. ADM used responses from the RDD survey to estimate the percentage of purchased bulbs that are installed in non-residential facilities. For these bulbs, HOU were estimated to be 3,253 based on EUL stipulations from the OKDSD. A corresponding CF of 0.55 is assumed. This has the effect of increasing annual energy savings and peak demand reduction for the percentage of bulbs estimated to be installed in non-residential settings. Non-Lighting Measures Savings calculations for non-lighting measures are outlined in the sections below. The detailed algorithms can be found in Appendix F. 30 This three-year period for achieving the full ISR is recommended by the DOE Uniform Methods Project Residential Lighting Evaluation Protocol. 31 The DOE Uniform Methods Project Residential Lighting Evaluation Protocol summarizes nine recent studies with HOU estimates ranging from 1.5 to 2.98 hours per day. See: Energy Efficiency Programs 85

89 ADM s evaluation consisted of (1) verifying the quantity of program eligible measures that were discounted in-store, (2) reviewing the assumptions and inputs associated with the deemed savings values and (3) verifying that the deemed per-unit impacts were applied appropriately. Room Air Purifiers Deemed kwh and peak demand kw savings values for room air purifiers were unavailable in the OKDSD; however, the Illinois TRM v5.0 has established deemed kwh savings and peak kw demand values that were used for this analysis 32,33. Verification For room air purifiers, ADM reviewed the program tracking database consisting of retailer transaction data. Important fields included: item description, number and type of room air purifier sold, Dust CADR, program and original retail pricing, retail location, and transaction period. This tracking data was compared to participating retailer/manufacturer invoices to verify the quantity of units sold and discounted through the program. The retailer/manufacturer invoices submitted to the program rebate processing center are based on actual sales transaction data from each retailer. Calculation of Gross Peak Demand Reduction Gross annual energy savings for discounted room air purifiers were calculated using the algorithm from the IL TRM v5.0 and can be found in Appendix F. Advanced Power Strips Deemed kwh and peak demand kw savings values for advanced power strips (APS) were unavailable in the OKDSD; however, the Arkansas TRM v5.0 has established deemed kwh savings and peak kw demand values that were used for this analysis 34. Verification For APS, ADM reviewed the program tracking database consisting of retailer transaction data. Important fields included: item description, number of APS, program and original retail pricing, retail location, and transaction period. Calculation of Gross Annual kwh Savings The PSO ESP program provided rebates for Tier 1 APS only. Deemed savings were calculated for Tier 1 by average complete system as the type of installation was unknown. 32 Illinois TRM version 5.0, June 1, Calculation for kwh savings and peak kw demand are based on the Mid-Atlantic TRM version 4.0. This specifies baseline kwh/year consumption and ENERGY STAR kwh/year consumption based on the Clean Air Delivery Rate (CADR) for ENERGY STAR room air purifier. 34 Arkansas TRM v6.0, August 31, Energy Efficiency Programs 86

90 Gross Impact Estimation Methodology: Downstream program Clothes Dryers Deemed kwh and peak demand kw savings values for clothes dryers (CD) were unavailable in the OKDSD; however, the Illinois TRM v5.0 has established deemed kwh savings and peak kw demand values that will be used for this analysis 35,36. Verification For CDs, ADM reviewed the program tracking database consisting of retailer transaction data. Important fields included: item description, number and type of CD sold, dryer type, vented/ventless, voltage, drum size, automatic termination controls, program and original retail pricing, retail location, and transaction period. Water Dispensers Deemed kwh and peak demand kw savings values for water dispensers (WD) were unavailable in the OKDSD; however, the PA TRM has established deemed kwh savings and peak kw demand values that will be used for this analysis 37. Verification For WDs, ADM reviewed the program tracking database consisting of retailer transaction data. Important fields included: item description, number and type of WD sold, type of storage, program and original retail pricing, retail location and transaction period. Bathroom Ventilation Fan Deemed kwh and peak demand kw savings values for bathroom ventilation fan (BVF) were unavailable in the OKDSD; however, the IL TRM v5.0 has established deemed kwh savings and peak kw demand values that will be used for this analysis 38. Verification For BVFs, ADM reviewed the program tracking database consisting of retailer transaction data. Important fields included: item description, number of BVFs sold, program and original retail pricing, retail location and transaction period. 35 Illinois TRM version 5.0, June 1, Calculation for kwh savings and peak kw demand are based on the Mid-Atlantic TRM version 4.0. This specifies baseline kwh/year consumption and ENERGY STAR kwh/year consumption based on the Clean Air Delivery Rate (CADR) for ENERGY STAR room air purifier. 37 Pennsylvania TRM June Illinois TRM v5.0 June 1, Energy Efficiency Programs 87

91 Heat Pump Water Heaters ADM checked HPWH model numbers listed in the program tracking system against ENERGY STAR databases to verify that each HPWH distributed in 2017 was ENERGY STAR certified and assigned the correct capacity and efficiency ratings. Deemed kwh savings values for HPWH were unavailable in the OKDSD; however, they were available in the Arkansas TRM v5.0. The variables that affect deemed savings are the following: storage tank volume, HPWH Energy Factor (EF), HPWH installation location (conditioned vs. unconditioned space) and weather zone. Weather zones were based on established zones in Arkansas. Similar weather zones have been established in Oklahoma that are commiserate with the numbered weather zones in Arkansas Net-to-Gross Estimation Lighting Program lighting measures were separated into two categories for NTG estimation. For LEDs distributed through local food pantries, the NTG ratio is assumed to be 100%. For the 25,000 LED packages (100,000 bulbs) distributed through local food banks, the 100% net-to-gross ratio is assumed because customers do not shop for the lighting products but rather are simply offered LEDs without prompting. Individuals who received LEDs through the food banks are also more likely to represent low income customers, potentially limiting their ability or willingness to purchase high efficiency lighting products. Overall, the LEDs giveaways represent less than 6.82% of reported energy savings from the ESP program lighting component. For LEDs discounted at participating retail stores, ADM estimated free-ridership as described throughout the rest of this section. Determining the net effects of the in-store retail discounts requires estimating the percentage of energy savings from efficient lighting purchases that would have occurred without program intervention. Ideally, participating retailers could provide light bulb sales data for non-program time periods and/or from similar non-program retail locations. This data would provide adequate information from which to calculate the lift in LED sales attributable to the program price markdowns. However, retailers are reluctant to release sales data for this purpose and non-program sales data was not made available to ADM. As a result, evaluating the net effects of the price discounts requires estimating free ridership without non-program sales data. Several methodologies have been used in similar evaluations across the country, all of which have certain advantages and disadvantages. For this evaluation of the PY2017 ESP program lighting component, ADM developed two separate estimates of free ridership, each using a different methodology. Energy Efficiency Programs 88

92 Table 3-38 provides a summary of the methodologies and their relative advantages and disadvantages. Details regarding each methodology follow below. Table 3-38: Free Ridership Estimation Methodologies Advantages and Disadvantages Methodology Advantages Disadvantages RDD General Population Survey Allows for a more truly random sample than intercept surveys. Allows for discussion of bulbs postinstallation. Large sample size more costeffective than intercept surveys. Relies on customer self-reporting of purchase decision making. Potential for recall bias is higher than intercept surveys (discuss purchases over past six months). This may also affect whether the respondent purchased program bulbs. Potential for bias in scoring algorithm. Consumer Demand Model Estimate is developed from actual sales data, eliminating potential biases that customer self-report data can exhibit. The model is estimated using program sales data only. While the model may fit program sales data well, it is possible that it does not predict sales levels at non-program prices well. Survey Based Methodology The first methodology is based on self-report surveys with a sample of customers aimed at understanding decision making for light bulb purchases. The goal of these surveys is to elicit information from which to estimate the number of bulbs that the customers would have purchased in the counterfactual scenario where LEDs were not discounted. Selfreport survey methods for determining free ridership are generally recognized as susceptible to certain biases and error. This may be especially true for upstream price markdown programs, where the counterfactual scenario of regular retail prices may be difficult to explain or grasp. The self-report methodologies also rely on specific scoring algorithms, which may bias the free ridership estimates if they do not accurately reflect the customer decision making process. This evaluation relies on self-report survey data from two surveying efforts: The survey-based effort for calculating free ridership was conducted using a Random Digit Dial (RDD) technique. The strength of this approach is the ability to obtain a random and relatively large sample size cost-effectively. It also allows for further questioning regarding the fate of recently purchased bulbs (e.g., installed immediately, stored for future use, location of installation, etc.). The biggest drawback to the approach is the potential for respondent recall bias. It may be difficult to get accurate responses to questions about the number of bulbs the Energy Efficiency Programs 89

93 respondent recently purchased and whether they were discounted through the program, for example. Survey respondents were asked a series of questions to elicit feedback regarding influences on their light bulb purchasing decisions. Each respondent was then assigned a free ridership score based on a consistent free ridership scoring algorithm. The free ridership scoring algorithm for the RDD surveys is shown in Figure 3-7. The behavior without discount scoring is the primary determinate of respondents free ridership scores. This section asked whether the respondent would have purchased the same light bulbs if they had cost the regular retail price. This may be a question that is particularly prone to social desirability bias the tendency to respond in a manner that might be viewed favorably by others. For this reason, a consistency check was performed. For the RDD survey, a consistency check was performed by asking each respondent to state light bulb characteristics that are important to them when choosing between available options. If a respondent listed price as the most important characteristic, but then went on to indicate that they would have still purchased efficient options at full retail price, a 50% reduction to the behavior without discount score was applied. For the RDD survey, responses were not weighted. That is, each response had equal weight in estimating the average free ridership level for the program. Energy Efficiency Programs 90

94 Figure 3-7: Free Ridership Scoring for RDD Survey Respondents Non-Survey Based Methodology The second estimate of free ridership was developed through the estimation of a price response model which was used to predict sales levels in the absence of the program. The program tracking data included package and bulb sales for each retailer, by model number and week. 39 For each retailer and model number combination, original retail price and program price data were available. As program price discounts and/or retailer original pricing changed throughout the year, the tracking data was updated, allowing for the comparison of same-bulb sales under slightly different pricing conditions. Price effects are the main program tool for encouraging the purchase of high efficiency lighting choices. However, there are also regular promotional events sponsored by PSO within participating retail locations. The dates, location, and duration of in-store promotional events were also tracked, allowing for estimation of their effects on sales levels as well. 39 The majority of bulb sales were recorded on a weekly basis. However, some retailer/manufacturer partners reported bulb sales bi-weekly or monthly. In order to produce weekly sales estimates for these bulbs, the bi-weekly sales were divided by two and monthly sales were divided by four. While this may not be entirely accurate over a given timespan, it is a reasonable assumption in the absence of weekly data. Energy Efficiency Programs 91

95 The final price response model is used to estimate a free ridership as described in the equation below: Where: Free ridership ratio = n i (E[Bulbs NoProgram i ] kwh i ) n(e[bulbs Programi ] kwh i ) i Equation 3-2: Estimation of Free Ridership E[Bulbs NoProgrami ] = the expected number of bulbs of type, i, purchased given original retail pricing (as predicted by the model). E[Bulbs Programi ] = the expected number of bulbs of type, i, given program discounted pricing (as predicted by the model). kwhi = the average gross kwh savings for bulb type, i. The price response modeling approach is advantageous in that it is built upon actual sales data from participating retailers (as opposed to relying on consumer self-report surveys). There are; however, a number of limitations for the approach. Most importantly, nonprogram sales data is unavailable for inclusion in the model. As a result, the modeling of price impacts may fit program sales data well, but it is uncertain whether those price effects apply well to prices outside of program ranges. Additionally, the lack of nonprogram sales data means that for many bulb types and time ranges, the available sales data lists zero sales. These zeroes in most cases do not actually represent zero sales, but rather a lack of information because program pricing was not in effect for a given bulb during a given week. This presents a challenge in modeling the sales data using typical time-series or panel data methods. Additionally, during the sales period analyzed there was only pricing variation for a subset of bulb models, limiting the ability of the model to predict price response effects in a robust manner. Finally, there are likely variables that affect sales levels for LEDs that are not captured by the program tracking data; thus, there is a risk of omitted variable bias in addition to the inherent amount of error from statistical modeling. Appendix H provides further technical details regarding the price response model development and results. Spillover and Market Effects It is worth noting that none of the methodologies used to estimate program free ridership include estimates of spillover or market effects. Spillover refers to savings that occur as a result of program influences on customers but for which an incentive or rebate is not given. In the context of a program for LED price markdowns, the following examples illustrate potential sources of spillover: Energy Efficiency Programs 92

96 Participant spillover: a customer who purchases program discounted bulbs is influenced to install additional (non-rebated) energy efficiency measures or change their energy usage behavior as a result of their program experience. Nonparticipant spillover: a customer notices PSO sponsored discounts or receives educational resources from an in-store promotional event. While they do not ultimately purchase program discounted bulbs, their interaction with the program encourages them to install other (non-rebated) energy efficiency measures or change their energy usage behavior. Market effects refer to changes in market structure or market actor behavior due to program influence that results in non-incented adoption of energy efficiency measures. In the context of a program for LED price markdowns, the following examples illustrate potential sources of market effects: Market pricing related effects: it is possible that the program sponsored discounts for certain lighting products cause downward pressure on prices for competing products (non-program bulbs). The competing products could potentially be LEDs at participating retailers or non-participating retailers. If pricing for these competing products is lowered in response to program discounts and a corresponding increase in purchases (and installations) occurs, then there may be additional savings attributable to program influences. Market manufacturing/stocking effects: it is possible that the program sponsored inducements cause bulb manufacturers and retailers to adjust their lighting product offerings. To the extent that the program causes lesser efficiency bulbs to be displaced with higher efficiency bulbs at the manufacturer/retailer level, there may be additional savings attributable to program influences. It is likely that some combination of these effects increase the savings attributable to the ESP lighting portion of the program. However, there is also reason to believe these effects may be small overall. Participant and non-participant spillover typically occurs through customer education. The ESP program component does include regular in-store promotional/educational event, but the number of customers reached relative to overall program sales is likely small. Additionally, the promotional events usually provide information designed to encourage customers to participate in other PSO energy efficiency programs, which would not constitute spillover if these customers ultimately did participate and receive a rebate. The implementor s field team educates customers regarding the incentives provided in the PSO ESP program; however, these are not explicitly quantified and therefore cannot provide reliable estimates of spillover. Market effects may exist to some extent but disaggregating the PSO program influences from other influences such as technological advances and other lighting discount programs across the country is difficult. The current ESP program component covers a Energy Efficiency Programs 93

97 substantial share of the bulbs sold in the PSO service territory, with no immediate plans for discontinuing the price markdowns. Overall, it should be noted that spillover and market effects likely remain a minor factor, and the net-to-gross estimate developed in this evaluation should be considered with these omitted effects in mind. APS, CDs, WDs, BVFs, RAPs and HPWHs PY2017 was the first year that the APS upstream measure and CD, WD, BVF and HPWH downstream measures were a part of the ESP program. As they made up a significantly smaller percentage of energy and demand savings from the ESP program and PSO s energy efficiency portfolio, NTG values were applied based on previously stipulated NTG ratios, with the exception of CDs. For CDs and RAPs, a meta-analysis was performed to calculate an average NTG ratio across similar programs nationwide and/or with similar rebates. There is limited information related to NTG estimates for APS without direct installation (i.e. sold through an upstream incentive); therefore, a net-to-gross ratio of 0.7 was applied. An ISR of 50% was assigned to APS to adjust gross savings. The reason for the adjustment is that most people do not install and utilize APS correctly, particularly, as an upstream measure. Ameren Missouri RebateSavers Impact and Process Evaluation: Program Year 2013 found that the installation rate of APS was 48% and customers were confused how to use them correctly 40. Since the APS were sold as an upstream measure in the ESP program, ISR was not possible to determine and so was set at 50%. ADM performed a meta-analysis on reported net-to-gross ratios for clothes dryers across different utility programs that sold electric clothes dryers through energy efficiency programs. Based on this meta-analysis, an average net-to-gross ratio of 0.66 was calculated and was used in PY2017 as shown in Table Table 3-39: Meta-analysis of Net-to-Gross estimates for Clothes Dryers Utility Report Date NTGR Penelec First Energy 11/23/ EmPower Maryland 9/15/ Rhode Island TRM PY ComEd 12/9/ Average NTGR Energy Efficiency Programs 94

98 Based on ComEd s PY10 net-to-gross recommendations, the suggested net-to-gross ratio for WD is based on a participant self-report survey in PY8. Additionally, based on ComEd s PY10 net-to-gross recommendations, the suggested net-to-gross ratio for BVFs is based on a participant self-report survey in PY8. For RAPs, ADM performed a meta-analysis on reported net-to-gross ratios. All of these were downstream rebate programs, but three of four of them reported the same rebate as PSO ($50). Based on this meta-analysis, an average net-to-gross ratio of 0.69 was calculated and was used in PY2017 as shown in Table Table 3-40: Meta-analysis of Net-to-Gross estimates for Room Air Purifiers Utility Incentive Amount NTGR Ameren Missouri $50 1 ComEd $ Efficiency Maine Appliance Rebate Program $ Ameren Illinois $ Average NTGR Impact Evaluation Findings Gross Energy and Peak Demand Impacts: Lighting Only The Energy Federation Inc. (EFI) tracking data for the ESP program lighting component identified a total of 375,846 packages of LEDs (65 packages of CFLs were sold as part of the PY2017 program in July 2016, but were not paid until PY2017) were discounted through participating retail stores. An additional 25,000 packages of LEDs were distributed free-of-charge through local food pantries. Table 3-41 shows the reported quantities and impacts of measures discounted or distributed free-of-charge through the ESP program during PY ngs/initial_recommendations/initial_comed_ntg_history_and_py10_recommendations_ xlsx 42 ngs/initial_recommendations/initial_comed_ntg_history_and_py10_recommendations_ xlsx Energy Efficiency Programs 95

99 Distribution Type Retail Discounts Table 3-41: Reported Measure Quantities and Impacts Lighting Only Measure Type Package Quantity Bulb Quantity Reported kwh Reported kw Standard CFL , Omni-directional LED 257,643 1,129,879 34,290,070 5, Directional LED 84, ,262 9,073,798 1, Discount1 Omni-directional LED 9,024 18, , Food Pantries Omni-directional LED 25, ,000 2,996, Verification Total 375,846 1,467,303 43,910,052 7,224,47 To verify the types and quantities of distributed measures, ADM performed a census review of all retailer/manufacturer invoices for LED sales. This review verified that the reported quantity of light bulbs sold through retail stores and distributed free-of-charge through local food pantries matched exactly with the invoices that PSO paid. ADM also reviewed the program tracking database to determine if energy and demand impacts were correctly calculated according to the Oklahoma Deemed Savings Document algorithms for each LED type. For PY 2017, ADM calculated verified energy and demand impacts based on OKDSD but used an adjusted HOU (960.61). ADM found that for the majority of light bulbs, reported impacts were calculated in accordance with the deemed savings algorithms. Each program eligible bulb was checked to determine the correct bulb wattage and ensure the correct lumen output and baseline wattage was applied. The discrepancies identified through the database review required adjustment for the actual wattages and/or baseline wattages used in the calculation of energy and demand impacts for some bulbs. In addition, there were some incorrect calculations of kwh and kw savings in the program tracking data. Table 3-42 shows each of these adjustments, which cumulatively decreased gross energy savings from program discounted light bulbs by 80,038 kwh. Energy Efficiency Programs 96

100 Table 3-42: Gross kwh Savings Adjustments Lighting Only Measure Type 6.1 W SPEC LED Retailer DIY 1 Quantity of Bulbs Reported kwh Adjustment to Verified kwh ,230 1, W SPEC LED 104 3, W SPEC LED 1,080 45,200 9, W SPEC LED W STD LED 72 1, DISCOUNT W SPEC LED 128 6, W STD LED 168 5, W STD LED 98 2, W SPEC LED Mass Merch , Reason for Adjustment Two bulb models, wattage specified as 6.1W when they should have been 6 and 8W. Baseline wattage for one model number was also incorrectly specified as 45W when it should have been 65W. For one bulb model, the wattage was specified as 7.2W when it should have been 7W. For one bulb model, the wattage was specified as 8.5W when it should have been 8W. For one bulb model, the wattage was specified as 4.5W when it should have been 4W. The baseline wattage was also incorrectly specified as 45W when it should have been 25W. For three bulb models, the wattages were specified as 7.5W when they all should have been 5.5W. For three bulb models, the wattages were specified as 8.5W when they should have been 8W, 9.5W and 9.5W. For two bulb models, the wattages were specified as 9.2W when they should have been 9.5W. For one bulb model, the wattage was specified as 9.8W when it should have been 9.5W. One bulb model, wattage specified as 13W when it should have been 12W. Energy Efficiency Programs 97

101 Measure Type 3.2 W SPEC LED Retailer DIY 2 Quantity of Bulbs Reported kwh Adjustment to Verified kwh 210 4, W SPEC LED W STD LED 4,660 95,875 1, W STD LED 5, ,669 4, W STD LED ,134 9, W SPEC LED W STD LED , W STD LED , W STD LED 1,274 64, W STD LED , W SPEC LED 56 3,662 2 Reason for Adjustment For one bulb model, the wattage specified as 3.2W when it should have been 3.1W. For one bulb model, the baseline wattage was specified as 29W when it should have been 45W. For one bulb model, the wattage was specified as 6W when it should have been 5.6W. For one bulb model, the wattage was specified as 6.5W when it should have been 5.6W. For two bulb models, the wattages were specified as 8.5W when they should have been 13.5 and 14.5W. The baseline wattages were also incorrectly specified as 43W when they should have been 72W. For three bulb models, the baseline wattages were specified for all three models as 55W when it should have been 65W. For one bulb model, the wattage was specified as 11.8W when it should have been 12.5W. For one bulb model, the wattage was specified as 15.5W when it should have been 15W. For one bulb model, the wattage was specified as 15.6W when it should have been 15W. For one bulb model, the baseline wattage was specified as 43W when it should have been 72W. For one bulb model, the baseline wattage was specified as 70W when it should have been 90W. Energy Efficiency Programs 98

102 Measure Type 4 W SPEC LED 4 W SPEC LED Retailer MASS MERCH 2 MASS MERCH 3 Quantity of Bulbs Reported kwh Adjustment to Verified kwh 3, ,651-67, , W STD LED 79 3, W SPEC LED W SPEC LED 423 7, W SPEC LED 89 1, W SPEC LED MASS MERCH 4 6 W STD LED 1,968 82,163-41,676 7 W SPEC LED , W SPEC LED 49 1, Reason for Adjustment For one bulb model, the wattage was specified as 4W when it should have been 4.2W. For one bulb model, the wattage was specified as 4W when it should have been 4.2W. For one bulb model, the wattage was specified as 10W when it should have been 16.8W. For one bulb model, the wattage was specified as 3.5W when it should have been 4W. For three bulb models, the wattages were specified as 4W when they should have been 4.2W. For one bulb model, the wattage was specified as 4.5W when it should have been 2.8W. For one bulb model, the baseline wattage was specified as 45W when it should have been 25W. For one bulb model, the baseline wattage was specified as 72W when it should have been 29W. For two bulb models, the wattages were specified as 7W when they should have been 6.5W. For a different bulb model, the baseline wattage was specified as 45W when it should have been 72W. For one bulb model, the wattage was specified as 7.5W when it should have been 7W. Energy Efficiency Programs 99

103 Measure Type 12 W SPEC LED Retailer Quantity of Bulbs Reported kwh Adjustment to Verified kwh Reason for Adjustment For two bulb models, the baseline wattages were specified as 55W when MASS they should have been 166 8, MERCH 4 65W. For one bulb model, the baseline wattage was specified as 55W when it should have been 72W. Total 23, ,435-80,038 Cumulative Effect The same discrepancies identified in Table 3-42 also affected peak demand reduction for light bulbs distributed through the program, resulting in a decrease of kw. After verifying that energy and demand impacts were calculated as stipulated in the OKDSD without an adjusted HOU, ADM calculated verified kwh and peak demand kw savings using the OKDSD with the adjusted HOU (960.61), and this is what is presented from this point forward. In-Service Rate Adjustments The 152 survey respondents purchased a total of 1,441 LED bulbs. These respondents indicated that 69.0% of the bulbs would be installed within a week, and ~94.9% of LEDs would be installed within a year. For the purpose of calculating program cost effectiveness, an average of the first-year ISR of 85% and the full year ISR of 97% was assumed (91%). This does not affect annual kwh savings estimates, as is assumed that 97% of the bulbs are installed within three years based on the stipulations in the deemed savings documents. 43 Leakage Adjustments ADM estimated leakage of light bulbs outside of PSO territory in PY2015. A total of 3.6% of bulbs leaked out of PSO territory into the Oklahoma and the surrounding states. The leakage estimate of 3.6% was used in PY2017. The verified gross savings developed by ADM include an adjustment for leakage using 3.6% as calculated in PY2015. This results in a decrease of 1,718,599 kwh for annual energy savings and kw for peak demand reduction. 43 Calculating cost-effectiveness requires an estimation of when the bulbs are installed in order to correctly discount future year savings. The cost-effectiveness estimates for the ESP program presented in this report assume that 85% of the bulbs are installed within the first year. By the third year, it is assumed that 97% of bulbs are installed, based on the deemed savings document. For the second year, 91% are assumed to be installed (a linear interpolation of years one and two). Energy Efficiency Programs 100

104 Cross Sector Sales Adjustments An adjustment to gross impacts was made to account for the proportion of program bulbs estimated to be installed in non-residential settings, where HOU and CF are typically higher than residential sockets. The general population RDD survey included a question related to cross sector sales. Respondents who indicated they had purchased LEDs in the past eight months were asked: We re any of the LEDs you purchased in the past eight months installed in a business or commercial setting? Of the 152 LED purchasers, just eight indicated that they installed bulbs in a non-residential setting. 44 The resulting non-residential allocation is therefore 5.3%. The result from the RDD survey is within the range of values that previous evaluations of residential lighting mark down programs have estimated. A recent meta-analysis of 23 recent evaluations of a similar nature found estimates ranging from 0.0% to 18.7%, with various methodologies used. 45 The average non-residential allocation estimate from these studies was 6.7%. Eight respondents installed light bulbs in a commercial setting, which is too small of a sample size to accurately calculate cross-sector sales estimates. ADM utilized the 5.0% non-residential allocation estimate from the average of the intercept and RDD surveys from PY2015. The verified gross savings developed by ADM included an adjustment for cross sector sales by using 3,253 HOU and a 0.55 CF for 5.0% of the bulbs (rather than the adjusted HOU of and 0.09 CF for residential installations). This resulted in an increase of 3,582,441 kwh for annual energy savings and 1, kw for peak demand reduction. Final Verified Gross Savings Estimates After considering leakage and cross-sector sales adjustments annual energy savings for the ESP program were estimated to be 50,489,094kWh. Verified peak demand savings were 8, kw. Table 3-43 compares reported and verified impact estimates for this program component. 44 These two respondents were further asked how many of the CFLs and/or LEDs they purchased were installed in non-residential settings. A sample size of two is too small to accurately calculate a cross sector sales estimate based on the number of bulbs Memo.pdf Energy Efficiency Programs 101

105 Table 3-43: ESP Program Impact Findings Lighting Only Distribution Measure Reported Verified Reported Verified Verified Quantity Type Type kwh kwh kw kw Standard CFL 114 5,354 5, Omnidirectional Retail Discounts LED 1,129,879 34,290,070 37,117,205 5, , Directional LED 219,262 9,073,798 9,782,561 1, , Omnidirectional Discount1 LED 18, , , Omnidirectional Food Pantries LED 100,000 2,996,600 2,996, Total 1,467,303 46,906,652 50,489,094 7, , Gross Energy and Peak Demand Impacts: Room Air Purifiers (RAPs) Only The Energy Federation Inc. (EFI) tracking data for the ESP program RAPs component identified a total of 633 RAPs sold at participating retail stores. Table 3-44 shows the reported quantities and impacts of RAPs through the ESP program during PY2017. Table 3-44: Reported Measure Quantities and Impacts RAPs Only Distribution Type Measure Type Total Quantity Reported kwh Reported kw Retail Discounts RAPs , Verification Total , To verify the types and quantities of distributed measures, ADM performed a census review of all retailer/manufacturer invoices for RAP sales. This review verified that the reported quantity of RAPs sold through retail stores matched exactly with the invoices that PSO paid. The invoice did; however, find a few small discrepancies relating to Dust CADR and associated kwh and peak demand kw savings. Energy savings for several RAPs were incorrectly assigned kwh and kw savings based on Dust CADR assigned in the program tracking as seen below in Table The adjustment to verified kwh savings was an increase of 24,403 kwh. Energy Efficiency Programs 102

106 Measure Type 83 Dust CADR 100 Dust CADR 106 Dust CADR 118 Dust CADR 190 Dust CADR 200 Dust CADR 300 Dust CADR Table 3-45: Gross kwh Savings Adjustments RAPs Only Retailer DIY 2 DIY 3 Quantity Reported kwh Adjustment to Verified kwh 13 2, ,498 1, ,807 1, ,611 3, ,247 4, ,253 5, ,519 6,523 Reason for Adjustment These 10 room air purifiers were assigned an incorrect value of 202 when it should be 293 kwh savings. These 15 room air purifiers were assigned an incorrect value of 202 when it should be 293 kwh savings. These 10 room air purifiers were assigned an incorrect value of 336 when it should be 488 kwh savings. These 26 room air purifiers were assigned an incorrect value of 336 when it should be 488 kwh savings. These 22 room air purifiers were assigned an incorrect value of 471 when it should be 683 kwh savings. These 26 room air purifiers were assigned an incorrect value of 471 when it should be 683 kwh savings. These 22 room air purifiers were assigned an incorrect value of 807 when it should be 1169 kwh savings. Total ,836 24,402 Cumulative Effect The same discrepancies identified in Table 3-45 also affected peak demand reduction for RAPs distributed through the program, resulting in an increase of 3.02 kw. There were no other adjustments specific to gross peak demand reduction estimation. Final Verified Gross Savings Estimates Table 3-46 compares reported and verified impact estimates for this program component. Table 3-46: ESP Program Impact Findings RAPs Only Distribution Type Retail Discounts Measure Type Verified Quantity Reported kwh Verified kwh Reported kw Verified kw RAPs , , Total , , Energy Efficiency Programs 103

107 Gross Energy and Peak Demand Impacts: Advanced Power Strips (APS) Only The Energy Federation Inc. (EFI) tracking data for the ESP program APS identified a total of 1,629 APS sold at participating retail stores. Table 3-47 shows the reported quantities and impacts of APS through the ESP program during PY2017. Table 3-47: Reported Measure Quantities and Impacts APS Only Distribution Type Measure Type Total Quantity Reported kwh Reported kw Retail Discounts APS 1, , Verification Total 1, , To verify the types, quantities and savings associated with distributed measures, ADM performed a census review of the program tracking data for APS sold through the program. This review found that all APS were assigned the correct kwh and kw savings based on Tier 1 APS specification in the AR TRM v5.0. The APS were sold as an upstream component, making it difficult to assess whether customers were installing APS correctly. To account for this, ADM applied an ISR of 0.5. The review of the program tracking data for APS; however, found a number of discrepancies for 243 APS from a single invoice. These were incorrectly assigned a kwh and kw savings of 0 as seen below in Table Measure Type Tier 1 APS Table 3-48: Gross kwh Savings Adjustments APS Only Retailer Quantity Reported kwh Adjustment to Verified kwh Reason for Adjustment These 243 APS were assigned DISCOUNT an incorrect value of 0 kwh ,339 2 when they should be kwh savings each. Total , Cumulative Effect Final Verified Gross Savings Estimates Table 3-49 compares reported and verified impact estimates for this program component. Table 3-49: ESP Program Impact Findings APS Only Distribution Type Retail Discounts Measure Type Verified Quantity Reported kwh Verified kwh Reported kw Verified kw APS 1, , , Total 1, , , This adjustment to verified kwh takes into account an ISR of 0.5. Energy Efficiency Programs 104

108 Gross Energy and Peak Demand Impacts: Clothes Dryers (CDs) Only The Energy Federation Inc. (EFI) tracking data for the ESP program CDs identified a total of 169 CDs sold at participating retail stores. Table 3-50 shows the reported quantities and impacts of CDs through the ESP program during PY2017. Table 3-50: Reported Measure Quantities and Impacts CDs Only Distribution Type Measure Type Total Quantity Reported kwh Reported kw Retail Discounts CDs , Verification Total , To verify the types, quantities and savings associated with distributed measures, ADM performed a census review of the program tracking data for CDs sold through the program. This review found that all CDs were assigned the correct kwh and kw savings based on CD specification in the IL TRM. Final Verified Gross Savings Estimates Table 3-51 compares reported and verified impact estimates for this program component. Distribution Type Retail Discounts Table 3-51: ESP Program Impact Findings CDs Only Measure Type Verified Quantity Reported kwh Verified kwh Reported kw Verified kw CDs ,040 27, Total ,040 27, Gross Energy and Peak Demand Impacts: Water Dispensers (WDs) Only The Energy Federation Inc. (EFI) tracking data for the ESP program WDs identified a total of 42 WDs sold at participating retail stores. Table 3-52 shows the reported quantities and impacts of WDs through the ESP program during PY2017. Table 3-52: Reported Measure Quantities and Impacts WDs Only Distribution Type Measure Type Total Quantity Reported kwh Reported kw Retail Discounts WDs 42 20, Verification Total 42 20, To verify the types, quantities and savings associated with distributed measures, ADM performed a census review of the program tracking data for WDs sold through the program. This review found that all WDs were assigned the correct kwh and kw savings based on WD specification in the PA TRM. Energy Efficiency Programs 105

109 Final Verified Gross Savings Estimates Table 3-53 compares reported and verified impact estimates for this program component. Distribution Type Retail Discounts Table 3-53: ESP Program Impact Findings WDs Only Measure Type Verified Quantity Reported kwh Verified kwh Reported kw Verified kw WDs 42 20,236 20, Total 42 20,236 20, Gross Energy and Peak Demand Impacts: Bathroom Ventilating Fans (BVFs) Only The Energy Federation Inc. (EFI) tracking data for the ESP program BVFs identified a total of 16 WDs sold at participating retail stores. Table 3-54 shows the reported quantities and impacts of BVFs through the ESP program during PY2017. Table 3-54: Reported Measure Quantities and Impacts BVFs Only Distribution Type Measure Type Total Quantity Reported kwh Reported kw Retail Discounts BVFs 16 1, Verification Total 16 1, To verify the types, quantities and savings associated with distributed measures, ADM performed a census review of the program tracking data for BVFs sold through the program. This review found that all BVFs were assigned the correct kwh and kw savings based on BVF specification in the IL TRM. There were two BVFs unaccounted for in the program tracking data as seen below in Table The adjustment to verified kwh savings was an increase of 180 kwh. Measure Type BVF Table 3-55: Gross kwh Savings Adjustments BVFs Only Retailer Quantity Reported kwh Adjustment to Verified kwh DIY Reason for Adjustment This one BVF was not accounted for in the program tracking data. DIY This one BVF was not accounted for in the program tracking data. Total Cumulative Effect The same discrepancies identified in Table 3-55 also affected peak demand reduction for BVFs distributed through the program, resulting in an increase of 0.02 kw. There were no other adjustments specific to gross peak demand reduction estimation. Energy Efficiency Programs 106

110 Final Verified Gross Savings Estimates Table 3-56 compares reported and verified impact estimates for this program component. Distribution Type Retail Discounts Table 3-56: ESP Program Impact Findings BVFs Only Measure Type Verified Quantity Reported kwh Verified kwh Reported kw Verified kw BVFs 16 1,240 1, Total 16 1,240 1, Gross Energy and Peak Demand Impacts: Heat Pump Water Heaters (HPWH) Only The Energy Federation Inc. (EFI) tracking data for the ESP program HPWHs identified a total of 1 HPWHs sold at participating retail stores. Table 3-57 shows the reported quantities and impacts of HPWHs through the ESP program during PY2017. Table 3-57: Reported Measure Quantities and Impacts HPWHs Only Distribution Type Measure Type Total Quantity Reported kwh Reported kw Retail Discounts HPWHs 1 1, Verification Total 1 1, To verify the types, quantities and savings associated with distributed measures, ADM performed a census review of the program tracking data for HPWHs sold through the program. This review found several discrepancies based on an incorrectly assigned weather zone which are noted below in Table The adjustment to verified kwh savings was a decrease of 95 kwh. In addition, as no information was recorded regarding where the HPWH was installed in the home (conditioned vs. unconditioned space) or the furnace type, unconditioned space and unknown furnace type values were stipulated from the AR TRM v5. Energy Efficiency Programs 107

111 Measure Type Table 3-58: Gross kwh Savings Adjustments HPWHs Only Retailer Quantity Reported kwh Adjustment to Verified kwh Reason for Adjustment HPWH DIY 3 1 1, This one HPWH was incorrectly assigned to weather zone 8, when the customer address was in zone 7. The estimated annual hot water use was also incorrectly assigned based on the incorrect weather zone (20,831 when it should have been 20,758). The average water supply temperature was incorrectly assigned due to the weather zone (66.1º F instead of 67.8 º F). The performance adjustment was incorrectly assigned due to the weather zone ( instead of ) Total 1 1, Cumulative Effect The same discrepancies identified in Table 3-45 also affected peak demand reduction for HPWHs distributed through the program, resulting in a decrease of 0.01 kw. There were no other adjustments specific to gross peak demand reduction estimation. Final Verified Gross Savings Estimates Table 3-59 compares reported and verified impact estimates for this program component. Distribution Type Retail Discounts Table 3-59: ESP Program Impact Findings HPWHs Only Measure Type Verified Quantity Reported kwh Verified kwh Reported kw Verified kw HPWHs 1 1,851 1, Total 1 1,851 1, Net-to-Gross Estimation Results Lighting Only The NTG analysis for the ESP program was conducted using the methodologies outlined in Section The results of this analysis are summarized below. Random Digit Dialing Free Ridership Estimate The second self-report survey used to estimate free ridership was conducted using Random Digit Dialing (RDD) to reach the general population of Oklahoma residents (PSO and non-pso service territory). A total of 152 respondents completed the RDD survey from September 2017 December 5, 2017 (100% of whom reported being PSO electric utility customers). Of those 152 survey respondents, 152 had purchased LEDs in the prior 8 months and these respondents were used to calculate free ridership as shown in Figure Energy Efficiency Programs 108

112 3-7 and Table Zip codes were cross-checked to confirm that respondents reported utility company was accurate based on PSO service territories. Table 3-60 below shows the results of this free ridership calculation by the type of bulbs respondents claimed to have recently purchased. Respondent Type Table 3-60: RDD Survey Free Ridership Estimate N Prior Experience Score Behavior w/o Program Score Free Ridership Estimate Mitigating Factor LED Purchasers Total The average free ridership score for all 152 respondents was 39.1%. This is 0.4% lower than the free ridership level estimated from the RDD survey in PY2016. One component of the RDD survey free ridership scoring algorithm that is somewhat subjective has to do with the behavior without discount category. Survey respondents were asked whether they would have still purchased the same program bulbs if they had been at regular retail price (the retail price was given to the respondent in the form of $X more ). For PY2017, an additional category was added to these questions to assign equal free ridership to each category using the principle of equal distribution of ignorance. The idea behind this principle is that we do not apriori know what percentage of free ridership should be assigned to any given category; therefore, we equally divide free ridership amongst all categories (Figure 3-7). Those who responded probably were assigned a free ridership score of 0.75 for that category, unless they also said they would have purchased fewer bulbs. Those who responded not sure were assigned a free ridership score of 0.5 for that category unless they also said they would have purchased fewer bulbs. Those who responded probably not were assigned a free ridership score of 0.25 for that category. For LEDs, 57.9% of customers would have definitely or probably still purchased LEDs, 14.5% of customers were not sure if they would have still purchased LEDs and 27.6% of customers definitely would not or probably would not still have purchased LEDs. In comparison, in PY2016, only 38.6% of customers would have definitely or probably still purchased LEDs and 55.4% of customers definitely would not or probably would not still have purchased LEDs. The difference in responses between PY2016 and PY2017 suggest that customer attitudes towards purchasing LEDs may be shifting towards purchasing a more energy efficient lighting technology. Table 3-61 shows the responses in the RDD surveys collected in PY2017. Energy Efficiency Programs 109

113 Table 3-61: RDD Survey Responses to Behavior Without Discount Category Questions If the particular LEDs you purchased in the past eight months had cost $5 more per bulb than they did, would you still have bought LEDs over other available light bulb types? Responses Percentage of Total Definitely would have still purchased LEDs % Probably would have still purchased LEDs % Not sure if I would have purchased LEDs % Probably would not have still purchased LEDs % Definitely would not have still purchased LEDs % Total % In addition to assessing free ridership, ADM also included a question on the survey to determine if the prior purchase of discounted bulbs affected the purchase of discounted bulbs in PY2017. The majority of customers surveyed in the RDD survey did not recall purchasing PSO discounted bulbs previously (95.1%). Customers that did respond that they had previously purchased light bulbs that were discounted by PSO (four total out of 82 customers that had previously purchased LEDs prior to 2017) said that their prior purchase of a PSO program discounted bulb was important in the decision they made to purchase a PSO program discounted bulb in PY2017 (average 4 out of a score of 5). Price Response Model Free Ridership Estimate Free ridership was also estimated using an econometric price response model that estimates the effect of program discounts and promotional events on bulb sales. Coefficients from the model were used to predict sales quantities at regular retail pricing and with an absence of program promotional events. The difference in model predictions for sales quantities under program and non-program conditions produces an estimate of free-rider (or naturally occurring) bulb sales. Multiplying the free-rider bulb sales quantities by SKU specific deemed gross savings estimates results in the final estimate of freeridership. The analysis resulted in a program level free ridership estimate of 38.9%, which closely aligned with the RDD estimation methodology (39.1%). The price response model also allows for estimating free ridership by bulb type. The estimated free ridership for specialty LEDs is slightly higher than for standard LEDs as shown below in Figure 3-8. Energy Efficiency Programs 110

114 Figure 3-8: Price Response Model Free Ridership Estimates by Bulb Type A detailed explanation of both the methodology and results from the price response model can be seen in Appendix H. Final Net-to-Gross Ratio The discussion above outlines the results of two efforts to understand the level of attribution appropriate for the energy savings resulting from the lighting bulb sales through the ESP program. The one self-report RDD survey methodology resulted in an estimate of free-ridership of 39.1%. The price response modeling resulted in a free ridership estimate of 38.9%. Energy Efficiency Programs 111

115 Ultimately, the two methods for calculating free ridership are within 0.2% of each other and as such, ADM decided to use the simple average of the free ridership estimate from the RDD survey and the price response model. The final free ridership ratio applied to retail discounted bulbs in this evaluation is therefore 39.0%. Ultimately, both a survey-based and non-survey-based methodology resulted in a similar estimate for free ridership that was slightly lower than the estimate in PY2016 (40.5%). The measure level net-to-gross ratios are calculated as 1- estimated free ridership. 47 The final net-to-gross ratios and associated net savings for each measure in the ESP program are shown in Table Similarly, LEDs distributed through the food bank giveaways are assumed to have a net-to-gross ratio of 1.0. The same NTGR was applied to the small percentage of CFLs rebated through the program in PY2017. Distribution Type Retail Discounts Table 3-62: Verified Gross and Net Impacts ESP Program Measure Type Verified Net Peak Verified kw NTGR Net kwh kwh kw Standard CFL 5, , Omni-directional LED 37,117,205 6, ,641,495 3, Directional LED 9,782,561 1, ,967,362 1, RAPs 348, , APS 136, , CDs 27, , WDs 20, , BVFs 1, HPWHs 1, , Discount1 Omni-directional LED 586, , Food Pantries Omni-directional LED 2,996, ,996, Total 51,024,824 8, ,569,214 5, Conclusions The following summarizes the key findings from the evaluation of the Energy Saving Products Program. The incentive approaches for heat pump water heaters, ENERGY STAR dryers, and APS are consistent with those used in other jurisdictions. Additionally, the planned sales targets are reasonable given the achieved market penetrations in other jurisdictions. Most customers who received rebates for appliances were satisfied with the program overall, the application process, and the wait time to receive the rebate. 47 This is sometimes referred to as a net-of-free-ridership ratio, as it excludes any estimation of spillover or market effects. Energy Efficiency Programs 112

116 One customer noted dissatisfaction with the time to receive the rebate. This incident may have been related to the initial rebate processing delays that that staff noted occurred and have since been addressed. Planned Program Changes The primary changes made or planned for 2018 are as follows: The program is reviewing adding new measures to the program including: Mail-in rebates for ENERGY STAR clothes washers, room air conditioners and refrigerators. 3.4 High Performance Homes Program Program Overview PSO s High Performance Homes program seeks to generate energy and demand savings for residential customers through the promotion of comprehensive efficiency upgrades to building envelope measures and HVAC equipment for both new construction homes and retrofits to existing homes. Offering PSO customers direct inducements for higher efficiency measures offsets the first cost obstacle, encouraging customers to choose the upgraded products. The program can essentially be divided into three components: New Homes, Multiple Upgrades, and Single Upgrades. The New Homes component of the program provides prescriptive incentives to builders of single-family homes. Builders receive $800 for construction that meets the following standards: 75% Energy Efficient Lighting Insulation 15 R-value blown walls; 38 R-value blown attic 13 R-value foam walls; 21 R-value foam attic HVAC 15 SEER Home infiltration reduction Duct infiltration reduction 100% ENERGY STAR Windows Additionally, bonus incentives are offered for: $1000 for meeting ENERGY STAR V 3.1 certification requirements $50 for performance of ACCA Manual J calculations $400 for installing 16 SEER HVAC systems Energy Efficiency Programs 113

117 $600 for installing 17 SEER HVAC systems $800 for installing 18+ SEER HVAC systems $1,000/ton geothermal $200 for duct infiltration less than 4% $200 for 95% CFL $400 for 95% LED Additionally, the program is also providing a design incentive of $3,000 to assist builders with designing program compliant homes. The program is promoted to builders of single-family dwellings and to customers buying new homes. In 2017 PSO marketed the program through various advertising and promotional events, including consumer and trade ally promotions. Supporting this effort, PSO s website 48 provided a comprehensive set of information to builders and customers showing the benefits of building to or beyond the ENERGY STAR standard. Key program activities include: Training homebuilders, sales staff, trade contractors, and other market allies; Increasing consumer awareness of and demand for ENERGY STAR qualified homes through various consumer marketing channels; Increasing homebuilder promotion of ENERGY STAR qualified homes through programprovided collateral items; encouraging the use of the ENERGY STAR brand. The Multiple Upgrades component of the program focuses on energy efficiency upgrades to existing residential homes. To qualify for the program in 2017, customers needed to install three or more eligible equipment upgrades. Eligible measures include: Central air-conditioning systems (CAC) SEER 15 or higher Air source heat pumps (ASHP) SEER 15 or higher Ground source heat pumps (GSHP) Duct system sealing (or replacement) Air Infiltration reduction measures Attic insulation Exterior wall, knee wall, and floor/crawlspace insulation 48 Energy Efficiency Programs 114

118 ENERGY STAR windows and glass doors Solar Screens Radiant barriers Electronically Commutated Furnace Fan Motor (ECM) The Multiple Upgrades component includes a walk-through assessment from a PSO approved contractor to help identify energy efficiency measures that could improve customers comfort level while reducing energy costs. Once the initial audit is performed, a PSO/ICF contracted employee, also referred to as PSO Home Energy Rater, will perform a diagnostic test on the home before and after installation of upgrades are made. This process is in place to measure and document efficiency gains from infiltration reduction and duct sealing measures. The Single Upgrades component of the program also focuses on energy efficiency upgrades to existing residential homes. To qualify for this component of the program, customers needed to install one or two eligible equipment upgrades. Eligible measures include: Central air-conditioning systems (CAC) SEER 16 or higher Air source heat pumps (ASHP) SEER 16 or higher Ground source heat pumps (GSHP) Attic insulation ENERGY STAR windows and glass doors Solar screens Electronically Commutated Furnace Fan Motor (ECM) High Efficiency Electric Water Heater Variable Speed Pool Pumps PY2017 performance metrics are summarized in Table Overall, reported energy savings exceeded projected values considerably. Energy Efficiency Programs 115

119 Table 3-63: Performance Metrics High Performance Homes Program Metric PY2017 Number of Customers 4,787 Budgeted Expenditures $8,872,010 Actual Expenditures $8,700,022 Energy Impacts (kwh) Projected Energy Savings 7,493,480 Reported Energy Savings 10,354,909 Gross Verified Energy Savings 7,832,371 Net Verified Energy Savings 6,800,037 Peak Demand Impacts (kw) Projected Peak Demand Savings 4, Reported Peak Demand Savings 4, Gross Verified Peak Demand Savings 3, Net Verified Peak Demand Savings 3, Benefit / Cost Ratios Total Resource Cost Test Ratio 1.56 Utility Cost Test Ratio 1.35 The remainder of this section details the EM&V methodologies and findings for the High Performance Homes program. The New Homes component is reported first in Section the Multiple Upgrades component in Section and the Single Upgrades component in Section Energy Efficiency Programs 116

120 3.4.2 New Homes Component EM&V Methodologies This section provides an overview of the gross and net impact evaluation of the New Homes component of the High Performance Homes program. The process evaluation for all program components is provided in Section Data Collection Data collection activities that supported the evaluation included builder interviews, verification site visits, and in-depth interviews with program staff at PSO and implementation. For each sampled home, ADM was provided project documentation and modeling files from the program implementer. The provided materials included REM/Rate simulation files, HERS rating certificates, AHRI certificates, Beacon PST Input/Output Summaries, DOE2 Input files, and DOE2 simulation files. The documentation reviews were supplemented by ride-along field visits with the implementation contractor, independent site verification visits by the evaluator. and interviews with builders. The ride along visits allowed ADM to observe the program data collection protocols and access the inputs to the simulation models. The builder interviews were used for the program attribution analysis and to obtain builder feedback about the program. Table 3-64 summarizes the data collection activities and sample size for the New Homes component of the program. Table 3-64: Samples Sizes for Data Collection Efforts New Homes Data Collection Activity Achieved Sample Size Sampling Plan New Homes: Engineering Reviews 29 New Homes: On-Site M&V 24 In-depth Interviews with Program Staff 2 Interviews with Participants (Builders) 3 The sample for the engineering review of building simulation models was designed to achieve ±10% relative precision or better at the 90% confidence interval. Table 3-65 below shows the achieved sample design. The achieved sample design resulted in verified gross kwh estimates with ±6.3% relative precision at the 90% confidence interval. Energy Efficiency Programs 117

121 Table 3-65: Sample Design High Performance New Homes Stratum Number Reported kwh Savings Strata Boundaries (kwh) Population Size CV 49 Sample Size 1 184,159 < 2, ,473 2,000-3, ,960 3,250-12, ,301 > 12, Total 1,324, Gross Impact Methodologies The main impact evaluation activity for the New Homes program was an engineering review of building simulation models and project documentation review for a sample of program rebated homes. The first step in conducting measurement of the New Homes component activity was to review program tracking data and identify the population of homes rebated through the program in The data tracking system was reviewed to ensure that the proper data fields required to support this evaluation as well as future evaluations were included. Furthermore, the tracking data was screened to ensure there were no duplicate entries or other inconsistencies. After reviewing the program tracking data, establishing a program population, and pulling a sample of rebated homes for further engineering review, ADM performed the following verification steps for the 2017 analysis: ADM Reviewed ex ante Model (Beacon PST TM ) The building characteristics that form the basis of the simulation modeling for each rebated home are initially submitted by participating builders/hers raters in the form of REM/Rate modeling files. The implementer s proprietary Beacon Predictive Savings Tool then converts the REM/Rate home model into a DOE2.1 simulation file. The DOE2.1 input and output files were reviewed for each sampled prototype home. Specifically, the differences between input files were compared to identify the modeling approach taken for above code improvements. This also included a thorough review of DOE2.1E parameters, their values, and their applications. These models were then updated as described in the bullet points below and used to derive the reported ex post verified savings values. 49 The coefficient of variation (CV) used for sample design purposes was the based on the variation in reported kwh savings within each stratum: CV = σ (kwh) / µ (kwh). Energy Efficiency Programs 118

122 ADM Developed Baseline 50 Values for Key Simulation Inputs The ex post reference home modeling assumptions are outlined in Table Values for the listed parameters were taken from either the 2009 building energy codes (IECC or IRC) or the Mortgage Industry National Home Energy Rating Systems Standards (RESNET). ADM also reviewed the values used for these parameters in the Beacon PST reference home models. ADM found most of the ex ante values to be consistent with the ex post values. ADM Developed Key Simulation Inputs for ex post Design Models ADM verified the modeling inputs for each sampled home in several ways. First, in our engineering desk review of the ex ante simulation files, we found several modeling methods that needed to be updated. These updates included recalculation of the floor heat transfer coefficient term to be consistent with the use of custom weighting factors, updated heating/cooling set-point schedules to be consistent across reference and rated home simulations (and with RESNET), and application of TMY3 weather data. In addition to the modeling changes, ADM also updated the key model parameters with values consistent with our on-site findings. 51 Model parameters are presented in Table Table 3-66: PY2017 Baseline Home Assumptions Input Ex Post Reference Home Source Roof Solar Absorptivity IECC Reference home, Table (1). Attic Insulation: R IRC Table N values. Cathedral Ceiling Insulation: R IRC section N requirements for ceilings without attic spaces. Ceiling Insulation Grade: 1 The overall U-factor of the ceiling assembly is calculated based on RESNET standards. Wall Construction: 2x4 (16-inch O.C.) Wall Insulation: R IRC Table N values. Wall Insulation Grade: 1 RESNET Standard Note that the term baseline in this instance refers to the "reference" home model. 51 ADM observed/verified as-built conditions through on-site data collection activities. Energy Efficiency Programs 119

123 Input Ex Post Reference Home Source Wall Sheathing: Plywood 2009 IRC Table N values. Door R: R IRC Table N fenestration requirements. Window U IRC Table N values. Window SHGC IRC Table N values. Infiltration F-L-A 2009 IECC Reference home, Table (1). Slab Edge Insulation None 2009 IRC Table N values. HVAC Equipment Type As proposed 2009 IECC Reference home, Table (1). Cooling Efficiency (SEER) 14 NAECA minimum values. Heating Efficiency (AFUE) 80 NAECA minimum values. DSE (Heating / Cooling) 0.8 ANSI-RESNET 301 Table 4.2.2(1) Percent Fluorescent Lighting 50% IRC 2009 Prescriptive Requirements ADM conducted an engineering analysis on 29 homes. Twenty-four homes were engineering reviews utilizing data from site verification visits. The remaining 5 were engineering desk reviews, using project documentation collected during the program implementation cycle. ADM performed post-processing on the ex post-simulation outputs to account for components that were not modeled in DOE2.1E. These components included duct system efficiency (listed in Table 3-66 as DSE). Verified energy and demand impacts for each home were calculated by taking the difference between the reference and rated home energy simulations. Verified gross peak demand reduction was calculated using the DOE2.1E output by taking the maximum difference concurrent demand from June through September. Realization rates for gross annual kwh savings and gross peak kw reduction were calculated for all sampled homes. Program results were derived by extrapolating the results from sampled homes to the population of participating homes according to the sample design. Energy Efficiency Programs 120

124 Net-to-Gross Estimation ADM interviewed 15 builders during the 2016 program evaluation and an additional 3 builders during These builders accounted for (92%) of the program participation in kwh. The interview results from both years were utilized to estimate an NTG ratio for the New Homes component of the High-Performance Homes program. The combing of the builder interview results from 2016 and 2017 data was decided upon to reduce interview fatigue, which could potentially bias the results. Fifty-three percent of the builders interviewed in the 2016 program were also participants in the 2017 program year. There was no significant change in the model mix of participating builders between the program years. NTG scores were developed for each interviewed builder by analyzing responses to three lines of questioning: program influence, building practices in the absence of the program, and co-participation in other rebate programs. Each line of questioning was used to account for 1/3 of the overall free ridership score for each respondent. That is Total Free Ridership = 1/3 x Program Influence FR + 1/3 x Building Practices in the Absence of the Program FR + 1/3 x Co-Participation FR. The scoring for each line of questioning is detailed below. Program Influence: Builders were asked to rate the influence of the program on their decision to build an energy efficient home. The ranking was recorded on a scale of one to five with one representing not at all influential and five representing very influential. Free ridership percentages were applied to the answer as follows; 5 = 100%, 4 = 75%, 3 = 50%, 2 = 25% and 1=0%. The builders were then asked to list all factors influencing their decision to build an above code energy saving home. In cases where builders reported the program as having very little or no influence, but also reported consideration of rebate reductions to building costs, guidance from raters or program staff, or competition with other program builders as being a contributing factor, the initial Net-to-Gross score was increased by 10 percent. Building Practices in the Absence of the Program: Builders were then asked about the percentage of homes they would have built to an above code energy standard if the PSO New Homes Program were not available. They were also asked to report the percentage of homes they would build to an above code standard if the program had never existed (to account for prior year program influence). The reported percentages from the two questions were averaged to determine a free ridership score for this line of questioning. Co-participation in other Rebate Programs: Builders were then asked about participation in the Federal Energy Star New Homes Tax Credit (which expired on December 31, 2016). If they did not participate in the Federal Energy Star New Energy Efficiency Programs 121

125 Homes Tax Credit a Net-to-Gross score of 100% was applied to this line of questioning. If they did participate in the Federal Energy Star New Homes Tax Credit program and the average square footage of the homes they built in the New Homes program was greater than 2000 feet, a Net-to-Gross score of 100% was applied. If they did participate in the Federal Energy Star New Homes Tax Credit program and the average square footage of the homes they built in the New Homes program was less than 2000 feet, a Net-to-Gross score of 75% was applied. Figure 3-9: New Homes Net-to-Gross Logic Chart Energy Efficiency Programs 122

126 Net-to-Gross Estimation Results Builder interviews were used to estimate net-to-gross ratios for the New Homes component of the High Performance Homes program. NTG ratios (ranging from zero for complete free ridership to one for no free ridership) were determined for interviewed home builders representing over 70% of the total program ex ante kwh savings in the last two years. NTG ratios were weighted by the builder s kwh savings contributions to the program. The final component level NTG ratio was 84.4% (commiserate free ridership score of 15.6%). Anecdotal evidence suggests that the magnitude of both participant and non-participant spillover is negligible. The program may have some market transformation effects, but no attempt was made to quantify these effects in terms of additional energy and demand impacts. Results from the builder interviews suggest that the new program design for the New Homes component has had a positive impact on free ridership levels. Impact Evaluation Findings Overall, verified gross energy savings are estimated at 1,278,873 kwh, as shown in Table 3-67, the difference in the reported and verified energy savings results can be attributed to some differing model inputs for the sample homes. Despite these discrepancies, correspondence between ex ante and ex post energy savings was high (realization rate of 97%). Verified gross peak demand reduction is estimated at 335 kw (realization rate of 92%). Table 3-67: Reported and Verified Gross Impacts - New Homes Component Reported Energy Savings (kwh) Reported Peak Demand Savings (kw) Verified Gross Energy Savings (kwh) Verified Gross Peak Demand Savings (kw) New Homes 1,324, ,278, The impact analysis found that, for sampled homes, ex ante simulation models generally reflected the building characteristics verified during field verification visits. However, some exceptions discrepancies were found, leading to mixed variances across the sampled homes as illustrated in Table Note that each + and (-) signifies a difference between the reported model input files and the verified model input files that either increased or decreased the savings. The primary difference between the ex ante and ex post savings stems from differences in differences in the inputs used for calculating lighting savings. The ex post lighting mythology remained consistent with previous year s evaluation. The ex ante algorithm was not applicable to the 2017 program year evaluation. The coefficients and algorithm Energy Efficiency Programs 123

127 were developed in Because of the rapid changes in the res lighting market and the tendency to move towards CFLs as a baseline, the methodology was not revised. Table 3-68: Changes to Reported Model Inputs by Sample Home Building Model Inputs Modeling Approach Sit e Ceiling Insulation Infiltrati on Lighting Calculation HVAC Efficien cy DS E EER Vs. SEE R TMY 3 U- EF F Site RR 1 (-) (-) (-) (-) (-) (+) (+) 95% 2 (-) (+) (-) (-) (+) (-) (+) (+) 116% 3 (-) (+) (-) (-) (-) (+) (+) 108% 4 (-) (+) (-) (-) (-) (+) (+) 139% 5 (-) (+) (-) (-) (-) (-) (+) (+) 95% 6 (-) (+) (-) (-) (-) (-) (+) (+) 89% 7 (-) (-) (-) (-) (+) (-) (+) (+) 111% 8 (-) (+) (-) (-) (-) (-) (+) (+) 99% 9 (-) (-) (-) (-) (-) (-) (+) (+) 106% 10 (-) (-) (-) (+) (-) (+) (+) 92% 11 (-) (-) (-) (-) (+) (-) (+) (+) 84% 12 (-) (-) (-) (-) (-) (+) (+) 71% 13 (-) (-) (-) (+) (-) (+) (+) 103% 14 (-) (-) (-) (-) (-) (+) (+) 53% 15 (-) (-) (-) (-) (+) (+) 54% 16 (-) (+) (-) (-) (-) (+) (+) 27% 17 (-) (-) (-) (-) (-) (+) (+) 84% 18 (-) (-) (-) (+) (-) (+) (+) 125% 19 (-) (-) (-) (-) (-) (+) (+) 79% 20 (-) (-) (-) (-) (-) (-) (+) (+) 98% 21 (-) (-) (-) (-) (+) (-) (+) (+) 108% 22 (-) (-) (-) (-) (+) (-) (+) (+) 109% 23 (-) (+) (-) (+) (-) (-) (+) (+) 104% 24 (-) (-) (-) (+) (-) (+) (+) 121% 25 (-) (-) (-) (-) (+) (+) 103% 26 (-) (-) (-) (-) (+) (+) 131% 27 (-) (-) (-) (-) (+) (+) 70% 28 (-) (-) (-) (-) (+) (+) 59% 29 (-) (-) (-) (+) (-) (+) (+) 123% Energy Efficiency Programs 124

128 Program Activity Twenty-nine builders participated in the program, with 483 qualifying homes being completed in Participants were given an average cash incentive of $2,253 per home with a total of $1,088,050 in incentives paid for the program year. Builders completed an average of 16.7 homes during the program year, however, the distribution of the number of homes built had a strong positive skew and the median number of homes built was four. The distribution in the number of homes completed per participant is shown in Figure Figure 3-10: Number of Homes Completed by Builders Energy Efficiency Programs 125

129 Similarly, the ten most active builders accounted for 83% of program ex ante kwh savings as shown in Figure 3-11 below. Figure 3-11: Percent of Expected kwh Savings by Builder Net Savings: Based on the impact evaluation results, the total ex post net energy savings for the New homes program component are 1,278,873 kwh, and the total ex post net peak demand savings are 335 kw. Table 3-69 provides a summary of New Homes attributable net savings. Table 3-69: Verified Gross and Net Impacts New Homes Program Component Verified Gross kwh Verified Gross Peak kw NTGR Net kwh Net Peak kw New Homes 1,278, % 1,079, Multiple Upgrades (MU) Component EM&V Methodology This section provides an overview of the gross and net impact evaluation of the Multiple Upgrades (MU) component of the High Performance Homes program. The process evaluation for all program components is provided in Section Energy Efficiency Programs 126

130 Data Collection The primary data collection activities for the Multiple Upgrades component of the program consisted of a participant telephone survey, a separate sample of on-site verification visits, in-depth interviews with program staff, and discussions with project auditors. Additional sources of information included in the data review were a census of program tracking data from PSO s tracking and reporting system, and when necessary, project documentation. Program tracking data for the MU component of the program included customer contact information and descriptions of the measures installed. Each MU participant was assigned a random number, and the list of customers was sorted by the random number to create a prioritized call list. Ultimately, 150 surveys were completed. On-site verification visits were conducted for 24 participants with an additional 10 ridealong visits with ICF. During the visits, ADM field staff verified measure installation and operational characteristics. For HVAC equipment, the type, capacity, efficiency ratings, and model numbers were recorded. For attic insulation, installation was visually verified, and pre- and post- insulation levels were measured where possible. Heating and cooling system types were also recorded along with an approximate measurement of square feet of insulation installed. For high efficiency windows/doors, the total count of new units was recorded along with heating and cooling system type. Where possible, pre-existing window type (dual/single pane) was recorded. Pictures were taken during all on-site visits to document measure installation and characteristics. The findings from the on-site verification visits were compared to information in the program tracking database to verify that input to savings calculations were correctly recorded. Program staff members from PSO and implementation contractor ICF were interviewed to elicit the program administrator perspective on program processes and operations. Sampling Plan Table 3-70 summarizes the sample size for each primary data collection activity. The random sample for verification was designed to achieve ±10% relative precision or better at the 90% confidence interval. Table 3-70: Sample Sizes for Data Collection Efforts Multiple Upgrades Data Collection Activity Achieved Sample Size Participant Survey 150 On-Site Verification Visits 24 In-Depth Interviews with Program Staff 2 Energy Efficiency Programs 127

131 For the calculation of sample size for survey completes, a coefficient of variation of 0.5 was assumed. 52 With this assumption, a minimum sample size of 68 participants was required, as shown in the following formula: Where: 2 Z CV n 0 = ( RP ) = ( ) = 68 Equation 3-3: Minimum Sample Size Formula for 90 Percent Confidence Level n 0 = minimum sample size Z = Z-statistic value (1.645 for the 90% confidence level) CV = Coefficient of Variation (assumed to be 0.5) RP = Relative Precision (0.10) ADM conducted phone surveys with 150 participants across the service territory. The additional survey completes were obtained to increase the chance of participation in all areas of program impact. The sample for in-home inspections was designed to achieve ±10% relative precision or better at the 90% confidence interval. The ex ante savings values were placed in one of five strata according to the amount of reported savings and the number of in-home inspections needed per stratum was calculated. Table 3-71 below shows the achieved sample design. 52 The coefficient of variation, cv(y), is a measure of variation for the variable to be estimated. Its value depends on the mean and standard deviation of the distribution of values for the variable (i.e., cv(y) = sd(y)/mean(y)). Energy Efficiency Programs 128

132 Table 3-71: Sample Design Multiple Upgrades Site Visits Stratum Number Ex Ante kwh Strata Boundaries (kwh) Population Size CV 53 Sample Size 1 770,690 < 2, ,322,345 2,500-5, ,375,025 5,000-7, ,202 7,500-12, ,035 > 12, Total 4,865,298-1, Gross Impact Methodologies The methodology used to calculate energy (kwh) and demand impacts (kw) consisted of: Reviewing a census of program tracking data. The tracking data was reviewed for a census of homes and measures. ADM verified there were not any duplicate project data entry errors. Verifying measure installation. ADM calculated installation rates (ISR) by measure for a sample of program participants utilizing data from on-site verification visits. Reviewing ex ante savings estimates for each measure. ADM reviewed ex ante savings calculations for all measures to provide an explanation of any savings discrepancies. Calculating ex post gross savings utilizing the Oklahoma Deemed Savings Document (OKDSD) and Arkansas TRM V6.0. A brief description of each measure calculation methodology is described in the sections below. Appendix F provides measure level algorithms and deemed savings values utilized for the ex post gross and kwh and kw savings calculations. Air Sealing Package: To determine ex post savings for the air sealing, or infiltration reduction, measure, ADM first compared deemed savings values provided in the OKDSD to those provided in the AR TRM. The kwh savings calculated from the OKDSD were found to be much higher than would be expected for this measure, while the AR TRM 53 The coefficient of variation (CV) used for sample design purposes was based on the variation in ex ante kwh savings within each stratum: CV = σ (kwh) / µ (kwh). Where σ is the standard deviation of saving and µ is the average of savings. Energy Efficiency Programs 129

133 produced values that were in the expected savings range. Therefore, using ADM s engineering judgment, the AR TRM was utilized to calculate energy and demand impacts of infiltration reduction measures. Savings are calculated by multiplying the air infiltration reduction (CFM), with the energy savings factor corresponding to the climate zone and HVAC type. The air infiltration reduction estimate in CFM is obtained through blower door testing performed by the program contractor for each home serviced. Only homes with electric cooling systems are eligible for the measure (CAC or room AC). Duct Sealing: This measure involves sealing leaks in ducts of the distribution system of homes with either a CAC or a ducted heating system. ADM utilized the OKDSD in conjunction with the duct leakage reduction results to calculate measure savings. The duct leaked reduction estimate in CFM is obtained through duct blaster testing performed by the program contractor for each home serviced. If a CAC was installed with this project, the SEER value of the installed unit is used in the savings calculations. Duct Insulation: Deemed savings for this measure are calculated for each weather zone for installation of a minimum R-value of R-6 to un-insulated metal supply and return ductwork, located in unconditioned space that previously had no or minimal existing insulation. The OKDSD was utilized to calculate energy and demand impacts for duct insulation. Deemed savings are based on the location of the ducts: attic or crawlspace. Savings are calculated by multiplying the deemed savings value for the corresponding area and weather zone by the square footage of the conditioned area of the home. Attic Insulation: To determine the most appropriate methodology for calculating savings for this measure, ADM performed a benchmarking analysis of the attic insulation measure. The benchmark analysis compared the deemed savings value in the OKDSD and AR TRM with two independent engineering calculations conducted by ADM. All calculation methodologies used similar assumptions regarding equipment efficiency and thermostat setpoints. However, ADM determined that the OKDSD simulations underestimate baseline construction material R-Values and found the AR TRM to provide more reasonable savings levels. For ex post savings calculations, the AR TRM values for Zone 9 are used directly for any homes within this climate Zone. Based on the review of the AR TRM values, as well as ADM s engineering calculations, it was determined that the savings values in Zone 6, Zone 7, and Zone 8 should be scaled by the ratio of each respective Zone to the deemed savings for Zone 9 in the OKDSD. Deemed savings provided in the AR TRM are based on the R-value of the baseline insulation. Savings are calculated by multiplying the applicable savings value by the square footage insulated. The savings algorithms require new insulation to meet a Energy Efficiency Programs 130

134 minimum R-value of R-38. Savings are calculated for both R-38 and R-49 insulation, depending on the final insulation levels installed in the home. Floor Insulation: Deemed savings values are calculated for each weather zone in accordance with the AR TRM. The savings algorithms require new insulation to meet a minimum R-value of R-19. Deemed savings provided in the AR TRM are based on the heating and cooling system type of the home. Savings are calculated by multiplying the corresponding savings value by the square footage insulated. Wall Insulation: Deemed savings values are calculated for each weather zone in accordance with the AR TRM. The savings algorithm requires new insulation to meet a minimum R-value of R-13. Deemed savings provided in the AR TRM are based on the heating and cooling system type of the home and the R-Value of the insulation installed. Savings are calculated by multiplying the corresponding savings value by the square footage insulated. Knee Wall Insulation: Deemed savings values are calculated for each weather zone in accordance with the AR TRM. The savings algorithms require new insulation to meet a minimum R-value of R-19. Deemed savings provided in the AR TRM are based on the heating and cooling system type of the home and the R-value of the installed insulation. Savings are calculated by multiplying the corresponding savings value by the square footage insulated. Electronically Commutated (ECM) Furnace Fan Motor: Deemed heating and cooling savings values are calculated for each weather zone in accordance with the OKDSD. Cooling and peak demand savings are eligible only if the existing cooling system is left in place when the ECM is installed. Savings are calculated by multiplying the corresponding savings value by the square footage of the conditioned space of the home. Central Air Conditioner: ADM utilized the savings algorithms found in the OKDSD. As per the updated federal minimum code 54, the baseline SEER value has been updated from 13 to 14. However, the OK DSD allows for a baseline SEER value of to be used in savings calculations to account for a mixture of replacing on burnout and early replacement installations. To be consistent with the methodology provided in the OKDSD, the ratio of to 13 is applied to the updated 14 SEER value and used in the ex post calculations. Heat Pump: ADM utilized the savings algorithms found in the OKDSD. As per the updated federal minimum code 55, the baseline HSPF value has been updated from 7.7 to 8.8. This HSPF has been updated and used in the savings calculations Ibid. Energy Efficiency Programs 131

135 Windows and Glass Doors: Deemed savings values are calculated for each weather zone in accordance with the OKDSD. Deemed savings listed are based on the heating and cooling type of the home and the existing window pane type. Savings are calculated by multiplying the corresponding savings value by the square footage of the installed window, including the frame and sash. Faucet Aerators: ADM utilized savings algorithms found in the AR TRM. ADM performed a benchmarking analysis for the faucet aerator measures. The analysis compared the deemed savings in the OKDSD and savings algorithm in the AR TRM. ADM determined the OKDSD required all kitchen and bathroom faucets be retrofitted to claim savings while the AR TRM has savings per faucet. Low Flow Showerheads: Deemed savings values are calculated for each shower head type in accordance with the OKDSD. Deemed savings listed are based on the number of showerheads per household, the number of showerheads retrofitted, and the type of water heating unit in the home. Omnidirectional LEDs: ADM utilizes savings algorithms found in the OKDSD with a modification to the hours of use. From a benchmark study performed by ADM in 2016, it was determined for the hours of use to be 960 hours per year 56. Net-to-Gross Estimation This section provides a summary of the methodology ADM used to score survey responses for free ridership and spillover. A sample of program participants were surveyed and asked a series of questions aimed at estimating program attribution. The attribution scoring system is broken down into three components: measure level free ridership score, project level free ridership score, and spillover score. Each component is described individually below, followed by a section discussing how the scores will be weighted to extrapolate the survey results to the program component level (all Multiple Upgrades projects). Measure Level Free Ridership Scoring: The survey was pre-populated with information about each respondent. Specifically, the list of energy efficiency measures the respondent installed through the program was known to the surveyor. The survey was programmed to ask a series of questions about the program s effect on deciding to purchase and install each measure. For example, if a respondent installed a new heat pump, ceiling insulation, and had their ducts sealed, they were asked a series of questions about each of those measures (but not other program-eligible measures which they did not install). The measure level free ridership questions were arranged as follows: 1. Indicator one: prior planning 56 ADM HOU memo, Energy Efficiency Programs 132

136 2. Indicator two: stated likelihood in absence of program recommendations 3. Indicator three: stated likelihood in absence of 50% bonus incentives 4. Indicator four: stated likelihood in absence of all program incentives 5. Mitigating factor one: reported efficiency/quantity increase 6. Mitigating factor two: timeliness of upgrade How these questions work together to determine a measure level free ridership score is displayed in Figure 3-12 on the following page. Note that the scoring algorithm requires the respondent to indicate a burden of proof that they are a free rider. They must have stated that either 1) they had prior plans to install the measure or 2) they would have likely installed the measure in the absence of the program. To be considered a full free rider for the specific measure, they must have gone on to say the program did not affect the level of efficiency (or quantity, where appropriate) they chose for their upgrade nor did it affect the timeliness of the upgrade. Energy Efficiency Programs 133

137 Figure 3-12: Measure Level Free Ridership Scoring Multiple Upgrades Energy Efficiency Programs 134

138 Project Level Free Ridership Scoring: The measure level free ridership scores for each respondent were weighted by the ex ante kwh savings by measure. For Multiple Upgrades, the measure level free-ridership score for each measure a respondent installed was first multiplied by the ex ante kwh savings that measure represents. Total measure level free-ridership kwh was then summed over the incentivized measures and divided by the total project ex ante kwh savings to get the respondent s free-ridership score. This means that if a respondent indicated free ridership for a low kwh impact measure, but no free ridership for a high kwh impact measure, the overall free ridership score will take this into account. Spillover Scoring: Spillover, as addressed in this survey, was restricted to energy efficiency measures that respondents report installing in their home without receiving additional incentives. Additionally, respondents must have indicated that the additional non-incentivized measures were installed based on program influence. Potential spillover respondents were identified using the question below: Following your participation in the PSO s Multiple Upgrades Service, did you install any additional energy efficiency upgrades in your home for which you did not receive a rebate or financial incentive? Customers who responded no to this question were determined to not be potential spillover candidates. If a respondent indicated that they have installed additional energy efficiency measures, they were then asked: Did your participation in PSO s Multiple Upgrades program influence your decision to install the additional energy efficient upgrades in your home? Customers who responded yes, strongly influenced or yes, somewhat influenced were considered potential spillover candidates, and were asked to identify the additional measures they have installed. The measures identified were then checked to ensure they had not received a corresponding PSO rebate through any program and that they actually represent measures with the potential to save energy. All measures that passed through this screening process were assigned a savings value based on the deemed savings documents. Verified spillover savings for a given customer were then divided by the project level ex ante savings for that customer. The resulting percentage is the total spillover score. The average spillover score per sampled customer was applied to the program level savings to determine program level spillover. Weighting of Respondent Scores and Determination of Program Level NTGR: The project level free ridership scores for each respondent was weighted by the ex ante kwh savings per project to determine the final weighted average free-ridership estimate per customer in the sample. This estimate along with the spillover estimate was used as the program level verified gross savings to in order to determine net savings. Energy Efficiency Programs 135

139 Net-to-Gross Estimation Results ADM surveyed 150 MU participants to determine the net-to-gross ratio for this program component. Survey respondents were asked a series of questions aimed at determining the program influence on the purchase and installation decisions for each installed measure. Each respondent was assigned a free ridership score (ranging from 0 for no free ridership to 1 for complete free ridership) based on their responses for each measure they installed. The free ridership scores for all survey respondents were then weighted by kwh savings and averaged to determine the program level free ridership rate. The resulting free ridership estimate is 11.70%. Survey respondents were also asked a series of questions to determine if they had installed any additional, non-rebated, energy efficiency measures because of their participation in the program (spillover). Twenty-nine respondents said they had installed additional measures, and that their participation in the program was influential in their decision to do so. However, only twenty-four of these respondents reported measures that were not rebated and had potential energy savings. The final spillover estimate was 2.25% for the MU program component. The final NTG ratio is calculated as 1 free ridership + participant spillover, which is estimated at 90.56%. This estimate is consistent with prior year estimates for the program (PY2014 = 85%, PY2015 = 88%, PY2016, 85%). Impact Evaluation Findings Table 3-72 provides a summary of gross impact savings. The savings estimates result in a gross annual kwh realization rate of 62% and a peak kw reduction realization rate of 80%. Ex ante savings were calculated utilizing the OKDSD. Energy Efficiency Programs 136

140 Table 3-72: Ex Ante and Ex Post Gross kwh and kw Measure Ex Ante kwh Ex Ante kw Ex Post Gross kwh Ex Post Gross kw kwh RR 57 kw RR Infiltration Reduction 360, , % 73% Attic-Ceiling Insulation 624, , % 213% Central AC 1,273, , % 58% Duct Insulation 58, , % 100% Duct System Sealing 1,828,108 1, ,410, % 70% Faucet Aerator 7, , % 65% Floor Insulation 2, % -12% Heat Pump 302, , % 65% Heating System ECM 125, , % 46% Knee Wall Insulation 98, , % 26% Omnidirectional LED 4, , % 100% Shower Heads 6, , % 100% Wall Insulation 107, , % 163% Window and Glass Door 65, , % 100% Total 4,865,298 2, ,038,577 1, % 80% The gross impact analysis consisted of verifying measure installation and checking the program tracking data to ensure that deemed savings algorithms were appropriately applied. ISRs for each measure type were developed based on the findings from the onsite visits. Findings from the on-site visits determined a 100% ISR for all measures in Multiple Upgrades for PY2017. A description of ex post gross findings for each measure type is included below: Infiltration Reduction: During site visits performed by ADM, four customers did not recall having this measure installed. This measure can be difficult for a homeowner to see, for example, if the plumbing fixture penetration under a kitchen sink were done a customer may not notice the measure, or if the windows were sealed using a clear caulking with a thin bead. ADM utilized data from the Sightline web portal and verified the measure was complete in all four of these residences. Based on these findings, an ISR of 100% was applied. ADM utilized the AR TRM to calculate the deemed savings for each home and determined the kwh realization rate to be 55% and the kw realization rate to be 73%. Discrepancies between ex ante and ex post gross savings is attributable to ex ante calculations utilizing values and algorithms from the OKDSD while ex post savings utilized the AR TRM. 57 SSRS Database was not updated to reflect changes to kwh and kw savings that were brought about by a change in methodology during the portfolio cycle. The evaluation, implementation, and utility decided to instead capture the changes in the realization rates. Energy Efficiency Programs 137

141 Duct Sealing: ADM calculated savings for each home with duct sealing and determined the measure level kwh realization rate to be 77% and the kw realization rate to be 70%. The difference between ex ante and ex post Gross savings is that ex ante calculations used default SEER values in the savings calculations when a new central air conditioner unit was installed. ADM determined if a central air conditioner unit was installed and used the SEER for the installed unit in the savings calculations. Duct Insulation: ADM calculated savings for each home with duct insulation and determined the measure kwh realization rate to be 100% and the kw realization rate to be 100% Attic Insulation: ADM calculated deemed savings for each home with attic insulation and determined the measure kwh realization rate to be 53% and the kw realization rate to be 213%. Deemed savings values were taken from the AR TRM with scaled values for Weather Zone 6, 7, and 8. The ex ante calculations utilized deemed values from the OKDSD. Discrepancies between ex ante and ex post gross savings is attributable to ex ante calculations utilizing values and algorithms from the OKDSD while ex post savings utilized the AR TRM. Floor Insulation: Realization rates for floor insulation measures were low (RR kwh: 3%, RR kw: -12%). However, floor insulation represented a small number of projects for the MU component (6 projects, 0.05% of ex ante kwh). The discrepancy between ex ante and ex post kwh stemmed primary from two of the projects not meeting the minimum R- Value requirements, resulting in 0 kwh savings. Furthermore, ex ante kw values were calculated using deemed values from the OKDSD (which are positive), while ex ante kw were calculated using values from the AR TRM (which are negative), resulting in a negative realization rate for gross kw savings. Despite these discrepancies, these findings have a minimal impact on overall program savings. Wall Insulation: ADM calculated deemed savings for each home with wall insulation and determined the measure kwh realization rate to be 55% and the kw realization rate to be 153%. The ex ante calculations utilized the OKDSD deemed savings values. The difference between the ex ante and ex post gross savings was the utilization of the energy savings factors from the OKDSD and the AR TRM. Knee Wall Insulation: ADM calculated deemed savings for each home with knee wall insulation and determined the measure kwh realization rate to be 20% and the kw realization rate to be 26%. The ex ante calculations utilized the OKDSD deemed savings values. The difference between the ex ante and ex post Gross savings was the utilization of the energy savings factors from the OKDSD and the AR TRM. Electronically Commutated (ECM) Furnace Fan Motor: ADM calculated deemed savings for each home that installed an ECM and determined the kwh realization rate to be 96% and the kw realization rate to be 46%. Discrepancies between ex ante and ex Energy Efficiency Programs 138

142 post gross savings stemmed primarily from whether or not participants installed a new CAC or HP system in conjunction with the ECM. Cooling savings and peak kw savings can only be claimed if the ECM is not installed with a new CAC or HP system. Eighteen projects claimed cooling and peak kw savings when a new CAC or HP was installed. Central Air Conditioner: ADM calculated savings for each home with a central air conditioner unit and determined a measure kwh realization rate of 54% and a kw realization rate of 58%. The difference between the ex ante and ex post Gross savings is due two things: the change in the federal minimum code for baseline SEER and the capacity of the new unit for the calculations will be equal to the existing capacity if the unit is installed in combination with other measures. The ex ante calculations used a SEER of rather than the updated code value of 14 and capacity of the new unit for calculations of all units installed. Heat Pump: ADM calculated savings for each home with a heat pump unit installed and determined a kwh realization rate of 23% and a kw realization rate of 65%. Discrepancies between ex ante and ex post gross savings is attributable to: (1) discrepancies in the baseline HSPF and (2) discrepancies in the installed capacity. The ex ante calculations used an antiquated HSPF of 7.7 rather than an updated value of 8.6. Furthermore, the installed capacity for the new unit should be equivalent to the capacity of the original unit when units are installed with multiple measures. Windows and Glass Doors: ADM calculated deemed savings for each home with windows and glass doors installed and determined the measure kwh realization rate to be 100% and the kw realization rate to be 100%. Faucet Aerators: ADM calculated deemed savings for each home with faucet aerators and determined the measure kwh realization rate to be 32% and the kw realization rate to be 65%. Discrepancies between ex ante and ex post gross savings is attributable to ex ante calculations utilizing values and algorithms from the OKDSD while ex post savings utilized the AR TRM. Low Flow Showerheads: ADM calculated deemed savings for each home with showerheads installed and determined the measure kwh realization rate to be 100% and the kw realization rate to be 100%. Omnidirectional LEDs ADM calculated deemed savings for each home with LEDs installed and determined the kwh realization rate to be 94% and the kw realization rate to be 100%. The difference in ex ante and ex post gross savings is the difference in the HOU. The ex ante calculations used the deemed hours of use from the OKDSD (1,023 hours) and ex post gross calculations used an HOU from the 2016 ADM HOU study (960 hours). Energy Efficiency Programs 139

143 Net Savings: Based on the impact evaluation results, the total ex post net energy savings for the MU program component are 2,751,735 kwh, and the total ex post net peak demand savings are 1, A summary of MU impact findings is shown in Table Table 3-73: Verified Gross and Net Impacts Multiple Upgrades Program Component Verified Gross kwh Verified Gross Peak kw NTGR Net kwh Net Peak kw Multiple Upgrades 3,038,577 1, % 2,751,735 1, Single Upgrades (SU) Component EM&V Methodologies This section provides an overview of the gross and net impact evaluation of the Single Upgrades component of the High Performance Homes program. The process evaluation for all program components is provided in Section Data Collection The primary data collection activities for the Single Upgrades component of the program consisted of a participant telephone survey, a separate sample of on-site verification visits, in-depth interviews with program staff, and discussions with project auditors. Additional data reviewed included a census of program tracking data from Sightline database, SQL Server Reporting Services (SSRS) and, when necessary, project documentation obtained from VisionDSM. Program tracking data for the SU component of the program included customer contact information and descriptions of the measures installed. Each SU participant was assigned a random number, and the list of customers was sorted by the random number to create a prioritized call list. Ultimately, 150 surveys were completed. On-site verification visits were conducted for 42 participants. During the visits, ADM field staff verified measure installation and operational characteristics. For HVAC equipment, the type, capacity, efficiency ratings, and model numbers were recorded. For attic insulation, installation was visually verified, and pre- and post- insulation levels were measured where possible. Heating and cooling system types were also recorded along with an approximate measurement of square feet of insulation installed. For high efficiency windows/doors, the total count of new units was recorded along with heating and cooling system type. Where possible, pre-existing window type (dual/single pane) was recorded. Pictures were taken during all on-site visits to document measure installation and characteristics. The findings from the on-site verification visits were Energy Efficiency Programs 140

144 compared to information in the program tracking database to verify that input to savings calculations were correctly recorded. Program staff members from PSO and implementation contractor ICF were interviewed to elicit the program administrator perspective on program processes and operations. Sampling Plan Table 3-74 summarizes the sample size for each primary data collection activity. The random sample for verification was designed to achieve ±10% relative precision or better at the 90% confidence interval. Table 3-74: Sample Sizes for Data Collection Efforts Single Upgrades Data Collection Activity Achieved Sample Size Participant Survey 150 On-Site Verification Visits 42 In-Depth Interviews with Program Staff 2 For the calculation of sample size for survey completes, a coefficient of variation of 0.5 was assumed. 58 With this assumption, a minimum sample size of 68 participants was required, as shown in the following formula: Where: Equation 3-4: Minimum Sample Size Formula for 90 Percent Confidence Level n 0 = minimum sample size Z = Z-statistic value (1.645 for the 90% confidence level) CV = Coefficient of Variation (assumed to be 0.5) RP = Relative Precision (0.10) ADM conducted phone surveys with 150 participants across the service territory. The additional survey completes were obtained to increase the chance of participation in all areas the program impacted. The sample for in-home inspections was designed to achieve ±10% relative precision or better at the 90% confidence interval. The ex ante savings values were placed in one of 58 The coefficient of variation, cv(y), is a measure of variation for the variable to be estimated. Its value depends on the mean and standard deviation of the distribution of values for the variable (i.e., cv(y) = sd(y)/mean(y)). Energy Efficiency Programs 141

145 five strata according to the amount of claimed savings and the number of in-home inspections needed per stratum was calculated. Table 3-75 below shows the achieved sample design. Table 3-75: Sample Design Single Upgrades Stratum Ex Ante kwh Strata Boundaries (kwh) Population Size CV Sample Size 1 635,140 < 1, ,790,399 1,000-2, ,049 2,000-3, ,480 3,000-4, ,650 > 4, Total 4,164,719-3, Gross Impact Methodologies The methodology used to calculate energy (kwh) and demand impacts (kw) consisted of: Reviewing a census of program tracking data The tracking data was reviewed for a census of homes and measures. ADM verified there were not any duplicate project data entry errors. Verifying measure installation ADM calculated installation rates (ISR) by measure for a sample of program participants utilizing data from onsite verification visits. Reviewing ex ante savings estimates for each measure ADM reviewed ex ante savings calculations for all measures to provide an explanation of any savings discrepancies. Calculating ex post Gross savings utilizing: Oklahoma Deemed Savings Document (OKDSD). Arkansas Technical Reference Manual v6.0 (AK TRM). A brief description of each measure calculation methodology has been described in the Multiple Upgrades section above except Ground Source Heat Pumps, Variable Speed Drive Pool Pumps, and Solar Screens. Those measures are described in the sections Energy Efficiency Programs 142

146 below. Appendix F includes the measure level algorithms and deemed savings values utilized for the ex post Gross kwh and kw savings calculations. Ground Source Heat Pumps: ADM utilized the savings algorithms found in the OKDSD for units that meet the minimum efficiency requirements. Additional savings are included for units that have a de-superheater for domestic water heating. Variable Speed Drive Pool Pumps: ADM utilized the savings algorithms found in the OKDSD. The savings algorithms inputs are dependent upon the pool capacity in gallons, horsepower of the motor, and the seasonal usage. Solar Screens: Deemed savings values are calculated for each weather zone in accordance to the OKDSD. Deemed savings listed based on the heating and cooling type of the home. Savings are calculated by multiplying the corresponding deemed savings value by the square footage of the solar screen. Net-to-Gross Estimation This section provides a summary of the methodology ADM used to score survey responses for free ridership and spillover. PSO customers who received rebates through the Single Upgrades component of the High Performance Homes program were surveyed and asked a series of questions aimed at estimating program attribution. The attribution scoring system is broken down into two components: free-ridership score and spillover score. Each component is described individually below, followed by a section discussing how the scores will be weighted to extrapolate the survey results to the program component level (all Single Upgrades projects). Free Ridership Scoring: The survey will be pre-populated with information about each respondent. Specifically, the list of energy efficiency measures the respondent installed through the program is known to the surveyor. The survey will be programmed to ask a series of questions about the program s effect on deciding to purchase and install each measure. For example, if a respondent installed a new heat pump they would be asked a series of questions specifically about the heat pump (but not other program-eligible measures, which they did not install). The free ridership questions are arranged as follows: 1. Indicator one: prior planning 2. Indicator two: stated likelihood in absence of program incentives 3. Mitigating factor one: reported efficiency/quantity increase 4. Mitigating factor two: timeliness of upgrade Energy Efficiency Programs 143

147 How these questions work together to determine a measure level free ridership score is displayed in Figure 3-13 on the following page. Note that the scoring algorithm requires the respondent to indicate a burden of proof that they are a free rider. They must state that either 1) they had prior plans to install the measure or 2) they would have likely installed the measure in the absence of the program. To be considered a full free rider for the specific measure, they must go on to say the program did not affect the level of efficiency (or quantity, where appropriate) they chose for their upgrade nor did it affect the timeliness of the upgrade. Energy Efficiency Programs 144

148 Figure 3-13: Measure Level Free Ridership Scoring Single Upgrades Energy Efficiency Programs 145

149 Spillover Scoring Spillover, as addressed in this survey, is restricted to energy efficiency measures that respondents report installing in their home without receiving additional incentives. Additionally, respondents must indicate that the additional non-incentivized measures were installed based on program influence. Potential spillover respondents are identified using the question below: 1. Following your participation in the PSO s Energy Efficiency Program, did you install any additional energy efficiency upgrades in your home for which you did not receive a rebate or financial incentive? Customers who respond no to this question will be determined to not be potential spillover candidates. If a respondent indicates that they have installed additional energy efficiency measures, they will then be asked: 2. Did your participation in PSO s Energy Efficiency Program influence your decision to install the additional energy efficient upgrades in your home? Customers who respond yes, strongly influenced or yes, somewhat influenced will be considered potential spillover candidates, and will be asked to identify the additional measures they have installed. The measures identified will then be checked to ensure they had not received a corresponding PSO rebate through any program and that they actually represent measures with the potential to save energy. A spillover ratio is calculated for the program that is equal to the sum of spillover savings identified by survey respondents divided by the sum of ex post Gross savings for all survey respondents. Net-to-Gross Estimation Results ADM surveyed 150 Single Upgrades participants to determine the net-to-gross ratio for this program. Survey respondents were asked a series of questions aimed at determining the program influence on the purchase and installation decisions for each installed measure. Each respondent was assigned a free ridership score (ranging from 0 for no free ridership to 1 for complete free ridership) based on their responses for each measure they installed. The free ridership scores for all survey respondents were then weighted by kwh savings and averaged to determine the program level free ridership rate. The resulting free ridership estimate is 23%. Survey respondents were also asked a series of questions to determine if they had installed any additional, non-rebated, energy efficiency measures because of their participation in the program (spillover). Seventeen respondents said they had installed additional measures, and that their participation in the program was influential in their decision to do so. However, only eight of these respondents reported measures that were Energy Efficiency Programs 146

150 not rebated and had potential energy savings. The final spillover estimate was 7.86% for the MU program. The final Single Upgrades net-to-gross ratio is calculated as 1 free ridership + participant spillover and is estimated at 84%. This estimate is higher than prior year estimates for the program (PY2014, 78%, PY2015, 80%, PY %) due to lower free ridership. Impact Evaluation Findings The ex post Gross and ex ante savings by measure are shown in Table The savings estimates result in a gross annual kwh realization rate of 86% and a peak kw reduction realization rate of 107%. Ex ante savings were calculated utilizing the OKDSD. Table 3-76: Ex Ante and Ex Post Gross kwh and Peak kw Measure Ex Ante kwh Ex Ante kw Ex Post Gross kwh Ex Post Gross kw kwh RR 59 Heat Pumps 277, , % 100% Central AC 2,044, ,549, % 100% Ground Source Heat Pumps 126, , % 100% Heating System ECM 484, , % 89% Attic-Ceiling Insulation 228, , % 245% Pool Pumps 385, , % 102% Solar Screen 1, , % 100% Windows and Doors 615, , % 99% Total 4,164,719 1, ,514,921 1, % 107% The gross impact analysis consisted of verifying measure installation and reviewing the program tracking data to ensure the deemed savings algorithms were appropriately applied. In-Service Rates (ISR) for each measure type were developed based on the findings from the on-site visits. Findings from on-site visits determined a 100% ISR for all measures in Single Upgrades for PY2017. A description of ex post findings for each measure type is included below: Heat Pumps: ADM calculated savings for each home with a heat pump unit installed and determined a kwh realization rate of 71% and a kw realization rate of 100%. The difference between the ex ante and ex post Gross savings is due to the change in the federal minimum code for baseline HSPF. The ex ante calculations used an HSPF of 7.7 rather than the updated code value of 8.6. kw RR 59 SSRS Database was not updated to reflect changes to kwh and kw savings that were brought about by a change in methodology during the portfolio cycle. The evaluation, implementation, and utility decided to instead capture the changes in the realization rates. Energy Efficiency Programs 147

151 Central Air Conditioner: ADM calculated savings for each home with a central air conditioner unit and determined a measure kwh realization rate of 76% and a kw realization rate of 100%. The difference between the ex ante and ex post Gross savings is due to the change in the federal minimum code for baseline SEER. The ex ante calculations used a SEER of rather than the updated code value of 14. Ground Source Heat Pump: ADM calculated savings for each home with a ground source heat pump unit installed and determined a measure kwh realization rate of 110% and a kw realization rate of 100%. Differences in ex ante and ex post Gross savings are due to three projects, POHIPS , POHIPS , and POHIPS , that are not claiming heating kwh savings. The unit meets minimum efficiency requirements and should have heating savings calculated. Electronically Commutated (ECM) Furnace Fan Motor: Electronically Commutated (ECM) Furnace Fan Motor: ADM calculated deemed savings for each home that installed an ECM and determined the kwh realization rate to be 98% and the kw realization rate to be 89%. Discrepancies between ex ante and ex post gross savings stemmed primarily from whether or not participants installed a new CAC or HP system in conjunction with the ECM. Cooling savings and peak kw savings can only be claimed if the ECM is not installed with a new CAC or HP system. Thirty two projects claimed cooling and peak kw savings when a new CAC or HP was installed. Attic Insulation: ADM calculated deemed savings for each home with attic insulation and determined the measure kwh realization rate to be 64% and the kw realization rate to be 245%. As described in the EM&V Methodologies section (Multiple Upgrades), the deemed savings values were taken from the AR TRM with scaled values for Weather Zone 6, 7, and 8. Discrepancies between ex ante and ex post gross savings is attributable to ex ante calculations utilizing values and algorithms from the OKDSD while ex post savings utilized the AR TRM. Variable Speed Drive Pool Pumps: ADM calculated savings for each home with a variable speed drive pool pump and determined the measure kwh realization rate to be 102% and the kw realization rate to be 102%. Eight projects use an EFlo value of 5 in the ex ante calculations, which does not come from the OKDSD. The EFlo value is a deemed value based on the horsepower range of the new pool pump. This discrepancy results in the 102% realization rates. Solar Screens: ADM calculated deemed savings for each home with solar screens installed and determined the measure kwh realization rate to be 100% and the kw realization rate to be 100%. Windows and Glass Doors: ADM calculated deemed savings for each home with windows and glass doors installed and determined the measure kwh realization rate to be 100% and the kw realization rate to be 99%. Energy Efficiency Programs 148

152 Net Savings: Based on the impact evaluation results, the total ex post net energy savings for the SU program component are 3,968,933 kwh, and the total ex post net peak demand savings are 1, A summary of SU impact findings is shown in Table Table 3-77: Verified Gross and Net Impacts Single Upgrades Program Component Verified Gross kwh Verified Gross Peak kw NTGR Net kwh Net Peak kw Single Upgrades 3,514,921 1, % 2,968,933 1, Impact Evaluation Findings Program level ex ante and ex post Gross savings by component are shown in Table 3-78 below. Table 3-78: Program Level Gross Savings Program Component Ex Ante kwh Ex Ante kw Ex Post Gross kwh Ex Post Gross kw New Homes 1,324, ,278, Multiple Upgrades 4,865, ,038, Single Upgrades 4,164, ,514, Total 10,354, ,832,371 3, Table 3-79 and Table 3-80 summarize the ex post net impacts of the complete High Performance Homes program. Table 3-79: Ex Post Gross and Net kwh Savings Program Component Free Ridership Participant Spillover NTG Ratio Ex Post Gross kwh Ex Post Net kwh New Homes 15.60% 0.00% 84.4% 1,278,873 1,079,369 Multiple Upgrades 11.70% 2.25% 90.56% 3,038,577 2,751,735 Single Upgrades 23.00% 7.86% 84.00% 3,514,921 2,968,933 Total 87.00% 7,832,371 6,800,037 Energy Efficiency Programs 149

153 Table 3-80: Ex Post Gross and Net Peak kw Reduction Program Component Free Ridership Participant Spillover NTG Ratio Ex Post Gross kw Ex Post Net kw New Homes 15.60% 0.00% 84.40% Multiple Upgrades 11.70% 2.25% 90.56% Single Upgrades 23.00% 7.86% 84.00% Total 87.00% 3, , Process Evaluation Activities Process Evaluation Activities ADM performed a process evaluation that assesses program documentation and primary data collected from program stakeholders. The process evaluation addressed the following general research questions. What were the energy savings achieved by the various components of the High Performance Homes Program in 2017 as compared to 1) 2016 energy savings and 2) 2017 energy savings goals? If there was a significant difference in either, what accounts for the changes? Did the program design or operations change in 2017? If so, what influenced the change(s) and how did the change(s) impact the program? Are there any planned changes to program design for 2018? To address these researchable issues, ADM administered participant surveys, completed interviews with program staff, reviewed program documentation, and analyzed the program tracking data. The data collection activities are summarized in Table Table 3-81: Data Collection Activities Data Collection Activity Program Staff Interviews Participant Survey (Multiple Upgrades and Single Upgrades) Process Evaluation Research Objectives Assess program staff perspectives regarding program operations, strengths, or barriers to success. Assess source of program awareness, benefits of improvements, satisfaction, and customer demographics and home characteristics Conclusions The following summarizes the key findings of the process evaluation of the High Performance Homes Program. Energy Efficiency Programs 150

154 The program exceeded its energy savings goals for The air sealing incentive structure changed and generally decreased the incentive payments for this measure. ICF staff stated that this change, as well decreased activity from a contractor who had relatively high acquisition costs for duct sealing, resulted in the program generating higher energy savings with the available incentive budget than planned. In comparison to the 2016 program year, the average incentive payment made to homes in multiple upgrades declined slightly and there was a notable increase in the average reported kwh savings (from 3,105 kwh to 3,963 kwh) achieved at each home. Additionally, the average number of measure types installed increased from 3.5 to 3.7. The program offered the New Green Appraiser training in 2017 and ICF staff indicated that this change resulted in an increase in the number of green appraisers in Oklahoma from 1 to 18. An improved quality control/assurance process will be instituted for Single Upgrades in All service providers will be required to either submit applications through the Power Rebate app or schedule a verification visit with a program Third Party Verifier. The Power Rebate app submissions will require photos of the improvement made and pre-installation conditions for applicable measures (i.e., insulation levels prior to adding insulation) and will geocode the submission. ICF indicated that their quality control and assurance processes will not change and that they will continue to perform verifications on 5-10% of projects. This change will be funded by discontinuing incentives for windows and applying the budget to fund third-party verifications and provide a bonus incentive to service providers for using the Power Rebate App. Staff chose to divert funds from windows because historically it has had higher free ridership than other measures. Two program improvements were in process for the new homes component. One change is to reduce the bonus for ENERGY STAR homes from $1,000 to $800. Second, the staff has identified a means to streamline the processing of REM/Rate files through bulk processing. This change should allow program staff to process builder submissions more frequently than once a month (as is currently done), allowing for faster payment and an improved experience with the program for builders. In the 2016 evaluation report, ADM noted that the acquisition cost for duct sealing was high relative to other measures and variable. In response to that finding, ADM recommended tying the incentive payment to factors related to the estimated kwh savings, namely leakage reduction. Staff considered this recommendation but decided that the benefit of tying the incentive payment to more predictable factors Energy Efficiency Programs 151

155 (number of supply and return vents) was more beneficial to the overall program goals than a performance-based incentive structure. Staff also considered instituting a cap on the incentives for duct sealing but decided against it because the duct sealing incentives get service providers to buy into the program and have benefits beyond the annual energy savings (improved home comfort). Nevertheless, staff reported that the high duct sealing acquisition cost was reduced through redirection of a service provider, who was achieving particularly low savings from the measure, towards other program measures. Almost every participant surveyed reported being somewhat or very satisfied (96%) with the multiple upgrades component overall, with 2% indicating they were neither satisfied nor dissatisfied, and 1% indicating they were dissatisfied. Similarly, almost every participant surveyed reported being somewhat or very satisfied (95%) with the single upgrades component overall. Planned Program Changes There are no planned program changes for the High Performance Homes Program. 3.5 Education Program Program Overview PSO s Education program sought to generate energy and demand savings by providing customers with energy conservation kits and educational materials about energy efficiency. During 2017, three types of energy efficiency kits were distributed through PSO s Education Program: Schools Program Kits, Residential Energy Conservation Kits and Non-Residential Energy Conservation Kits. In 2017, the program distributed a total of 26,212 kits to customers, which included 8,535 Schools Program Kits, 16,175 Residential Energy Conservation Kits, and 1,502 Non-Residential Energy Conservation Kits. Schools Program Kits The distributed kits provided materials related to energy efficiency to fifth-grade teachers within PSO s service territory. After teachers and students completed lessons on how to save energy, students were provided with an energy conservation kit which contained the measures listed in Table This provided teachers, students and parents an interactive opportunity to explore how to use energy efficiently in their homes. In addition to the kit students also receive the Take-Home Workbook. The workbook guides the students through both measure installation-related activities and the self-analysis of the savings their home receives from their installation actions. Thus, educating two generations on the benefits of energy efficiency. Energy Efficiency Programs 152

156 Residential Energy Conservation Kits Active PSO residential customers were sent direct mail fliers to encourage them to request a Residential Energy Conservation Kit which contained the measures listed in Table The contents of the kit were carefully selected ensuring that they were easy to install when customers received them. The residential kits were designed to educate residential customers on energy consumption and provided solutions to move toward energy efficiency in their homes. PSO Non-Residential Conservation Kits PSO s small-business customers were contacted through phone solicitation and encouraged to request a Residential Energy Conservation Kit which contained the measures listed in Table The non-residential kit contained products that provided immediate savings. This allowed PSO small business customers to learn and adopt energy saving practices that would help reduce their energy use and save money. Table 3-82: Summary of Kit Content Quantity by Kit Type Kit Contents Quantity School Kits 9-watt LED Light bulbs 4 7-Plug Advanced Power Strips 1 FilterTone Alarms 1 Digital Thermometers 1 LED Night Lights 1 Residential 9-watt LED light bulbs 3 15-watt LED light bulbs 2 7-plug advanced power strip 1 FilterTone alarm 1 LED night light 1 Digital thermometer 1 Non-Residential 9-watt LED light bulbs 2 11-watt BR30 LED light bulb 1 7-plug advanced power strip 1 Energy Efficiency Programs 153

157 PY2017 performance metrics for all three kits are summarized in Table Table 3-83: Performance Metrics Education Program Metric PY2017 Number of Customers 26,212 Budgeted Expenditures $2,113,500 Actual Expenditures $1,872,941 Energy Impacts (kwh) Projected Energy Savings 4,949,066 Reported Energy Savings 7,948,862 Gross Verified Energy Savings 7,906,421 Net Verified Energy Savings 7,177,779 Peak Demand Impacts (kw) Projected Peak Demand Savings Reported Peak Demand Savings 1, Gross Verified Peak Demand Savings 1, Net Verified Peak Demand Savings 1, Benefit / Cost Ratios Total Resource Cost Test Ratio 4.19 Utility Cost Test Ratio EM&V Methodologies This section provides a brief overview of the data collection activities, gross and net impact calculation methodologies and process evaluation activities that ADM employed in the evaluation of the Energy Education program. Data Collection To determine the quantity of measures delivered, ADM reviewed all entries in the tracking system to ensure there were no duplicate entries. Additionally, ADM verified that each of the kits was delivered using a database of unique shipping tracking numbers for each kit provided. Energy Efficiency Programs 154

158 The primary data collection for the evaluation consisted of a telephone survey to verify program participation as well as self-report data collection from return-by-mail surveys included in the kits. The total surveys conducted for each kit can be seen in Table Two in-depth interviews with PSO program staff were also conducted to gain insight into the process evaluation. Table 3-84: Data Collection and Sample Size Effort by Survey Achieved Sample Size Data Collection Activity Data use Schools Residential Non- Residential Participant Survey (Return by Mail and Phone) NTG ISR Implementation Staff Interviews Process 2 Gross Impact Methodologies The methodology used to calculate energy (kwh) and demand impacts (kw) consisted of: A review of census program tracking data. The tracking data was reviewed for a census of kits. ADM verified that there were no duplicate project data entry errors. A verification of measure installation. ADM calculated installation rates (ISR) by measure for a sample of program participants utilizing data from phone and return by mail survey responses. A review of ex ante savings estimates for each measure. The implementation team utilized the per kit kwh and kw savings reported in the 2016 EM&V Report. The values were slightly reduced (around 2%) to add a degree of conservatism to account for potential variation in evaluation ISRs and NTG scores, which can fluctuate by program year. A calculation of ex post verified savings utilizing the following: Oklahoma Deemed Savings Document, (OKDSD); Arkansas Technical Reference Manual V6 (AR TRM) Pennsylvania Technical Reference Manual V5 (PA TRM). Energy Efficiency Programs 155

159 ADM s approach for estimating savings was unique to each measure type installed. Primarily, algorithms and deemed savings from the Oklahoma Deemed Savings Document and the AR TRM were used to determine verified gross energy and demand impacts; however, when a measure was not found in either Technical Reference Manual, ADM utilized savings algorithms or deemed savings values from the PA TRM. For each measure in the program, total gross energy and demand savings were calculated as a product of the number of measures verified as installed and the deemed savings per measure. A brief description of each measure calculation methodology is described in the sections below. Appendix F includes the measure level algorithms and deemed savings values utilized for the ex post and kwh and kw savings calculations. ENERGY STAR LEDs For residential LED installed ADM used the algorithm in the OKDSD with HOU from ADM s 2016 HOU Memo. Advanced Power Strips For residential advanced smart strips installed ADM used deemed savings values from the OKDSD 60. Deemed savings were applied based on smart strip installation location. During the survey effort, participants asked about the installation location. They were given the option of home entertainment center, home office, other or not installed. Deemed savings were applied to appliances in the other category that effectively switch off the slave outlets when not in use. FilterTone Alarm Neither the AR TRM nor the OKDSD provided an algorithm for FilterTone Alarms. Thus, ADM utilized the algorithm for calculating deemed savings for FilterTone Alarms provided in the Pennsylvania Technical Reference Manual (PA TRM). 61 LED Night Light Neither the AR TRM nor the OKDSD provided an algorithm for the savings for LED Night Lights. LED Night lights savings were calculated using algorithm assumptions from the PA TRM. (see Appendix F). There is no peak demand reduction associated with this measure. 60 The most recent version of the Oklahoma Deemed Savings Document can be found here: 61 Algorithm Source: 2016 Pennsylvania TRM, Page 72. Energy Efficiency Programs 156

160 Digital Thermometer No kwh savings or kw demand reduction were claimed for the digital thermometers distributed in the Schools Program and Residential Energy Conservation Kits Net-to-Gross Estimation PSO customers who receive Residential Energy Conservation kits through the Education Program were surveyed by ADM to determine a program attribution estimation for the NTG calculation. The attribution scoring system was broken down into three components: a measure-level free-ridership score, a project-level free ridership and a spillover score as described below. For the Non-Residential kits, a NTG ratio of 85% was applied. ADM calculated the 85% NTG from an analysis of data from 125 commercial kits collected during the 2016 program year. A review of commercial kit NTG scores was performed and determined 85% was a reasonable value for continued use during the 2017 evaluation. For the School kits, ADM applied a NTG ratio of 100%. The objective of the net-to-gross analysis for the residential kit component was to estimate the share of program activity that would have occurred in the absence of the program. To accomplish this, ADM administered a survey to residential kit program participants that contained questions regarding the participants plans to implement the kit items and the likelihood of implementing those measures had they not been provided through the program. Program participants were asked questions regarding: Whether they had plans to purchase and install the kit item; When they would have implemented the kit item in the absence of the program; The likelihood of purchasing and installing the kit item had they not received it for free. Measure Level Free Ridership Score Participant responses to the survey questions were used to calculate two scores corresponding to the presence of prior plans and the likelihood of installing the items in the absence of the program. Prior Plans Score The prior plans score was calculated as follows: Respondents who indicated that they did not have plans to install the kit item were scored as 0. Respondents who indicated that they did have plans to install the kit item were scored as 1. Energy Efficiency Programs 157

161 This score was adjusted based on the quantity of the number of the items the participant planned to install and the timing of that planned installation. The quantity adjustment was based on the share of items sent that the respondent planned to install. That is if the respondent indicated that they installed one of the three LEDs, a score of 1 was multiplied by The timing adjustment was based on when they would have likely installed the items. For respondents that indicated they would have likely installed the items within the next six months, no timing adjustment was made. Respondents who indicated that they would have likely installed the item in the next 6-12 months, the plans score was multiplied by 0.5. For those that indicated they would install in 12 months, the planned score was set to 0. Likelihood of Project Completion Score The score reflecting the likelihood of completing the project in the absence of the program was based on the following question: How likely or unlikely would you have been to purchase and install the kit items if you had not received them for free? This question was assigned a score for each response as follows: Very likely: 1 Somewhat likely:.75 Neither particularly likely or unlikely:.5 Somewhat unlikely:.25 Very unlikely: 0 Final Free Ridership Score The final free ridership score is equal to the following: Free Ridership = Average (Plans Score, Likelihood Score) * Previous experience adjustment The previous experience adjustment was based on a question related to whether a respondent had similar items currently installed in the home. The free ridership scores for those that answered No to this question were multiplied by 0.5. The free ridership questions were arranged as follows: 1. Indicator one: prior planning 2. Indicator two: stated likelihood in absence of program incentives 3. Mitigating factor one: reported prior experience with energy conservation measure Energy Efficiency Programs 158

162 Measure level free ridership was calculated based on the flow diagram shown in Figure Note that the scoring algorithm required the respondent to indicate a burden of proof that they were a free rider. They must have stated that either 1) they had prior plans to install the measure or 2) they would have likely installed the measure in the absence of the program. Figure 3-14: Free Ridership Scoring Education Program Weighting of Respondent Scores and Determination of Program Level NTGR The free ridership scores for each respondent will be weighted by the ex post kwh savings per kit type to determine the final weighted average free-ridership estimate per customer in the sample. This estimate will be applied to the program level verified gross savings to determine net savings Process Evaluation Activities The process evaluation for the program year 2017 was limited in scope. The process evaluation activities included a review of updated program materials, participant surveys, and interviews with program staff. Table 3-85 below summarizes the data collection activities and corresponding process evaluation research objectives. Energy Efficiency Programs 159

163 Table 3-85: Process Evaluation Data Collection Activities Summary Data Collection Activity Program Staff Interviews Process Evaluation Research Objectives Assess program staff perspectives regarding program operations, strengths, or barriers to success. Participant Survey (Residential) Assess source of program awareness, impact on knowledge and opinions of energy efficient technologies, awareness of PSO rebates, satisfaction with the program and kit measures Impact Evaluation Findings Table 3-86 below displays the gross savings by kit for the program. Table 3-86: Gross Electric Savings Summary by Kit Type for PY2017 Gross Electric Savings (kwh) Kit Number of Kits Realized Ex Post Lifetime Energy Savings Residential 8,535 3,315,918 50,379,322 Schools 16,175 4,015,910 53,780,327 Non-Residential 1, ,593 8,246,608 Total 26,212 7,906, ,406,257 Gross Demand Savings (kw) Kit Number of Kits Realized Residential 8, Schools 16, Non-Residential 1, Total 26,212 1, Table 3-87 below displays the gross savings by kit for the program. Energy Efficiency Programs 160

164 Table 3-87: Net Electric Savings Summary by Kit Type for PY2017 Kit Net-to-Gross Ratio Ex Ante Net Ex Post Net Demand Savings (kw) Ex Ante Net Ex Post Net RR Net Lifetime Energy Savings (kwh) Residential 3,285,285 2,673, % 40,807,251 Schools 4,125,326 4,015, % 53,780,327 Non- Residential 533, , % 7,009,617 Total 7,943,848 7,177,779 1, , % 101,597,194 The difference in ex ante and ex post savings for the residential kits was due to the lower NTG ratio calculated from 2017 survey data. The NTG ratio for residential kits was 87 percent in 2016 and 81 percent in For Schools kits, the difference in the ex ante and ex post savings calculations can be attributed to slightly lower installation rates reported by 2016 and 2017 survey participants. The main variation in ISR was for the 9W LED bulbs. Respondents reported installing 75 percent of the bulbs in 2016 and 60 percent of the bulbs in Table 3-88 presents the verified ex post energy savings (kwh) results of the 2017 Education Program Residential Energy Conservation Kit by measure. Table 3-88: 2017 Residential Kit Gross kwh & KW Savings Summary by Measure Measure Number of measures per Kit ISR Ex Post Energy Savings (kwh) Ex Post Lifetime Energy Savings (kwh) Ex Post Peak Demand Savings (kw) 9-watt LED Light Bulbs 3 86% 718,592 14,371, watt LED 2 73% 679,660 13,593, Plug Advanced Power Strip 1 88% 782,516 7,825, LED Night Light 1 97% 217,164 1,737,310 - FilterTone Furnace Filter Alarm 1 60% 917,987 12,851, Total 3,315,918 50,379, Energy Efficiency Programs 161

165 Table 3-89 shows net savings by measure. Measure (3) 9-watt LED Light Bulbs (2) 16-watt LED (1) 7-Plug Advanced Power Strip (1) LED Night Light (1) FilterTone Furnace Filter Alarm Net-to- Gross Ratio Table 3-89: Residential Kit Net Savings Summary Net Energy Savings (kwh) Net Ex Ex Ante Post Net Demand Savings (kw) Net Ex Ex Ante Post Net RR Net Lifetime Energy Savings (kwh) 81% 713, , % 11,587,320 81% 658, , % 10,959,535 81% 796, , % 6,309,044 81% 209, , % 2,451,240 81% 907, , % 5,921,028 Total 3,285,285 2,673, % 81% The difference in ex ante and ex post saving for the residential kits was due to the lower NTG ratio calculated from 2017 survey data. The NTG ratio for residential kits was 87 percent in 2016 and 81 percent in Table 3-90 shows the verified ex post energy savings (kwh) results of the 2017 Education Program School Kit, by measure. The difference in the ex ante and ex post savings calculations can be attributed to slightly lower installation rates reported by 2016 and 2017 program participants. Table 3-90: 2017 School Kit Gross kwh and kw Savings Summary by Measure Measure Number of Measures in Kit ISR Ex Post Energy (kwh) Savings Ex Post Lifetime Energy Savings (kwh) Ex Post Peak kw Savings 7-Plug Advanced Power Strip 1 76% 1,230,660 9,845, LED Night Light 1 80% 340,109 2,380, Furnace Filter Alarm 1 42% 1,224,756 17,146, watt LED 4 60% 1,220,385 24,407, Total 4,015,910 53,780, Energy Efficiency Programs 162

166 The School kits program had a NTG of 100%. Net savings are shown in Table Table 3-91: School Kit Net Savings Summary by Measure for PY2017 Measure Net-to- Gross Ratio Net Energy Savings (kwh) Ex Ante Net Ex Post Net Demand Savings (kw) Ex Ante Net Ex Post Net RR Net Lifetime Savings 7-Plug Advanced Power Strip LED Night Light Furnace Filter Alarm 100% 1,142,742 1,230, % 9,845, % 325, , % 2,380, % 1,201,119 1,224, % 17,146,590 9-watt LED 100% 1,456,004 1,220, % 24,407,692 Total 100% 4,125,325 4,015, % 53,780,326 Table 3-92 shows the verified ex post energy savings (kwh) results of the 2017 Education Program Non-Residential Kit by measure. Table 3-92: 2017 Non-Residential Gross Electric Savings Summary by Measure Measure Number of Measures per Kit ISR Realized kwh Savings Ex Post Lifetime Energy Savings (kwh) Realized Peak kw Savings 9-Watt LED Light Bulbs 2 85% 289,658 4,344, BR30 11-Watt Directional LED 1 100% 210,478 3,157, Plug Advanced Power Strip 1 81% 74, , Total 574,593 8,246, Energy Efficiency Programs 163

167 The overall kit NTG ratio was 85%. 62 Net savings are shown in Table Table 3-93: Non-Residential Kit Net Savings Summary Measure Net Energy Net Demand Net Lifetime Net-to- Savings (kwh) Savings (kw) Energy Gross Net RR Net Ex Net Ex Savings Ratio Ex Ante Ex Ante Post Post (kwh) (2) 9-watt LED Light Bulbs 84% 288, , % 3,654,033 BR30 Dimmable 11-Watt LED 89% 172, , % 2,816,195 7-Plug Advanced 81% 72,483 60, % 604,592 Power Strip Total 85% 533, , % 7,074,821 ADM recommends that PSO continue to cross-promote other programs through the kits. Feedback suggests that the kits do generate a moderate amount of awareness of other programs. Consider more detailed information about programs, such as retail locations where discounted products/appliance can be purchased Conclusions The School Kits component continued to operate smoothly and did not have any challenges meeting its goals. Staff noted that program participation is currently limited by available budget and interest in the component exceeds what can be supported by the available funding. The Residential Kits Program achieved its distribution goals without difficulty. The Non-residential Kits component did not meet its annual goal of 3,000 kits distributed. Based on the program participation in PY2016 and PY2017, PSO decided to reduce the goal to 1,000 kits, which PSO was able to meet in Twenty-percent of respondents reported that they were aware of the PSO discounts and incentives and learned of them from information provided in the kit. Participants who received a residential kit were satisfied with the program overall (94%) and the time it took to get the kit (87%). 62 For the Non-Residential kits, ADM applied a NTG ratio by measure from the 2016 evaluation effort%. ADM calculated the 85% NTG from an analysis of data from 125 commercial kits collected during the 2016 program year. ADM performed a literary review of commercial kit NTG scores and determined the value to be reasonable for continued use during the 2017 evaluation. Energy Efficiency Programs 164

168 3.6 Behavioral Modification Program Overview PY2017 was the first year PSO administered the Behavioral Modification Program. ICF implemented the program for PSO. The program first deployed to selected customers starting 10/25/2017. The program was designed to generate greater awareness of energy use and ways to manage energy use through energy efficiency education in the form of Home Energy Reports (HERs). The HERs provide customers with energy conservation tips. It is expected that through this education, customers will adopt energy conservation tips that will lead to more efficient energy use in their homes. Customers who had HERs delivered by could choose to opt out if the customer no longer wanted to receive the HERs. Energy Efficiency Programs 165

169 Table 3-94: Performance Metrics Behavioral Modification Program Metric PY2017 Number of Customers 92,939 Budgeted Expenditures $1,223,500 Actual Expenditures $451,869 Energy Impacts (kwh) Projected Energy Savings 14,000,000 Reported Energy Savings 0 Gross Verified Energy Savings 0 Net Verified Energy Savings 0 Peak Demand Impacts (kw) Projected Peak Demand Savings 4, Reported Peak Demand Savings 0 Gross Verified Peak Demand Savings 0 Net Verified Peak Demand Savings 0 Benefit / Cost Ratios Total Resource Cost Test Ratio Utility Cost Test Ratio NA NA EM&V Methodologies This section provides a brief overview of the data collection activities, gross and net impact calculation methodologies, and process evaluation activities that ADM employed in the evaluation of the Energy Education program. To determine annual kwh savings and kw reduction, ADM performed an analysis of the billing data for participants in the program utilizing panel regression modeling. The data cleaning steps and methodology for the panel regression approach are presented in this following section. Energy Efficiency Programs 166

170 Preparation of Data ADM incorporated several types of data into the preparation of the dataset that was utilized in the regression analysis outlined in this section: 1. PSO provided raw monthly billing data for all treatment and control group participants for the period January 1, 2016, through February 18, Regional weather data 3. Participant information 4. Home energy reports delivery data a. Start date for HERs program b HERs distribution data 5. A dual enrollment dataset compiled by ADM of participants in PSO s other residential programs. ADM performed the following steps to prepare the dataset that was utilized to determine the verified energy savings for the 2017 Home Energy Reports Program. 1. Verified that participants were sent HERs during Merged this dataset with the raw billing data provided by PSO. 3. Cleaned the data for duplicate bills, outliers, and string characters in the monthly consumption column. 4. Removed billing data with negative consumption on their monthly bill 5. Removed billing data with number of billing days less than 25 or greater than Removed customers that did not have both pre-program and program year data 7. Removed customers that participated in PSO s other residential programs Methodology for Regression Approach ADM utilized the mixed effects panel regression model specified in Equation 3-5 to determine daily average electricity savings for treatment group members. AEC i,t = β 1 CDD i,t + β 2 HDD i,t + β 3 Post i,t + β 4 Treatment i,t + β 5 Post i,t Treatment i,t + α i Customer i + E i,t Equation 3-5: Mixed Effects Panel Regression Model Where the subscript i denotes individual customers and t = 1,,T(i) serves as a time index, where T(i) is the number of bills available for customer i. The model is defined as mixed effects because the model decomposes its parameters into fixed-effects (i.e. HDD, CDD, Energy Efficiency Programs 167

171 Post, Treat, and its various interactions) and random effects (i.e. the individual customer s base usage). A fixed effect is assumed to be constant and independent of the sample, while random effects are assumed to be sources of variation (other than natural measurement error) that are uncorrelated with the fixed effects. The variables included in the regression model are specified in Table The program implementer provided ADM with a dataset that included the participation start date for each treatment group member and their corresponding control group. Ordinarily, the first billing period after the beginning of treatment is considered the deadband period. Observations that occur in the deadband period are not included in the mixed effects panel regression. For the treatment and control group members, the post period begins in the first billing period following the deadband period. However, because the HERs program started late in the year in 2017, the deadband period was eliminated from the analysis to provide for more post period billing data. The post period for 2017 began in the first billing period following the start of the program. The post variable is defined as a 0 in the billing periods prior to the beginning of treatment and a 1 for billing periods following the beginning of treatment. Heating degree day (HDD) and cooling degree day (CDD) were the metrics used in the model to control for energy demand based on outside temperature. HDD is derived from the difference between 65 degrees (the outside temperature at which it is assumed that a building needs no heating) and the actual outside air temperature. CDD is derived from the difference between the actual outside air temperature and 75 degrees (the outside temperature at which it is assumed that a building needs no heating). If HDD and CDD go negative, there s a minimum value of 0 applied. Table 3-95: Description of Variables Used in the Regression Model Variable Average Electricity Consumption (AEC i,t ) Customer Cooling Degree Days (CDD) Heating Degree Days (CDD) Post Treatment Et Variable Description Average daily use of electricity for period t for a customer (determined by dividing total usage over a billing period by number of days in that period) A panel of dummy variables that is a 1 if customer i is the i in AEC i,t or a 0 otherwise. The mean cooling degree days per day during the billing period. The mean heating degree days per day during the billing period. Post is a dummy variable that is 0 if the monthly period is before the customer received their first HER and 1 if the monthly period is after the customer received their first HER. Treatment is a dummy variable that is 0 if the customer is a member of the control group and a 1 if the customer is a member of the treatment group. Et is an error term Table 3-96 describes the coefficients that were determined by using the mixed effects panel model shown in Equation 3-5. Energy Efficiency Programs 168

172 Table 3-96: Description of the Coefficients Estimated by the Regression Model Coefficient α 1 β 1 β 2 β 3 β 4 β 5 Coefficient Description α 1 is a coefficient that represents the grand mean (mean of the unique customer specific intercepts). The customer specific intercepts control for any customer specific differences. β 1 is a coefficient that adjusts for the customer s cooling season weather-sensitive usage. β 2 is a coefficient that adjusts for the customer s heating season weather-sensitive usage. β 3 is a coefficient that adjusts for whether customer i s monthly billing data in period t is in the pre or post period β 4 is a coefficient that adjusts for whether customer i is in the treatment group or the control group. β 5 is a coefficient that adjusts for the interactive effect between whether customer i s monthly billing data in period t is in the pre or post period and whether customer i was in the treatment or control group during period t. The value of β5 is the kwh savings per customer per day if it is significant. Calculation of kwh Savings The kwh savings was determined by multiplying the per-participant kwh savings value for the post period treatment groups (β 5 ) by the number of customers who were participants in the treatment group. Calculation of kw Peak Reduction The kw peak reduction was determined by dividing the kwh savings by the number of hours in the year (8,760) Net-to-Gross Estimation Because the program is administered using an RCT design, free riders are equally likely to be distributed in both the treatment and control group. Additional controls are put in place to control for cross-program participation program participants who are found to have participated in other energy efficiency program are removed from the regression analysis. Therefore, it is assumed that the savings detected is attributable directly to induced savings due to the receipt of the home energy report and NTG is assumed to be Impact Evaluation Findings The following section reports the findings for 2017 kwh savings and kw peak reduction. Data Review ADM reviewed the billing data and the number of participants following each data cleaning step. The results of this process are reported in Table The step that removed the Energy Efficiency Programs 169

173 largest number of participants (except for cross participation) was the verification that a participant had valid pre and post period data. Table 3-97: Number of Participants Remaining After Each Data Cleaning Step Data Cleaning Step Treatment Group Control Group Pre Post Pre Post Verified billing data 104,999 92,957 24,000 21,217 Filtered less than 0 or greater than 10,000 kwh billing consumption and less than 25 or greater than 35 billing days 104,999 92,939 24,000 21,214 Removed duplicate bills 104,999 92,939 24,000 21,214 Limited data to only those that have both pre and post period data Removed cross participants in other PSO programs 92,939 92,939 21,214 21,214 83,268 83,268 18,990 18,990 The number of participants with valid billing data and the average daily consumption for both the treatment and control groups during the pre-program period was determined. This step was performed to ensure that the average daily pre-treatment consumption was similar for both the treatment and control groups. The results are reported in Table Table 3-98: Pre-Treatment Average Daily Consumption Treatment Group Control Group Count Average Daily Pre-Treatment kwh Count Average Daily Pre-Treatment kwh 92, , ADM also evaluated the percentage of treatment participants that cross-participated in other PSO programs and reported the findings in Table Table 3-99: Cross Participation with other PSO Programs Treatment Group Count Count of Treatment Group in Other DSM Programs Percent of Treatment Group in Other DSM Programs 92,939 9, % Energy Efficiency Programs 170

174 Gross Impact Estimation Calculated Energy Savings (kwh) ADM attempted to conduct the analysis on these customers bills in order to identify any initial findings. However, ADM found that very few bills met the time constraint of beginning after the enrollment date and ending before the end of PY2017. The results of the analysis were not statistically significant, and therefore ADM did not apply any savings to these customers for PY2017. However, these customers will be evaluated as a typical treatment group beginning in PY2018. Table provides the results of the mixedeffects panel regression modeling. Table 3-100: Results of Mixed Effect Panel Regression Modeling Variable Coefficient p-value Intercept <0.01 HDD <0.01 CDD <0.01 Post <0.01 Treat Post x Treat R-squared 0.82 Although the program was administered in late 2017, the program has not been administered long enough for savings to be detectable, ADM will revisit in Energy Efficiency Programs 171

175 4 Demand Response Programs PSO s demand response (DR) portfolio in 2017 consisted of one program that targeted commercial and industrial customers. Results of the evaluation, measurement, and verification efforts are shown below in Table 4-1. Table 4-1: Peak Demand Reduction Demand Response Programs Gross Peak Demand Reduction (MW) Net Impacts Program Projected Reported Verified Gross Realization Rate NTG Ratio Net Peak Demand Reduction (MW) Peak Performers % 100% Demand Response Totals % 100% As shown in Table 4-2, PSO did not report annual energy savings for demand response programs. This program s sole aim is to provide load reduction capabilities during times of high demand. However, because of participants voluntary load reductions during event days, there are energy savings associated with the program. These energy savings are not persistent, in the sense that an energy efficient equipment installation provides energy savings for the life of the equipment, while energy savings from DR programs only occur during event days. Table 4-2: Annual Energy Savings Demand Response Programs Gross Annual Energy Savings (MWh) Net Impacts Program Projected Reported Verified Gross Realization Rate NTG Ratio Net Annual Energy Savings (MWh) Peak Performers N/A 100% Demand Response Totals N/A 100% Peak Performers Overview The Business Demand Response program, also referred to as Peak Performers, is a Demand Response (DR) program for commercial and industrial customers in the PSO service territory. Nonresidential PSO customers enroll in the program and are notified when a load reduction event is initiated. Participants have the option of participating in Demand Response Programs 172

176 each event individually, and are paid incentives based on average reduction over the course of all events. Incentives are set at $32 per average kw reduction over all event hours, and participants receive a 5% payment bonus if they opt to participate in all reduction events throughout the year (two events in 2017). There is no direct penalty for opting out of specific event days. The program is active during summer months, when average demand typically approaches designated capacity thresholds. During the summer of 2017, a total of 1,427 premise account numbers participated in two DR events. The two events lasted from 2-6 PM (7/19/2017 and 7/20/2017). ADM s evaluation developed verified demand reduction estimates that were lower than reported values. Both reported and verified peak demand reduction represent the average kw reduction for each customer over all 8 event hours (two event days, four hours for each event), summed across participants. Demand Response Programs 173

177 Table 4-3: Performance Metrics Peak Performers Metric PY2017 Number of Customers 221 Budgeted Expenditures $3,296,900 Actual Expenditures $2,594,726 Energy Impacts (kwh) Projected Energy Savings 255,000 Reported Energy Savings 0 Gross Verified Energy Savings 541,927 Net Verified Energy Savings 541,927 Peak Demand Impacts (kw) Projected Peak Demand Savings 51,000 Reported Peak Demand Savings 64,259 Gross Verified Peak Demand Savings 45,739 Net Verified Peak Demand Savings 45,739 Benefit / Cost Ratios Total Resource Cost Test Ratio 7.79 Utility Cost Test Ratio Impact Evaluation The section below covers ADM s impact evaluation methodology and results for the 2017 Peak Performers Program. Demand Response Programs 174

178 PSO Methodology for Estimating Customer Baselines For the purposes of financial settlement with Peak Performer participants, PSO uses a top 3-of-10 baseline days methodology to estimate participants baseline load, or the demand that participants would have used had no Peak Performer event been called. Reported program impacts were calculated based on this baseline estimation methodology. For each premise, one applies the following algorithm: 1. For an event day D, let D(h) be the participant s actual electric demand at hour h on D. 2. Starting with the day before D, take the most recent 10 non-weekend, non-holiday, non-peak Event days. These are the eligible baseline days. 3. For each of the eligible baseline days, calculate the average midday electric demand during the hours corresponding to the Peak Event (usually 2 PM 6 PM, but can be any two to four-hour period between 1 PM and 7 PM). Rank the eligible baseline days in descending order of this average peak time demand. 4. Take the top 3 days from the previous step and average their loads hour by hour. This is the unadjusted baseline, B(h). 5. If, on average, the ratio of B(h)/D(h), between 10 AM and 12 PM, is less than 1 (that is, the baseline is too low), multiply B(h) by the reciprocal of that ratio so that the baseline and event loads match prior to the event. The most B(h) can be adjusted upward is 30%; no downward adjustments are made. Reported demand reduction and payments made to Peak Performers participants depend on the difference, B(h)-D(h). PSO provided hourly interval data for all the facilities involved in the Peak Performers program. PSO staff also provided internal audits for all the events, which are produced by a database script that implements the 3-of-10 baseline. ADM used these audits and interval data to independently verify that the baseline loads reported by PSO were calculated according to the algorithm described above. ADM Baseline Methodology In the case of evaluating demand reduction impacts associated with the Peak Performers program, baselines, or counterfactuals, should represent what participant s usage would have been if the event had not occurred. In 2017, ADM employed multiple baseline methodologies and selected the best fitting models (i.e. models that produced load profiles which best represented participant s usage in absence of the program as determined by objective statistical test) for each premise number. These methodologies included the following models: Demand Response Programs 175

179 Table 4-4: ADM Models Model Name 3 of 10 Unadjusted 3 of 10 Adjusted 3 of 10 Weather Sensitive 5 of 10 Unadjusted 5 of 10 Adjusted 5 of 10 Weather Sensitive 9 of 10 Adjusted 9 of 10 Unadjusted 9 of 10 Weather Sensitive Description Model described in section 0 without the adjustment described in step 4. Model described in section 0, but allows for a ±30% adjustment. The 3 of 10 unadjusted model with a weather sensitivity adjustment based on temperature s impact on energy usage for each premise from June to August. Model described in the preceding in section 0, but with 5 baseline days selected and without the adjustment described in step 4. Model described in section 0, but with 5 baseline days selected and allows for a ±30% adjustment. The 5 of 10 unadjusted model with a weather sensitivity adjustment based on temperature s impact on energy usage for each premise from June through August. Model described in section 0, but with 9 baseline days selected and allows for a ±30% adjustment. Model described in section 0, but with 9 baseline days selected and without the adjustment described in step 4. The 9 of 10 unadjusted model with a weather sensitivity adjustment based on temperature s impact on energy usage for each premise from June through August. ADM identified candidate baseline best fits using residual root mean squared error (RRMSE) scores 63 from the five highest usage weekdays during the program year during typical demand response hours (8/15/2017, 8/16/2017, 8/17/2017, 8/18/2017, 8/21/2017) for each premise number. These days were chosen as high usage days serve as a good proxy for event days. Although ADM cannot determine for certain that the two event days 63 ADM also tested best fits of models using mean absolute error. This returned results consistent with using RRMSE. Demand Response Programs 176

180 would have been amongst some of the higher usage days of the summer, it is a safe assumption as they were the two hottest days of the year in Tulsa and many program participant s energy usage is HVAC-driven. It has been our experience that baseline estimation methodologies often produce generally consistent results, but in some cases, these estimations may produce divergent results. To minimize calculation bias, we combined results as a weighted average of the best three models for each premise number. The weights were the inverse squares of the model RRMSEs. For example, if the three best fitting models have RRMSEs of 5%, 11%, and 52% respectively, their relative weights will be 79%, 20%, and 1% respectively. ADM Baseline Methodology for Small Sites Baselines with a day of adjustment were excluded from ex post analysis of smaller sites as customers were notified a day ahead of an event and could have taken energy reducing or increasing steps prior to the actual event such as shutting down machinery, sending employees home, or precooling a building that would have impacted the reduction calculation. This potentially could have led to biased baseline energy usage estimations. The modified selection of models was then compared to the five highest usage weekdays using RRMSE with the three best fitting models being selected and weighted in the way described in the previous section. ADM Baseline Methodology for Large Sites For the twenty sites with the largest kw reductions in the program, ADM chose to modify the models considered for RRMSE testing based on premise level information such as business type and pre-event energy usage. Weather sensitive models were dropped if a premise s energy usage was determined to not be weather dependent. Adjusted models were added if the premise did not show an abnormal dip or spike pre-event. The modified selection of models was then compared to the five highest usage weekdays using RRMSE with the three best fitting models being selected and weighted in the way described in the previous section. The table below shows the action taken regarding models for all twenty sites. Demand Response Programs 177

181 Table 4-5: Large Site Model Selection Premise Name Reported kw Weather Dependent Energy Usage? Abnormal Pre- Event Usage? What Models were Added/Removed A 481 Yes No Add adjusted models B 1,087 Yes No Add adjusted models C 466 Yes No Add adjusted models D 708 Yes No Add adjusted models E 639 No No F 1,377 No No Add adjusted models and remove weather sensitive models Add adjusted models and remove weather sensitive models G 750 Yes Yes No Change H 1,506 Yes Yes No Change I 3,611 No Yes Remove weather sensitive models J 2,389 No Yes Remove weather sensitive models K 1,438 No Yes Remove weather sensitive models L 693 No Yes Remove weather sensitive models M 466 No Yes Remove weather sensitive models N 3,215 No Yes Remove weather sensitive models O 1,148 No Yes Remove weather sensitive models P 5,227 No Yes Remove weather sensitive models Q 1,097 No Yes Remove weather sensitive models R 3,490 No Yes Remove weather sensitive models S 1,176 No Yes Remove weather sensitive models T 2,191 No Yes Remove weather sensitive models Review of Program Interval Data ADM reviewed program interval data found on the PSO s SSRS for both completeness and accuracy. Additional attention was given to sites where data from the SSRS was amended in past evaluations. ADM found that all event day data was present in PSO s SSRS for all but one participant. In a small minority of cases, data within the 10-day baseline period was missing, but it never exceeded 3 days for any premise and did not pose issues in the development of a counterfactual baseline. Demand Response Programs 178

182 All sites that previously required additional interval data besides that found in the SSRS appear to no longer require it. The graphs below are of several sites that exemplify this. Figure 4-1: Premise F Event Graphs Figure 4-2: Premise R Event Graphs Demand Response Programs 179

183 Figure 4-3: Premise I Event Graphs The three sites show a reduction at the time of the event indicating that a shift in timestamp is not present. This finding, in addition to an examination of a random sample of 30 other participants, led ADM to conclude that no additional interval data was required to calculate program energy impacts. Net-to-Gross Methodology Demand response programs are not likely to have NTG effects because customers are unlikely to curtail load in absence of the program. An NTG ratio of 100% was assumed for this program. Impact Evaluation Results This section presents the results of ADM s impact evaluation of the Peak Performers Program Program Level Graphs The graph below presents the aggregated results of each model averaged for each premise account for all nonevent, nonholiday June through August weekdays. Demand Response Programs 180

184 Figure 4-4: Average Weekday Usage for All Models The nine of ten models appear to perform the best compared to the average summer day while the ex ante model appears to overestimate more so than other models. This is useful information; however, how models perform when compared to high usage nonevent days is more important as the intention of event days is to reduce demand during the highest usage time periods of the year. The following graph presents the aggregated results of each model averaged for each participant for the five highest usage summer weekdays during the typical event period (8/15/2017, 8/16/2017, 8/17/2017, 8/18/2017, 8/21/2017). Demand Response Programs 181

185 Figure 4-5: Five Highest Weekday Usage for All Models On these days, which serve as a good substitute for event days, the five out of ten models tend to perform best while the ex ante model still overestimates savings by the largest amount. The following graph presents the aggregated results of actual usage, ex ante modeled usage, and ex post modeled usage for all nonevent, nonholiday June through August weekdays. Demand Response Programs 182

186 Figure 4-6: Average Actual, Ex Ante, Ex Post Weekday Usage For the average summer nonevent day, ADM s baseline performs better than the ex ante model, but would appear to overestimate energy usage. The following graph presents the aggregated results of actual usage, ex ante modeled usage, and ex post modeled usage five highest usage summer weekdays during the typical event period. Demand Response Programs 183

187 Figure 4-7: Five Highest Usage Days Average Actual, Ex Ante, Ex Post Weekday Usage On the five highest usage days, which serve as a proxy for event days, ADM s baseline may overestimate energy usage slightly but provides a good fit for actual usage. The following graph presents aggregated results of actual usage, ex ante modeled usage, and ex post modeled usage for event one on 7/19/2017. Demand Response Programs 184

188 Figure 4-8: Actual, Ex Ante, and Ex Post Usage for Event 1, 7/19/2017 ADM s baseline replicates actual energy usage well during nonevent periods and appears to serve as a reasonable counterfactual during the event period. The following graph presents aggregated results of actual usage, ex ante modeled usage, and ex post modeled usage for event two on 7/20/2017. Demand Response Programs 185

189 Figure 4-9: Actual, Ex Ante, Ex Post Usage for Event 2, 7/20/2017 Both models fit the actual event day usage well during nonevent periods; however, the ADM baseline does not follow an increase in usage at around 10 am. This jump in usage could be due to participants ramping-in to an event by precooling their facility. Because this activity is caused by the event, it would not be desirable for a baseline model to replicate it. Customer X Results This year s analysis of Customer X (Sites I, K, N, Q, and R) was done in the same way as the other fifteen largest sites in the program; however, ADM evaluation staff determined that it would be valuable to include details on the customer in the program memo as a reference point from past analysis. The table below presents the customer s ex ante and ex post reduction. Demand Response Programs 186

190 Table 4-6: Customer X Peak Demand Reduction Premise Name Gross Peak Demand Reduction (MW) Ex Ante MW Ex Post MW Realization Rate I % R % N % K % Q % Total % Customer X was again a large portion of program demand reduction representing about 20% of ex ante impacts. Below is a graph of the customer s usage during event 1. Figure 4-10: Customer X Actual, Ex Ante, and Ex Post Usage Event 1, 7/19/2017 Below is a graph of the customer s usage during event 2. Demand Response Programs 187

191 Figure 4-11: Customer X Actual, Ex Ante, and Ex Post Usage Event 2, 7/20/2017 As the figures above show, the counterfactual event day usage is difficult to determine due to Customer X s unpredictability in day-to-day usage. Peak Performers Peak Demand Reductions Demand response event impacts are estimated by comparing the event day demand curves with the estimated baseline demand curves; the difference between the two is the estimated peak demand reduction. As described in PSO Methodology for Estimating Customer Baselines and ADM Baseline Methodology sections, ADM used hourly interval data to recreate the baseline estimations used to determine reported impacts. The process was then repeated, this time using ADM s baseline methodology described in ADM Baseline Methodology section and represented by ADM Adjusted Baseline in the graphs in the previous section. Below are ADM s peak demand reduction estimates for Demand Response Programs 188

192 Table 4-7: Peak Demand Reduction Peak Performers Gross Peak Demand Reduction (MW) Net Impacts Program Projected Reported Verified Gross Realization Rate NTG Ratio Net Peak Demand Reduction (MW) Peak Performers % Peak Performers Annual Energy Savings The Business Demand Response Program is designed primarily as a resource for procuring peak demand savings during periods of high demand. As such, the program does not report annual kwh savings. However, the program does generate energy impacts during and surrounding called events. These impacts are not lasting, in the sense that kwh savings from a lighting retrofit might last the lifetime of the installed lighting fixtures. When a peak demand event is called usually hours to a full day before the actual event period participants have several options. They might decrease electric energy usage immediately in anticipation of the upcoming event, or they might increase usage for the remaining pre-event hours in anticipation of future usage reduction. Additionally, the post-event hours are of interest because it may take several hours for facilities to restore electric energy usage to pre-event operation levels. Facilities might also increase electric energy usage immediately after the conclusion of an event to make up for previously reduced usage. ADM chose to use the full event day to evaluate kwh savings for 2017 to capture before, during, and after event energy usage behavior. Verified kwh savings presented below represent the net difference in energy consumption (between the estimated baseline and the observed usage) summed over the event days (48 total hours). It is possible that some facilities shifted event related load outside of the event day due to their reduction; however, given the post-event survey findings, investigating the entire event day appears to be sufficient. The following table presents ADM s annual energy savings estimates for Table 4-8: Annual Energy Savings Estimates Gross Annual Energy Savings (MWh) Net Impacts Program Projected Reported Verified Gross Realization Rate NTG Ratio Net Peak Demand Reduction (MWh) Peak Performers N/A Demand Response Programs 189

193 4.1.2 Process Evaluation This section contains a summary of process evaluation. Process Evaluation Activities The process evaluation was designed to answer the following research questions that relate to the delivery of the program. What changes, if any, were made to the program design or implementation procedures? How did new participants learn of the program? What factors motivated their decision to participate? Were new program participants satisfied with their experience? What was the level of satisfaction with the event notification process and other aspects of program participation? What actions did participants take to reduce load? What were the key successes and challenges during the 2017 program year? To address these questions, ADM completed an interview with the program manager and administered a post-event survey to participants and a new participant survey. For all program participants, ADM administered a brief web-based survey shortly after the first demand reduction event of the year. The survey will cover the following topics: Sufficiency of time given for notice of the event; How the participant was notified of the event; and Actions taken, if any, to reduce load during the event. The purpose of administering this survey was to get a better understanding of the types of actions that customers take to reduce their load and to get timely feedback on the notification process. By administering these surveys shortly after the time when the event is called, customer responses should be less likely to be adversely affected by poor recall. A second survey was administered after the peak period season to customers who were new to the program in The purpose of this survey will be to understand: Sources of program awareness; Factors that influenced participation; Awareness of energy use monitoring tools provided through Vision; Likelihood of participating again in 2018; and Program satisfaction. Demand Response Programs 190

194 Ex Post MW Reductions Process Evaluation Findings The program design has remained consistent since The 2013 Annual Report contains a comprehensive discussion of the program design. Program Participation Location of Participant Facilities: Table 4-9 displays the number of participating premise accounts by district. The share of sites that participated from the Lawton district increased from 19% in 2016 to 27% in Staff noted that the increase was due to the completion of the installation of AMI meters in that region and an outreach campaign that targeted this region. Table 4-9: Distribution of Premise Accounts Across Districts District Percent of Premise Accounts Percent of Business Establishments* Lawton 27% 22% McAlester 18% 11% Tulsa 38% 54% Tulsa Northern 16% 13% *Source: U.S. Census, County Business Patterns: 2014, ZIP Code Industry Detail File Figure 4-12 shows the growth in the number of program participants and the year-to-year changes in MW reductions that have been facilitated by the high retention rate and recruitment of new participants MW Reductions Figure 4-12: Number of Premise Accounts and MW Reductions Program Operations Perspective ADM completed an interview with the PSO program manager in November of The purpose of the interview was to understand any changes made to the programs and the intended and realized impacts of the changes. Overall, the program remained largely consistent in its design and procedures. Nevertheless, staff noted some continued Demand Response Programs 191

195 improvements made to the program operations. The following material summarizes the main findings from these interviews. The installation of AMI meters, which are required to participate in the program, in the Lawton region was completed at the end of The program focused on enrolling new customers in the Lawton region and staff noted that, for example, 15 new schools in the Lawton region enrolled. Direct outreach through PSO account managers was the primary means of recruiting new customers, but these efforts were supplemented with a direct mail campaign that included information on energy efficiency opportunities in addition to information on Peak Performers. Additionally, two articles on the program were posted to Questline in March and June of The program also developed a video posted on YouTube that explains how the program works and actions that customers can engage in to reduce their loads. The video featured a cartoon superhero taking steps to reduce energy use after receiving a notification that an event was occurring. Overall, the notification process went smoothly this year. Two issues noted were that a change in AEP staff resulted in some glitches during the testing period that were worked out prior to the first event notification. An efficiency improvement was made to the enrollment process. A new enrollment process for enrolling customers with several accounts was developed that reduced the number of enrollment forms to be completed. Staff also noted that the delivery of results to participating customers was reduced to four days and that this process went well in Each year, the program hosts an awards ceremony for program participants. For the first time, staff had participating customers speak about their experiences with the program at these events. Staff reported that customers greatly appreciated the opportunity to hear from other customers and that the sharing of these personal experiences are particularly valued by their customers. New Participant Survey Results An survey was administered to new program participants in October and November ADM identified new participants by comparing project numbers and company names listed in the 2017 program data with data from the program years. The survey was conducted to collect data on how participants learned of the program, motivations for participating, potential barriers to participation, and overall experience with the program. Demand Response Programs 192

196 The survey was distributed to 29 new program participant contacts. Contacts were sent an initial invitation and two follow-up s. Table 4-10 summarizes the response to the survey. Table 4-10: New Participant Survey Response Response Metrics Number of Participant Contacts Number of Contact s 29 delivery Failures 0 Completions 15 Completion Rate 52% Table 4-11 summarizes the roles of survey respondents in their organization s participation in the program. Enrollment Table 4-11: Survey Respondent Program Role Role in Participation in Program Percent of Respondents (n = 12) Signed up for Peak Performers 67% Communicate to others in organization that an event will be called 50% Manage energy use during event 50% Informing others about Peak Performers 42% Don't know 8% Direct outreach by a program representative, whether by phone, in-person, or , was the source of awareness for 75% of the respondents (Figure 4-13). Additionally, two other respondents reported learning of the program through the program website. Demand Response Programs 193

197 Figure 4-13: Initial Source of Program Awareness The motivations for participating in the program were consistent with motivations provided by new participants in The primary motivation was to reduce their energy bill, cited by 83% of respondents. Additionally, 42% of respondents cited the incentive as a reason for participating. Sustainability concerns were noted by 67% of respondents. Demand Response Programs 194

198 Figure 4-14: Motivations for Participating in Peak Savers None of the participants reported that they had initial concerns about participating. Participation in Events One participant reported that they had not participated in an event and the reason given was that the school the respondent worked for was not in session. Fifty-eight percent of respondents reported that the number of events called during the peak season was about what they were expecting and 25% said it was fewer than expected. This respondent was expecting five events. The findings on the comparison of actual to expected number of events were consistent with 2016 findings. It is likely the case that most participants reported fewer than expected events because two events were called when up to 12 may be called during a season. Number of Events as Compared to Expected Table 4-12: Expected Number of Events Percent of Respondents (n = 12) Average Number of Expected Events (n = 1) More than expected 0% n/a Fewer than expected 25% 5 About the number expected 58% NA Don't know 17% NA Demand Response Programs 195

199 Awareness of Available Information on Achieved Reductions Additional questions on participants awareness of summary information were added after the survey was launched. Because they were added late, two of the twelve respondents were asked these questions. One of the two respondents stated that they were aware of the that summarized the reductions and incentive payments for the premise and the website provides the final verified results and a graphical presentation of the results. This respondent also stated that they had viewed the website. The other respondent did not recall if they had seen the . Table 4-13: Awareness of Summary Information of Reductions during Events Response Awareness of Summary (n = 2) Awareness of Website with Results (n = 1) Viewed Website (n = 1) Yes 50% 100% 100% No 0% 0% 0% Don't know 50% 0% 0% Future Intentions to Participate and Satisfaction Most participants (75%) reported that they were very likely to participate in the opportunity during One respondent indicated that they were very unlikely to participate. This respondent did not provide a reason for why they were unlikely to participate in 2018 and did not report any dissatisfaction with the program. Likelihood of future participation Table 4-14: Likelihood of Future Participation Percent of Respondents (n = 12) Very unlikely 8% Somewhat unlikely 0% Neither particularly unlikely nor likely 0% Somewhat likely 8% Very likely 75% Don't know 8% Demand Response Programs 196

200 Figure 4-15: Satisfaction with Peak Savers One respondent provided an additional comment and noted that they did not see any savings as a result of their participation. Peak Event Survey Results An survey was administered to program participants shortly after the July 19 th, 2017 event. The survey was conducted to collect data about participant decision-making, preferences, and opinions of the Peak Performers program. The survey was distributed to 279 program participant contacts. Contacts were sent an initial invitation and one follow-up . Table 4-15 summarizes the survey response. Table 4-15: Peak Event Survey Response Response Metrics Number of Participant Contacts Number of Contact s 279 Delivery Failures 51 Completions 60 Completion Rate 36% Demand Response Programs 197

201 Reduction Strategies Eighty-seven percent of respondents stated that they acted to reduce their load during the July 19th event, while 11% did not know if they did anything to reduce their usage. Table 4-16 displays the share of respondents reporting various actions to reduce load during the peak event. The action taken that was most commonly reported by participants was to adjust the temperature setting or to shut off the air conditioner. Eighty-seven percent of respondents reported that they had taken this action. Several of the actions reported by survey respondents may have resulted in electricity savings in addition to load reductions during the event. Customers that turned off lights (77%) and closed window blinds (28%) likely saved electricity in addition to reducing load. Some actions taken by participants may have resulted in shifting of load from the peak event to another period. As such, the action would not have resulted in energy savings even if peak load was reduced. Respondents that reported these types of actions were asked follow-up questions to determine if they shifted the load or if their actions may have resulted in energy savings. As summarized below and in Table 4-15, many of the respondents did not report behaviors consistent with load shifting. 29% of respondents that adjusted the air conditioner temperature setting or turned off the air conditioner reported that that they precooled the building. Additionally, some load shifting impacts may have occurred after the event if the air conditioning system ran for additional time to cool the building to the post-event set point. 9% of respondents that shut off non-industrial equipment stated that they increased use of that equipment at another period. 33% of respondents that reduced their industrial/manufacturing operations increased or extended operations at another time. 18% of respondents that closed early or sent staff home early reported that they extended business hours at another time. Demand Response Programs 198

202 Table 4-16: Actions Taken to Reduce Electricity Usage During Peak Event Peak Demand Reduction Action Percent of Respondents (n = 47)* Load Shifting Follow-Up Question Load Shifting Response Adjusted air conditioner temperature setting / Turn off air conditioner 87% Did you cool the building to a lower temperature than usual in preparation for the peak event? Turned off lights 77% N/A Shut off non-industrial equipment 47% Did you increase use of that non-industrial equipment at another time? Reduced or shut down industrial / manufacturing operations 32% Did you increase or extend manufacturing/industrial operations during another time to make up for the reduced operations or shutdown? Closed window blinds 28% N/A Closed early / sent staff home early 23% Did you extend your business hours / ask staff to make up the hours at another time? Other 15% N/A * Some respondents reported taking more than one action. Event Notifications and Intent to Participate in Future Events Ninety-eight percent of respondents reported that they received sufficient notification of the event, while two percent (one customer) said they did not receive sufficient notification of the event. This customer stated that they would prefer 24-hour notice of the event. Notification of the event was sent approximately 23 hours before the event. Therefore, this customer may not have received the notification or not recalled receiving it. All respondents reported that they intended to participate in future events called during the year. Demand Response Programs 199

203 4.1.3 Conclusions The following conclusions were developed from the evaluation findings. Overall the program operations remained consistent in its design and operations during Staff continues to make incremental improvements to notification and results delivery procedures that likely improved customers experience with the program. The outreach efforts in Lawton were successful in increasing the share of sites from this region that participated in the program. All new participants in the program were satisfied with the program overall and none reported dissatisfaction with the program. Seventy-five percent said they were very likely to participate again next year. As was the case in 2016, the number of events called were consistent with the majority of respondents expectations, though a few noted that fewer events were called than they were expecting. The ex post kw reductions remained largely consistent with the results from 2016, although the number of participating sites increased. The per-site reduction in kw is likely a function in the change in the ex post savings approach rather than a change of effort by participants. As was the case in 2016, customers reported employing a mixture of load shedding strategies that did not result in increased demand at another period and load shifting strategies that do generate higher demand at another period. The former types of actions reported include actions such as shutting off lights and closing blinds. The latter includes actions such as pre-cooling the building or making up for slowed operations at another period of the day. Demand Response Programs 200

204 Appendix A. Glossary Cash Inducement Costs: Refers to customer and service provider rebate/incentive costs incurred by PSO in the implementation of a program. Coincidence Factor (CF): For energy efficiency measures, the CF represents the fraction of connected load reduction that occurs during the peak demand period. Deemed Savings: A savings estimate for relatively homogeneous measures. Generally, an assumed average savings across many rebated units is applied to each individual unit installed. Effective Useful Life (EUL): The number of years (or hours) that an energy-efficient technology is estimated to function. Also, referred to as measure life. EM&V Administrative Costs: EM&V administrative costs include all costs associated with evaluation, measurement and verification of reported energy and demand impacts resulting from the implementation of a program. Ex Ante: Refers to estimates of energy savings and peak demand reduction developed before program evaluation. Equivalent to reported impacts. Ex Ante: Refers to estimates of energy savings and peak demand reductions developed from program evaluation. Equivalent to verified impacts. Free-ridership: Percentage of participants who would have implemented the same energy-efficiency measures in a similar timeframe even in the absence of the program. Gross Impacts: Changes in energy consumption/demand that result directly from program-promoted actions regardless of the extent or nature of program influence on these actions. Impact Evaluation: Impact evaluation is the verification and estimation of gross and net impacts resulting from the implementation of one or more energy-efficiency or demand response programs. Measure: An energy-efficiency measure refers to any action taken to increase energy efficiency, whether through changes in equipment, control strategies, or behavior. Net Savings: The portion of gross savings that is directly attributable to the actions of an energy-efficiency or demand response program. Net-to-Gross Ratio (NTGR): A factor representing net program savings divided by gross program savings that is applied to gross program impacts to convert them into net program impacts. Generally calculated as 1 (free-ridership %) + (Spillover %). Non-Cash Inducement Costs: Non-cash inducement costs include third party implementation costs and advertising costs incurred by PSO in the implementation of a program. PSO earns no incentives on advertising costs. Appendix A: Glossary 201

205 PSO Glossary Non-Energy Benefits: Non-energy benefits refer to any benefits PSO customers may experience due to their participation in PSO programs beyond energy savings. Examples include improved comfort, aesthetic enhancements, better indoor air quality, improved security, better employee productivity, etc. Non-EM&V Administrative Costs: Non-EM&V administrative costs include PSO staff labor costs and overhead costs associated with implementing a program. Oklahoma Deemed Savings Documents (OKDSD): Refers to the Oklahoma Deemed Savings, Installation & Efficiency Standards and associated work papers for small commercial and residential energy efficiency measures. These documents were originally submitted to the OCC as part of Cause # PUD and approved for use as part of Order # In 2013, the documents were update to reflect more recent and applicable baseline conditions. Participant Cost Test (PCT): The PCT examines the cost and benefits from the perspective of the customer installing the energy efficiency measure. Costs include incremental costs of purchasing and installing the efficient equipment, above the cost of standard equipment. Benefits include customer bill savings, incentives received from the utility, and any applicable tax credits. Peak Demand: For the purposes of this report peak demand refers to the average metered demand during the peak period, defined as 2 to 9 PM during the summer months, June through September, excluding weekends and holidays. Note that for the Business Demand Response program, peak demand reduction is calculated as the average reduction during event hours. Process Evaluation: A systematic assessment of an energy efficiency program for documenting program operations at the time of examination and identifying potential improvements that can be made to increase the programs efficacy or effectiveness. Projected, Reported, and Verified Savings: Projected impacts refer to the energy savings and peak demand reduction forecasts submitted to the OCC as part of PSO s initial portfolio filing on July 1, Reported impacts refer to energy savings and peak demand reduction estimates based on actual program participation in PY2016, before program evaluation activities. Finally, verified impacts refer to energy savings and demand reduction estimates for PY2016 developed through independent program evaluation, measurement, and verification (EM&V). Ratepayer Impact Measure (RIM): The RIM examines the impact of energy efficiency programs on utility rates. Reduced energy sales can lower revenues and put upward pressure on retail rates as the remaining fixed costs are spread over fewer kwh. Costs include overhead and incentive payments and the cost of lost revenue due to reduced 64 Cause No. PUD , Direct Testimony of Eric Raines. Appendix A: Glossary 202

206 PSO Glossary sales. Benefits include cost savings associated with not delivering energy to customers. These avoided costs include generation, transmission, and distribution costs. Realization Rate: The ratio of verified (ex post) impacts to reported (ex ante) impacts. Societal Cost Test (SCT): The SCT includes the same costs and benefits as the TRC, but uses a lower discount rate to reflect the overall benefit to society over the long term. Spillover: Energy and/or demand savings caused by a program, but for which the utility did not have to provide cash inducements. Total Resource Cost Test (TRC): The TRC measures the net benefits of the energy efficiency program for the region as a whole. Costs included in the TRC are incremental costs of purchasing and installing the efficient equipment, above the cost of standard equipment and overhead cost associated with implementing the program. Benefits include cost savings associated with not delivering energy to customers. These avoided costs include generation, transmission, and distribution costs. Utility Cost Test (UCT): The UCT examines the costs and benefits of the energy efficiency program from the perspective of the utility company. Costs include overhead (administration, marketing, EM&V) and incentive costs. Benefits include cost savings associated with not delivering energy to customers. These avoided costs include generation, transmission, and distribution costs. This test is also often referred to as the Program Administrator Cost Test (PACT). Appendix A: Glossary 203

207 Appendix B. Portfolio Cost-Effectiveness This appendix provides an overview of each programs participation, verified reduction in peak load, verified kwh savings, annual admin costs, total program costs, as well as a summary of the cost effectiveness analysis. B.1 Cost Effectiveness Summary This appendix covers all verified electricity and peak demand savings, and associated program costs incurred in the implementation of PSO s 2017 energy efficiency and demand response portfolio from January 1, 2017 through December 31, The cost-effectiveness of PSO s 2017 programs was calculated based on reported total spending, verified energy savings, and verified demand reduction for each of the energy efficiency and demand response programs. All spending estimates were provided by PSO. The methods used to calculate cost-effectiveness are informed by the California Standard Practice Manual. 65 The demand reduction (kw) and energy savings (kwh) presented throughout this appendix represent net savings at the generator by applying program level net-to-gross (NTG) ratios and adjusting for line losses. Program level NTG ratios for the 2017 programs were estimated by ADM as part of the portfolio impact evaluation. Verified savings estimates at the meter were adjusted to account for line losses using a line loss adjustment factor of For gas savings estimates, a gas loss factor was included. To calculate the cost-effectiveness of each program, measure lives were assigned on a measure-by-measure basis. When available, measure life values came from the Oklahoma Deemed Savings Documents (OKDSD). When not available in the OKDSD, measure life values came from the Arkansas TRM. 66 Additionally, assumptions regarding incremental/full measure costs were necessary. These costs were taken directly from the portfolio plan or project specific invoices. 65 California Standard Practice Manuel: Economic Analysis of Demand Side Management Programs, October Available at: _Electricity_and_Natural_Gas/CPUC_STANDARD_PRACTICE_MANUAL.pdf 66 Appendix B: Portfolio Cost Effectiveness 204

208 Avoided energy, capacity, transmission/distribution, and CO2 costs used to calculate cost-effectiveness were provided by PSO and are found in Section B.4 of this appendix. Residential and commercial rates used to estimate certain cost-effectiveness tests were also provided by PSO. Table B-1 lists each program included in this analysis, along with the projected savings estimates and projected budget. Impacts show in Table B-1 are net-at-generator, reflecting the NTG projections and line losses. Table B-2 lists each program included in this analysis, along with the final verified savings estimates, total expenditures, Utility Cost Test (UCT) 67 results, and Total Resource Cost Test (TRC) results. Impacts shown in Table B-2 are net-at-generator, reflecting NTG assumptions and line losses as described above. Results from the UCT and TRC are focused on in this summary for the following reasons: The UCT results are a direct input to the shared savings component of the Demand Side Management Cost Recovery Rider (DSM Rider) as described in Oklahoma Corporate Commission PUD Oklahoma Administrative Code (OAC) 165: lists the goals of energy efficiency and demand response programs as (1) minimize the long-term cost of utility service, and (2) avoid or delay the need for new generation, transmission, and distribution investment. The TRC test best reflects these goals, as it looks at benefits and costs from the perspective of all utility customers in the utility s service territory (participants and non-participants). In addition to UCT and TRC results, results from the Ratepayer Impact Measure (RIM), Participant Cost Test (PCT) and Societal Cost Test (SCT) are included in the body of this appendix. Based on verified program impacts and spending during PY2017, PSO s overall portfolio is cost-effective based on both the UCT and TRC. 67 The UCT is also referred to as the Program Administrator Cost Test (PACT). 68 Cause No. PUD , Direct Testimony of Earlyne Reynolds. Appendix B: Portfolio Cost Effectiveness 205

209 Table B-1: Projections by Program, 2017 (Impacts are Net, at Generator) Projected Projected Program Peak Annual Annual Gas Total Program Demand Energy Savings Expenditures Reduction Savings (Therms) (kw) (kwh) High Performance Business 7,550 45,416, ,820 $10,549,799 Home Weatherization 1,305 4,116, ,419 $3,923,754 Energy Saving Products 2,605 22,505, ,235 $3,687,883 High Performance Homes 3,514 6,494, ,229 $8,872,010 Education 597 4,216,905-25,392 $2,113,500 Behavioral Modification 4,779 14,911, ,939 $1,223,500 Conservation Voltage Reduction $29,600 Total EE Programs 20,349 97,661,833 1,250,141 $30,400,046 Business Demand Response 54, ,594 0 $3,296,900 Total DR Programs 54, ,594 0 $3,296,900 Total - Overall Portfolio 74,668 97,933,427 1,250,141 $33,696,946 Table B-2: Cost-Effectiveness by Program, 2017 (Impacts are Net, at Generator) Program Verified Verified Annual Peak Annual Total TRC UCT Gas Demand Energy Program (b/c (b/c Savings Reduction Savings Expenditures ratio) ratio) (Therms) (kw) (kwh) High Performance Business 9, ,875, $10,176, Home Weatherization 1, ,231, , $3,808, Energy Saving Products 5, ,561, $3,877, High Performance Homes 3, ,215, , $8,700, Education 1, ,739, $1,872, Behavioral Modification $451, Conservation Voltage Reduction $66, Total EE Programs 22, ,623, , $28,954, Business Demand Response 48, , $2,594, Total DR Programs 48, , $2,594, Total - Overall Portfolio 70, ,198, , $31,549, Appendix B: Portfolio Cost Effectiveness 206

210 B.2 Energy Efficiency Programs PSO s energy efficiency portfolio in 2017 consisted of six programs with a verified net peak demand reduction of 22,118 kw and verified net annual energy savings of 110,623,302 kwh (including line-loss estimates of 6.11%). Total spending in 2017 equaled $28,954,911. Table B-3 provides a summary of program participation and verified net impacts for each of the energy efficiency programs. Table B-4 provides a summary of program costs in Table B-3: Energy Efficiency Programs Verified Impacts (Net, at Generator) Program Number of Participants in 2017 Verified Peak Demand Reduction (kw) Verified Annual Energy Savings (kwh) Gas Savings (Therms) High Performance Business 1,210 9,797 55,875,440 0 Home Weatherization 2,239 1,619 5,231, ,720 Energy Saving Products 69 1,469,793 5,727 34,561,350 0 High Performance Homes 70 4,786 3,504 7,215, ,468 Education 26,212 1,470 7,739,895 0 Behavioral Modification Conservation Voltage Reduction Total EE Programs 1,504,240 22, ,623, , The Energy Saving Products consists of the number of upstream LED bulbs and appliances discounted. For the downstream portion of the program, determining the number of participants is straight forward. For the upstream bulb discounts, the number of bulb packages sold is listed instead of number of participants. 70 The number of participants for High Performance Homes reflects to total number of customers and the total number of homes in the New Homes portion of the program. Appendix B: Portfolio Cost Effectiveness 207

211 Program Table B-4: Energy Efficiency Programs Reported Costs Annual Non- EM&V Admin Costs ($) 71 Annual EM&V Admin Costs ($) Annual Cash Inducement Costs ($) 72 Annual Non-Cash Inducement Costs ($) 73 Marketing Costs ($) High Performance Business $276,961 $430,055 $6,225,445 $2,929,325 $314,620 Home Weatherization $88,523 $138,311 $3,404,461 $74,737 $102,894 Energy Saving Products $74,281 $99,661 $2,742,284 $788,590 $173,024 High Performance Homes $155,048 $177,785 $5,801,486 $2,299,707 $265,996 Education $53,912 $74,765 $1,691,194 $21,050 $32,020 Behavioral Modification $50,559 $40,203 $0 $361,107 $0 Conservation Voltage Reduction $7,570 $45,265 $0 $14,072 $0 Total EE Programs $706,855 $1,006,045 $19,864,871 $6,488,588 $888,553 In the tables that follow, total costs and benefits, and cost-effectiveness test results are provided for each energy efficiency program in the PY2017 portfolio. B.2.1 High Performance Business Program Table B-5: High Performance Business Benefit/Cost Tests Metric Total Ratepayer Utility Cost Societal Participant Resource Impact Test Cost Test Cost Test Cost Test Measure Benefit/Cost Ratio Net Benefits ($000s) Total Benefits ($000s) Total Costs ($000s) Non-EM&V Admin Costs include PSO staff labor costs and overhead costs. 72 Cash inducement costs refer to customer rebate costs. 73 Non-cash inducement costs include third party implementation costs. Appendix B: Portfolio Cost Effectiveness 208

212 B.2.2 Home Weatherization Program Table B-6: Home Weatherization Benefit/Cost Tests Metric Total Ratepayer Utility Cost Societal Participant Resource Impact Test Cost test Cost Test Cost Test Measure Benefit/Cost Ratio Net Benefits ($000s) 2, , (3,630.93) 7, , Total Benefits ($000s) 6, , , , , Total Costs ($000s) 3, , , , , B.2.3 Energy Saving Products Program Table B-7: Energy Saving Products Benefit/Cost Tests Metric Total Ratepayer Utility Cost Societal Participant Resource Impact Test Cost test Cost Test Cost Test Measure Benefit/Cost Ratio Net Benefits ($000s) 17, , , , , Total Benefits ($000s) 21, , , , , Total Costs ($000s) 3, , , , , B.2.4 High Performance Homes Program Table B-8: High Performance Homes Benefit/Cost Test Metric Total Ratepayer Utility Cost Societal Participant Resource Impact Test Cost test Cost Test Cost Test Measure Benefit/Cost Ratio Net Benefits ($000s) 2, , (5,824.42) 8, , Total Benefits ($000s) 11, , , , , Total Costs ($000s) 8, , , , , Appendix B: Portfolio Cost Effectiveness 209

213 B.2.5 Education Program Table B-9: Education Benefit/Cost Test Metric Total Ratepayer Utility Cost Societal Participant Resource Impact Test Cost test Cost Test Cost Test Measure Benefit/Cost Ratio Net Benefits ($000s) 4, , (3,112.64) 6, , Total Benefits ($000s) 6, , , , , Total Costs ($000s) 1, , , , , B.3 Demand Response Programs PSO s demand response portfolio in 2017 consisted of one demand response program with a verified net peak demand reduction of MW and verified net energy savings of MWh. 74. Total spending in 2017 equaled $2,594,726. Table B-10 provides a summary of program participation and verified net impacts for the 2017 demand response portfolio. Table B-11 provides a summary of 2017 program costs. Table B-10: Demand Response Programs Verified Impacts (Net, at Generator) Number of Verified Peak Verified Annual Gas Program Participants Demand Energy Savings in 2017 Reduction (kw) Savings (kwh) (Therms) Business Demand Response , ,039 0 Total DR Programs , ,039 - Program Table B-11: Demand Response Programs Reported Costs Annual Non- EM&V Admin Costs ($) Annual EM&V Admin Costs ($) Annual Cash Inducement Costs ($) Annual Non- Cash Inducement Costs ($) Marketing Costs ($) Business Demand $49, $82, $2,145, $221, $95, Total DR $49, $82, $2,145, $221, $95, In the table that follows, total costs and benefits, and full cost-effectiveness test results are provided for the Business Demand Response program. 74 The verified peak demand reduction shown here for the Business Demand Response program includes an adjustment for line-losses (6.11%). Appendix B: Portfolio Cost Effectiveness 210

214 B.3.1 Business Demand Response Program Table B-12: Business Demand Response Benefit/Cost Test Metric Total Ratepayer Utility Cost Societal Participant Resource Impact Test Cost test Cost Test Cost Test Measure Benefit/Cost Ratio Net Benefits ($000s) 5, , , , , Total Benefits ($000s) 7, , , , , Total Costs ($000s) 2, , B.4 Avoided Costs The avoided costs in the table below were developed for energy, capacity, T&D, and CO2 during the portfolio design process (PUD ). Appendix B: Portfolio Cost Effectiveness 211

215 SPP - Energy Year Table B-13: Avoided Costs from PSO Portfolio Plan Natural SPP Capacity T&D Costs CO2 Gas $/MWh $/MW-day $/kw-yr $/kw-yr ($/metric ($/Mcf) tonne) 2017 $46.36 $ $ $17.53 $0.00 $ $47.64 $ $ $17.80 $0.00 $ $49.17 $ $ $18.08 $0.00 $ $51.25 $ $ $18.35 $0.00 $ $54.12 $ $ $18.63 $1.26 $ $62.48 $ $ $18.91 $15.10 $ $64.07 $ $ $19.19 $15.29 $ $66.39 $ $ $19.48 $15.49 $ $68.59 $ $ $19.77 $15.69 $ $69.91 $ $ $20.07 $15.90 $ $72.18 $ $ $20.37 $16.10 $ $74.00 $ $ $20.67 $16.31 $ $75.92 $ $ $20.98 $16.52 $ $78.07 $ $ $21.30 $16.74 $ $80.38 $ $ $21.53 $16.96 $ $83.77 $ $ $21.82 $17.18 $ $85.54 $ $ $22.11 $17.40 $ $81.01 $ $ $22.39 $17.62 $ $83.93 $ $ $22.68 $17.86 $ $85.22 $ $ $22.97 $18.09 $ $86.54 $ $ $23.26 $18.33 $ $87.89 $ $ $23.55 $18.58 $ $89.26 $ $ $23.83 $18.83 $ $90.66 $ $ $24.12 $19.08 $ $92.09 $ $ $24.41 $19.33 $ $93.56 $ $ $24.70 $19.59 $ $95.05 $ $ $24.99 $19.83 $ $97.14 $ $ $25.54 $20.26 $8.81 Appendix B: Portfolio Cost Effectiveness 212

216 Appendix C. Identification of Program Implementers Table C-1 identifies program implementation contractors, their associated contact information, and the 2017 programs they were involved in. Table C-1: Program Implementer Identification Program(s) Implementation Contractor Contact Contact Title Contact Address Contact Phone Contact Business Demand Response PSO Jeff Brown Consumer Programs Manager 212 E. 6th St. Tulsa, OK Education Resource Action Programs Mike Gross Director of Program Management 976 United Circle, Sparks, NV tion.com Energy Saving Products CLEAResult Karen Miller Program Manager 146 Chestnut Street, Springfield, MA sult.com High Performance Business, High Performance Homes ICF International ICF International Janine Pittman Andrea Palmer Program Manager Program Manager 907 S. Detroit Ave. Suite 505 Tulsa, OK S. Detroit Ave. Suite 505 Tulsa, OK i.com com Home Weatherizatio n Titan ES, LLC Rebuilding Tulsa Together Bradley Cockings Jennifer Barcus - Schafer President Chief Executive Officer 9700 S. Pole Road, Tulsa, OK P.O. Box 52201, Tulsa, OK bcockings@titanes.u s jennifer.barcus_shaf er@rebuildingtogeth ertulsa Program Marketing Services VI Marketing and Branding Judi Startzman Vice President of Strategic Marketing 125 Park Avenue, Suite 200, Oklahoma City, OK jstartzman@thevibra nd.com Appendix C: Identification of Program Implementers 213

217 Appendix D. Training and Customer Outreach During 2017, PSO conducted several service provider recruitment and training events. Additionally, PSO sponsored various customer outreach events and stakeholder presentations. Table D-1 summarizes the in-store retail lighting promotional events; Table D-2 summarizes service provider recruitment and training events, customer outreach events, and other non-lighting promotion events throughout PY2017. Table D-1: Summary of In-Store Retail Lighting Promotional Events Date Event Name Location Sponsored By 1/13/2017 Lighting Event Hobart PSO/CLEAResult 1/19/2017 Lighting Event Lawton PSO/CLEAResult 1/20/2017 Lighting Event Elk City PSO/CLEAResult 1/26/2017 Lighting Event Elk City PSO/CLEAResult 1/28/2017 Lighting Event Tulsa PSO/CLEAResult 1/29/2017 Lighting Event Tulsa PSO/CLEAResult 2/3/2017 Lighting Event Hobart PSO/CLEAResult 2/4/2017 Lighting Event Lawton PSO/CLEAResult 2/11/2017 Lighting Event Lawton PSO/CLEAResult 2/25/2017 Lighting Event Tulsa PSO/CLEAResult 2/26/2017 Lighting Event Tulsa PSO/CLEAResult 3/2/2017 Lighting Event Lawton PSO/CLEAResult 3/18/2017 Lighting Event Hobart PSO/CLEAResult 3/25/2017 Lighting Event Lawton PSO/CLEAResult 3/26/2017 Lighting Event Tulsa PSO/CLEAResult 3/31/2017 Lighting Event Tulsa PSO/CLEAResult 4/1/2017 Lighting Event Lawton PSO/CLEAResult 4/27/2017 Lighting Event Hobart PSO/CLEAResult 4/29/2017 Lighting Event Elk City PSO/CLEAResult 4/29/2017 Lighting Event Tulsa PSO/CLEAResult 4/30/2017 Lighting Event Tulsa PSO/CLEAResult 5/6/2017 Lighting Event Hobart PSO/CLEAResult 5/13/2017 Lighting Event Lawton PSO/CLEAResult 5/19/2017 Lighting Event Grove PSO/CLEAResult 5/27/2017 Lighting Event Tulsa PSO/CLEAResult 5/28/2017 Lighting Event Tulsa PSO/CLEAResult 6/10/2017 Lighting Event Lawton PSO/CLEAResult 6/17/2017 Lighting Event Lawton PSO/CLEAResult 6/24/2017 Lighting Event Tulsa PSO/CLEAResult 6/25/2017 Lighting Event Tulsa PSO/CLEAResult 7/1/2017 Lighting Event Lawton PSO/CLEAResult 7/22/2017 Lighting Event Hobart PSO/CLEAResult Appendix D: Training and Customer Outreach 214

218 PSO Training and Customer Outreach Date Event Name Location Sponsored By 7/29/2017 Lighting Event Broken PSO/CLEAResult 7/30/2017 Lighting Event Tulsa PSO/CLEAResult 8/12/2017 Lighting Event Lawton PSO/CLEAResult 8/24/2017 Lighting Event Lawton PSO/CLEAResult 8/25/2017 Lighting Event Hobart PSO/CLEAResult 8/26/2017 Lighting Event Lawton PSO/CLEAResult 8/26/2017 Lighting Event Owasso PSO/CLEAResult 8/26/2017 Lighting Event Tulsa PSO/CLEAResult 8/27/2017 Lighting Event Tulsa PSO/CLEAResult 8/30/2017 Lighting Event Elk City PSO/CLEAResult 8/30/2017 Lighting Event Tulsa PSO/CLEAResult 9/9/2017 Lighting Event Tulsa PSO/CLEAResult 9/16/2017 Lighting Event Choute PSO/CLEAResult 9/17/2017 Lighting Event Tulsa PSO/CLEAResult 9/23/2017 Lighting Event Lawton PSO/CLEAResult 9/23/2017 Lighting Event Tulsa PSO/CLEAResult 9/30/2017 Lighting Event Broken PSO/CLEAResult 9/30/2017 Lighting Event Lawton PSO/CLEAResult 10/1/2017 Lighting Event Tulsa PSO/CLEAResult 10/7/2017 Lighting Event Broken PSO/CLEAResult 10/8/2017 Lighting Event Lawton PSO/CLEAResult 10/13/2017 Lighting Event Tulsa PSO/CLEAResult 10/20/2017 Lighting Event Hobart PSO/CLEAResult 10/21/2017 Lighting Event Lawton PSO/CLEAResult 10/21/2017 Lighting Event Tulsa PSO/CLEAResult 10/28/2017 Lighting Event Tulsa PSO/CLEAResult 10/28/2017 Lighting Event Vinita PSO/CLEAResult 10/29/2017 Lighting Event Tulsa PSO/CLEAResult 11/5/2017 Lighting Event Owasso PSO/CLEAResult 11/8/2017 Lighting Event Tulsa PSO/CLEAResult 11/9/2017 Lighting Event Lawton PSO/CLEAResult 11/11/2017 Lighting Event Lawton PSO/CLEAResult 11/18/2017 Lighting Event Grove PSO/CLEAResult 11/18/2017 Lighting Event Hobart PSO/CLEAResult 11/18/2017 Lighting Event Tulsa PSO/CLEAResult 11/19/2017 Lighting Event Bartles PSO/CLEAResult 11/19/2017 Lighting Event Tulsa PSO/CLEAResult 12/2/2017 Lighting Event Grove PSO/CLEAResult 12/2/2017 Lighting Event Tulsa PSO/CLEAResult 12/3/2017 Lighting Event Tulsa PSO/CLEAResult Appendix D: Training and Customer Outreach 215

219 PSO Training and Customer Outreach Date Event Name Location Sponsored By 12/8/2017 Lighting Program Promotion Tulsa PSO/CLEAResult 12/9/2017 Lighting Event Lawton PSO/CLEAResult 12/16/2017 Lighting Event Coweta PSO/CLEAResult 12/16/2017 Lighting Event Elk City PSO/CLEAResult Table D-2: Service Provider Recruitment & Training Events, Customer Outreach Events, and Other Non-Lighting Promotional Events Date Event Name Location Sponsored By # of Attendees Estimated 1/9/2017 PowerForward Overview McAlester PSO /11/2017 PowerForward Overview OSU - Tulsa PSO /12/2017 PowerForward Overview Grove PSO /13/2017 HPB Lunch and Learn Grove PSO /18/2017 Service Provider Training Tulsa ICF 30 1/21/2017 PowerForward Overview PSO Office PSO /24/2017 Service Provider Training Weatherford ICF 11 1/25/2017 PowerForward Overview McAlester PSO /25/2017 Service Provider Training Bartlesville ICF 12 1/26/2017 Service Provider Training McAlester ICF 11 2/7/2017 Service Provider Training Idabel ICF 6 2/8/2017 Service Provider Training Hugo ICF 14 2/8/2017 HPB Service Provider Training PSO Office PSO /9/2017 Service Provider Training McAlester ICF 13 2/15/2017 Service Provider Training OKC ICF 36 2/22/2017 PowerForward Overview PSO Office PSO /26/2017 PowerForward Overview WOSU PSO /28/2017 PowerForward Overview Bartlesville PSO /1/2017 PowerForward Overview Sand Springs PSO /14/2017 Service Provider Training Tulsa ICF 5 4/13/2017 PowerForward Overview Idabel PSO /19/2017 PowerForward Overview PSO Office PSO /26/2017 Mechanical Seminar Tulsa ICF 225 4/26/2017 PowerForward Overview PSO Office PSO /10/2017 PowerForward Overview PSO Office PSO /11/2017 PowerForward Overview Tulsa Mid Metro PSO /17/2017 Service Provider Training Tulsa ICF 21 5/23/2017 HPB Lunch and Learn PSO Office PSO /13/2017 PowerForward Overview Clinton PSO /26/2017 PowerForward Overview PSO Office PSO Appendix D: Training and Customer Outreach 216

220 PSO Training and Customer Outreach Date Event Name Location Sponsored By # of Attendees Estimated 7/13/2017 PowerForward Overview Hugo PSO /20/2017 PowerForward Overview McAlester PSO /9/2017 PowerForward Overview McAlester PSO /18/2017 PowerForward Overview Lawton PSO /1/2017 PowerForward Overview Lawton PSO /3/2017 PowerForward Overview PSO Office PSO /6/2017 PowerForward Overview PSO Office PSO /23/2017 PowerForward Overview PSO Office PSO /23/2017 PowerForward Overview PSO Office PSO /24/2017 PowerForward Overview PSO Office PSO /26/2017 PowerForward Overview PSO Office PSO /1/2017 PowerForward Overview Tulsa PSO /2/2017 Customer Meeting/Mechanical Seminar Lawton PSO/ICF 90 11/8/2017 PowerForward Overview Krebs PSO /10/2017 PowerForward Overview Apache Casino PSO /8/2017 PowerForward Overview PSO Office PSO /8/2017 PowerForward Overview Bartlesville PSO 0-10 Appendix D: Training and Customer Outreach 217

221 Appendix E. Marketing Synopsis The following pages of this appendix provide examples of marketing materials used to promote PSO s Demand Side Management portfolio in Appendix E: Marketing Synopsis 218

222 Appendix E: Marketing Synopsis 219

223 Appendix E: Marketing Synopsis 220

224 Appendix E: Marketing Synopsis 221

225 Appendix E: Marketing Synopsis 222

226 Appendix E: Marketing Synopsis 223

227 Appendix E: Marketing Synopsis 224

228 Appendix E: Marketing Synopsis 225

229 Appendix E: Marketing Synopsis 226

230 Appendix E: Marketing Synopsis 227

231 Appendix E: Marketing Synopsis 228

232 Appendix E: Marketing Synopsis 229

233 Appendix E: Marketing Synopsis 230

234 Appendix E: Marketing Synopsis 231

235 Appendix E: Marketing Synopsis 232

236 Appendix E: Marketing Synopsis 233

237 Appendix E: Marketing Synopsis 234

238 Appendix E: Marketing Synopsis 235

239 Appendix E: Marketing Synopsis 236

240 Appendix E: Marketing Synopsis 237

241 Appendix E: Marketing Synopsis 238

242 Appendix E: Marketing Synopsis 239

243 Appendix E: Marketing Synopsis 240

244 Appendix E: Marketing Synopsis 241

245 Appendix E: Marketing Synopsis 242

246 Appendix E: Marketing Synopsis 243

247 Appendix E: Marketing Synopsis 244

248 Appendix E: Marketing Synopsis 245

249 Appendix E: Marketing Synopsis 246

250 Appendix E: Marketing Synopsis 247

251 Appendix E: Marketing Synopsis 248

252 Appendix E: Marketing Synopsis 249

253 Appendix E: Marketing Synopsis 250

254 Appendix E: Marketing Synopsis 251

255 Appendix F. OKDSD, AR, & IL TRM Deemed Savings and Algorithms F.1 Energy Efficiency Programs F.1.1 High Performance Business (HPB) Program ADM s approach to project level savings analysis depends largely on the types of measures installed. Whenever possible, deemed savings and prescribed algorithms from the Arkansas TRM will be used to determine ex post gross savings. Care will be taken to assure any assumptions are reasonable and current, and that there are no errors in the algorithms. Additionally, where engineering calculations from the Arkansas TRM v6.1 are applicable to measures installed through the HPB Program, those algorithms may also be used. Care will be taken to ensure that weather conditions and other factors that may vary from AR to OK will be considered when applying these algorithms. The following discussion describes, in general, our plan for analyzing savings from different measure types: Analyzing Savings from Lighting Measures: Lighting measures may include retrofits of existing fixtures, lamps and/or ballasts with energy efficient fixtures, lamps and/or ballasts. These types of measures reduce demand, but operating hours for fixtures are generally the same pre- and post-retrofit. Also examined are any proposed lighting control strategies that might include the addition of energy conserving control technologies, such as motion sensors or day-lighting controls. These measures typically involve a reduction in hours of operation and/or lower current passing through the fixtures. New construction lighting projects are also included in the High Performance Business program, which requires calculating savings in comparison to applicable building codes instead of preretrofit conditions. ADM analyzes the savings from lighting measures using data for new/retrofitted fixtures on (1) wattages before and after retrofit and (2) hours of operation before and after the retrofit. Fixture wattages are generally taken from a table of standard wattages or cut sheets when feasible, with corrections made for non-operating fixtures. Prescriptive algorithms for calculating energy savings and demand reductions from the Arkansas TRM or other relevant program sources will be used. Additionally, HVAC interactive effects will be accounted for using partially deemed algorithms from the AR TRM v6.0 dependent upon heating and cooling systems serving areas where lighting systems are installed. Analyzing Savings from HVAC Measures: For the analysis of non-prescriptive HVAC and control measures, we develop estimates of the savings through simulations with our energy analysis models (e.g., DOE-2, equest). Before making the analytical runs for each sample site with these measures, we prepare a Model Calibration Run. Calibration is Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 252

256 based on actual billed usage during actual weather conditions. Once the analysis model has been calibrated for a particular facility, there are three steps in our procedure for calculating estimates of energy savings for HVAC measures installed or to be installed at the facility. First, we perform an analysis of energy use at a facility under the assumption that the energy efficiency measures are not installed. Second, we analyze energy use at the facility with all conditions the same but with the energy efficiency measures now installed. Third, we compare the results of the analyses from the preceding steps to determine the energy savings attributable to the energy efficiency measure. The compared analysis runs are normalized to a typical meteorological weather year (TMY3). We use monitoring data to verify set points and operating characters and to calibrate the simulations as necessary. Analyzing Savings from Motor and VFDs: Estimates of energy savings from the use of non-prescriptive high efficiency motors or VFDs are derived through an "after-only" analysis. With this method, energy use is measured for the high efficiency motor or VFD and after it has been installed. We (1) make one-time measurements of voltage, current, and power factor of the VFD/motor and (2) use ACR loggers to conduct continuous measurements of amps or watts over a period of time in order to obtain the data needed on operating schedules. The data thus collected is then used in estimating what energy use would have been for the motor application if the high efficiency motor or VFD had not been installed. ADM field staff participate in annual safety training to ensure that safety best practices are used. Analyzing Savings from Process Improvements: Analysis of savings from process improvements (including air compressors, process machines, etc.) is inherently projectspecific. Because of the specificity of such processes, analyzing the processes through simulations is generally not feasible. Rather, we rely on engineering analysis of the process affected by the improvements. Major factors in our engineering analysis of process savings are operating schedules and load factors. We develop the information on these factors through short-term monitoring of the affected equipment, be it pumps, heaters, compressors, etc. The monitoring is done after the process change, and the data gathered on operating hours and load factors are used in the engineering analysis to define before conditions for the analysis of savings. Retro-commissioning and Enhanced O&M: As is the case for custom measures, the methods used to verify project gross energy impacts will be dependent on the specifics of each site and the availability of data. However; the gross savings analysis for each site will be more involved based on the additional data and documentation that must be included in the savings calculations. Methods will include the range of International Performance Measurement & Verification Protocols, as shown in the table below. An emphasis will be placed on Option D (Building Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 253

257 simulation) for commercial facilities and Options B (pre/post monitoring) & C (Billing analysis) for industrial facilities. Often, multiple approaches are used to minimize uncertainty in the ex post verified energy savings estimates. The preceding descriptions of typical gross savings estimation methods by measure type may well be used for retrocommissioning projects as well. International Performance Measurement & Verification Protocols M&V Options M&V Option A. Partially Measure Retrofit Isolation B. Retrofit Isolation C. Whole Facility D. Calibrated Simulation How Savings Are Calculated Engineering calculations using short term or continuous postretrofit measurements and stipulations. Engineering calculations using short term or continuous measurements. Analysis of whole facility utility meter or sub-meter data using techniques from simple comparison to regression analysis. Energy use simulation, calibrated with hourly or monthly utility billing data and/or end-use metering. ADM s approach to project level savings analysis depends largely on the types of measures installed. Whenever possible, deemed savings and prescribed algorithms from the Arkansas TRM will be used to determine ex post gross savings. Care will be taken to assure any assumptions are reasonable and current, and that there are no errors in the algorithms. Additionally, where engineering calculations from the Arkansas TRM V6.1 are applicable to measures installed through the HPB Program, those algorithms may also be used. Care will be taken to ensure that weather conditions and other factors that may vary from AR to OK will be considered when applying these algorithms. The following discussion describes, in general, our plan for analyzing savings from different measure types: Analyzing Savings from Lighting Measures: Lighting measures may include retrofits of existing fixtures, lamps and/or ballasts with energy efficient fixtures, lamps and/or ballasts. These types of measures reduce demand, but operating hours for fixtures are generally the same pre- and post-retrofit. Also examined are any proposed lighting control strategies that might include the addition of energy conserving control technologies, such as motion sensors or day-lighting controls. These measures typically involve a reduction in hours of operation and/or lower current passing through the fixtures. New construction lighting projects are also included in the High Performance Business program, which Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 254

258 requires calculating savings in comparison to applicable building codes instead of preretrofit conditions. ADM analyzes the savings from lighting measures using data for new/retrofitted fixtures on (1) wattages before and after retrofit and (2) hours of operation before and after the retrofit. Fixture wattages are generally taken from a table of standard wattages or cut sheets when feasible, with corrections made for non-operating fixtures. Prescriptive algorithms for calculating energy savings and demand reductions from the Arkansas TRM or other relevant program sources will be used. Additionally, HVAC interactive effects will be accounted for using partially deemed algorithms from the AR TRM v6.0 dependent upon heating and cooling systems serving areas where lighting systems are installed. Analyzing Savings from HVAC Measures: For the analysis of non-prescriptive HVAC and control measures, we develop estimates of the savings through simulations with our energy analysis models (e.g., DOE-2, equest). Before making the analytical runs for each sample site with these measures, we prepare a Model Calibration Run. Calibration is based on actual billed usage during actual weather conditions. Once the analysis model has been calibrated for a particular facility, there are three steps in our procedure for calculating estimates of energy savings for HVAC measures installed or to be installed at the facility. First, we perform an analysis of energy use at a facility under the assumption that the energy efficiency measures are not installed. Second, we analyze energy use at the facility with all conditions the same but with the energy efficiency measures now installed. Third, we compare the results of the analyses from the preceding steps to determine the energy savings attributable to the energy efficiency measure. The compared analysis runs are normalized to a typical meteorological weather year (TMY3). We use monitoring data to verify set points and operating characters and to calibrate the simulations as necessary. Analyzing Savings from Motor and VFDs: Estimates of energy savings from the use of non-prescriptive high efficiency motors or VFDs are derived through an "after-only" analysis. With this method, energy use is measured for the high efficiency motor or VFD and after it has been installed. We (1) make one-time measurements of voltage, current, and power factor of the VFD/motor and (2) use ACR loggers to conduct continuous measurements of amps or watts over a period of time in order to obtain the data needed on operating schedules. The data thus collected is then used in estimating what energy use would have been for the motor application if the high efficiency motor or VFD had not been installed. ADM field staff participate in annual safety training to ensure that safety best practices are used. Analyzing Savings from Process Improvements: Analysis of savings from process improvements (including air compressors, process machines, etc.) is inherently projectspecific. Because of the specificity of such processes, analyzing the processes through Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 255

259 simulations is generally not feasible. Rather, we rely on engineering analysis of the process affected by the improvements. Major factors in our engineering analysis of process savings are operating schedules and load factors. We develop the information on these factors through short-term monitoring of the affected equipment, be it pumps, heaters, compressors, etc. The monitoring is done after the process change, and the data gathered on operating hours and load factors are used in the engineering analysis to define before conditions for the analysis of savings. Retro-commissioning and Enhanced O&M: As is the case for custom measures, the methods used to verify project gross energy impacts will be dependent on the specifics of each site and the availability of data. However; the gross savings analysis for each site will be more involved based on the additional data and documentation that must be included in the savings calculations. Methods will include the range of International Performance Measurement & Verification Protocols, as shown in the table below. An emphasis will be placed on Option D (Building simulation) for commercial facilities and Options B (pre/post monitoring) & C (Billing analysis) for industrial facilities. Often, multiple approaches are used to minimize uncertainty in the ex post verified energy savings estimates. The preceding descriptions of typical gross savings estimation methods by measure type may well be used for retrocommissioning projects as well. International Performance Measurement & Verification Protocols M&V Options M&V Option A. Partially Measure Retrofit Isolation B. Retrofit Isolation C. Whole Facility D. Calibrated Simulation How Savings Are Calculated Engineering calculations using short term or continuous post-retrofit measurements and stipulations. Engineering calculations using short term or continuous measurements. Analysis of whole facility utility meter or sub-meter data using techniques from simple comparison to regression analysis. Energy use simulation, calibrated with hourly or monthly utility billing data and/or end-use metering. F.1.2 Home Weatherization Program This section includes the measure level algorithms and deemed savings values utilized for the ex post and kwh and kw savings calculations. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 256

260 Infiltration Reduction (AR TRM) Savings are calculated by multiplying the air infiltration reduction (CFM) with the energy savings factor corresponding to the climate zone/hvac type. The air infiltration reduction estimate in CFM is obtained through blower door testing performed by the program contractor for each home serviced. Only homes with electric cooling systems are eligible for the measure (central AC or room AC). The algorithms for energy savings listed in the Arkansas TRM are: kwh Savings = CFM X ESF Where: kw Savings = CFM X DSF CFM ESF DSF = Air infiltration reduction in Cubic Feet per Minute at 50 Pascal = The energy savings value corresponding to the climate zone and heating and cooling type in the following table = The demand savings value corresponding to the climate zone and heating and cooling type in the following table Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 257

261 Infiltration Control Deemed Savings Values Infiltration Control Deemed Savings Impact per CFM50 Reduction Equipment Type kwh Savings (ESF) kw Savings (DSF) Therm Savings (GSF) Peak Therms (GPSF) Zone 9 Electric AC with Gas Heat Gas Heat Only (no AC) NA Elec. AC with Resistance Heat NA NA Heat Pump NA NA Zone 8 Electric AC with Gas Heat Gas Heat Only (no AC) NA Elec. AC with Resistance Heat NA NA Heat Pump NA NA Zone 7 Electric AC with Gas Heat Gas Heat Only (no AC) NA Elec. AC with Resistance Heat NA NA Heat Pump NA NA Zone 6 Electric AC with Gas Heat Gas Heat Only (no AC) NA Elec. AC with Resistance Heat NA NA Heat Pump NA NA Duct Sealing: ADM utilized the Oklahoma Deemed Savings Document in conjunction with the duct leakage reduction results in order to calculated measure savings. ADM modified to the default SEER value used in the algorithm. The default SEER value is 13, but ADM utilized a value of 11.5 SEER because the measure is being implemented qualified income homes which tend to be older. The 11.5 SEER value is the average of Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 258

262 U.S. DOE minimum allowed SEER for air conditioners form (10 SEER) and after January 23, 2006 (13 SEER). The algorithms for cooling and energy saving listed in the Oklahoma Deemed Savings Document for duct sealing are as follows: Where: DLpre DLpost EFLHc h ρout kwh savings,c = (DL pre DL post )X EFLH C X (h out ρ out h in ρ in ) X X SEER = Pre-improvement duct leakage at 25 Pa (ft3/min) = Post-improvement duct leakage at 25 Pa (ft3/min) = Equivalent full load cooling hours, from table = Outdoor/Indoor seasonal specific enthalpy (Btu/lb), from table = Density of outdoor air (lb/ft3), from table ρin = Density of conditioned air at 75 F (lb/ft3) = = Constant to convert from minutes to hours 1,000 = Constant to convert from W to kw SEER = Seasonal Energy Efficiency Ratio of existing system (Btu/W hr) = The algorithms for heating (heat pump) and energy saving listed in the Oklahoma Deemed Savings Document for duct sealing are as follows: Where: DLpr DLpost kwh savings,c = (DL pre DL post ) X 60 X 0.77 X HDD X 24 X X HSPF = Pre-improvement duct leakage at 25 Pa (ft3/min = Post-improvement duct leakage at 25 Pa (ft3/min) 60 = Constant to convert from minutes to hours = Volumetric heat capacity of air (Btu/ft3 F) 0.77 = Factor to correlated design load hours to EFLH under actual working conditions (to account for the fact that people do not always operated their heating system when the outside temperature is less than 65 F) HDD = Heating Degree Day 1,000 = Constant to convert from W to kw 75 Average of US DOE minimum allowed SEER for new air conditioners from (10 SEER) and after January 23,2006 (13 SEER). Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 259

263 HSPF = Heating Seasonal Performance Factor of existing system (Btu/W hr) = 7.7 (default) Heating Savings (Electric Resistance): Where: kwh savings,c = (DL pre DL post ) X 60 X 0.77 X HDD X 24 X.018 3,412 DLpre DLpost = Pre-improvement duct leakage at 25 Pa (ft3/min = Post-improvement duct leakage at 25 Pa (ft3/min) 60 = Constant to convert from minutes to hours = Volumetric heat capacity of air (Btu/ft3 F) HDD = Heating Degree Days 3,412 = Constant to convert from Btu to kwh Ceiling Insulation: Deemed savings values are calculated, in accordance to the Oklahoma Deemed Savings Document, for each weather zone. Deemed savings are based on the R-value of the baseline insulation. Retrofit insulation must meet a minimum R-value of R-38 in order to be considered for savings. Savings are then calculated by multiplying the corresponding savings value by the square footage. The following table is an example of the deemed values. Example Deemed Savings Table - Ceiling Insulation Zone 8B Ceiling Insulation Deemed Savings Ceiling Insulation Base R-value AC/Gas Heat kwh Gas Heat (no AC) kwh Gas Heat Therms AC/Electric Resistance kwh Heat Pump kwh Summer Peak kw Savings (per sq. ft.) (per sq. ft.) (per sq. ft.) (per sq. ft.) (per sq. ft.) (per sq. ft.) R-0 to R R-5 to R R-9 to R R-15 to R Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 260

264 Water Heater Jackets: For water heater jackets, a review of the tracking system showed that conservative assumptions were used to inform the use of the deemed savings. Savings values corresponding to 2 thick jackets on 40-gallon tanks were used for all sites. The deemed saving for this measure depend on 1) insulation thickness and 2) water heater tank size. The table below shows the deemed savings for water heater jackets installed on electric water heaters. Deemed Savings Electric Water Heater Jacket Electric Approximate Tank Size Energy Savings (kwh) Peak Savings (kw) " WHJ savings kwh " WHJ savings kwh Water Heater Pipe Insulation: Water heater pipe insulation involves insulating of all hot and cold vertical lengths of pipe, plus the initial length of horizontal hot and cold water pipe, up to three feet from the transition, or until wall penetration, whichever is less. The Oklahoma Deemed Savings Document specifies deemed values below for energy and demand impacts of water heater pipe insulation measures. Deemed Savings Electric Water Heater Pipe Insulation Elec. Water Heater Pipe Insulation Gas Water Heater Pipe Insulation Annual Peak kwh Peak kw Therm Therm Savings Savings Savings Savings Per home Per Home Per home Per Home LED Light Bulbs: The Oklahoma Deemed Savings Document specifies the following formula for use in calculating energy and demand impacts of ENERGY STAR Omni- Directional LEDs measures. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 261

265 kwh savings = ((W base W post )/1000) X Hours X ISR X IEF E Where: Wbase Wpost = Based on wattage equivalent of the lumen output of the purchased LED Omni-directional lamp and the program year purchased/installed; for Omni-directional LED, use the following base wattages = Wattage of LED purchased/installed Hours = Average hours of use per year (960.6) 76 ISR IEFE = In-Service Rate or percentage of rebate units that get installed, to account for units purchased but not immediately installed. (0.96, ADM calculated) = Interactive Effects Factor to account for cooling energy savings and heating energy penalties (see table below) ENERGY STAR Omni-Directional LED EISA Baseline 2007 Minimum Lumens Maximum Lumens Incandescent Equivalent 1 st Tier (W base) , ,050 1, ,490 2, ENERGY STAR Omni-Directional LED Interactive Effects Factor for Cooling Energy Savings and Heating Energy Penalties Heating Type Interactive Effects Factor (IEF E) Gas 1.15 Electric Resistance 0.44 Heat Pump 0.84 Heating Unknown 0.96 Where: kw savings = ((W base W post )/1000) X CF X ISR X IEF D CF = summer peak coincidence factor for measure, 9% indoor and 0% outdoor 76 ADM HOU Memo, Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 262

266 IEFD = Interactive Effects Factor to account for cooling demand savings and heating demand penalties; this factor also applies to outdoor and unconditioned spaces (see table below) ENERGY STAR Omni-Directional LED Interactive Effects Factor for Cooling Demand Savings and Heating Energy Penalties Heating Type Interactive Effects Factor (IEF D) Gas Electric Resistance Heat Pump 1.53 F.1.3 Energy Saving Products Program This section includes the measure level algorithms and deemed savings values utilized for the ex post Gross kwh and kw savings calculations. LED Bulbs Gross annual energy savings for discounted LEDs were calculated using the algorithm from the Oklahoma Deemed Savings Documents shown below, except for the Hours of Use (HOU). Where: Wbase = Wpost = Baseline wattage equivalent for the lumen output of purchased bulb Wattage of purchased bulb Hours = Average hours of use per year (960.61) ISR = IEFE = In Service Rate, or percentage of discounted bulbs that get installed (97%) Interactive Effects Factor to account for cooling energy savings and heating energy penalties (0.96 for unknown heating fuel type). Peak demand savings for LEDs discounted through the program were also calculated using the algorithm from the Deemed Savings Documents, shown below: Where: Wbase = Baseline wattage equivalent for the lumen output of purchased bulb Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 263

267 Wpost = ISR = Wattage of purchased bulb In Service Rate, or percentage of discounted bulbs that get installed (97%) CF = Summer Peak Coincidence Factor (9%) IEFD = Interactive Effects Factor (1.53) Room Air Purifiers Gross annual energy savings for discounted room air purifiers were calculated using the algorithm from the IL TRM v5.0, shown below: Where, Room Air Purifier kwh savings = kwh Base kwh ESTAR kwhbase = Baseline kwh consumption per year; based on the table below kwhestar = ENERGY STAR kwh consumption per year; based on the table below Clean Air Delivery Rate (CADR) kwh Per Year Usage Based on Clear Air Delivery Rate CADR used in calculation Baseline Unit Energy Consumption (kwh/year) ENERGY STAR Unit Energy Consumption (kwh/year) ΔkWH CADR CADR CADR CADR CADR Over The peak demand kw savings for room air purifiers were calculated using the algorithm from the IL TRM v5.0, shown below: Where, Room Air Purifier peak kw demand = ΔkWh Hours CF ΔkWh = Gross customer annual kwh savings for the measure Hours = Average hours of use per year, CF = Summer Peak Coincidence Factor for measure, Consistent with ENERGY STAR Qualified Room Air Clean Calculator; 16 hours a day, days a year. 78 Assumes appliance use is equally likely at any hour of the day or night. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 264

268 Advanced Power Strips Clothes Dryers Peak kw Demand Based on Clear Air Delivery Rate Clean Air Delivery Rate ΔkW CADR CADR CADR CADR CADR Over Advanced Power Strip Deemed Savings in Residential Applications APS Type System Type Peripheral Device kw Savings kwh Savings Tier 1 Average Whole System Average The kwh savings for clothes dryers (CD) were calculated via the following formula: Where: CD kwhsavings = ( Load CEF base Load CEF eff ) N cycles %Electric Load = The average total weight of clothes per drying cycle. Standard = 8.45 lbs and Compact = 3 lbs. CEF base = Combined energy factor (CEF) of the baseline unit is based on existing federal standards energy factor and adjusted to CEF as performed in the ENERGY STAR analysis. CEFbase by Product Class Product Class CEF (lbs/kwh) Vented Electric, Standard ( 4.4 ft 3 ) 3.11 Vented Electric, Compact (120 V) (<4.4 ft 3 ) 3.01 Vented Electric, Compact (=240 V) (<4.4 ft 3 ) 2.73 Ventless Electric, Compact (=240 V) (<4.4 ft 3 ) 2.13 Vented Gas 2.84 CEF eff = CEF of the ENERGY STAR unit based on ENERGY STAR requirements. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 265

269 CEFeff by Product Class Product Class CEF (lbs/kwh) Vented Electric, Standard ( 4.4 ft 3 ) 3.93 Vented Electric, Compact (120 V) (<4.4 ft 3 ) 3.80 Vented Electric, Compact (=240 V) (<4.4 ft 3 ) 3.45 Ventless Electric, Compact (=240 V) (<4.4 ft 3 ) 2.68 Vented Gas 3.48 N cycles = Number of dryer cycles per year (283 cycles) %Electric = The percent of overall savings coming from electricity (100% for electric dryers and 16% for gas dryers) Demand savings will be calculated via the following formula: Where: CD kwsavings = kwh savings Hours Hours = Annual run hours of clothes dryer (283 hours) CF = Summer peak coincidence factor (3.8%) Water Coolers CF The kwh and demand kw savings will be calculated as shown below. Default Savings for ENERGY STAR Water Coolers Cooler Type kwh savings kw peak Cold Only Hot & Cold Storage Hot & Cold On-Demand Bathroom Ventilation Fans The kwh savings for bathroom ventilation fans (BVF) will be calculated via the following formula and is set at 88.6 kwh: Where: BVF kwhsavings = CFM 1 η Baseline CFM = Nominal Capacity of the exhaust fan (50 CFM) 1 η Efficient 1000 η Baseline = Average efficicay for baseline fan (3.1 CFM/Watt) η Efficient = Average efficancy for efficient fan (8.3 CFM/Watt) Hours Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 266

270 Hours = Assumed annual run hours (8,766 for continuous ventilation) Demand savings will be calculated via the following formula and is set at kw: Where: BVF kwhsavings = CFM 1 η Baseline 1 η Efficient 1000 CF = Summer peak coincidence factor (continuous operation, 1). Heat Pump Water Heaters Hours Thus, kwh savings for heat pump water heaters (HPWH) were calculated via the following formula: HPWH kwh savings ρ X C p X V (T SetPoint T Supply ) ( 1 ( EF pre = 3,412 Btu/kWh Where: ρ = Water density = 8.33 lb/gal Cp = Specific heat of water = 1 BTU/lb F 1 (EF post (1+PA%)) X Adj)) V = Estimated annual hot water use (gal) (shown in table below) TSetPoint = Water heater set point (default value = 120 F) TSupply = Average supply water temperature (shown in table below) EFpre = Baseline Energy Factor (shown in table below) EFpost = Energy Factor of new HPWH PA% = Performance Adjustment to adjust the HPWH EF relative to ambient air temperature per DOE guidance = Tamb Tamb Tamb Tamb = Ambient temperature dependent on location of HPWH (Conditioned or Unconditioned Space) and Weather Zone. Adj = HPWH-specific adjustment factor to account for Cooling Bonus and Heating Penalty on an annual basis, as well as backup electrical resistance heating which is estimated at 0.92 EF. Adjustment factors are listed in the table below. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 267

271 Estimated Annual Hot Water Use (gal) Weather Zone Tank Size (gal) of Replaced Water Heater Fayetteville 18,401 20,911 25,093 30,111 8 Fort Smith 18,331 20,831 24,997 29,996 7 Little Rock 18,267 20,758 24,910 29,892 6 El Dorado 17,815 20,245 24,293 29,152 Average Water Main Temperature Weather Zone Average Water Main Temperature ( F) 9 Fayetteville Fort Smith Little Rock El Dorado 70.1 Water Heater Replacement Baseline Energy Factors (Calculated) Minimum Required Energy Factors by NAECA After 4/16/2015 Fuel Type Natural Gas or Propane Electric Average Ambient Temperatures (Tamb) by Installation Location Weather Zone Conditioned Unconditioned Space Space 9 Fayetteville Fort Smith Little Rock El Dorado Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 268

272 HPWH Adjustment Weather Zone 9 Fayetteville Water Heater Location Furnace Type Conditioned Space Gas Heat Pump Elec.Resistance Unconditioned Space N/A Weather Zone 8 Fort Smith Water Heater Location Furnace Type Conditioned Space Gas Heat Pump Elec.Resistance Unconditioned Space N/A Weather Zone 7 Little Rock Water Heater Location Furnace Type Conditioned Space Gas Heat Pump Elec.Resistance Unconditioned Space N/A Weather Zone 6 El Dorado Water Heater Location Furnace Type Conditioned Space Gas Heat Pump Elec.Resistance Unconditioned Space N/A Demand savings were calculated via the following formula: Where: Peak kw Ratio Annual kwh = Peak kw kwsavings = kwh savings Ratio Annual kwh F.1.4 High Performance Homes Program New Homes, Multiple Upgrades, and Single Upgrades Components This section includes the measure level algorithms and deemed savings values utilized for the ex post Gross kwh and kw savings calculations. Infiltration Reduction: ADM utilized the AR TRM for the savings algorithms and deemed savings values. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 269

273 Annual Energy Savings: kwh Savings = CFM X ESF Coincident Peak Demand Reduction: kw Savings = CFM X DSF Where: CFM ESF DSF = Air infiltration reduction in Cubic Feet per Minute at 50 Pascal = The energy savings value corresponding to the climate zone and heating and cooling type in the following table = The demand savings value corresponding to the climate zone and heating and cooling type in the following table Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 270

274 Infiltration Reduction Deemed Savings Values Infiltration Reduction Deemed Savings Impact per CFM50 Reduction Equipment Type kwh Savings (ESF) kw Savings (DSF) Therm Savings (GSF) Peak Therms (GPSF) Zone 9 Electric AC with Gas Heat Gas Heat Only (no AC) NA Elec. AC with Resistance Heat NA NA Heat Pump NA NA Zone 8 Electric AC with Gas Heat Gas Heat Only (no AC) NA Elec. AC with Resistance Heat NA NA Heat Pump NA NA Zone 7 Electric AC with Gas Heat Gas Heat Only (no AC) NA Elec. AC with Resistance Heat NA NA Heat Pump NA NA Zone 6 Electric AC with Gas Heat Gas Heat Only (no AC) NA Elec. AC with Resistance Heat NA NA Heat Pump NA NA Duct Sealing: ADM utilized the OKDSD for the savings algorithms. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 271

275 Annual Energy Savings Cooling: Where: DLpre DLpost EFLHc h ρout kwh savings,c = (DL pre DL post )X EFLH C X (h out ρ out h in ρ in ) X X SEER = Pre-improvement duct leakage at 25 Pa (ft3/min) = Post-improvement duct leakage at 25 Pa (ft3/min) = Equivalent full load cooling hours, from table = Outdoor/Indoor seasonal specific enthalpy (Btu/lb), from table = Density of outdoor air (lb/ft3) ρin = Density of conditioned air at 75 F (lb/ft 3 ) = = Constant to convert from minutes to hours 1,000 = Constant to convert from W to kw SEER = Seasonal Energy Efficiency Ratio of existing system (Btu/W hr) = 13 (default) Annual Energy Savings Heating (Heat Pumps): Where: DLpr DLpost kwh savings,c = (DL pre DL post ) X 60 X 0.77 X HDD X 24 X X HSPF = Pre-improvement duct leakage at 25 Pa (ft3/min = Post-improvement duct leakage at 25 Pa (ft3/min) 60 = Constant to convert from minutes to hours.77 = Factor to correlate design load hours to EFLH under actual working conditions = Volumetric heat capacity of air (Btu/ft3 F) HDD = Heating Degree Days 1,000 = Constant to convert from W to kw HSPF = Heating Seasonal Performance Factor of existing system (Btu/W hr) = 7.30 (default) Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 272

276 Annual Energy Savings Heating (Electric Resistance): Where: DLpre DLpost kwh savings,c = (DL pre DL post ) X 60 X 0.77 X HDD X 24 X.018 3,412 = Pre-improvement duct leakage at 25 Pa (ft3/min) = Post-improvement duct leakage at 25 Pa (ft3/min) 60 = Constant to convert from minutes to hours.77 = Factor to correlate design load hours to EFLH under actual working conditions = Volumetric heat capacity of air (Btu/ft3 F) HDD = Heating Degree Days 3,412 = Constant to convert from Btu to kwh Coincident Peak Demand Reduction: Where: kwhsavings,c EFLHc SEER kw savings,c = kwh savings,c EFLH c = Calculated kwh savings for cooling X SEER EER = Equivalent full load cooling hours, from table X CF = Seasonal Energy Efficiency Ratio of existing system (Btu/W hr) = 13 (default) EER = Energy Efficiency Ratio of existing system (Btu/W hr) 79 CF = Coincidence Factor = 0.87 (default) Duct Insulation: ADM utilized the OKDSD for the savings algorithms and deemed savings values. 79 EER = -.02 X SEER X SEER Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 273

277 Weather Zone Duct Insulation Deemed Savings Values - Attic ac_gas_heat per sq. ft. Duct Insulation Deemed Savings - Attic gas_heat per sq. ft. Gas Heat (no AC) Therms per sq. ft. ac_electric per sq. ft. heat_pump per sq. ft. AC Peak Savings kw per sq. ft a b Weather Zone Duct Insulation Deemed Savings Values - Crawlspace ac_gas_heat per sq. ft. Duct Insulation Deemed Savings - Crawlspace gas_heat per sq. ft. Gas Heat (no AC) Therms per sq. ft. ac_electric per sq. ft. heat_pump per sq. ft. AC Peak Savings kw per sq. ft a b Attic Insulation: ADM utilized the AR TRM for the savings algorithms and scaled deemed savings values. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 274

278 Baseline Insulation R- Value AC/Gas Heat kwh Deemed Savings for R-38 Ceiling Insulation Gas Heat (No AC) kwh Ceiling Insulation R-38 Impact per sq. ft. Gas Heat Therms AC/Electric Resistance kwh Zone 9 Heat Pump kwh AC Peak Savings kw Peak Gas Savings Therms 0 to to to to to Zone 8 0 to to to to to Zone 7 0 to to to to to Zone 6 0 to to to to to Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 275

279 Baseline Insulation R- Value AC/Gas Heat kwh Deemed Savings for R-49 Ceiling Insulation Gas Heat (No AC) kwh Ceiling Insulation R-49 Impact per sq. ft. Gas Heat Therms AC/Electric Resistance kwh Zone 9 Heat Pump kwh AC Peak Savings kw Peak Gas Savings Therms 0 to to to to to Zone 8 0 to to to to to Zone 7 0 to to to to to Zone 6 0 to to to to to Floor Insulation: ADM utilized the AR TRM for the savings algorithms and deemed savings values. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 276

280 Deemed Savings Values for Floor Insulation Floor Insulation Impact per sq. ft. Equipment Type kwh Savings kw Savings Zone 9 Therm Savings Peak Therms Electric AC with Gas Heat Gas Heat Only (no AC) NA Elec. AC with Resistance Heat NA NA Heat Pump NA NA Zone 8 Electric AC with Gas Heat Gas Heat Only (no AC) NA Elec. AC with Resistance Heat NA NA Heat Pump NA NA Zone 7 Electric AC with Gas Heat Gas Heat Only (no AC) NA Elec. AC with Resistance Heat NA NA Heat Pump NA NA Zone 6 Electric AC with Gas Heat Gas Heat Only (no AC) NA Elec. AC with Resistance Heat NA NA Heat Pump NA NA Wall Insulation: ADM utilized the AR TRM for the savings algorithms and deemed savings values. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 277

281 Deemed Savings Values for Wall Insulation Wall Insulation Impact per sq. ft. Equipment kwh Savings kw Savings Therm Savings Peak Therm Savings R-13 R-23 R-13 R-23 R-13 R-23 R-13 R-23 Zone 9 Electric AC with Gas Heat Gas Heat Only (no AC) NA NA Elec. AC with Resistance Heat NA NA NA NA Heat Pump NA NA NA NA Zone 8 Electric AC with Gas Heat Gas Heat Only (no AC) NA NA Elec. AC with Resistance Heat NA NA NA NA Heat Pump NA NA NA NA Zone 7 Electric AC with Gas Heat Gas Heat Only (no AC) NA NA Elec. AC with Resistance Heat NA NA NA NA Heat Pump NA NA NA NA Zone 6 Electric AC with Gas Heat Gas Heat Only (no AC) NA NA Elec. AC with Resistance Heat NA NA NA NA Heat Pump NA NA NA NA Knee Wall Insulation: ADM utilized the AR TRM for the savings algorithms and deemed savings values. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 278

282 Insulation Level Installed AC/Gas Heat kwh Deemed Savings Value for Knee Wall Insulation Gas Heat (No AC) kwh Knee Wall Insulation Gas Heat Therms Impact per sq. ft. AC/Electric Resistance kwh Heat Pump kwh AC Peak Savings kw Peak Gas Savings Therms Zone 9 R R Zone 8 R R Zone 7 R R Zone 6 R R Electronically Commutated Motors: ADM utilized the OKDSD for the savings algorithms and deemed savings values. ECM Deemed Savings Values Weather Zone Electronically Commutated Furnace Fan Motor Impact per SqFt Energy Savings in Heating Mode (kwh) Energy Savings in Cooling Mode (kwh) Peak Demand Savings (kw) a b Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 279

283 Air Conditioner and Air Source Heat Pump Retrofits: ADM utilized the OKDSD for the savings algorithms. Annual Energy Savings Cooling: 1 kwh savings,clg = (Cap base X CAP SEER AC X Base Annual Energy Savings Heating: 1 kwh savings,htg = (Cap base X CAP HSPF AC X Base Coincident Peak Demand Reduction: Where: 1 kw savings = (Cap base X CAP EER AC X Base 1 SEER post,ac ) X 1 HSPF HP ) X 1 EER post,ac/hp ) X 1 kw 1,000 W X EFLH C 1 kw 1,000 W X EFLH H 1 kw 1,000 W X CF Cap base = Rated equipment cooling capacity of the existing unit (BTU/hr) Cap AC/HP = Rated equipment cooling/heating capacity of the new unit (BTU/hr) 80 SEER Base SEER post EER Base EER post EFLH C EFLH H HSPF Base HSPF post, = Season Energy Efficiency Ratio of existing cooling equipment = Season Energy Efficiency Ratio of installed cooling equipment = Energy Efficiency Ratio of the existing equipment = Energy Efficiency Ratio of the installed equipment = Equivalent full load hours for cooling = Equivalent full load hours for heating = Heating Seasonal Performance Factor for existing heating equipment = Heating Seasonal Performance Factor for installed heating equipment CF = Coincidence Factor = 0.87 (default) High Efficiency Windows: ADM utilized the OKDSD for the savings algorithms and deemed savings values. 80 Rated capacity of the new unit shall no exceed capacity of the existing unit; if completing this with other measures, use existing unit capacity. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 280

284 Climate Zone Deemed Values for Energy Star Windows Window and Glass Door Improvements Deemed Savings Gas Gas Existing Heat Heat (no Window ac_gas_heat (no ac_electric heat_pump AC) Pane Type AC) Therms kwh AC Peak Savings kw 9 Single Pane Double Pane a Single Pane a Double Pane b Single Pane b Double Pane Single Pane Double Pane Single Pane Double Pane Faucet Aerators ADM utilized the AR TRM for the savings algorithms and deemed savings values. Annual Energy Savings: Where: ρ kwh Savings = ρ X C px V X (T mixed T Supply )X ( 1 RE ) Conversion Factor = Water density = 8.33 lb/gal C p = Specific heat of water = 1 BTU/lb * F V = gallons of water saved per year per faucet T mixed = Mixed water temperature = F T Supply RE = Average supply water temperature = Recovery Efficiency = 0.98 for Electric Water Heater = 2.2 for Heat Pump Water Heater = 0.79 for Natural Gas Water Heater Conversion Factor = 3,412 BTU/kWh for Electric Water Heating = 100,000 BTU/Therm for Gas Water Heating Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 281

285 Coincident Peak Demand Reduction: Where: Ratio Peak kw = Annual kwh kw Savings = kwh Savings X Ratio Peak kw Annual kwh Low Flow Showerheads: ADM utilized the OKDSD for the savings algorithms and deemed savings values. Deemed kwh Savings Low-Flow Showerhead kwh Savings for 1.5 GPM # shower heads retrofitted One Showerhead Low Flow Shower Heads Electric Water Heater Showerheads per household One Two Three Four Two Showerheads Three Showerheads Four Showerheads Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 282

286 Deemed kw Savings Low-Flow Showerhead kw Savings for 1.5 GPM # shower heads retrofitted One Showerhead Low Flow Shower Heads Electric Water Heater Showerheads per household One Two Three Four Two Showerheads Three Showerheads Four Showerheads 0.03 Omnidirectional LEDs: ADM utilized the OKDSD for the savings algorithms and deemed savings values. Annual Energy Savings: Where: kwh savings = ((W base W post )/1000) X Hours X ISR X IEF E Wbase Wpost = Wattage of baseline lamp = Wattage of LED purchased/installed. Hours = Average hours of use per year. (960.6) 81 ISR = In-Service Rate. 97% IEFE = Interactive Effects Factor to account for cooling energy savings and heating energy penalties. See table below. 81 ADM HOU Memo, Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 283

287 ENERGY STAR Omni-Directional LED Interactive Effects Factor for Cooling Energy Savings and Heating Energy Penalties Heating Type Interactive Effects Factor (IEF E) Gas 1.15 Electric Resistance 0.44 Heat Pump 0.84 Heating Unknown 0.96 Coincident Peak Demand Reduction: Where: CF IEFD kw savings = ((W base W post )/1000) X CF X ISR X IEF D = summer peak coincidence factor for measure, 9% indoor and 0% outdoor = Interactive Effects Factor to account for cooling demand savings and heating demand penalties; this factor also applies to outdoor and unconditioned spaces ENERGY STAR Omni-Directional LED Interactive Effects Factor for Cooling Demand Savings and Heating Energy Penalties Heating Type Interactive Effects Factor (IEF D) Gas Electric Resistance Heat Pump Ground Source Heat Pump: ADM utilized the OKDSD for the savings algorithms and deemed savings values. Annual Energy Savings: kwh savings,clg = Cap X kwh savings,htg = Cap X kw 1,000 W X EFLH 1 C X ( EER Base 1 kw 3,412 Btu X EFLH 1 H X ( COP Base 1 EER GSHP ) 1 COP GSHP ) Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 284

288 Coincident Peak Demand Reduction: Where: CAP EFLHC EFLHH EERbase EERGSHP COPBase COPGSHP kw savings = CAP X 1 kw 1,000 W X ( 1 EER Base 1 EER post,ac/hp ) X CF = Rated equipment cooling capacity of the new unit (Btu/hr) = Equivalent full load hours for cooling = Equivalent full load hours for heating = Energy Efficiency Ration of the baseline cooling equipment = Energy Efficiency Ration of the installed GSHP = Coefficient of Performance for the baseline heating equipment = Coefficient of Performance of the GSHP CF = Coincidence Factor = 0.87 Solar Screens: ADM utilized the OKDSD for the savings algorithms and deemed savings values. Deemed Savings for Solar Screens Solar Screens Impact per SqFt Weather Zone AC/Gas Heat kwh Gas Heat (no AC) kwh Gas Heat Terms AC/Electric Resistance kwh Heat Pump kwh AC Peak Savings kw A B Pool Pumps: ADM utilized the OKDSD for the savings algorithms and deemed savings values. Annual Energy Savings: Coincident Peak Demand Reduction: kwh savings = (Wh base Wh post ) X Days/1000 Wh base Wh post kw savings = ( X CF 1000 X Hrs base ) ( X CF base 1000 X (Hrs lo + Hrs hi ) post) Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 285

289 Where: Capacitygal Wh base = Capacity gal X #Turnovers / EF base Wh post = Capacity gal X#Turnovers/[(EF lo X = Capacity of swimming pool in gallons = 22,000 gal (default) # Turnovers = Number of pool turnovers per day = 1 (default) EF Hrsbase Hrslo Hrshi Days Hrs lo Hrs Total ) + (EF hi X Hrs hi HRS total )] = Energy Factor (EF) of pump, motor, and speed combination in gallons/wh (see table below) = Average hours a single-speed pump and motor needs to turn over the volume in a pool = 5.7 hours (default) = Average hours motor and pump spent in low speed for filtering = 7.4 hours (default) = Average hours motor and pump spent in high speed for operation for pool cleaners and other additional pool features = 2 hours (default) = Days of pool operation per year Year Round = 365 days Summer only = 167 days Energy Factors for Baseline and Change Case VSD Pumps Energy Factor (EF) - gallons/wh Horsepower (HP) Single-Speed (EFbase) VSD (EFlo / EFhi) EFlo - Low Speed (1725 rpm) <1.0 HP 6.34 >= 1.0 HP and <= 2.0 HP >2.0 HP EFhi - High Speed (3450 rpm) <1.0 HP >=1.0 HP and <= 2.0 HP >2.0 HP Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 286

290 F.1.5 ENERGY STAR LEDs Education Program The energy savings for ENERGY STAR LEDs are calculated by using the following equations. 82 Inputs for lighting calculations were determined from the data from the participant surveys in combination with algorithms and inputs found in the Arkansas TRM. Energy Savings: Demand Reduction: Where: kwh savings = (( Watts)/1,000) x Hours x ISR x IEF E kw demand reduction = (( Watts)/1,000) x CF x ISR x IEF D Watts = Average delta watts for the specified measure. Delta watts for LEDs are determined by the difference in watts between an EISA compliant baseline bulb and the distributed LED. Baseline wattages will be determined based on the lumen range of the measure and the EISA baseline standards. ISR = In-service rate, the percentage of LEDs distributed that are installed. Determined though participant survey data. Hours = Average hours of use per year, assumed to be 960 hours. 83 IEF E = Interactive effects factor to account for cooling energy savings and heating energy penalties. IEF D = Interactive effects factor to account for cooling energy savings and heating energy penalties. Advanced Power Strips For nonresidential advanced smart strips installed, ADM utilized the deemed savings value for small business from the Arkansas TRM. 82 Algorithm source: Arkansas TRM Vol Page Based on 2016 Energy Saving Products Memo. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 287

291 Advanced Power Strips-Demand and Annual Energy Savings System Type kw Demand Reduction kwh Savings Residential Home Entertainment System Home Office Other Devices Small Business Small Business Whole System FilterTone Alarm Energy Savings: kwh savings = [ EFLH Heat + EFLH Cool ] kw motor EI ISR Where, EFLH Heat = Assumed to be 1,337 hours. 84 EFLH Cool = Assumed to be 1,069 hours. kw motor = Average motor full load electric demand (kw), assumed to be 0.5 kw. EI = Efficiency improvement, assumed to be 15% or ISR = In-service rate, or percentage of units that get installed. Demand Reduction: kw demand reduction = kw motor EI ISR CF Where, kw motor = Average motor full load electric demand (kw), assumed to be 0.5 kw. EI = Efficiency improvement, assumed to be 15% or ISR = In-service rate, or percentage of units that get installed. CF = Coincidence factor, assumed to be EFLH Heat and EFLH Cool were modeled for PSO s service territory by ADM in support of PSO s PY2016 to PY2018 DSM Portfolio Plan. 85 Coincidence factor for demand reduction associated with air conditioning units provided in Arkansas TRM Vol 6. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 288

292 LED Night Light ADM utilized the algorithm for calculating were calculated using the following equation from the PA TRM. 86 Where: Hours 365 days year kwh savings = [(W base W post ) ( 1000 W )] ISR kw Wbase = Assumed to be a 7-watt incandescent nightlight. Wpost = Assumed to be a 1-watt LED nightlight. Hours = The nightlight is assumed to operate 12 hours per day. ISR = In-Service Rate, or percentage of delivered units that get installed. 86 Algorithm source: 2016 Pennsylvania TRM, Page 28. Appendix F: OKDSD & AR TRM Deemed Savings and Algorithms 289

293 Appendix G. M&V Safety ADM is dedicated to creating a safe work environment and to provide training for our employees. All ADM employees undergo general safety training. Our field technicians and engineers undergo additional safety training related to fieldwork. We encourage all our employees to be responsible and alert to identify hazardous conditions wherever they may exist be it in transportation to the customer or at the customer s facility. If hazardous conditions are found, they are to report them immediately to their supervisor or the ADM Safety Officer. Never are they to proceed to work in an identified hazardous situation. ADM follows Cal/OSHA rules and guidelines for safety in the workplace and these rules are as or more stringent than the federal OSHA rules. Personal Protective Equipment (PPE) is provided and the procedures to use it as appropriate for the work expected. Our field staff is provided training to safely conduct activities they may encounter. Specifically, this includes the use of ladders and the rules associated with working at heights. Three points of contact on ladders are required at all times. It is trained that body harnesses are required when being lifted by a man lift or bucket, although we also train to avoid the use of lifts. If rooftops need to be accessed, our field staff is trained to identify if it is safe to be there and the requirements for perimeter protection. For those that will make electrical measurements, electrical safety training is given for new hires and periodically reviewed for all employees working in such conditions. Electrical safety training includes the use of PPE and the voltage the PPE is appropriate for use around. Arc flash training reinforces the reason for using PPE. ADM does not conduct any measurement activity on systems over 500 Volts. Other training includes exposure to asbestos, lead, and hydrogen sulfide. Employees are trained to follow safety procedures and there are consequences for not following proper procedures which can include termination of employment. Appendix G: M&V Safety 290

294 Appendix H. Lighting Discounts Price Response Model Details H.1 Introduction In order to develop one estimate of free ridership for discounted LED bulb sales, ADM developed a price response model using sales data and associated information provided by the program administrator. This approach to free ridership estimation uses econometric techniques to estimate the effect of price changes on the number of bulb packages sold. The model uses variation in bulb package pricing over time to estimate price elasticity of demand (the change in quantity demanded as prices change). The model is used to predict what level of sales would have occurred under the counterfactual scenario where no discount program is offered, and bulbs are sold at their original retail prices. H.2 Data Sources and Processing The program administrator provided ADM with weekly sales data 87 for the Lighting Discounts component of the ESP program. The data included sales quantities separated by retailer and bulb model number. Data was used for all of PY2017 (48 weeks) and PY2016 (4 weeks). Additional records regarding dates and retail locations of in-store promotional events was combined with the sales tracking data to create the final dataset used for the price response modeling. For each unique combination of retailer, model number, and price discount, the dataset contained the following field used for the econometric model: Original retail price Target retail price (price after any manufacturer incentives and program sponsored discounts) Rated lumens and wattage Bulb type designation (omni-directional LED, directional LED) Promotional events for given retailers in given time periods Month in which the product was sold 87 The majority of the sales data was reported in weekly increments. However, for some retailer/manufacturer combinations sales data was reported bi-weekly or monthly. In these instances, the bi-weekly or monthly sales data were allocated equally into weekly allotments. While this may introduce some measurement error, the number of instances where this was required is small relative to the total population of bulb sales. Appendix H: Lighting Discounts Price Response Model Details 291

295 Summary statistics for the final dataset used to estimate the price response model are provided in the tables below. Bulb Type Count of SKUs by Bulb Type and Store Type Bulb Type Discount DIY Mass Merchant Total LED Omni-directional LED directional Total Summary Statistics by Bulb Type Total Packages Sold Total SKUs Average Retail Price Per Package Average Program Discount Per Package LED Omni-directional 265, $15.45 $4.60 LED directional 84, $15.66 $5.71 Total 350, $15.56 (average) $5.16 (average) H.2.1 Price Response Model Development and Specification The econometric approach used to estimate the price response model was informed by past evaluations of residential lighting programs in Maine 88 and Michigan. 89 Program sales data are, by their nature, non-negative integer values (i.e., count data). Typical ordinary least squares (OLS) estimation procedures are designed to deal with continuous dependent variables that are normally distributed. Count data dependent variables can be adapted for OLS estimation through logarithmic or square root transformations, but these models may produce nonsensical predictions, such as negative sales. ADM chose instead to use a negative binomial model 90 based on the prior research in Maine and summary statistics of the available data. The program sales data can be arranged as a panel, with a cross-section of program packages modeled over the 52 weeks for which there is information. However, the large number of zeroes introduced by missing sales data presents a problem for estimating a model with good fit and predictive power. There are econometric techniques for modeling excessive zeros (hurdle models, zero-inflated models) but the theoretical justification for 88 Report_FINAL.pdf 89 Lighting_Price-Elasticity-Model.pdf 90 A negative binomial regression is a type of generalized linear model that is implemented using maximum likelihood estimation. For a detailed description of the negative binomial regression, see Cameron, A.C. and P.K. Trivedi (2013), Regression Analysis of Count Data, Second Edition, Cambridge University Press. Appendix H: Lighting Discounts Price Response Model Details 292

296 these techniques does not align with a situation where the zeroes represent sales data that does not exist (no sales data that week) or an incentive was not available. Instead of preserving the panel structure of the data by leaving the zeroes in the model, ADM opted to estimate a cross-sectional negative binomial regression, omitting any instances of zero sales. That is, rather than modeling sales over a 52-week period, each weekly package sales quantity was modeled as if it was sold during the same time period, with zero sales instances removed from the model. 91 Seasonal effects on sales quantities were controlled for through a set of monthly dummy variables. After determining the general modeling approach 92, ADM tested a number of different specifications to determine program impacts on standard LED and specialty LED demand. Ultimately, a model similar to the final model for the Michigan evaluation was chosen, as it provided the best statistical fit to the program sales data with the best predictive power of the models compared. The model assumes that three broad factors affect bulb sales: prices, the presence of promotional events and seasonal trends. The final model uses dummy variables to control for seasonal effects (month dummies) and bulb type (model number dummies). A separate model was run for each bulb type (omnidirectional LED and directional LED). The basic equation of the price response model was estimated as follows (for bulb model i, in period t): Where: ln(q it ) = β 1 + β 2 ln(p it ) + β 3 EventDummy it + β π ModelNumberDummy i ln = natural logarithm + β γ MonthDummy t + ε it γ Q = quantity of bulb packs, i, sold during week t P = retail price (after markdown) for package of bulbs, i, during week t EventDummy = a binary variable equaling 1 if a promotional event occurred at the retailer selling bulb pack, i, during week t; 0 otherwise ModelNumberDummy = a binary variable equaling 1 for each unique model number; 0 otherwise π 91 By omitting all zeroes, some instances of truly zero sales are ignored. However, a review of the data indicates that true zeros are a very small proportion of the omitted data. The vast majority represent missing sales data due to non-program pricing. 92 Hurdle models, Poisson models, and zero-inflation models were all considered. However, the nature of the zero sales quantities eliminated hurdle and zero-inflation models. Overdispersion eliminated the Poisson model from consideration. Appendix H: Lighting Discounts Price Response Model Details 293

297 MonthDummy = a binary variable equaling 1 in a given month; otherwise The β2 coefficient in the model represents average price elasticity of demand holding the effects of all other independent variables constant. The β3 coefficient captures the impact of promotional events on bulb sales. Under the counterfactual scenario where no program exists, the EventDummy variable is always zero, indicating the absence of program sponsored promotional events. In some cases, there were multiple promotional events at a given retailer during a single sales period; however, ADM used a binary indicator variable to indicate promotional events in all cases. There was additional data available regarding product placement (e.g., end caps, wing stacks, etc.) that was not included in the model. To the extent that the program influenced positive product placement, there may have been additional sales independent of price changes. Therefore, the free ridership values estimated through this model may be conservative because they do not account for the effects of the featured placements. The βπ and βγ coefficient captures the impact of light bulb model and seasonality on sales volume, respectively. The figure below shows total package sales by during each month of PY2017and demonstrates clear demand fluctuation across months. The sales volume variation is partly due to naturally occurring seasonality in bulb sales, and partly due to variations in program intensity (i.e., funding, discount levels). Inclusion of the month indicator variables may capture some of the sales volume variation attributable to the program intensity, thus potentially biasing the free ridership estimate upwards. The alternative specification (leaving the month indicator variables out of the model) could potentially attribute naturally occurring sales increases to the program. Since both approaches have inherent uncertainty, the more conservative approach (in terms of free ridership estimation) was used by including the month indicator variables. Appendix H: Lighting Discounts Price Response Model Details 294

298 PY2017 Package Sales by Month The tables below show the estimated coefficients and related measures of fit for the final model by bulb type (omni-directional LED and directional LED). Using the coefficients from the model, ADM was able to estimate bulb sales under various conditions. To estimate a free ridership ratio, ADM used the model to estimate what bulb sales would have been at the original retail price and absent any in-store promotional events. Appendix H: Lighting Discounts Price Response Model Details 295

299 Negative Binomial Regression - Price Response Model for Standard LEDs (Dependent Variable: Bulb Packages Sold / Week) Variable Coefficient 95% Confidence Standard p- z Interval for Error value Estimated Coefficients Constant ln(price) EventDummy August December February January July June March May November October September ModelNumberDummies OMITTED Appendix H: Lighting Discounts Price Response Model Details 296

300 Negative Binomial Regression - Price Response Model for Specialty LEDs (Dependent Variable: Bulb Packages Sold / Week) Variable Coefficient 95% Confidence Standard p- z Interval for Error value Estimated Coefficients Constant ln(price) EventDummy August December February January July June March May November October September ModelNumberDummies OMITTED Negative Binomial Regression - Price Response Model for all Bulb Types Summary Statistics Bulb Type Loglikelihood Omnidirectional LED Directional LED Null Deviance Null Degrees of Freedom AIC BIC Residual Deviance Residual Degrees of Freedom 38, ,302-15, , , , ,122 20, ,747-14, , , , ,545 The figure below shows actual weekly package sales vs. model fitted quantities for standard omni-directional LEDs and directional LEDs. Included is a linear regression fit of the total number of packages versus the fitted number of packages based on the price response model by bulb type (the gray area around the line of fit represents the 95% standard error). Appendix H: Lighting Discounts Price Response Model Details 297

301 Actual Packages vs. Fitted Package Sales Price Response Model H.2.2 Free Ridership Estimation Results Free ridership ratios were calculated for the program as follows. First, the price response model was used to estimated bulb package sales under program and non-program pricing scenarios. The non-program scenario represents pricing at original retail levels along with the absence of any program sponsored promotional events. Bulb package sales under both scenarios were then multiplied by the number of bulbs per package to arrive at total bulb sales under the program and non-program scenarios. Finally, deemed savings Appendix H: Lighting Discounts Price Response Model Details 298