Public Service Company of Oklahoma 2016 Energy Efficiency & Demand Response Programs: Annual Report

Transcription

1 Public Service Company of Oklahoma 2016 Energy Efficiency & Demand Response Programs: Annual Report Prepared for: Oklahoma Corporation Commission In accordance with annual reporting requirements: Title 165: Oklahoma Corporation Commission Chapter 35. Electric Utility Rules Subchapter 41. Demand Programs 165: Reporting June 22, 2017

2 Table of Contents 1. EXECUTIVE SUMMARY PROGRAM OFFERINGS SUMMARY OF ENERGY IMPACTS SUMMARY OF PEAK DEMAND IMPACTS SUMMARY OF PORTFOLIO BENEFIT-COST RATIOS REDUCED EMISSIONS AND WATER CONSUMPTION CUMULATIVE PORTFOLIO PERFORMANCE SUMMARY OF PROCESS EVALUATION FINDINGS INTRODUCTION HISTORY OF PSO S ENERGY EFFICIENCY AND DEMAND RESPONSE PROGRAMS PROGRAM DESCRIPTIONS MILESTONES ACHIEVED IN MARKET TRANSFORMATION PROGRAMS ANNUAL UTILITY GROWTH METRICS AND PORTFOLIO RATIOS HIGH-VOLUME ELECTRICITY USER OPT OUT FUEL SWITCHING IMPACTS PROGRAM IMPLEMENTATION & STRATEGIC ALLIANCES TRAINING AND CUSTOMER OUTREACH EVALUATION MEASUREMENT AND VERIFICATION ENERGY EFFICIENCY PROGRAMS HIGH PERFORMANCE BUSINESS PROGRAM HOME WEATHERIZATION PROGRAM ENERGY SAVING PRODUCTS PROGRAM HIGH PERFORMANCE HOMES PROGRAM EDUCATION PROGRAM DEMAND RESPONSE PROGRAMS BUSINESS DEMAND RESPONSE APPENDIX A. GLOSSARY APPENDIX B. PORTFOLIO COST-EFFECTIVENESS COST EFFECTIVENESS SUMMARY ENERGY EFFICIENCY PROGRAMS DEMAND RESPONSE PROGRAMS i

3 5.4 AVOIDED COSTS APPENDIX C. IDENTIFICATION OF PROGRAM IMPLEMENTERS APPENDIX D. TRAINING AND CUSTOMER OUTREACH APPENDIX E. MARKETING SYNOPSIS ii

4 1. Executive Summary This report presents an evaluation of the performance of the energy efficiency and demand response programs, also known as the Demand Portfolio, offered by Public Service Company of Oklahoma (PSO) in PSO is submitting this report to fulfill the requirements outlined in Title 165: Oklahoma Corporation Commission Chapter 35. Electric Utility Rules Subchapter 41. Demand Programs 165: On July 1, 2015, PSO submitted a comprehensive portfolio of energy efficiency and demand response programs to the Oklahoma Corporation Commission (OCC) for Program Years This portfolio was approved by the OCC in Cause No. PUD by Order No on December 1, The focus of this report is participation during the first program year (PY2016) of the implementation cycle, spanning from January 1, 2016, to December 31, Program Offerings PSO offered customers six programs, four residential and two commercial/industrial in Program names, PY2016 start dates, and customer sector targeted are shown in Table 1-1. Table 1-1: Program Start Dates Program Sector Start Date Energy Efficiency Programs High Performance Business Commercial & Industrial January 01, 2016 Home Weatherization Low-Income Residential January 01, 2016 Energy Saving Products Residential December 01, High Performance Homes Residential January 01, 2016 Education Residential & Commercial January 01, 2016 Demand Response Programs Business Demand Response Commercial & Industrial January 01, 2016 Program impacts including projected, reported, and verified annual energy savings and peak demand reduction during PY2016 are summarized in the following sections. For the purposes of this report, projected, reported, and verified impacts are defined as follows: 1 All the programs represent program participation from January 1, 2016 December 31, 2016 except the ESP program. The reported savings for CFL and LED retail discounts span the time period of December 1, 2015 November 30, This month offset allows for reconciliation of retail sales data and manufacturer/retailer invoices Energy Efficiency & Demand Response Programs 1

5 PSO Executive Summary Projected Impacts refer to the energy savings (kwh) and peak demand reduction (kw) estimates submitted to the OCC as part of PSO s portfolio filing approved on December 1, Reported Impacts refer to energy savings (kwh) and peak demand (kw) reduction estimates based on actual customer participation in PY2016, before program evaluation activities. Verified Impacts refer to energy savings (kwh) and peak demand (kw) reduction estimates for PY2016 developed through independent program evaluation, measurement, and verification (EM&V). Verified impacts reflect actual program participation (as opposed to projected participation) and adjust for any findings from ADM s independent evaluation, which includes a detailed review of program materials, interviews with program participants, and in some cases detailed on-site data collection. A glossary of these and other energy efficiency and demand response evaluation specific terms is provided in Appendix A of this report. All impacts presented in this report represent energy savings or peak demand reduction at-the-meter. The only exception Section 1.4 and Appendix B, where impacts at-thegenerator are adjusted using an estimated line loss factor of for calculating program cost effectiveness. 1.2 Summary of Energy Impacts At the portfolio level, reported energy savings in PY2016 (123,446 MWh) exceeded projections by 23%. Total gross verified energy savings developed through EM&V of PSO s portfolio of programs is estimated at 128,080 MWh. This represents a realization rate of 104% as compared to reported energy savings. The portfolio-level Net-to-Gross (NTG) ratio, indicating the percentage of gross savings directly attributable to program influences, is estimated to be 82%, resulting in net verified energy savings of 104,437 MWh. 2 Order number for Cause No. PUD , 2016 Energy Efficiency & Demand Response Programs 2

6 PSO Executive Summary Table 1-2 summarizes the energy impacts of PSO s energy efficiency and demand response programs during PY2016. Table 1-2: Summary of Energy Impacts PY Gross Annual Energy Savings (MWh) Gross Program Projected Reported Verified Realization Rate Energy Efficiency Programs Net Impacts Net Annual NTG Energy Ratio Savings (MWh) High Performance Business 51,795 51,932 53, % ,009 Home Weatherization 3,794 5,726 5,537 97% ,537 Energy Saving Products 32,043 51,123 51, % ,830 High Performance Homes 7,541 8,406 8, % ,711 Education 4,949 6,260 7, % ,496 Energy Efficiency Totals 100, , , % ,583 Demand Response Programs Business Demand Response NA Demand Response Totals NA Portfolio Totals 100, , , % , Summary of Peak Demand Impacts Reported peak demand reduction in PY2016 exceeded the portfolio level projections by 18%. The 2016 EM&V study estimates verified gross peak demand of MW. This represents a realization rate of 92% as compared to reported demand reduction. The portfolio-level Net-to-Gross (NTG) ratio, indicating the percentage of gross reduction directly attributable to program influences, is estimated to be 94% percent, resulting in net verified peak demand reduction of MW. A complete list of peak demand impacts resulting from the implementation of PSO s energy efficiency and demand response programs during PY2016 is provided in Table Rounding may affect totals and net-to-gross ratio multiplication/division in table Energy Efficiency & Demand Response Programs 3

7 PSO Executive Summary Table 1-3: Summary of Demand Impacts PY Gross Peak Demand Reduction (MW) Gross Program Projected Reported Verified Realization Rate Energy Efficiency Programs Net Impacts Net Peak NTG Demand Ratio Reduction (MW) High Performance Business % Home Weatherization % Energy Saving Products % High Performance Homes % Education % Energy Efficiency Totals % Demand Response Programs Business Demand Response % Demand Response Totals % Portfolio Totals % Summary of Portfolio Benefit-Cost Ratios ADM calculated the cost-effectiveness of PSO s 2016 programs based on reported total spending, verified net energy savings, and verified net demand reduction for each of the energy efficiency and demand response programs. Additional inputs to the cost effectiveness testing include estimates of natural gas savings, line-loss adjustments, emissions reductions, measure lives, discount rates, participant costs, and avoided costs. All program spending inputs were provided by PSO as shown in Appendix B. The total portfolio spend was $30,425,693. The methods used to calculate costeffectiveness are informed by the California Standard Practice Manual. 5 The specific tests used to evaluate cost-effectiveness are the Utility Cost Test, Total Resource Cost Test, Ratepayer Impact Measure, Societal Cost Test, and Participant Cost Test. PSO s 2016 portfolio of programs passes these tests except the ratepayer 4 Rounding may affect totals and net-to-gross ratio multiplication/division in table. 5 California Standard Practice Manual: Economic Analysis of Demand Side Management Programs, October Available at: CE56ADF8DADC/0/CPUC_STANDARD_PRACTICE_MANUAL.pdf 2016 Energy Efficiency & Demand Response Programs 4

8 PSO Executive Summary impact measure. Few programs pass the ratepayer impact measure, so the latter finding is not unusual. The benefit-cost ratios developed through the cost-effectiveness analysis are presented in Table 1-4. Detailed cost-effectiveness assumptions and findings are in Appendix B. Table 1-4: Benefit-Cost Ratios Program Utility Cost Test Total Resource Cost Test Ratepayer Impact Measure Societal Cost Test Participant Cost Test Energy Efficiency Programs High Performance Business Home Weatherization Energy Saving Products High Performance Homes Education Energy Efficiency Total Demand Response Programs Business Demand Response Demand Response Total Portfolio Total Another way to view portfolio performance for 2016 is on a levelized dollar per kwh savings or dollar per peak kw reduction basis. Energy efficiency programs are designed to reduce energy usage while providing the same or improved service to the end-user in an economically efficient way. This includes energy usage at any time, inclusive of peak and non-peak periods. Energy savings occur for the lifetime of the energy efficiency measures installed. As such, it makes sense to assess program performance on a levelized dollar per kwh basis for energy efficiency programs. Levelized cost in $/kwh is calculated as shown in the formula below: Levelized Cost (in $/kwh) = C x Capital Recovery Factor / D Capital Recovery Factor = [A*(1+A)^(B)]/[(1+A)^(B)-1] Where: A = Discount rate (5%) 2016 Energy Efficiency & Demand Response Programs 5

9 PSO Executive Summary B = Estimated measure life in years 6 C = Total program costs D = Annual kwh savings Table 1-5 shows how PSO s portfolio of energy efficiency programs performed on a levelized cost basis for PY2016. The verified net lifetime energy savings in Table 1-5 include a line loss adjustment factor of Table 1-5: Levelized $/kwh for Energy Efficiency Programs Program Year Total Costs Verified Net Lifetime Energy Savings (kwh) Levelized $/kwh 2016 Residential Commercial EE Programs $18,023, ,486, $0.030 $10,116, ,917, $0.015 $28,140,036 1,263,404, $0.022 A study conducted by the American Council for an Energy Efficient Economy (ACEEE) 9 in 2014 compared levelized $/kwh for electric energy efficiency programs in 20 different states between 2009 and Levelized costs estimated as part of this study ranged from $0.016 to $0.048 with an average of $0.028 (2011$). The levelized $/kwh for PSO s energy efficiency programs are more favorable as compared to the 20 states from the ACEEE analysis. Demand response programs are designed to encourage customers to change their normal consumption patterns during periods when prices are high or system reliability is potentially constrained. These programs encourage load reduction during a short period of time, usually a limited number of days during the summer. As such, it makes sense to assess demand response program performance on a peak kw reduction per dollar 6 Calculated as described in Appendix B. 7 Residential Programs include: Home Weatherization, High Performance Homes, Energy Savings Products, and Education Residential and School Kits 8 Commercial Programs include: High Performance Business and Education Non-Residential Kits 9 Molina, Maggie, The Best Value for America s Energy Dollar: A National Review of the Cost of Utility Energy Efficiency Programs. ACEEE, March Energy Efficiency & Demand Response Programs 6

10 PSO Executive Summary basis. Table 1-6 shows how PSO s portfolio of demand response programs performed on a $/kw reduction basis for PY2016. The verified net peak demand reduction in Table 1-6 includes a line loss adjustment factor of Table 1-6: $/kw for Demand Response Programs Program Year Total Costs Verified Net Peak Demand Reduction from DR (kw) $/kw 2016 $2,285,657 46,707 $48.94 Overall, the $48.94 per kw of load reduction is lower than the $58.57 per kw from program year 2013, the $52.60 per kw in PY2014, and the $56.84 per kw in Reduced Emissions and Water Consumption Reduced emissions occur because of energy savings achieved through PSO s Demand Portfolio displacing marginal fossil fuel based electric generation. The EPA s Emissions and Generation Resource Integrated Database (egrid) is a comprehensive source of emissions data related to the electric power sector in the U.S. The tenth edition of egrid, based on 2014 data, was released in January of Included in the database are estimates of non-baseload emission rates for various greenhouse gasses in different sub regions of the country. The PSO service territory falls into egrid sub region SPP South (SPSO). Table 1-7 below lists the most recent egrid non-baseload output emission rates for SPSO. Table 1-7: egrid GHG Annual Output Emission Rates egrid Sub region Annual non-baseload output emission rates Carbon dioxide Methane Nitrous oxide (CO2) (CH4) (N2O) (lb/mwh) (lb/gwh) (lb/gwh) SPP South (SPSO) 1, Using the egrid emission rates and lifetime energy savings for measures installed through the PSO Demand Portfolio in 2016 results in the estimated emissions reductions listed in Table Energy Efficiency & Demand Response Programs 7

11 PSO Executive Summary Table 1-8: Emission Reduction Estimates Lifetime Energy Savings 11 Carbon dioxide reduction Methane reduction Nitrous oxide reduction (Net at Generator) (CO2) (CH4) (N2O) (MWh) (tonnes) (tonnes) (tonnes) 1,263, , Reductions in water consumption at participant homes/facilities resulting from PSO s 2016 portfolio of programs were not tracked. Many of the energy efficiency measures commonly associated with water savings in the residential sector were not included in the portfolio design because of the high prevalence of natural gas water heating in the PSO service territory (faucet aerators, low-flow shower heads, efficient clothes washers and dishwashers, etc.). The High Performance Business program does offer inducements for certain measures that have water saving potential for C&I customers (e.g., variable frequency drives on pump motors). The effects on water consumption for these measures were not quantified for PY2016. There are also water savings associated with reduced energy generation attributable to PSO s energy efficiency and demand response programs. PSO s generation fuel mix is made up of coal (~15%), natural gas (~21%), purchased power (~41%) and wind (~22%) 12. Most the purchased power comes from combined cycle goal. All the non-wind generation fuel sources are used in thermoelectric power plants which boil water to create steam, which in turn drives turbines. After the steam passes through a turbine, it is cooled so that it condenses and the water can be reused. The process of cooling the steam accounts for almost all the water use in most thermoelectric power plants, as the steam itself circulates in a closed system. A portion of the water used for this cooling process is lost to evaporation. The specifics regarding how much water is consumed in the process depend largely on the technologies used in each power plant (once-through water cooling, recirculating water cooling, dry-cooling). A 2003 report by the National Renewable Energy Laboratory (NREL) 13 developed estimates of water consumption per kwh of energy consumed for all U.S. states. The estimate for thermoelectric generation in Oklahoma is 0.51 Gallons per kwh consumed. Using the NREL water consumption estimates and lifetime energy savings for measures installed through the PSO Demand Portfolio in 2016 results in the lifetime water savings estimates listed in Table Lifetime energy savings listed are based on measure lives from the OK Deemed Savings Documents or the AR TRM, annual net energy savings estimated through EM&V of the 2016 portfolio, and a line-loss adjustment factor of Source: Energy Efficiency & Demand Response Programs 8

12 PSO Executive Summary Table 1-9: Water Savings Estimates, Thermoelectric Generation Lifetime Energy Overall Savings Generation Water Consumption Lifetime Water (Net at Generator) Percentage per kwh Consumed Savings (MWh) Thermoelectric (Gallons/MWh) (Gallons) 1,263,404 78% ,582, Cumulative Portfolio Performance Program year 2016 was the first year of implementation for the Demand Side Management (DSM) portfolio. Energy and demand impacts from program activity, starting on January 1, 2016, are shown in Table Table 1-10: 2016 Portfolio Performance Verified Energy and Peak Demand Impacts Program Year Energy Efficiency Programs Verified Gross Annual Energy Savings (GWh) Verified Net Annual Energy Savings (GWh) Verified Gross Peak Demand Reduction (MW) Verified Net Peak Demand Reduction (MW) Cumulative EE Totals Demand Response Programs Cumulative DR Totals Cumulative Portfolio Totals Summary of Process Evaluation Findings This section presents the high level findings and recommendations developed through process evaluation activities for the PY2016 portfolio. The results of the process evaluation research are largely positive. Program participants, service providers, and program staff were largely satisfied with the PY2016 Demand Program offerings. Key process related findings and recommendations from the PY2016 evaluation are summarized below: Key Overall Process Findings High levels of program participant satisfaction. 93% or more of participants were satisfied with their participation in High Performance Homes, Home Weatherization, High Performance Business, and Peak Performers. Teachers 2016 Energy Efficiency & Demand Response Programs 9

13 PSO Executive Summary that participated in the Education Program all agreed that the kit products were easy to use and that the materials were written clearly and well organized. Moreover, nearly all said they would participate again (99%) and that they would recommend the program to others (98%). Overall, these responses suggest that the programs are meeting customer needs and expectations, and that there are few to no perceivable issues with program delivery to participants. Table 1-11: Customer Satisfaction per program Program Percent Satisfied Multiple Upgrades (n = 148) 99% Single Upgrades (n = 146) 94% New Construction (n = 14) 93% Weatherization (n = 177) 95% HPB - SBES (n = 44) 100% HPB - Custom/Prescriptive (n = 135) 94% Education Nonresidential Kit Recipients (n= 124) 97% Education Residential Kit Recipients (n = 123) 99% Peak Performers (n = 17) 94% * Percent satisfied is the share of participants that said they were "very satisfied" or "somewhat satisfied" Key Process Findings for High Performance Business Custom and Prescriptive Staff continue to focus on achieving regional diversity of projects. Staff made additional changes to the business programs to ensure participation from customers throughout the service territory. A key change that should assist with regional diversity was the introduction of Small Business Efficiency Solutions. Small Business Solutions assigns service providers to portions of the service territory and requires that the service providers meet savings targets. That said, staff noted that the high demand for program incentives from businesses in the Tulsa region is a challenge to meeting regional diversity objectives. In response to market conditions, program staff has taken deliberate steps toward increasing non-lighting uptake. To diversify the measure types implemented through High Performance Business, staff has begun offering mechanical training for service providers and customers, and increased marketing materials that focus on non-lighting measures Energy Efficiency & Demand Response Programs 10

14 PSO Executive Summary The new Fulcrum tool has streamlined the verification processes. To facilitate the recording of information collected during the verification visits, staff introduced a new electronic tool to record data collected during those visits. This new tool is software referred to as Fulcrum and it allows for customized forms to be developed to aid data collection during site visits. The Fulcrum application interfaces electronically with the VisionDSM system used to track and manage program activity. Through this electronic interface, the Fulcrum tool improves the efficiency of the program by reducing data entry requirements. Key Process Findings for High Performance Business Small Business Energy Solutions (SBES) The program is well designed to mitigate barriers to energy efficiency in most small business types. The program provides energy assessments along with incentives that cover up to 70% of the measure costs. The 70% incentive is consistent with best practice design for small business programs and the energy assessments provide information about efficient equipment options, cost, and payback. The program targets refrigeration measures in addition to lighting and uses the assignment of sections of the service territory to service providers to garner participation from small businesses throughout the PSO service territory. Lighting service providers noted that the $0.295 per kwh saved target created two barriers to participation. First, that target largely ruled out LED retrofits and according to some service providers, some customers were only interested in LED retrofits. To address this barrier, program staff adjusted the incentive to cover 50% of the cost of LED retrofits in Second, the cost requirement largely ruled out participation from customers with few hours of operation, such as churches. While these types of organizations remain a poor fit for the program, they also generate the fewest energy benefits for the available incentive dollars. The range of measures available through PSO s program compares well with other small business programs. The program covers the types of lighting and refrigeration measures that are likely to account for most of the energy saving opportunities in small businesses. It does not, however, cover water heating measures (e.g., low-flow pre-rinse spray valves, hot water pipe wrap), vending misers, or strip curtains. The paperless process used in the program contributes to the efficiency of the process by reducing data entry needs and transcription errors. The Audit Direct Install (ADI) software is used by service providers to complete assessments, develop proposals, and submit project information. The ADI system allows for standardized audits and includes measure cost information for calculation of customer costs, incentives, as well as project savings and payback. The system also utilizes data on eligible customers provided by PSO to enable 2016 Energy Efficiency & Demand Response Programs 11

15 PSO Executive Summary contractors to qualify and target customers. Service providers generally thought that ADI was easy to use though one service provider commented that they do not receive a notification through ADI or an if program staff reject a project proposal. Fulcrum is used to capture information during verification visits, including discrepancies from the reported project, notes, and images. Both ADI and Fulcrum interface with the Vision system to track projects completed through the program. Service provider training has been sufficient to support program processes. All service providers reported that the program provided training on the use of the ADI software all thought this training was sufficient and none identified additional training needs. Additionally, service providers provided a favorable assessment of the software. Verification and quality control processes are sufficient to mitigate evaluation risk. Ten percent of projects completed by program service providers are subject to quality control verification visits. Staff reported that during the first few months of program operations, most projects were inspected. Additionally, PSO staff attended inspection visits and were satisfied with the procedures. Overall, the rate of verification visits and the data collected during visits are sufficient to support quality program delivery. Key Process Findings for Home Weatherization Two changes were made to the program in The household income threshold requirement increased to $45,000 or less per year (from $35,000) and direct install LED light bulbs were included. The increased income threshold requirement expands the share of PSO customers eligible for no-cost weatherization improvements. The addition of LED light bulbs addresses lighting efficiency needs not previously met by the program. Key Process Findings for Energy Saving Products Two primary changes made to Energy Saving Products were the addition of ENERGY STAR air purifiers to the program and a greater focus on LED lighting. Sales of ENERGY STAR air purifiers exceeded program expectations, despite the discounts introduction at mid-year. The focus on LED lighting was largely driven by changes in the lighting market and the reduced availability of CFLs. Additionally, a Costco location was added to the program and added significantly to the total program sales. Store visits found that air purifier discounts were not always visible. ADM visited two stores that offered discounts on air purifiers. At one retailer location, ADM did not identify any signs noting that the air purifiers were discounted. At 2016 Energy Efficiency & Demand Response Programs 12

16 PSO Executive Summary the other location, ADM found three ENERGY STAR models of the same manufacturer without a sign noting the discount two of the three models were models that received program discounts. While it cannot be concluded from these visits that discount signage was inconsistently displayed all 15 stores that sold discounted air purifiers, these two observations suggest that review of sign placement for air purifiers may be an area for future program development. Interviews with retailer contacts found that most retailers are satisfied with the program and confirmed in store quality control activities. Retailer contacts confirmed that program staff teach them about the program, talk to customers about the program, check pricing, and deliver signs and stickers. However, additional training on the purchase limits may be needed as more than one-half of interview respondents (62%) either did not know that there was a limit to the number of bulbs purchased or did know the correct number of bulbs that could be purchased. Key Process Findings for High Performance Homes Multiple and Single Upgrades The Multiple and Single Upgrade components are mature, well-established offerings. Through these components, PSO customers have access to rebates for a comprehensive range of energy efficiency improvements. Higher incentives are offered for implementing multiple measures to encourage customers to implement deeper efficiency improvements. Few service providers suggested additional measures to include in the programs. Recommended measures included incentives for multiple speed pool pumps, HVAC tune-ups, energy recovery ventilators, and variable refrigerant flow HVAC systems. The program provides services to a wide range of housing types including in-unit multifamily improvements (common area improvements are eligible for incentives through PSO s business programs). Single family and duplex rental properties receive no-cost efficiency improvements to overcome split incentive barriers resulting from the tenants payment of the utility costs. Payments for duct sealing projects exceeded $500 for 49% of projects and more than $1000 for 19% of the project. Duct sealing accounted for 40% of expected savings for Multiple Upgrades. The cost per kwh saved was $0.49 per expected kwh saved. Although this amount was comparable to other program measures such as central air conditioner replacements and door replacements and less than most insulation measures, it is a higher per kwh saved rate than found in other programs. For example, the cost per kwh of duct sealing for 2016 Energy Efficiency & Demand Response Programs 13

17 PSO Executive Summary Entergy New Orleans was $ While regional differences may account for some of this difference, the difference is sufficiently large to warrant review of the current incentive structure. Moreover, it is not uncommon for programs to cap duct sealing incentives in the range of $200 to $400. Multiple Upgrades regional budget allocations were effective at ensuring the program component met regional savings objectives. Budget allocations for specific measures in Single Upgrades were effective for managing the incentives provided through the program. Increased use of electronic tools has enhanced program efficiency. The program now uses an application (Fulcrum) that records site inspection information and transfers this information to the program database. Additionally, Single Upgrade applications can be submitted using an online application. Program changes made to improve the service provider experience. Key changes include the use of Automated Clearing House (ACH) payments that have reduced payment time by several days. Key Process Findings for High Performance Homes - New Homes Builders prefer new prescriptive incentives to incentives based on assessed efficiency. The prescriptive incentives applied to any home permitted after February 1 st, The intent of the prescriptive incentives was to enable builders to know exactly what incentive they will receive. Under the performancebased incentive approach, builders were faced with uncertainty if the finished home s efficiency was not as high as intended. Feedback provided by builders suggest that they prefer the prescriptive incentives and that they provide greater certainty when making decisions about equipment and design features. Additionally, the new incentives may have increased the number of participating builders. Fifty-three builders that participated in the program in 2016, up from 37 in Seven percent of the builders that participated in 2016 had not completed a home since 2014 and 2% had not completed a home since Program ENERGY STAR bonus likely increased the number of homes built to ENERGY STAR specifications. The New Homes Program offered a $1,000 incentive for receiving ENERGY STAR v3.1 certification. In total, 119 homes, or 11% of the completed homes, received ENERGY STAR certification. While a minority of homes completed through the program received the ENERGY STAR certification, the 119 homes compare very favorably to data reported by 14 ADM Associates (2016). Evaluation of PY5 Energy Efficiency Programs Portfolio Energy Efficiency & Demand Response Programs 14

18 PSO Executive Summary ENERGY STAR that indicates only 4 homes built in Oklahoma during 2015 received ENERGY STAR certification. Key Process Findings for Peak Performers Few Changes in Program Design or Operations. Overall, Peak Performers has remained relatively unchanged. The most significant change in 2016 was the change in procedures for managing program data. Under the new data management procedures, PSO is directly contracted with AEG to manage the data. Staff report that the process is streamlined and more highly automated than the process in place during the previous cycle. High Retention Continues. The beginning of the new portfolio cycle required reenrollment of customers that participated during Program staff provided a high level of assistance to customers to facilitate completion of the enrollment paperwork. This effort was successful as the number of accounts increased again with the continued participation of past participants and the addition of new accounts. PSO fully deployed AMI meters in 2015 and the first half of These communicating interval meters for all customers improved the efficiency of enrolling customers and allowed customers to view their data in near real time with the My Energy Advisor web platform. Previously, if a customer didn t have an interval meter, which most did not, PSO had to install one. Key Process Findings for Energy Education The School Kits component continues to operate well and no issues with program operations and procedures were identified. Teachers provided very favorable feedback on the program and nearly all stated that they would participate again and that they would recommend the program to colleagues. All parents stated that the materials were easy for them and their child to use. The Residential Kits component met its distribution goal, despite a late start, but the Non-Residential Kits Component did not. The primary reasons for why the non-residential kits component did not meet goal was that the distribution of kits started late in the program year and the response to the direct mail campaign was less than anticipated. Additionally, electronic marketing to business customers may have been less impactful than the approach used for residential kits. Specifically, business kits were promoted as an item in a newsletter and residential kits were promoted through an blast that focused specifically on the kits. As of late January, the Nonresidential Kits component was on track to meets its distribution goal and Residential Kits was ahead of its target goal Energy Efficiency & Demand Response Programs 15

19 PSO Executive Summary The distribution process for kits has sufficient quality assurance and control procedures in place. Additionally, the program tracks issues with kits such as delivery failures and returns. There have been no unresolved issues. Key Process Recommendations The evaluators provide the following recommendations for consideration: High Performance Business Custom/Standard: Include instructions in the Retrofit Lighting Rebate Application to submit a completed Retrofit Lighting Worksheet and list it under the application checklist. The application does not state that this is required and one customer noted that it was not clear that this needed to be submitted along with applications for rebates of more than $1,500. High Performance Business SBES: Consider Modifications to ADI to Notify Service Providers of Project Rejections. One service provider noted that they had had projects rejected by staff but were not aware of this. Staff should consider modifying ADI to notify contractors and potential PSO program coordinators when a status change occurs for a project. High Performance Business SBES: Emphasize the Small Impact on Business Operations in Outreach and Marketing Materials. Several program participants stated that they were initially concerned about participating because it would be disruptive to their business. Staff should discuss project timelines and time commitments at outreach events, in marketing materials, and sales pitches made by service providers. Additionally, case studies featuring participant testimonials of the minimal business disruptions may also be useful in alleviating this potential barrier to participation. Home Weatherization Consider additional LED types. In 2016, the program only offered direct installation of 9.2 W LED lamps (60W incandescent equivalents). While that wattage would cover most household lighting use, staff should consider adding other lamp wattages so that replacements can be matched to the existing lamp wattages found in homes. The experience of some customers may be improved if the amount of light provided by the efficient light bulbs was matched to the amount of light from the replaced light bulbs. ESP Review Protocols to Verify that Program Discounts are Consistently Displayed. The implementation vendor for PSO currently performs onsite visits to verify that signs are on display during site visits but additional review may be needed given the identification of stores that did not display signs. High Performance Homes Multiple and Single Upgrades: Adjust Duct Sealing Incentives. The current incentive structure allows for relatively large incentive payments for this measure and larger incentive payments do not 2016 Energy Efficiency & Demand Response Programs 16

20 PSO Executive Summary necessarily result in greater program savings. Staff should consider either basing incentives on leakage reduction rather than the number of supply and return registers or instituting a cap on incentives for the measure. High Performance Homes New Homes: To maximize net savings opportunities among production builders, the program should focus on recruiting new builders into the program and on increasing the efficiency of the homes built by legacy builders. Interview results suggest that these builders may continue to incorporate efficient designs and equipment in the absence of program support. High Performance Homes New Homes: Promote ENERGY STAR design services as a means of increasing the number of homes that receive ENERGY STAR certification. Although completion of 119 ENERGY STAR certified homes is a noteworthy achievement given that only four were completed in the state of Oklahoma in 2015, there may be opportunities to grow this number. Several builders indicated that they were not aware of the design services available and increased awareness of these services may improve builders perceptions of the feasibility of completing ENERGY STAR homes. Energy Education: Track the kits as a source of awareness for participants in the residential and small business programs to monitor the effect of the program in increasing participation in the rebate programs. Additional detail regarding these and other process evaluation findings and recommendations can be found in the body of this report Energy Efficiency & Demand Response Programs 17

21 2. Introduction This report presents an evaluation of the performance of the energy efficiency and demand response programs offered by Public Service Company of Oklahoma (PSO) in PSO is submitting this report to fulfill the requirements outlined in Title 165: Oklahoma Corporation Commission Chapter 35. Electric Utility Rules Subchapter 41. Demand Programs 165: History of PSO s Energy Efficiency and Demand Response Programs In August of 2008, because of the OCC s final order in Cause No. PUD , PSO implemented a portfolio of seven Demand Side Management (DSM) Quick Start programs. Those seven programs were the starting point for many of the programs PSO implemented through The first year of implementation of the Quick Start Programs provided PSO with valuable information and lessons learned. This led to the development of a more robust DSM portfolio in The OCC ultimately approved a DSM portfolio of 17 programs through the final order in Cause No. PUD on January 14 th, This approved DSM portfolio remained in effect through December 31, PSO closely monitored the performance of the DSM portfolio through annual program evaluations in 2010, 2011, and Results from these evaluations were used as inputs to program design for a set of six streamlined DSM programs for the implementation cycle. The reduction in the number of programs from seventeen to six does not reflect a reduction in DSM investment, but rather a consolidation of programs that were previously competing for similar customers and the discontinuation of some efficiency measures that did not prove cost-effective. The portfolio of programs was approved by the OCC through the final order in Cause No. PUD on November 15, PSO continued to monitor the performance of the DSM portfolio through program evaluations in 2013, 2014, and The programs showed continued improvements over the three year cycle that ended December 31, Due to the continued improvement and successes in the portfolio, PSO designed a portfolio for consisting of a similar design of the previous evaluation cycle. The six main programs remain in place with some minor changes and additions, while two more programs will be added in the 2017 and 2018 program years. A Behavioral modification program will be added in 2017 and a Conservation Voltage Reduction will be added in The portfolio plan was submitted the OCC in Cause No. PUD on July 1, The plan was approved by the OCC through the final order (Order No ) in Cause No on December 1, This report focuses on implementation during the first year of the portfolio cycle (PY2016) Energy Efficiency & Demand Response Programs 18

22 PSO Introduction Program Descriptions Energy Efficiency Programs High Performance Business: PSO s High Performance Business program seeks to generate energy savings for commercial and industrial (C&I) customers through the promotion of high efficiency electric end use products and processes. The program provides PSO s C&I customers with flexibility in choosing how to participate, by either self-sponsoring or by working through a third-party implementer to leverage technical expertise. Small commercial customers with high potential for energy savings from the lighting and refrigeration end-uses can also qualify for a direct install program component. The Small Business Energy Solutions program will provide implementation of products and prices to allow for greater incentive and participation in rural areas of the service territory. In 2016, the High Performance Business program provided cash inducements for 1,188 projects completed by customers representing 950 unique participants. Of the total population, 1,002 projects representing 767 unique participants came from the prescriptive and custom portion of the program. The Small Business portion of the program had 186 projects completed by 183 unique participants Home Weatherization: PSO s Home Weatherization Program seeks to generate energy and demand savings for limited income residential customers through the installation of a wide range of cost effective weatherization and other measures in eligible dwellings. The purpose of the Home Weatherization Program is to provide PSO s limited income residential customers the financial assistance they need to make their homes more energy efficient, increase comfort levels, and reduce their utility bills. The program has opened to renters and increased the income qualification limits to $45,000 per household. The Home Weatherization Program reported energy and demand impacts for a total of 2,157 projects completed in High Performance Homes: PSO s High Performance Homes Program seeks to generate energy and demand savings for residential customers through the promotion of standalone and comprehensive efficiency upgrades for both new homes and retrofits. The purpose of the High Performance Homes Program is to provide PSO residential customers with inducements for increasing building envelope efficiencies and installing items such as high efficiency appliances and HVAC equipment. The program has opened to renters and multifamily homes. The program has put greater emphasis on participation outside the metro areas of the service territory. In 2016, the program had a total of 4,895 measures installed as part of 1,258 unique projects through the Multiple Upgrades component of the program and 8,510 measures installed as part of 2,598 unique projects through the Single Upgrades component of the program. Additionally, participating builders completed 677 rebated homes through the New Homes component Energy Efficiency & Demand Response Programs 19

23 PSO Introduction Energy Savings Products: PSO s Energy Saving Products Program seeks to generate energy and demand savings for residential customers through the promotion of ENERGY STAR qualified CFLs, LEDs, and other products. The purpose of this program is to provide PSO residential customers inducements for purchasing products that meet high efficiency standards. The Energy Saving Products Program reported energy and demand impacts for a total of 504,758 packages of CFLs/LEDs (1,638,212 individual bulbs) discounted through participating retailers or distributed in partnership with local food pantries. Additionally, 228 room air purifiers were discounted through participating retailers. Education: PSO s Education Program seeks to generate energy and demand savings while increasing awareness for PSO s 2016 to 2018 Demand Program offerings. In addition to educating customers about energy efficiency, the program will also be an effective customer engagement tool and will direct customers to additional energy conservation and demand reduction programs offered by PSO. The program has three components: school education kits, residential kits, and non-residential kits Demand Response Programs Business Demand Response: The Peak Performers program is a demand response program targeted to commercial and industrial customers served by PSO capable of reducing demand on short notice. After receiving a curtailment message from PSO, facility operators shed electric load in many ways such as shutting down motors, pumps, compressors, air conditioning equipment, and lighting. Some customers are even able to maintain normal production levels while temporarily reducing electric demand. During the summer of 2016, 186 customers (representing a total of 1,179 premise account numbers) participated in three DR events. Two events lasted from 2-6 PM ( and ). The third event lasted from 2-4 PM ( ). 2.3 Milestones Achieved in Market Transformation Programs While all five of PSO s energy efficiency programs are designed primarily as energy efficiency resource acquisition programs, there are some market transformation characteristics, briefly summarized below. Energy Saving Products (ESP) Program: The main component of the ESP program in 2016 was retail markdowns of certain CFL and LED light bulbs. The goal of the markdowns is to increase sales to customers who would have otherwise purchased less efficient options in the absence of the price discount. These programs have long been considered to have market transformation effects in terms of retailer stocking decisions and manufacturer shipment decisions. Much research has been conducted on these effects in different regions across the country. In 2012, the PSO program was active in 144 store locations across seven retailers. A total of 320,061 CFL bulbs were marked down through the program in In 2013, the program was expanded considerably 2016 Energy Efficiency & Demand Response Programs 20

24 PSO Introduction by adding retail locations, additional bulbs eligible for markdown, and additional markdown funding. Over 1.4 million bulbs were marked down through the program at 177 retail locations during PY2013. In PY2014, 1.4 million bulbs were discounted at 180 participating stores. PY2015 stayed consistent with previous years in terms of total bulbs discounted at 1.4 million bulbs, though the total number of participating stores dropped to 167. PY million bulbs were discounted in 86 participating stores. There was considerable expansion in terms of the types of eligible bulbs marked down. In PY2013, the program included 48 different model numbers that were marked down at participating retailers which increased to 139 model numbers in PY2014. In PY2015, 209 different model numbers were discounted as part of the program. Additionally, the number of LEDs sold through the program increased to nearly 100,000 bulbs up from 21,000 in PY2014. For the first time, the program offered discounts on ENERGY STAR air purifiers in 2016.The program targeted sales of 175 discounted units and exceeded this goal with 228 units sold. High Performance Homes New Homes: PSO s New Homes component of the High Performance Homes program, a 2011 ENERGY STAR Partner of the Year Award winning program, continues to expand its influence in the market for new homes. PSO was recognized again by ENERGY STAR in 2012, 2013, 2015, and 2016 receiving Sustained Excellence Awards each year. Fifty-three builders and seven HERS raters participated in the New Homes component of the program during This represents an increase in the number of builders and HERS raters from previous years. A total of 677 new homes were rebated through the program in 2016, down slightly from PSO continues to expand the program's influence through educational opportunities for sales agents, homeowners, and builders. PY2016 also included targeted training for home appraisers aimed at explaining the benefits of energy efficiency and how they can be included in home appraisals. This year PSO changed the program incentive structure from performance incentives based on kwh savings to prescriptive incentives based on meeting program design specifications and for making efficiency improvements beyond the base requirements. Service Provider Recruitment and Training: PSO s High Performance Business and High Performance Homes programs include service provider training opportunities that focus on increasing awareness and knowledge of building science approaches to energy efficiency. This aspect of the programs has potential market transformation effects beyond the energy savings induced through the program. For a complete list of service provider training events refer to Appendix D. Service provider participation continues to grow for the High Performance Business program. 2.4 Annual Utility Growth Metrics and Portfolio Ratios The Oklahoma Title 165: reporting rules provide guidance for providing context on the utility load growth and the Demand Portfolio relative to load and revenue. Table 2016 Energy Efficiency & Demand Response Programs 21

25 PSO Introduction 2-1 shows weather normalized annual growth rates for PSO s total utility energy sales, distribution, and peak demand as well as three-year compound growth rates. Table 2-1: Utility Growth Year Energy at Peak Net Sales Sales Energy Demand Generator Demand (GWh) Growth Growth Growth (GWh) (MW) , % 19, % 4, % , % 19, % 4, % , % 19, % 4, % Compound Growth Rate - 0.8% - 0.2% - 0.3% Table 2-2 shows weather normalized annual growth rates and three year compound growth rates for utility energy sales by customer class. Table 2-2: Weather Normalized Retail Meter Sales Residential Commercial Industrial Other Retail Total Retail FERC Total System Year GWh %Chg GWh %Chg GWh %Chg GWh %Chg GWh %Chg GWh %Chg GWh %Chg , % 5, % 5, % 1, % 17, % % 17, % , % 5, % 5, % 1, % 17, % % 17, % , % 5, % 5, % 1, % 18, % 9 0.0% 18, % Compound Growth Rate -0.8% - 0.9% - 2.8% % - 0.8% % - 0.8% Table 2-3 provides a comparison of program costs to operating revenue. Table 2-3: Demand Portfolio Funding 2016 Demand Portfolio Program Cost ($M) $ Operating Revenues ($M) $1,272 Program Cost as % of Utility Operating Revenue 2.4% Table 2-4 provides a comparison of energy savings to total utility energy sales. Table 2-4: Demand Portfolio Energy Savings 2016 Energy Efficiency & Demand Response Programs 22

26 PSO Introduction 2016 Demand Portfolio Energy Savings (GWh) Metered Energy Sales (GWh) 18,158 Savings as % of Utility Sales 0.6% 2.5 High-Volume Electricity User Opt Out The Oklahoma Title 165: rules allow for High-Volume Electricity Users to opt out of some or all energy efficiency or demand response programs by submitting a notice of such decision to the director of the Public Utility Division and to the electric utility. A High-Volume Electricity User is defined as any single customer that consumes more than 15 million kwh of electricity per year, regardless of the number of meters or service locations. The number of customers eligible for High-Volume Electricity User opt out, their aggregate load as a percentage of total sales, the number of such customers that opted out of energy efficiency programs for PY2016, and the percentage of total energy sales that they comprise are listed in Table 2-5. Table 2-5: High Volume Electricity User Opt Out Energy Efficiency Metric Opt out eligible 2016 Chose to opt out -EE Number of accounts 6,537 4, Electric Sales (GWh) 6,737 6,437 Aggregate load as a percentage of total sales 34.7% 33.1% Table 2-6 provides the same information regarding customers who opted out of demand response programs. Table 2-6: High Volume Electricity User Opt Out Demand Response Metric Opt out eligible 2016 Chose to opt out -DR Number of accounts 6,537 4, Electric Sales (GWh) 6,737 6,257 Aggregate load as a percentage of total sales 34.7% 32.2% 2016 Energy Efficiency & Demand Response Programs 23

27 PSO Introduction 2.6 Fuel Switching Impacts PSO did not provide any inducement for installation of electric heating or electric water heating to replace natural gas fueled equipment during PY2016. A review of the program tracking data found no instances, in which natural gas equipment was replaced with electric equipment that was rebated through a PSO program. 2.7 Program Implementation & Strategic Alliances PSO had eight full-time employees dedicated to the implementation of energy efficiency and demand response programs in Additionally, PSO entered contracts with several energy services companies (ESCOs) and contractors to aid in program implementation. A complete list of implementation contractors, including contact name, title, business address, phone number, address, and program associations, is provided in Appendix C. ICF International (ICF) was contracted to implement the High Performance Business and High Performance Homes programs. CLEAResult was contracted to implement the Energy Saving Products program. The Home Weatherization program was largely implemented by Titan ES, LLC, with some program participation also coming through Rebuilding Together Tulsa, a volunteer organization working to preserve and revitalize low-income homes and communities. PSO also continued a partnership with Oklahoma Natural Gas on 145 homes to extend the funds available for Home Weatherization. PSO contracted with Resource Action Programs to provide energy efficiency kits distributed through the Education program. Finally, the Business Demand Response program was implemented in-house by PSO, with database support provided by AEG. Additional customer engagement materials and services for the entire portfolio of programs were provided by VI Marketing and Branding. Examples of marketing materials used during 2016 to promote PSO s energy efficiency and demand response programs are provided in Appendix E. For most programs in the 2016 portfolio, service providers were recruited to participate by submitting rebate applications on behalf of customers implementing qualifying energy efficiency measures. PSO s website contains lists of registered service providers and the associated products/services they provide. 2.8 Training and Customer Outreach PSO regularly conducts various service provider training and customer outreach events, which are summarized in Appendix D. During 2016, PSO s energy efficiency and demand response programs sponsored: 74 in-store residential lighting promotional events 102 other customer outreach and service provider training events, including: Portfolio overview presentations 2016 Energy Efficiency & Demand Response Programs 24

28 PSO Introduction Program specific service provider training One-on-one presentations with potential participants Trade show and event booths promoting the portfolio 2.9 Evaluation Measurement and Verification PSO contracted with ADM Associates, Inc. (ADM) to perform comprehensive program evaluation, measurement, and verification (EM&V) for PY2016. ADM s evaluation findings for each 2016 energy efficiency program are provided in Chapter 3 of this report. Similarly, evaluation findings for the demand response program are provided in Chapter Energy Efficiency & Demand Response Programs 25

29 3. Energy Efficiency Programs PSO s energy efficiency portfolio in 2016 consisted of five programs, two commercial/industrial and four residential (a small component of the Education program distributed kits to small businesses). As shown in Table 3-1, reported gross annual energy savings of 123,446 MWh. Program evaluation findings resulted in gross verified savings of 127,226 MWh, a 103% realization rate as compared to reported energy savings. The program evaluations also estimated program level net-to-gross ratios, resulting in verified net savings of 103,583 MWh. Table 3-1: Annual Energy Savings Energy Efficiency Programs Program Energy Efficiency Programs Gross Peak Annual Energy Savings (MWh) Projected Reported Verified Gross Realization Rate Net Impacts NTG Ratio Net Annual Energy Savings (MWh) High Performance Business 51,795 51,932 53, % ,009 Home Weatherization 3,794 5,726 5,537 97% ,537 Energy Saving Products 32,043 51,123 51, % ,830 High Performance Homes 7,541 8,406 8, % ,711 Education 4,949 6,260 7, % ,496 Energy Efficiency Totals 100, , , % ,583 Reported peak demand reduction of MW also exceeded projections of MW. The EM&V study resulted in gross verified peak demand reduction of MW. Applying the evaluation estimated program level net-to-gross ratios results in total verified net peak demand reduction of MW, as shown below in Table 3-2. Energy Efficiency Programs 26

30 PSO High Performance Business Program Program Energy Efficiency Programs Table 3-2: Peak Demand Reduction Energy Efficiency Programs Gross Peak Demand Reduction (MW) Projected Reported Verified Gross Realization Rate NTG Ratio Net Impacts Net Peak Demand Reduction High Performance Business % Home Weatherization % Energy Saving Products % High Performance Homes % Education % Energy Efficiency Totals % In the sections that follow, evaluation findings for each of the PY2016 PSO energy efficiency programs are presented. Program performance metrics, evaluation methodologies, energy and demand impacts, and process related findings for each program are detailed. 3.1 High Performance Business Program HPB Program Overview PSO s High Performance Business (HPB) program provided rebates for a total of 1,188 projects completed by 950 unique participants in The program seeks to generate energy and demand savings for small and large commercial and industrial customers, schools, and municipalities through the promotion of high efficiency electric end use products including (but not limited to) lighting, HVAC, and Variable Frequency Drives (VFDs) for motors. For 2016, the program was expanded to include a Small Business Energy Solutions (SBES) rebate program. To participate in the SBES program, businesses muse use 220,000 kwh or less annually and used a PSO approved service provider. On the other hand, PSO s C&I customers have flexibility in choosing how to participate, by self-sponsoring or working through a program service provider to leverage technical expertise. Reported expenditures for HPB fell short of projections by approximately $700,000, representing 93% of the budget. Reported energy savings represented just over 100% of program level goals. ADM s verified savings estimates for HPB resulted in a 103% realization rate for gross energy savings and a 107% realization rate for gross peak demand reduction. Table 3-3 provides projected, reported, and verified demand impacts as well as other program performance metrics. (MW) Energy Efficiency Programs 27

31 PSO High Performance Business Program Table 3-3: Performance Metrics High Performance Business Program Metric PY2016 Number of Customers 950 Budgeted Expenditures $10,711,689 Actual Expenditures $10,009,915 Energy Impacts (kwh) Projected Energy Savings 51,795,048 Reported Energy Savings 51,932,323 Gross Verified Energy Savings 53,460,025 Net Verified Energy Savings 51,009,197 Peak Demand Impacts (kw) Projected Peak Demand Savings 8,669 Reported Peak Demand Savings 10,350 Gross Verified Peak Demand Savings 11,021 Net Verified Peak Demand Savings 10,471 Benefit / Cost Ratios Total Resource Cost Test Ratio 2.22 Utility Cost Test Ratio 4.11 HPB Prescriptive and Custom Overview PSO s HPB prescriptive and custom projects provided rebates for a total of 1,002 projects completed by 767 unique participants in Lighting system retrofit applications continued to be the main source of program impacts, representing 63% of reported kwh savings. With the addition of new construction lighting and custom lighting applications, lighting end use measures make up 78% of prescriptive and custom level reported kwh savings. Custom non-lighting applications account for 21% of reported savings and the remaining reported energy savings for PY2016 represent prescriptive HVAC measures. Overall, the number of rebated projects in HPB increased from 711 (PY2015) to 1002 (PY2016). Reported energy savings decreased from 59,452 MWh to 47,878 MWh. Energy Efficiency Programs 28

32 PSO High Performance Business Program Verified gross energy savings likewise decreased from 53,449 MWh to 49,037 MWh. This resulted in a gross energy savings realization rate of 102% and a gross peak demand reduction realization rate of 107%. The estimated net-to-gross ratio also increased, from 94% in PY2015 to 95% in PY2016. Table 3-4 provides projected, reported, and verified energy and demand impacts as well as other program performance metrics for the High Performance Business program. Table 3-4: Performance Metrics High Performance Business Program Prescriptive and Custom Metric PY2016 Number of Customers 767 Energy Impacts (kwh) Reported Energy Savings 47,878,154 Gross Verified Energy Savings 49,036,679 Net Verified Energy Savings 46,609,598 Peak Demand Impacts (kw) Reported Peak Demand Savings 9,259 Gross Verified Peak Demand Savings 9,947 Net Verified Peak Demand Savings 9,403 Benefit / Cost Ratios Total Resource Cost Test Ratio 2.25 Utility Cost Test Ratio 4.02 HPB SBES Overview PSO s HPB SBES projects provided rebates for a total of 186 projects completed by 183 unique participants in Projects completed by lighting-only contractors are the main source of program impacts, representing 98% of reported kwh savings. Refrigeration contractor applications account for 2% of reported savings. In the inaugural year for this program reported energy savings is 4,054 MWh, and verified gross energy savings is 4,423 MWh. This resulted in a gross energy savings realization rate of 109%. The program reported peak demand reductions is 1,091 kw Energy Efficiency Programs 29

33 PSO High Performance Business Program and verified peak demand reductions is 1,074 kw. This resulted in a gross peak demand reduction realization rate of 101%. The estimated net-to-gross ratio is 99.5%. Table 3-5 provides projected, reported, and verified energy and demand impacts as well as other program performance metrics for the SBES portion of the HPB program. Table 3-5: Performance Metrics High Performance Business Program Small Business Energy Solutions Metric PY2016 Number of Customers 183 Energy Impacts (kwh) Reported Energy Savings 4,054,170 Gross Verified Energy Savings 4,423,345 Net Verified Energy Savings 4,399,600 Peak Demand Impacts (kw) Reported Peak Demand Savings 1,091 Gross Verified Peak Demand Savings 1,074 Net Verified Peak Demand Savings 1,068 Benefit / Cost Ratios Total Resource Cost Test Ratio 1.97 Utility Cost Test Ratio High Performance Business EM&V Methodologies This section provides a brief overview of the data collection activities, gross and net impact calculation methodologies, and process evaluation activities that ADM employed in the evaluation of the High Performance Business program, including Prescriptive and Custom projects, as well as Small Business Energy Solutions projects. Data Collection Data for the analysis were collected through review of program materials, on-site inspections, end-use metering, and interviews with participating customers and service providers. Based on program tracking data provided by PSO through the VisionDSM database and SQL Server Reporting Services (SSRS), a sample design was developed Energy Efficiency Programs 30

34 PSO High Performance Business Program for on-site data collection. The VisionDSM system is a central database where program activities are tracked and project documentation is stored. The SSRS reporting system pulls data from VisionDSM and presents the information in useful summary reports that can be designed to meet specific program tracking needs. The on-site verification and data collection samples were drawn from EM&V specific SSRS reports to provide gross impact estimates with 10% precision or better at the 90% confidence level for the program. Separate samples were drawn for Prescriptive and Custom projects and SBES projects. On-site visits were used to collect data for gross impact calculations, to verify measure installation, and to determine measure operating parameters. Facility staff members were interviewed to determine the operating hours of the installed systems and provide any additional operational characteristics relevant to calculating energy savings. For a subset of sampled projects, lighting equipment, HVAC equipment, or motors/variable Frequency Drives (VFDs) were monitored to obtain accurate operational profiles. Energy Efficiency Programs 31

35 PSO High Performance Business Program Table 3-6 below shows the sample design that was used for prescriptive and custom projects. Stratum classifications were based on verified measure installations. The 74 projects that were sampled for on-site measurement and verification account for approximately 26% of reported program kwh savings. Energy Efficiency Programs 32

36 PSO High Performance Business Program Table 3-6: Sample Design for the High Performance Business Program Prescriptive and Custom Stratum Name Reported kwh Savings Strata Boundaries (kwh) Population of Projects Design Sample Size Custom 1 1,946,499 <127, Custom 2 3,377, , , Custom 3 4,659,231 >417, HVAC 1 64,894 < 3, HVAC 2 126,238 HVAC 3 137,143 3,104-8,128 8,129-19, HVAC 4 370,418 > 19, NC Lighting 1 1,407,169 < 60, NC Lighting 2 1,260,197 NC Lighting 3 2,304,698 60, , , , NC Lighting 4 1,861,791 > 412, Retrofit Lighting 1 6,975,069 < 41, Retrofit Lighting 2 7,678,908 Retrofit Lighting 3 6,582,511 41, , , , Retrofit Lighting 4 9,125,989 >286, Total 47,878,154 N/A 1, Energy Efficiency Programs 33

37 PSO High Performance Business Program Table 3-7 below shows the sample design that was used for small business projects. Stratum classifications were based on verified measure installations. The 22 projects that were sampled for on-site measurement and verification account for approximately 21% of reported program kwh savings. Note that only 16 of the 22 sample sites were visited. The remaining six sites were analyzed through a detailed desk review of project documentation. Table 3-7: Sample Design for the High Performance Business Program Small Business Energy Solutions Stratum Name Reported kwh Savings Strata Boundaries (kwh) Population of Projects Design Sample Size Lighting 1 1,070,090 9, Lighting 2 740,371 Lighting 3 1,424,862 Lighting 4 303,221 Lighting 5 420,215 9,407-21,233 21,234-41,875 41,876-94,601 94, , Non-Lighting 95,410 32, Total 4,054, As an additional on-site data collection activity for small business, ADM conducted ride along visits with three of the participating contractors. ADM gained valuable knowledge of the process flow of the program through these visits. ADM attended five audit visits and observed how proposals are created. ADM staff also conducted one joint post inspection with the implantation contractor. In addition to the small business on-site data collection effort, customer surveys provided self-report data for the net-to-gross analysis and process evaluation. A total of 140 customer decision makers who completed custom or prescriptive incentive projects completed the survey. Another 44 customers that completed SBES projects also completed the survey. In-depth interviews with four PSO and implementation staff members were conducted to provide additional perspectives for the process evaluation. Finally, 15 service providers that assisted customers with custom or prescriptive incentive projects were interviewed Energy Efficiency Programs 34

38 PSO High Performance Business Program and provided additional insight regarding program processes in All six of the service providers that work with the SBES program were also interviewed. Table 3-8 shows the achieved sample sizes for the different types of data collection employed for this study. Table 3-8: Sample Sizes for Data Collection Efforts High Performance Business Data Collection Activity Achieved Sample Size Custom/Prescriptive SBES On-Site M&V visits Customer Decision Maker Survey Service Provider Survey 15 6 In-depth Interviews with Program Staff 2 2 Gross Impact Estimation The evaluation of gross energy savings and peak demand reduction from projects rebated through the High Performance Business program can be broken down into the following steps: First, the program tracking database was reviewed to determine the scope of the program and to ensure there were no duplicate project entries. The tracking database was used to define a discrete set of rebated projects that made up the PY2016 program population. A sample of projects was then drawn from the population established in the tracking system review. Next, a detailed desk review was conducted for each project sampled for on-site verification and data collection. The desk review process includes a thorough examination of all project materials including invoices, equipment cut sheets, preand post-inspection reports, and estimated savings calculators. This review process informed ADM s fieldwork by identifying potential uncertainties, missing data, and sites where monitoring equipment was needed to verify key inputs to the reported savings calculations. Additionally, the review process involved assessing the reasonableness of deemed savings values and calculation input assumptions. After reviewing the project materials, on-site verification and data collection visits were scheduled for each sampled project. The visits were used to collect data for savings calculations, to verify measure installation, and to determine measure operating parameters. Energy Efficiency Programs 35

39 PSO High Performance Business Program Next, the data collected during the on-site verification visits was used to revise savings calculations as necessary. For example, if the reported savings calculations relied on certain measure operating hours that were determined inaccurate based on the facilities actual schedule, changes were made to more accurately reflect actual operating conditions. Finally, after determining the verified savings impacts for each sampled project, results were extrapolated to the program population using project-specific sampling weights. This allows for the estimation of program level gross verified energy (kwh) savings with a given amount of sampling precision and confidence. For the High Performance Business program, the sample was designed to ensure ±10% or better relative precision at the 90% confidence level for kwh reductions. Net-to-Gross Estimation The purpose of net savings analysis is to determine what portion of gross savings achieved by PSO customers is the direct result of program influence. The savings directly attributable to the program are the net savings attributable to the program. Net savings may be less than gross savings because of free ridership impacts, which arise to the extent that participants in a program would have adopted energy efficiency measures and achieved the observed energy usage changes in the absence of the program. Free riders for a program are defined as those participants that would have installed the same energy efficiency measures without the program. Conversely, net savings may be greater than gross savings due to energy saving spillovers attributable to the program. Participants or non-participants may implement energy efficiency measures due to the influence of the program, without receiving program inducements for implemented measures. Information collected from a sample of program participants through a customer decision maker survey was used for the net-to-gross analysis. These responses were reviewed to assess the likelihood that participants were free riders. Several criteria were used for determining what portion of a customer s savings for a project should be attributed to free ridership. The criteria are organized into the following three factors: Plans and intentions of the firm to install a measure even without support from the program; Influence that the program had on the decision to install a measure; and A firm s previous experience with a measure installed under the program. For each of these factors, rules were applied to develop binary variables indicating whether a participant showed free ridership behavior. The first required step was to Energy Efficiency Programs 36

40 PSO High Performance Business Program determine if a participant stated that his or her intention was to install an energy efficiency measure without the help of the program inducements. Two binary variables were constructed to account for customer plans and intentions: one, based on a more restrictive set of criteria that may describe a high likelihood of free ridership, and a second, based on a less restrictive set of criteria that may describe a relatively lower likelihood of free ridership. The first, more restrictive criteria indicating customer plans and intentions that likely signify free ridership are as follows: The respondent answered, yes to the following two questions: Did you have plans to install the [Equipment/Measure] before participating in the program? and Would you have gone ahead with this planned installation of the measure even if you had not participated in the program? The respondent answered, definitely would have installed to the following question: If the rebates from the program had not been available, how likely is it that you would have installed [Equipment/Measure] anyway? The respondent answered, did not affect the timing of purchase and installation to the following question: We would like to know whether the availability of information and rebates through the program affected the timing of your purchase and installation of [Equipment/Measure]. Did you purchase and install [Equipment/Measure] earlier than you otherwise would have without the program? The respondent answered, no, the program did not affect the level of efficiency that we chose for equipment in response to the following question: Did you choose equipment that was more energy efficient than you would have chosen had you not participated in the program? The respondent answered, no, the program did not affect timing of project in response to the following question: Did you install the [Equipment/Measure] earlier than you otherwise would have because of the information and rebates from PSO s program? The second, less restrictive criteria indicating customer plans and intentions that likely signify free ridership are as follows: The respondent answered yes to the following two questions: Did you have plans to install the [Equipment/Measure] before participating in the program? and Would you have gone ahead with this planned installation of the measure even if you had not participated in the program? Either the respondent answered, definitely would have installed or probably would have installed to the following question: If the rebates from the program Energy Efficiency Programs 37

41 PSO High Performance Business Program had not been available, how likely is it that you would have installed [Equipment/Measure] anyway? Either the respondent answered, did not affect timing of purchase and installation to the following question: Did you purchase and install [Equipment/Measure] earlier than you otherwise would have without the program? or the respondent indicated that while program information and rebates did affect the timing of equipment purchase and installation, in the absence of the program they would have purchased and installed the equipment within the next two years. The respondent indicated that no, the program did not affect level of efficiency that we chose for equipment. The second factor is determining if a customer reported that a recommendation from a program representative or experience with the program was influential in the decision to install a piece of equipment or measure. This criterion indicates that the program s influence may lower the likelihood of free ridership when either of the following conditions is true: The respondent answered, very important to the following question: How important was previous experience with PSO energy efficiency programs in making your decision to install the [Equipment/Measure] at your facility? The respondent either answered probably would not have or definitely would not have installed the equipment if it was recommended by a program staff member or the energy consultant they worked with (for SBES participants). The third factor is determining if a participant in the program indicated that he or she had previously installed an energy efficiency measure like one that they installed under the program without an energy efficiency program incentive during the last three years. A participant indicating that he or she had installed a similar measure is considered to have a higher likelihood of free ridership, due to the potential influence of the prior experience. The criteria indicating that previous experience may signify a higher likelihood of free ridership are as follows: The respondent answered yes to the following question: Before participating in the program, had you installed any equipment or measure similar to [Equipment/Measure] at your facility? The respondent answered yes, purchased energy efficient equipment but did not apply for financial incentive. to the following question: Has your organization purchased any energy efficient equipment in the last three years for which you did not apply for a rebate through the program? The four sets of rules just described were used to construct four different indicator variables that address free ridership behavior. For each customer, a free ridership Energy Efficiency Programs 38

42 PSO High Performance Business Program value was assigned based on the combination of variables. With the four indicator variables, there were 12 applicable combinations for assigning free ridership scores for each respondent, depending on the combination of answers to the questions creating the indicator variables. Table 3-9 shows these values. Table 3-9: Free Ridership Scores for Combinations of Indicator Variable Responses Had Plans and Intentions to Install Measure without C&I Program? (Definition 1) Had Plans and Intentions to Install Measure without C&I Program? (Definition 2) C&I Program had influence on Decision to Install Measure? Had Previous Experience with Measure? Free Ridership Score Y Y Y Y 100% Y Y N N 100% Y Y N Y 100% Y Y Y N 67% N Y N Y 67% N Y N N 33% N Y Y N 0% N Y Y Y 33% N N N Y 33% N N N N 0% N N Y N 0% N N Y Y 0% The customer decision maker survey also included a series of questions used to analyze whether there were potential spillover effects associated with non-rebated purchases by program participants. 15 Specifically, survey respondents were asked: We would like to know if you have installed any additional energy efficient equipment because of your experience with the program that you DID NOT 15 The spillover analysis is limited to participant spillover. Non-participant spillover effects may exist for the program, but they are not estimated and therefore assumed to be zero. Energy Efficiency Programs 39

43 PSO High Performance Business Program receive an incentive for. Since participating in the program has your organization installed any ADDITIONAL energy efficiency measures at this facility or at your other facilities within PSO's service territory that did NOT receive incentives through PSO's program? Customers who indicated yes were identified as potential spillover candidates. Potential spillover candidates were additionally asked to identify the type of additional equipment installed and provide information about the equipment for use in estimating energy savings. For each type of equipment that respondents reported installing, respondents were asked the following two questions used to assess if any savings resulting from the additional equipment installed were attributable to the program. [SP1] How important was your experience with the program in your decision to install this [Equipment/Measure]? [Rated on a scale where 0 meant not at all important and 10 meant very important] [SP2] If you had NOT participated in the program, how likely is it that your organization would still have installed this [Equipment/Measure]? [Rated on a scale where 0 meant not at all likely and 10 meant very likely] A spillover score was developed based on these responses as follows: Spillover Score = Average(SP1, 10-SP2) The energy savings of equipment installations associated with a spillover score of greater than five were attributed to the program. Survey timing is an important consideration when estimating participant spillover effects. The customer decision maker surveys were conducted during January and February of 2016, which may not have given participating customers the necessary time to implement additional energy efficiency measures that may be influenced by their past participation in the HPB program. As a result, the participant spillover estimates developed in this evaluation may be an underrepresentation, depending on the extent to which future energy efficiency choices are made because of past HPB program influences. Process Evaluation Activities The process evaluation is designed to address the following general questions: How well did PSO staff, implementation staff, service providers and participating customers work together? Were the program participants satisfied with their experience? What do service providers like about the program? What would they like to change about the program? Energy Efficiency Programs 40

44 PSO High Performance Business Program What share of projects is associated with specific service providers? How are savings distributed across service providers? Are there any differences in opinion between active and less active service providers? What types of buildings/facilities participated in the program? Could certain facility types be targeted more effectively? What customer barriers to participation do service providers see? How can these be mitigated? Which marketing methods were most effective? Were there any significant changes or new obstacles during the 2016 program year? What barriers to completion of incentive projects do customers that receive program sponsored technical assistance or facility audits perceive? To address these questions, ADM s process evaluation activities included a review of program materials, customer decision maker surveys, service provider, and in-depth program manager interviews with staff at PSO and implementation contractor Impact Evaluation Findings To estimate kwh savings and peak kw reductions for the program, data were collected and analyzed for samples of rebated projects. The data were analyzed using the methods described in Section 0 to estimate project energy savings and peak kw reductions and to determine realization rates for both program components. The results of that analysis are reported in this section. Prescriptive and Custom Verified Gross Energy Savings (kwh) The verified gross kwh savings for the PY2016 High Performance Business Program are summarized by sampling stratum in Table Overall, the verified gross savings of 49,036,679 kwh are equal to 102% of the reported savings for the prescriptive and custom measures in the High Performance Business program. Energy Efficiency Programs 41

45 PSO High Performance Business Program Table 3-10: Reported and Verified Gross kwh Savings by Sampling Stratum Prescriptive and Custom Stratum Reported kwh Savings Verified Gross kwh Savings Gross kwh Realization Rate Custom 1 1,946,499 2,001, % Custom 2 3,377,398 3,315, % Custom 3 4,659,231 4,154,006 88% HVAC 1 64, , % HVAC 2 126, , % HVAC 3 137, , % HVAC 4 370, , % NC Lighting 1 1,407,169 1,955, % NC Lighting 2 1,260, ,910 75% NC Lighting 3 2,304,698 2,216,399 98% NC Lighting 4 1,861,791 1,517,560 82% Retrofit Lighting 1 6,975,069 7,538, % Retrofit Lighting 2 7,678,908 6,494,347 85% Retrofit Lighting 3 6,582,511 9,119, % Retrofit Lighting 4 9,125,989 8,419,822 92% Total 47,878,154 49,036, % The achieved sample design resulted in verified gross kwh estimates with ±9.3% relative precision at the 90% confidence interval. 16 Verified gross energy savings were relatively close to the original reported values at the program level (102% realization rate). There was, however, a wide range of kwh realization rates at the sample project level. 16 That is, we are 90% confident that the true verified gross savings are between 44,486,243 and 53,587,115 kwh based on the uncertainty introduced by sampling. Energy Efficiency Programs 42

46 PSO High Performance Business Program Lighting Projects Lighting projects were included in two strata categories; retrofit (Retrofit Lighting 1-4), and new construction (NC Light1-4) lighting. The four retrofit lighting strata had a combined realization rate of 104%, with strata realization rates varying from 85% TO 136%. The new construction stratum showed more variability, resulting in strata realization rates varying from 75% to 137%. The four new construction lighting strata combined for a realization rate of 97%. The smallest new construction projects, those with reported savings below 100,000 kwh, resulted in a realization rate of 137%. In contrast, the largest new construction lighting strata resulted in a realization rate of 82%. Much of the discrepancy between reported and verified energy savings is explained by measure operation schedule assumptions that differed from on-site verification findings. When estimating energy savings for rebate calculation purposes, it is often convenient and advisable to use participant estimated hours of operation for measures such as lighting. Requiring more detailed schedules or monitoring data can often impede program performance by increasing barriers to participation. While there is some level of inherent uncertainty in the use of these estimated annual operating hours, the uncertainty is usually outweighed by the additional program participation encouraged by an easy application process. Of the retrofit lighting sites with energy savings realization rates greater than 125% and lower than 75%, the differences on all but one can be attributed to hours of use. ADM installed monitoring equipment to capture the hours of operation for nine of the sampled lighting projects. For lighting projects where monitoring was not performed, ADM used lighting schedules from detailed interviews with facility staff. Lighting settings from Energy Management Systems (EMS), timers, and photocells were also used where appropriate based on the field work findings. The 103% kwh realization rate across all lighting projects points to the fact that for many of the sample projects the reported hours of use were accurate, while slightly underestimated on average. Of the 37 sample projects that were lighting improvements, the average realization rate came in at 112%, again pointing to the overall accuracy of reported annual operating hours for lighting systems. In some cases, the reported energy savings (and peak demand reduction) for lighting projects did not identify an adjustment for heating and cooling interactive factors. The installation of efficient lighting means less waste heat, requiring less cooling in the summer, and more heating in the winter. In instances where retrofit lighting was installed in a location that was space conditioned, ADM applied Heating and Cooling Interactive Factors (HCIF) from the Arkansas TRM (1.05 for energy savings and 1.10 for peak demand reduction). ADM recommends that HCIF adjustments are made for all future lighting projects in conditioned space. Energy Efficiency Programs 43

47 PSO High Performance Business Program Aside from annual operating hours and HCIF, there was one retrofit lighting instance where baseline wattage was not reported and could not be determined when on-site. This resulted in a low realization rate for this project. Finally, there were three new construction lighting projects in which discrepancies were found in baseline lighting power density (LPD), building square footage, fixture counts, and hours of use. This resulted in a realization rate for new construction lighting projects of 91%. Non-Lighting Projects All non-lighting strata had an overall realization rate of 93%. Non-lighting projects were divided into two strata classifications, Custom (Custom 1-3), and HVAC (HVAC 1-4). These two strata classifications showed considerable variation. The custom strata realization rates varied from 88% to 103%. However, over 50% of the custom projects in the sample represented realization rates greater than 125% or less than 75%. Individual custom project realization rates varied from 3% to 220%. The HVAC strata also showed considerable variation between the four strata classifications. The HVAC strata realization rates varied from 40% to 740%. Of the 13 sampled HVAC projects, 70% presented realization rates greater than 125% or less than 75%. Analysis on the HVAC projects in the sample resulted in a realization rate of 186%. This realization rate for the 13 projects was driven by higher gross ex post energy savings determined for rooftop HVAC units based energy savings algorithms presented the Arkansas TRM v5.0. The annual equivalent full load hours applied to the ex-ante savings were underestimated. ICF has taken appropriate steps to correct this for The variance in realization rates for custom projects vary by measure and savings algorithm implemented. Measures reviewed include gaskets, strip curtains, air compressors, and VFDs. Projects that included gaskets and strip curtains resulted in average realization rates greater than 200%. This was due to conservative reported energy savings and ex post energy savings being reported based on an ACEEE (American Council for an Energy- Efficient Economy) study 17. This measure was discontinued part way through 2016 due to uncertainty in energy savings algorithms and the collection of information regarding the baseline condition. Other noticeable difference stemmed from the use of billing regression analyses and monitoring for the ex post gross savings. Varying realization rates (both high and low) were presented for air compressor projects in which ADM utilized power data acquired through monitoring of each air compressor ACEEE Summer Study on Energy Efficiency in Building (ACEEE Study) Energy Efficiency Programs 44

48 PSO High Performance Business Program A low realization rate was also determined for a VFD project, in which case gross ex post energy savings were determined through energy simulation. Overall, the kwh realization rate for all non-lighting projects is lower than lighting sites. In both cases, high realization rates for individual strata drove the program level realization rate over 100%. The strata with high sample level realization rates include: HVAC 1, HVAC 2, HVAC 3, HVAC 4, New Construction Lighting 1, Retrofit Lighting 1, and Retrofit Lighting 3. These are presented in table format in Table SBES Verified Gross Energy Savings (kwh) The verified gross kwh savings for the Small Business Energy Solutions portion of the PY2016 High Performance Business Program are summarized by sampling stratum in Table Overall, the verified gross savings of 4,423,345 kwh are equal to 109% of the reported savings for the High Performance Business program. Table 3-11: Reported and Verified Gross kwh Savings by Sampling Stratum - SBES Stratum Reported kwh Savings Verified Gross kwh Savings Gross kwh Realization Rate L 4 1,070,090 1,247, % L 2 740, , % L 3 1,424,862 1,424, % L 1 303, , % L 5 420, , % NL 1 95,410 76,058 80% Total 4,054,170 4,423, % The achieved sample design resulted in verified gross kwh estimates with ±8.6% relative precision at the 90% confidence interval. 18 Verified gross energy savings were relatively close to the original reported values at the program level (109% realization rate). There was, however, a wide range of kwh realization rates at the sample project level. Lighting Contractor Projects 18 That is, we are 90% confident that the true verified gross savings are between 3,707,108 and 4,401,232 kwh based on the uncertainty introduced by sampling. Energy Efficiency Programs 45

49 PSO High Performance Business Program For the lighting projects in the program, the realization rate for annual kwh savings is 110%. The wattage table used in the ex ante calculations is not an exact match for the Arkansas TRM, but the discrepancies are negligible. For all sampled sites, ADM verified measure counts to be accurate. There were minor discrepancies between claimed hours of use and verified hours, but these discrepancies nearly cancelled each other out. The only systematic discrepancy identified was that the ex ante calculations did not apply interactive effects for energy (1.09 in most cases) when applicable. The main reason for the greater-than-100% realization rate is the interactive effects, though the other discrepancies mentioned do have a small impact. Refrigeration Contractor Projects Realization rates by measure for sampled sites are presented in Table 3-12 below. Table 3-12: Reported and Verified Gross kw Reduction by Measure for Refrigeration Contractor Projects Stratum Reported Peak kw Reduction Verified Gross Peak kw Reduction Gross kw Realization Rate EC Motors 12,696 12,087 95% Evaporator Fan Controls Refrigerated Case Lighting Door Heater Controls Beverage Cooler Controls Total Sampled Site Savings 18,124 11,556 64% 14,506 14, % 11,770 8,560 73% 5,502 3,051 55% 62,598 49,905 80% As shown in the table above, the measures most responsible for the stratum level realization rate of 80% are evaporator fan controls, door heater controls, and beverage cooler controls. For all three of these measures, the source of inputs to savings calculations are not clear from the project documentation. In all cases, ADM used verified site-specific conditions or Arkansas TRM default assumptions. Without more specific documentation, ADM cannot determine the exact inputs that cause the discrepancies. Energy Efficiency Programs 46

50 PSO High Performance Business Program Prescriptive and Custom Verified Peak Demand Reduction (kw) The verified gross kw reduction for the PY2016 High Performance Business program is summarized by sampling stratum in Table Overall, the verified gross reduction of 9,947 kw is equal to 107% of the reported reduction for the program. Energy Efficiency Programs 47

51 PSO High Performance Business Program Table 3-13: Reported and Verified Gross kw Reduction by Sampling Stratum Stratum Reported Peak kw Reduction Verified Gross Peak kw Reduction Gross kw Realization Rate Custom % Custom , % Custom % HVAC % HVAC % HVAC % HVAC % NC Lighting % NC Lighting % NC Lighting % NC Lighting % Retrofit Lighting 1 Retrofit Lighting 2 Retrofit Lighting 3 Retrofit Lighting 4 1, % 1,415 1,344 95% 1,425 2, % 1,987 2, % Total 9,259 9, % The achieved sample design resulted in verified gross kw estimates with ±17.78% relative precision at the 90% confidence interval. 19 The high level of uncertainty 19 That is, we are 90% confident that the true verified gross peak demand reduction is between 8,178 and 11,715 kw based on the uncertainty introduced by sampling. Energy Efficiency Programs 48

52 PSO High Performance Business Program associated with peak kw reductions is due to the significant amount of variance from project to project. The relative precision of the sample in PY2016 is improved from PY2015, and the realization rate is above 100%. Much of the difference between reported and verified demand reduction, as in past program years, is explained by either 1) calculation error in reported demand reduction, 2) use of stipulated coincidence factors (CF) that did not align well with actual equipment schedules or 3) varying peak demand reduction from the defined peak period of 1 5 PM, weekday non-holidays, June through September as described in the PY2016 EM&V Plan. Lighting Projects Retrofit lighting projects resulted in a peak demand realization rate of 103%. There was significant variation in the peak demand realization rates among the four lighting retrofit strata with project level realization rates ranging from 0% (four projects) to 233% for one project. Of the 23 projects sampled in these four strata, only 7 had peak demand realization rates between 75% and 125%. The most common reason for a reduction in verified peak demand reduction was the usage of coincidence factors (CFs) not aligned with the facilities operational schedule and the defined peak period. ADM evaluation used a peak period from 1 5 PM, weekday non-holidays, June through September. This variation is particularly evident in outdoor lighting applications where photocells are utilized for dusk-to-dawn lighting. Several instances where reported peak demand reduction was based off building type resulted in a high realization rates, resulting in the retrofit lighting peak demand reduction realization rate close to 100%. The four new construction lighting strata resulted in an overall realization rate of 113%. Of the 14 sampled projects within the new construction lighting strata, only 5 had peak demand reduction realization rates between 75% and 125%. The reasons for variation realization for new construction lighting projects are like those for retrofit lighting projects. However, one low realization rate was due to a misclassification of the building type in the reported ex ante analysis. The largest new construction lighting project in the sample benefited from a high realization rate due to an analysis conducted using the space-by-space type algorithms and deemed values as opposed to whole building algorithms and deemed values. The overall realization rate for all lighting projects in the sample was 117%. Extrapolated to the population, this resulted in a total realization rate of 104%. Realization rates by strata can be found in Table The variation in realization rates on the project level basis was improved from PY2015. In general, the most common reason for the variation in peak demand reduction realization rates is due to the application of coincidence factors. Non-Lighting Projects Verified peak demand reduction for sampled non-lighting projects also differed from reported values. The three custom strata had an overall realization rate of 128%, while Energy Efficiency Programs 49

53 PSO High Performance Business Program the prescriptive HVAC strata resulted in a peak demand realization rate of 91%. The total realization rate for all non-lighting strata was 86%. Selected examples are provided below for prescriptive and custom projects. The custom non-lighting sites had a high amount of variability in the peak demand realization rates as individual projects ranged from 0% to 420%. One project that resulted in zero peak demand savings was due to incorrect calculations, and baseline conditions in a large heat pump application. Other projects with zero peak demand savings were building controls and a custom project that included exterior lighting. Only 4 of the 24 sampled custom sites had peak demand realization rates between 75% and 125%. The variation in realization rates was typically caused by different methods of calculation. However, one measure where prescriptive deemed values were applied, electric combination ovens, could not be explained. Other discrepancies were the product of using monitored data, energy simulations, and billing regression analyses. Additionally, several custom projects calculate peak demand savings based on peak loads or the highest kw reduction possible between the old and new equipment, regardless of hours of use or normal operating characteristics. Peak demand savings should be based on how the equipment operates during PSO s peak demand period. Verified peak demand reduction for the HVAC strata resulted in realization rates varying from 22% to 261%. Only 5 of the 13 projects had realization rates between 75% and 125%. Variability for HVAC projects was less than custom projects as most sample projects consisted of similar rooftop units. The overall realization rate for HVAC projects was 91%. Due to the variability in reported peak demand reduction values, ADM recommends reviewing savings calculation tools and technical review protocols to ensure consistent peak demand calculations across measure types per the defined peak period. SBES Verified Peak Demand Reduction (kw) The verified gross kwh savings for the Small Business Energy Solutions portion of the PY2016 High Performance Business Program are summarized by sampling stratum in Table Overall, the verified gross demand reductions of 1,074 kw are equal to 98% of the reported savings for the High Performance Business program. Energy Efficiency Programs 50

54 PSO High Performance Business Program Table 3-14: Reported and Verified Gross kw Reductions by Sampling Stratum - SBES Stratum Reported kw Reductions Verified Gross kw Reductions Gross kw Realization Rate L % L % L % L % L % NL % Total 1, , % The achieved sample design resulted in verified gross kwh estimates with ±19.4% relative precision at the 90% confidence interval. 20 Verified gross energy savings were relatively close to the original reported values at the program level (98% realization rate). There was, however, a wide range of kw realization rates at the sample project level. Lighting Contractor Projects For the lighting projects in the program, the realization rate for peak kw reductions is 98%. The wattage table used in the ex ante calculations is not an exact match for the Arkansas TRM, but the discrepancies are negligible. For all sampled sites, ADM verified measure counts to be accurate. There were likely minor discrepancies between claimed coincidence factors (CF) and verified CF, but the claimed CF is not shown in the ex ante calculations. It appears that, unlike the annual kwh savings calculation, the interactive effects were considered for demand, but, again, that figure is not shown in the project documentation. At the program level, these discrepancies have a minor impact resulting in the realization rate of 98%. Refrigeration Contractor Projects Realization rates by measure for sampled sites are presented in Table 3-15 below. 20 That is, we are 90% confident that the true verified gross demand reductions are between and 1, kw based on the uncertainty introduced by sampling. Energy Efficiency Programs 51

55 PSO High Performance Business Program Table 3-15: Reported and Verified Gross kw Reduction by Measure for Refrigeration Contractor Projects Stratum Reported Peak kw Reduction Verified Gross Peak kw Reduction Gross kw Realization Rate EC Motors % Evaporator Fan Controls Refrigerated Case Lighting Door Heater Controls Beverage Cooler Controls Total Sampled Site Savings % % % % As shown in the table above, the measures that differ most from the claimed savings are evaporator fan controls and door heater controls. For both measures, the source of inputs to savings calculations are not clear from the project documentation. In all cases, ADM used verified site-specific conditions or Arkansas TRM default assumptions. Without more specific documentation, ADM cannot determine the exact inputs that cause the discrepancies Net-to-Gross Estimation Results The data used to assign free ridership scores were collected through a survey of customer decision makers for projects rebated through the HPB program during PY2016. Free ridership was estimated using the methodology described in Section HPB Custom and Prescriptive Table 3-16 shows percentages of total gross ex post savings associated with different combinations of free ridership indicator variable values for the custom and prescriptive incentive component Five survey respondents were dropped from the free ridership analysis because they gave don t know responses to key questions. Energy Efficiency Programs 52

56 PSO High Performance Business Program Table 3-16: Estimated Free-ridership for PY2016 HPB Program Custom and Prescriptive Had Plans and Intentions to Install Measure without C&I Program? (Definition 1) Had Plans and Intentions to Install Measure without C&I Program? (Definition 2) C&I Program had influence on Decision to Install Measure? Had Previous Experience with Measure? Percentage of Total Ex Post Gross kwh Savings Free Ridership Score N N N N 30% 0% N N Y N 24% 0% N Y N N 4% 33% Y Y N N 3% 100% Y Y N Y 1% 100% N N N Y 1% 33% Required program to implement measures. 35% 0% Total 100% 6.0% Overall, the estimated percentage of program free ridership is 6%. This level of free ridership is the same as it was in Customer decision maker survey responses were also analyzed to estimate participant spillover effects. Overall, five of the 140 respondents reported installing efficient equipment that met the attribution criterion and for which energy savings could be estimated. Four of these respondents reported replacing lamps with LEDs and one installed two ENERGY STAR commercial refrigerators. The final net-to-gross ratio for the program is calculated as 1 free-ridership + participant spillover. This results in a NTGR of 95.1% for kwh savings and 94.5% for peak demand reductions. The gross and net verified energy savings and peak demand reduction of the HPB program during PY2016 are summarized by program in Table Table 3-17: Summary of Verified Gross and Net Impacts Program Verified Gross kwh Savings Verified Net kwh Savings Net-to- Gross Ratio Verified Gross kw Reduction Verified Net kw Reduction High Performance Business - Custom and Prescriptive 49,036,679 46,609, % - kwh 94.5% - kw 9,947 9,403 Energy Efficiency Programs 53

57 PSO High Performance Business Program HPB SBES Table 3-18 shows percentages of total gross ex post savings associated with different combinations of free ridership indicator variable values for the custom and prescriptive incentive component. 22 Table 3-18: Estimated Free-ridership for PY2016 HPB Program Small Business Energy Solutions Had Plans and Intentions to Install Measure without C&I Program? (Definition 1) Had Plans and Intentions to Install Measure without C&I Program? (Definition 2) C&I Program had influence on Decision to Install Measure? Had Previous Experience with Measure? Percentage of Total Ex Post Gross kwh Savings Free Ridership Score N N N Y 2% 33% N N N N 19% 0% N N Y Y 1% 0% Required program to implement measures. 78% 0% Total 100% 0.5% Overall, the estimated percentage of program free ridership is 0.5%. This level of free ridership is the same as it was in Customer decision maker survey responses were also analyzed to estimate participant spillover effects. None of the respondents reported installing efficient equipment that met the attribution criterion and for which energy savings could be estimated. The final net-to-gross ratio for the program is calculated as 1 free-ridership + participant spillover. This results in a NTGR of 99.5% for kwh savings and 99.5% for peak demand reductions. The gross and net verified energy savings and peak demand reduction of the SBES component during PY2016 are summarized by program in Table Eight survey respondents were dropped from the free ridership analysis because they gave don t know responses to key questions. Energy Efficiency Programs 54

58 PSO High Performance Business Program Table 3-19: Summary of Verified Gross and Net Impacts Program Verified Gross kwh Savings Verified Net kwh Savings Net-to-Gross Ratio Verified Gross kw Reduction Verified Net kw Reduction High Performance Business - SBES 4,423,345 4,399, % 1,074 1,068 Program Participation Prescriptive and Custom Process Evaluation Findings An analysis of the program tracking database was completed to identify the characteristics of the projects completed during the program year. Program Activity during the Program Year Figure 3-1 displays the accrual of reported energy savings in the High Performance Business program during PY2016. As shown, savings associated with project approval dates accumulated at a consistent rate during the year. Figure 3-1 Accrual of Reported kwh Savings during the Program Year Program Activity by Project Type Prescriptive retrofit lighting projects accounted for the largest share of program reported savings (63%) followed by custom (20%). Although custom projects may include a variety of end-uses, these results indicate that lighting accounted for at least 77% of program savings. Energy Efficiency Programs 55

59 PSO High Performance Business Program Table 3-20: Summary of Program Activity by Measure Type Project Type Number of Projects Reported kwh Savings Percent of Reported Savings Total Rebate Dollars Incentive Dollars per kwh Saved Retrofit Lighting ,362,478 63% $2,703,185 $0.09 Custom 62 9,657,935 20% $792,423 $0.08 New Construction Lighting 89 6,833,855 14% $440,405 $0.06 Unitary HVAC ,115 1% $498,897 $1.09 High Efficiency Equipment and Controls ,192 1% $67,432 $0.21 HVAC VFD 5 242,578 1% $9,575 $0.04 Total 1,002 47,878, % $4,511,917 $0.09 Program Activity by Location Table 3-21 summarizes the share of reported savings by district. As shown, Tulsa accounted for 77% of program reported savings. Table 3-21 District Share of Reported kwh Savings District Percent of Reported kwh Savings Eastern 9% Tulsa District 77% Tulsa Northern District 7% Western 8% Figure 3-2 displays the location of projects based on geocoded street addresses. Energy Efficiency Programs 56

60 PSO High Performance Business Program Figure 3-2 Distribution of Custom and Prescriptive Projects Program Operations and Design This section summarizes the findings from the review of the program design and operations. The focus of the process evaluation of the custom and prescriptive incentive components of the program was on program changes made for the 2016 program year. The material presented below was developed from a review of program documentation and interviews with program staff. The following lists the program documents reviewed by ADM: General service provider training presentation; Marketing materials and application forms; PSO work authorization for the implementation contractor; and Quality control procedures. ADM completed interviews with the PSO program managers and the ICF program managers. These interviews were completed in September/October of 2016 and in February Implementer Staffing Changes: The program manager for ICF left the position in 2016 and was replaced by another member of the implementation team. Additionally, the ICF program manager for the small business component left during the year and was replaced by a new staff member. Overall, PSO staff indicated that these transitions were generally smooth, though the small business program required more PSO staff support during the launch period than originally planned. More generally, parties stated that the ICF and PSO teams continue to work well together in delivering the High Performance Business Program. Energy Efficiency Programs 57

61 PSO High Performance Business Program Regional and Measure Diversity: PSO made a concerted effort to quantify and monitor the portfolio for regional (reaching all the service area) and measure (lighting, insulation, mechanical, etc.) diversity for the current program cycle. Specifically, the regional participation targets are based on district and summarized in Table These targets were based on the number of non-residential customers in each district. A staff member noted that the introduction of the small business component will be a key factor in enabling the program to reach more rural customers because a significant portion of the businesses operating outside of Tulsa are small business customers. Table 3-22 Regional Savings Targets District Share of Program Savings and Participants Lawton (Western) 20% McAlester (Eastern) 12% Tulsa 63% Tulsa Northern 5% The measure diversity target is intended develop savings from less prominent measures. To support the measure diversity, the program is offering additional training that focuses on mechanical systems. Staff noted that they saw increased interest in commissioning projects during 2016 that may help diversify the incentivized measures. These projects were facilitated by a tool for benchmarking building energy use. New Information Tools: The program is now utilizing new tools as part of the program management and delivery process. One of these tools is referred to as the Strategic Intelligence Management System (SIMS). SIMS is intended to allow for improved targeting of specific customer groups. Currently, the program has used the tool to generate maps demonstrating where program savings are occurring and staff is seeking new ways to leverage it. The program implemented an application called Fulcrum to facilitate verification visit data collection and documentation. The Fulcrum application allows staff to develop forms for collecting project verification information and capturing images. The information is then uploaded to the Vision data system used to track project activity. Implementation staff also discussed the Microsoft Dynamics Customer Relations Management (CRM) tool that was introduced in This tool facilitates reporting of program leads and activity as well as captures history. Energy Efficiency Programs 58

62 PSO High Performance Business Program Automated Clearing House (ACH) Payments for Service Providers Introduced: The program introduced ACH payments to service providers in 2016 as means of speeding up payment for projects. This development parallels the introduction of these payments for service providers working with the residential programs. Service Provider Training: The program continued to offer training related to the program offerings and administrative aspects in This training covered: Program offerings; Introduction to program staff and their respective roles; Overview of prescriptive and custom program guidelines and incentive amounts; The application process; Demonstration of the online application; and Benefits of being a registered service provider. Table 3-23 summarizes the number of offered by location. Table 3-23 Summary of Service Provider Training Date Location Number of Attendees 2/9 Tulsa 19 2/10 McAlester 4 2/11 Oklahoma City 18 5/5 Bart 6 5/12 Oklahoma City 13 5/18 Tulsa 32 7/28 Tulsa 22 8/10 Oklahoma City 16 Total 130 Service Provider Feedback ADM completed in-depth interviews with service providers that completed custom and standard incentive projects during the 2016 program year. During these interviews, service providers were asked to discuss the following topics: Benefits of program; Energy Efficiency Programs 59

63 PSO High Performance Business Program Satisfaction with any program training attended in the last year; Feedback on program provided marketing support and program direct marketing to customers; Feedback on program guidelines and application process; Barriers to program participation; Interactions with program staff; and Satisfaction with the program and suggestions for improvement. In total, fifteen service providers were interviewed about the program, one service provider started but did not complete the survey. The comments they shared are included. Respondent Firmographics As shown in Table 3-24, most respondents provide services to each of the three regions of the PSO service territory. The Tulsa area was the most heavily served 87% of respondents indicated that they provided services in that region. Table 3-24: Areas Served Areas served Percent of Respondents (n=15) Lawton 73% McAlester 73% Tulsa 87% *Totals may be more than 100%, respondents could choose more than one option Lighting services were the most commonly reported service provided, with 67% of service providers providing this service (Table 3-25). Additionally, 20% of respondents provided HVAC services, and 27% performed other types of services. Of the four service providers that stated they performed other services, two stated that they provided RCx services, two stated that they provided consultation services, and one stated that they performed energy modeling services. Energy Efficiency Programs 60

64 PSO High Performance Business Program Table 3-25: Services Provided Types of Service Percent of Respondents (n=15) HVAC 20% Lighting 67% Other, please specify 27% Don t know 7% Three service providers reported completing HVAC projects through the program during the program year. Of these, only one was aware of the redesigned incentives for unitary HVAC systems for the program year. This service provider stated that since the incentives changed, they had seen increased interest in installing energy efficient HVAC equipment through the program. Aside from one contractor who provides services to public sector organizations, service providers reported that they did not specialize in providing services to any specific business type. Program Awareness and Benefits of Participation Table 3-26 displays how respondents first learned of the PSO High Performance Business program. As shown, 27% of respondents heard about the program through a PSO representative. The same share of respondents heard about the program. Table 3-26: Initial Source of Program Awareness Source of Initial Awareness of Program Percent of Respondents (n=15) From a PSO representative 27% From a customer 27% Through your firms own research 20% From a distributor, business partner, or other contractor 13% Don t know 13% Energy Efficiency Programs 61

65 PSO High Performance Business Program Fourteen of the 15 service providers that stated they had seen benefits from participating in the program. The benefits service providers discussed are summarized below. As shown, the benefits reported included both benefits to the service provider and benefits to participating customers. Six stated that the program had resulted in more jobs or business for the company; Four referenced the benefit of the incentive in lowering the project cost for their customers; Two referenced energy cost savings for their customers; Two stated that the program simplified projects; One stated that the program increased energy savings for their customers; and One stated that it created more visibility for the company resulting in increased business. Only one service provider stated that they had not seen any benefits to participating in the program. This service provider stated that they had a negative experience with the program overall, and did not see any benefits to participating in the program. This contractor stated that they had issues with the training provided and the documentation requirements. Program Marketing and Promotion About a quarter (27%) of service providers stated that they had used program marketing materials to promote the program incentives. These respondents stated that they had used program brochures, fliers, and directed customers to the program website. When asked to rate the effectiveness of these materials, all contractors who used them rated them as a three or above on a five-point scale. Most service providers that had not used program materials were not aware that there were program marketing materials available for their use. This is consistent with findings from the 2015 evaluation that indicated several service providers were not aware of the availability of the materials. Of those service providers that were aware of but did not use the program marketing materials, three stated that they did not need to use the marketing materials because they used their own, and one stated that they had requested program materials but had not heard back from program staff about receiving them. Service providers were asked if they had any recommendations for how the program could help them be more effective in promoting the program. Most service providers did not have any recommendations. One contractor stated that they would like to be Energy Efficiency Programs 62

66 PSO High Performance Business Program provided with marketing materials, and two stated that they would like to see more interactions with program staff. One of these two service providers stated that ICF representatives used to call on them monthly and now they only see them every six months. They would like to see ICF representatives more often to ensure that they are keeping up with program changes. The other contractor stated that they would like to see program staff take a more active role in marketing the program to potential customers, as this may lend more credence to the program. Barriers to Participation Service providers discussed a variety of issues related to potential barriers to participation. Lack of program awareness is one potential barrier to participation. Thirteen percent of service providers stated that none of their customers were aware of the program and another 27% suggested that less than one-half of their customers were aware of the program. These findings suggest that lack of awareness of the program incentives may limit participation. Service providers reported that they recommend efficient equipment options with more than one half stating that they always recommend efficient equipment options and the remainder indicating that the recommendation will depend on the customer needs. Additionally, 43% of service providers stated that they present non-qualifying equipment options in addition to efficient equipment recommendations. Service providers also reported on the concerns that customers raise about participation. Three service providers stated that some customers raise concerns about the application process and rebate payment timelines, or the effort required to complete the application. One service provider raised the cost of efficient equipment as an issue, and one stated that some potential customers are concerned about the legitimacy of the program. The service provider that noted the perceived legitimacy issue also stated that they use their own marketing materials rather than those provided by PSO. This may affect how their customers view the program. Program Impact on Services Provided and Equipment Recommendations When asked how influential the incentives were on the service providers decision to recommend energy efficient equipment, 72% stated that it was influential or very influential, and two stated that they did not have much influence on their recommendations of efficient equipment. These two also stated that they always recommended efficient equipment before they worked with the program, so the program did not change their practices in this case. Energy Efficiency Programs 63

67 PSO High Performance Business Program One service provider stated that they had made changes to the products or services that they offer because of the program. Additionally, two service providers stated that they had increased staffing because of the program. Measure Comprehensiveness Two of the 15 service providers recommended additional equipment that should receive incentives under the program. These service providers recommended dimmer switches and LED exit signs. However, it should be noted that the program does provide prescriptive incentives for LED exit sign and staff noted that this measure is not offered through small business because the market is saturated. Additionally, three service providers stated that higher incentives for high bay LED lighting would be appropriate. Program Training Half of the respondents stated that they had personally attended at least one program training during the past year. Thirty-six percent of respondents also stated that someone else at their company had attended trainings. Twenty-eight percent of respondents stated that either nobody in their company had attended trainings, or they did not know if anyone had attended trainings. As seen in Table 3-27, 72% of service providers that attended training stated that they were satisfied or very satisfied with the program training. One service provider stated that they were neither satisfied nor dissatisfied, and one service provider stated that they were somewhat dissatisfied with the program training. Table 3-27: Satisfaction with Program Training Overall, how satisfied or dissatisfied are you with the training that has been provided through the program? Percent of Respondents (n=7) 1 - Not at all satisfied 0% 2 14% 3 14% 4 29% 5 - Very satisfied 43% Energy Efficiency Programs 64

68 PSO High Performance Business Program Respondents were asked to provide suggestions for improvements to the training provided through the program. Three respondents provided suggestions. Two of these service providers stated that they would like to see more tailored training based on the service providers experience with the program. For example, one respondent suggested that PSO offer a refresher training for service providers that had completed projects through the program in the past and another training for those who were new to the program. Another service provider stated that the program should offer training more specific to the types of incentives offered such as custom versus prescriptive incentives. Application Process and Interactions with Program Staff All but one respondent indicated that the guidelines on customer and equipment qualifications were clear. The respondent that stated that they were not clear did not have any specific recommendations on how to improve the guidelines. When asked if there were any program requirements that prevented certain types of customers from participating in the program, one service provider stated that there were types of participants that may be prevented from participating. This service provider stated that they had issues with government buildings participating in the program. Two service providers stated that they had at least one project rejected because the proposed measures did not qualify for the program. PSO recently introduced an online program application. Five service providers (36%) had used the online application. Only one service provider indicated that the instructions and guidance were unclear. This service provider stated that they were not computer savvy, and the instructions for the online application were not sufficient in helping them navigate the process. When asked to rate their satisfaction with the online application, four service providers stated that they were either satisfied or very satisfied with the online application, and one service provider stated that they were neither satisfied nor dissatisfied with the online application. Overall Satisfaction Service providers were generally satisfied with all program elements. When asked how they felt about the overall program, Three-quarters of service providers stated that they were satisfied or very satisfied. Two service providers stated that they were neither satisfied nor dissatisfied, and one stated that they were very dissatisfied with the program. About 75% of the services providers were somewhat or very satisfied with the program overall. The program elements of the program that service providers were most satisfied with were the quality of measures and the range of measures offered by the program. All service providers rated that they were either satisfied or very satisfied with these program elements. Energy Efficiency Programs 65

69 PSO High Performance Business Program PSO service providers were less satisfied with the application process overall, and PSO s support of service provider marketing efforts were the lowest rated program aspect. However, more than 70% of services providers were satisfied or very satisfied with PSO s support of their marketing efforts. Figure 3-3 Program Satisfaction Respondents were asked to provide suggestions for improving the program. Most service providers did not have any suggestions for improvement, but the suggestions made were as follows: One service provider stated that they would like the program to discontinue the self-install options; One service provider stated that they would like a simpler application process and quicker processing of applications and payments; and One service provider would like to see only regional service providers allowed to participate in the program rather than allowing national and international contractors able to compete with local contractors for customers through the program. Summary of Key Findings and Conclusions The following conclusions are based on feedback from service providers who assisted customers in completing projects through the High Performance Business Program. Most Service Providers Satisfied with the Program: Most service providers gave favorable scores to the overall program and each element of the program. Energy Efficiency Programs 66

70 PSO High Performance Business Program There is a Lack of Awareness of Program Marketing Materials: Most service providers do not use or are unaware of the marketing materials provided by the program for their use. Service Providers are Generally Satisfied with Program Training: Threequarters of service providers stated that they were satisfied or very satisfied with the program training. The Online Application Process is Underutilized: Only 36% of respondents indicated that they had used the online application process in the last year. More than one-half of service providers were very satisfied with the online application process. One service provider stated that the online application process was overly complicated and should be simplified and that it was difficult to complete the application without the customer account number. This contractor stated that they were not comfortable with online technology which made the online process difficult to use. Participant Feedback A total of 140 customer decision makers completed a survey about their experience with the High Performance Business Program in As shown in Table 3-28, survey respondents worked for a variety of organizational types and office (20%), retail (17%), religious facilities (15%) and industrial/manufacturing (14%) were the most common. Energy Efficiency Programs 67

71 PSO High Performance Business Program Table 3-28: Survey Respondent Organization Types Building Type Percent of Respondents (n = 138) Office 20% Retail 17% Religious Facility 15% Industrial/Manufacturing 14% School 6% Health facility 5% Lodging 4% Warehouse 4% Grocery / convenience 3% Multifamily 3% University 2% Restaurant (not fast food) 1% Restaurant (fast food) 1% Other 5% Source of Program Awareness To gauge the effectiveness of various approaches to promoting the program, survey respondents were asked how they learned of the High Performance Business program. Service providers were the most often cited source for learning about the program with almost half of respondents reporting that they learned of the program from a vendor or building contractor (34%) or from an architect, engineer, or energy consultant (10%). Friends and colleagues were cited by 15% of respondents, indicating that word of mouth is an important source for information about the program incentives. Program outreach and advertising were the first sources of program awareness for 31% of program participants, with 12% of those customers learning of the program from an account representative and the remainder learning of it from a letter or brochure, a program representative or a program sponsored advertisement. Energy Efficiency Programs 68

72 PSO High Performance Business Program Table 3-29: Sources of Awareness for the PSO Equipment Incentives How Participants Learned of Program Rebates Percent of Respondents (n = 139) From an equipment vendor or building contractor 34% Friends or colleagues 15% From a PSO Account Representative 12% From an architect, engineer or energy consultant 10% Received an informational brochure or newsletter 7% Through past experience with the program 6% A letter from a PSO representative 4% From a PSO program representative 4% TV / radio ads sponsored by PSO 4% Other 4% Use of My Energy Advisor PSO offers customers an online service called My Energy Advisor that helps customers identify energy saving opportunities and rebates that they are eligible for. Nineteen percent of respondents were aware of the service and six customers (4%) had used the service. PSO expects to market the service more in Table 3-30 Use and Awareness of My Energy Advisor Awareness and Use of My Energy Advisor (n = 44) Aware of My Energy Advisor 19% Use of My Energy Advisor 4% Of those customers that had used the service, four had used it two or more times in the last six months and one reported not using in the past six months. Additionally, one customer reported using it one time in the last six months but that the energy information presented did not seem to correspond to their actual usage. As shown in Table 3-31, four customers rated the tool as very or somewhat useful for managing their organization s energy use. Energy Efficiency Programs 69

73 PSO High Performance Business Program Table 3-31 Usefulness of My Energy Advisor How useful is the My Energy Advisor for helping your organization manage its energy use? Percentage of Respondents (n= 6) Very useful 33% Somewhat useful 33% Slightly useful 33% Not at all useful 0% Don't know 0% Influences on Participation and Projects Survey respondents reported on who initiated the decision to participate in the High Performance Business program. The findings were like those for 2015 and suggest that apart from the participating organizations themselves, vendors and contractors were primary drivers of program activity during 2016: 35% stated that their organization initiated the discussion and 35% reported that a vendor or contractor initiated the discussion. Another 15% of respondents indicated that the decision to participate arose in discussion between their organization and their vendor or contractor. Thirteen percent of respondents stated that a representative from PSO initiated the discussion, and 2% did not know. Table 3-32: Who Initiated Decision to Participate Who Initiated the Decision to Participate in the Program Percent of Respondents (n = 139) Your organization initiated the discussion 35% Your vendor or contractor initiated the discussion 35% A PSO program representative initiated the discussion 13% The idea arose in discussion between your organization and your vendor or contractor 15% Don't know 2% Energy Efficiency Programs 70

74 PSO High Performance Business Program Figure 3-4 displays the share of potential influencers on equipment decisions that had a moderate to large or a critical effect on participants decisions to retrofit their equipment with energy efficient options. A large share of participants (46%) reported that the contractor or installer had an important influence on the decision to install energy efficient equipment. Vendors were also influential for a large share of respondents (52%). About one-third of respondents stated that ICF program representatives and/or PSO staff members influenced the project decision. The results indicate that while vendors and contractors are most often cited as key influences on customer decision making, program representatives and PSO account representatives are also important in assisting customers with choosing energy efficient equipment. Figure 3-4: Degree of Influence on Projects Participation Process Survey respondents were asked several questions about their experience with the program participation process. These questions covered topics such as who was involved in completing the application, participants assessments of the application materials, the equipment installation process, the rebate amount, and post-installation verifications. Sixty-three percent of survey respondents indicated that they worked on completing the application materials to receive the program rebates. Respondents also reported that equipment vendors (35%) and contractors (29%) assisted with completing the applications. Survey respondents who worked on completing the application materials provided their assessment of these materials. Table 3-33 summarizes ratings of the clarity of instructions on how to complete the application materials. As shown, overall, most Energy Efficiency Programs 71

75 PSO High Performance Business Program participants (72%) indicated that the application instructions were completely or mostly clear. In the table, clarity ratings are presented separately for online submissions because this means of submitting the application is being promoted broadly for use by customers and contractors. As shown, ratings were comparable for online submissions and submissions made by other means. Table 3-33: Ratings of the Clarity of the Instructions Clarity of Instructions All Respondents Online Application Submitted by , Fax or Postal Mail (n = 84) (n = 20) (n = 64) 1 - Not at all clear 1% 0% 2% 2 5% 0% 6% 3 13% 20% 11% 4 30% 30% 30% 5 - Completely clear 42% 50% 39% Don't know 10% 0% 13% Average* *Excludes respondents that indicated they did not know how the application was submitted Respondents that noted a lack of clarity in the instructions were asked what information could be made clearer. Of these twenty respondents, the most commonly reported issue, mentioned by seven respondents, was with the technical aspects of attachments and/or completing the online form. Three respondents stated that it was not clear what equipment qualified for the program, and two stated that they were not made sufficiently aware of the timeline. It is also important to note that one participant that had difficulty completing the application stated that they contacted program staff who provided the assistance they needed to complete the form. Moreover, 81% of participants that worked on the application stated that they knew where to get additional assistance. Figure 3-5 summarizes respondent s ratings of the acceptability of the application process. As shown, very few respondents were dissatisfied with any aspect of the process or the application overall. Additionally, only 8% of respondents reported some dissatisfaction with the online portal, despite lower ratings of the clarity of instructions on how to complete the application by participants that used it. In general, these responses indicate that customers have little difficulty with the application process. Energy Efficiency Programs 72

76 PSO High Performance Business Program Figure 3-5: Acceptability of Elements of Application Process Most respondents (73%) reported that someone other than their own staff installed the equipment. Respondents most frequently reported that a contractor they had previously worked with installed the equipment or efficiency upgrades. The tendency of participants to work with a contractor they had previously worked with underscores the importance of ensuring that many contractors who work in the service territory are aware of the PSO incentives. Relatively few of the respondents (9%) indicated that they worked with a program registered service provider. However, it is possible that respondents were not aware that the contractor they worked with was a registered service provider and so the number of projects installed by registered service providers may have been underreported. Energy Efficiency Programs 73

77 PSO High Performance Business Program Table 3-34: Who Installed the Energy Efficient Equipment Who Installed the Program Equipment Percent of Respondents (n = 139) Your own staff 26% A contractor you've worked with before 53% A contractor approved by your PSO High Performance Business program (registered trade ally/service provider) 9% A new contractor that someone else recommended 7% Other 6% Don't know 1% As shown Figure 3-6, in a large majority of survey respondents reported that they were satisfied with the equipment installed, the quality of the installation, and their experience with the service provider. Figure 3-6: Satisfaction with Equipment Installation and Service Provider Forty percent of survey respondents said their project received a post-inspection. None of these respondents reported that they were dissatisfied with the inspection process and 79% reported that they were completely satisfied with the inspection. The high Energy Efficiency Programs 74

78 PSO High Performance Business Program levels of satisfaction with the inspection process are important because not overburdening participants with undue verification processes likely increases willingness to participate with future efficiency upgrades and can lead to a positive program reputation through word-of-mouth. Table 3-35: Satisfaction with Post-Project Inspection Satisfaction with the Inspection Process Percent of Respondents (n = 58) 1 - Not at all satisfied 0% 2 0% 3 3% 4 14% 5 - Completely satisfied 79% Don't know 3% Seventy-five percent of the respondents reported that the rebate amount was about what they expected. Fifteen percent of the respondents reported that the rebate amount was either somewhat less than they were expecting (12%) or much less than they were expecting (3%). Similarly, 8% reported that the rebate was somewhat more and 2% reported that it was much more than they were expecting. Tracking data indicated that three of the customers that reported that the rebate differed from what they were expecting completed custom incentive projects. Review of the survey responses and tracking data did not lead to a specific explanation for why 25% of respondents reported that they received rebates that differed from their expectations. There was no consistency in terms of who completed the application or how it was submitted, and the prescriptive application forms list the per equipment rebate amounts. However, 68% of respondents that received a rebate that was less than they were expecting reported that they were satisfied or very satisfied with the amount of the rebate. Additionally, two customers that received a rebate that was less than what they were expecting reported that they were somewhat or very dissatisfied with the amount. Energy Efficiency Programs 75

79 PSO High Performance Business Program Table 3-36: Amount of Rebate Compared to Expectations How the Rebate Compared to Expectations Percent of Respondents (n = 139) It was much less 3% It was somewhat less 12% It was about the amount expected 75% It was somewhat more 8% It was much more 2% Thirty-seven percent of respondents indicated that they had interactions with PSO program staff while completing their project. Most of these respondents reported that the staff was very knowledgeable (84%) about the issue they discussed. Figure 3-7 displays respondents level of satisfaction with their PSO program staff interactions. Most respondents reported that they were satisfied or very satisfied with how long it took staff to address their questions or concerns (96%) and how thoroughly these were addressed (96%). Figure 3-7: Satisfaction with Staff s Response to Question or Concern Overall Satisfaction Ninety-two percent of participants were satisfied or very satisfied with the program overall and 2% expressed dissatisfaction with it, as shown in Figure 3-8. Additionally, most respondents indicated satisfaction with the various elements of the program. No Energy Efficiency Programs 76

80 PSO High Performance Business Program single aspect of the program experience stood out as something participants were particularly satisfied or dissatisfied with. In total, ten respondents indicated dissatisfaction with one or more elements of the program or the program overall. The reasons for these customers dissatisfaction are summarized below: Program staff lost application paperwork or information was incorrect; Information about the process was unclear; The requirements changed or were difficult to meet; The process was difficult; and Desired equipment did not qualify. Examples of these types of comments include the following. It all moves too slow. Too many hoops to jump through. The address listed for our company is not correct and delayed the arrival of the rebate check. Rebate checks for our lighting program are also still forthcoming. We had to do most of the paperwork twice because my PSO rep lost or misplaced it. The program requirements for submitting invoices right away. We order our equipment 8 weeks ahead of construction and by the time construction is completed the invoices are 6 months old. That was a problem when I filed. The spreadsheet online that is necessary for rebate - is not mentioned on the paper application - it is difficult and requires PSO help to complete Energy Efficiency Programs 77

81 PSO High Performance Business Program Figure 3-8: Program Satisfaction Respondents were provided an opportunity to provide additional comments about their experience with the High Performance Business program. Most of the comments made by the respondents indicated satisfaction with the program. Many of these comments were general in nature and did not refer to any specific aspect of the program. Some examples of comments that typify this kind of response are: I appreciate financial incentives to become more fuel efficient. We could not have afforded the upgrade without some assistance from PSO/AEP and now I feel like we are able to make a difference in our power consumption... Thanks for this program! It works!! It is a good incentive and did help offset part of the cost of the upgrade. It was a great experience and PSO was very helpful before, during and after the project. Three participants made suggestions to improve the programs. These suggestions were top offer an incentive to upgrade from CFL to LED, offer more direct install measures, and offer incentives for a greater range of measures SBES Process Evaluation Findings This section presents the key process related findings derived from the analysis of program materials, literature review, customer decision maker surveys, service provider interviews, and interviews with PSO program staff. Program Participation An analysis of the program tracking database was completed to identify the characteristics of the projects completed during the program year. Energy Efficiency Programs 78

82 PSO High Performance Business Program Overall, 186 projects were completed during the year. The average cash incentive paid equaled $6,926 and the average reported savings for projects was 21,797 kwh. Figure 3-9 displays weekly and cumulative savings resulting from projects completed during the program year. As shown, the accrual of project savings began mid-year after implementation contracts were signed and increased significantly during the latter part of November and in December. Figure 3-9 Accrual of Reported kwh Savings during the Program Year Contractor Summary Table 3-37 summarizes program activity by service provider. As shown, the savings varied across service providers. This variation largely corresponds to the MWh goals set for each service provider, although performance still varied when the savings goal was accounted for. Energy Efficiency Programs 79

83 PSO High Performance Business Program Table 3-37 Summary of Projects by Service Provider Contractor Number of Projects Total Reported kwh Savings Incentive Costs Project Costs Lighting ,530,083 $473,978 $708,639 Lighting ,251 $133,038 $197,957 Lighting ,706 $237,534 $395,733 Lighting ,527 $331,498 $498,622 Lighting ,193 $82,950 $119,714 Refrigeration 5 95,410 $29,281 $41,832 Total 186 4,054,170 $1,288,279 $1,962,498 Table 3-38 summarizes the average number of days between when the audit was completed and when installation began, and the days between the start of installation and when installations were completed. As shown, the period of between audits and installations was about equal to two months on average. This time likely includes the time for customers to commit to a project and to procure equipment and schedule the installation. Generally, the installation time was brief four days was the average installation time although some contractors averaged longer installation times. Table 3-38 Summary of Project Timelines Contractor Number of Projects Days Between Audit and Start of Installation Days Between Start of Installation and Completion Lighting Lighting Lighting Lighting Lighting Refrigeration All Contractors Energy Efficiency Programs 80

84 PSO High Performance Business Program Program Activity by Measure Table 3-39 summarizes savings by measure type. Although some service providers noted in interviews that the incentive dollar per kwh saved target limited the number of LED projects, LED lighting accounted for 69% of program savings. Table 3-39 Savings by Measure Type Measure Project Count Reported kwh Savings Percent of Reported kwh Savings LED Fixture 102 1,428,374 35% Fluorescent 89 1,051,301 26% LED Linear ,588 16% LED Exterior Lights ,574 12% LED Screw-ins ,028 6% Abandoned Fluorescent 13 39,033 1% LED Exit Sign 58 37,671 1% Evaporative/Compressor Controls 5 28,038 1% Other controls & EC Motors 5 27,042 1% Cooler Door Heater Controls 3 25,819 1% LED Case Lights 3 14,511 <1% Occupancy Sensor 22 6,362 <1% Abandoned CFL 2 4,014 <1% Abandoned HID 3 2,813 <1% Total 558 4,054, % Program Activity by Location Table 3-40 displays the share of SBES savings by district. Because of the assignment of territories and budgets to service providers, two-thirds of program savings resulted from projects completed outside the Tulsa district. Energy Efficiency Programs 81

85 PSO High Performance Business Program Table 3-40 District Share of Reported kwh Savings District Percent of Reported kwh Savings Eastern District 39% Tulsa District 32% Tulsa Northern District 3% Western District 27% Figure 3-10 displays the location of SBES project across the service territory. Figure 3-10 Distribution of Small Business Energy Solutions Projects Program Operations and Design This section summarizes the detailed findings from the review of the program design and operations. The material presented below was developed from a review of program documentation and interviews with program staff. The program documents reviewed by ADM were: General service provider training presentation; Marketing flyers; Subcontract agreements with firms providing assessments and installations; Program manual; Energy Efficiency Programs 82

86 PSO High Performance Business Program PSO work authorization for the implementation contractor; Customer participation agreement and certificate of completion; Examples of program management reporting; Quality control procedures; and Technical review procedures. ADM completed interviews with the PSO program manager and the ICF program manager. These interviews were completed in September 2016 and in February Figure 3-11 summarizes the program logic. This summary was developed by the evaluator based on the reviews of documents and interviews with program staff. Energy Efficiency Programs 83

87 Figure 3-11 Program Logic Model Energy Efficiency Programs 84

88 Program Goals and Design The SBES program is intended to encourage energy efficient lighting and refrigeration improvements in small businesses. The barriers to efficiency faced by small businesses include limited financial resources and time, limited knowledge of efficient products, and lack of access to skilled contractors with expertise in efficient technologies. Key elements of the program design include: Incentives that cover 70% of the installed equipment costs (inclusive of material and labor); Use of contractors to manage projects and minimize the customer s paperwork burden; Direct one-on-one outreach to businesses by service providers; and No-cost or obligation energy assessments. Customers annual electricity usage must be 220,000 kwh or less to participate in the program. Additionally, customers with facilities in multiple locations cannot have annual usage across accounts that sum to more than 2,000,000 kwh. The 2016 electricity savings goal was 7,438,816 kwh. The program fell short of achieving this goal, primarily because the program only operated for one-half of the program year after a delayed launch. In addition to the energy savings goal, the program targeted a revised 8% of savings from refrigeration projects. Other objectives of the program included: Cap lighting incentive costs at $0.295 per kwh saved and refrigeration incentive costs at $0.40 per kwh saved. To clarify, these are not filed goals, but program implementation guidelines developed to manage cost effectiveness. Achievement of savings targets by individual service providers. Quality installation work and customer satisfaction. Collaboration among service providers and sharing of referrals to one another and the custom and standard incentive programs, as appropriate for customer needs. Achieving participation from customers through the PSO service territory. Five lighting service providers and one refrigeration service provider are under a subcontract with the implementation contractor. Each is assigned a section of the utility service territory. Two of the service providers were assigned portions of the western service territory based on where they were established. The remaining three lighting service providers selected zip codes from the remainder of the PSO s service territory. Service providers are assigned an electricity savings goal and a corresponding budget based on the potential market for the region. The assignment of service territories and Energy Efficiency Programs 85

89 PSO High Performance Business Program budgets contrasts with more openly designed programs that allow for a set of approved service providers to recruit any qualifying customers in the service territory. Program staff stated the assignment of service providers to specific regions of the service territory served multiple purposes. First, this approach was chosen to ensure that rural customers were solicited and had an opportunity to participate. The concern was that if the program was open, most service providers would concentrate on the Tulsa region where the density of businesses is greatest. Second, staff wanted consistent pricing across the program and consistent product quality. Third, staff stated that this model has the potential to improve program quality by having a single point of contact assigned to any given customer. The program intends that service providers will work together collaboratively and make referrals to one another, as appropriate. There are two primary types of referrals that service providers may make: Referrals from lighting contractors to the refrigeration contractor and vis-a-versa; and Referrals from lighting contractors who have a customer that has business locations in more than one service providers assigned service territory. Staff noted that two issues hampered the development of refrigeration projects resulting from referrals made by lighting service providers. First, small businesses often do not have sufficient funds to complete both lighting and refrigeration projects. As such, lighting contractors compete with the refrigeration contractor for customers. To address this issue, the program is considering increasing the customer annual kwh usage limit for refrigeration projects to create a segment of small businesses that are eligible for refrigeration but not lighting measures. Second, during 2016, the refrigeration service provider representative had essentially disengaged from the program. However, staff reported that this issue has since been remedied and that the refrigeration company has a representative that is actively engaged in the program. Referrals from one lighting service provider to another may occur if a service provider is completing a project for a customer that has another location in another service provider s service territory. Overall, service providers appear to be working well together and either referring customers to the service provider assigned to that location or getting permission from the assigned service provider to complete the project. However, staff reported that were a few incidents where a lighting service providers reported that another service provider had completed an assessment in their service territory. Staff reported that service providers have been coached on adhering to the assigned service territories and warned that it could result in the program not paying for projects completed outside of the assigned service territories. Additionally, staff considered altering the software used for completing assessments and projects (described below) Energy Efficiency Programs 86

90 PSO High Performance Business Program to prevent service providers from completing projects in another territory but rejected this approach because of the cost of making the programming change. Program Management The following sub-sections describe the program management in terms of roles and responsibilities, the development of qualified measures and costs, and the software tools used to manage the implementation process. Roles and Responsibilities PSO is responsible for overall management of the program and oversite of the implementation contractor. In addition to this broad role, PSO staff have supported the program in the following ways. During the design phase, PSO staff provided input into program design, budget and savings goals. PSO develops qualified customer lists used to screen customers for eligibility. PSO (in coordination with their marketing firm, VI Marketing) reviews and approves marketing materials. Additionally, the PSO program manager has regular contact with the program contractors. These discussions typically involve discussing any issues with the program or design, answering questions, and checking on the status of the project in the pipeline. ICF s is the program implementation contractor. Their role in the program includes providing key program infrastructure (e.g., software tools), providing marketing materials to contractors, providing savings goals and program budgets to service providers, and oversight of subcontractor performance and ensuring conformance with program rules. Additionally, ICF performs verification visits and reviews of project files. Service providers also fill a significant role in the delivery of the program. Service provider obligations include: Providing marketing services, including direct customer outreach; Completing comprehensive energy assessments (specific to measure types); Identifying opportunities for customers to participate in the program and providing leads to the contractor staff; Using the Audit Direct Install (ADI) software to complete energy assessments and submit project information; Managing customer relationships including following up with customers that receive assessments; Achieving assigned savings goals with the assigned budget; Energy Efficiency Programs 87

91 PSO High Performance Business Program Overseeing work performed by their firms or subcontractor firms; and Recycling waste material. Project Measures and Costs Program staff has developed a list of approved equipment as well as the material and labor costs that service providers charge for the installation of the equipment. A set cost list was developed to ensure consistency across the region, control program costs, and prevent contractors from overcharging customers. Program staff worked with the subcontracted service providers to develop a list of relevant measures for small businesses that service providers had access to and to set the amounts that could be charged for installation of the equipment. The set costs for the equipment were intended to cover removal and recycling of replaced equipment in addition to the material and installation costs. Staff reported that while the costs were not the lowest market cost for equipment and labor because some compensation is required for the service provider s administrative role, they are also not outside of what a customer would likely pay if they were independently contracting the work. Program incentives were calculated as 70% of the installed costs (i.e., the customer s co-pay is 30%). Software Tools The SBES program uses paperless processes for completing assessments, submitting project proposals, and completing verifications. The SBES program uses a tool called Audit Direct Install (ADI) to complete energy assessments, develop project proposals, and submit documentation for the program. This tool provides standardization to the assessment process by collecting consistent information on facility characteristics, as well as baseline and efficient equipment. The tool is also used to enter a proposal. Staff indicated that proposals can be modified within the tool as needed. Once a customer signs a proposal, the Vision system used to track data for all PSO portfolio programs is used to track the SBES project. In the Vision system, projects status is tracked from submission to payment processing. Post-project inspection visits are completed using forms developed within a software called Fulcrum. The tool captures information on business hours, specifications and quantities of installed equipment, and additional notes and comments. Reported variances document discrepancies in quantities and equipment types and include photographs of the issue. Upon completion of the inspection, an inspection report is uploaded to the Vision system. This report documents the details of the inspection including the inspector, business and project information, and the status of inspection. Energy Efficiency Programs 88

92 PSO High Performance Business Program Marketing and Outreach As is typical of small business direct install programs, program service providers are key components of the SBES marketing strategy. The program service provider s agreement states that contractors are obligated to market the program to customers in the assigned territory and to maintain a dedicated telephone line for the program. To facilitate service providers marketing of the program and to manage the representation of PSO and the SBES Program, the program provides service providers with materials and artwork for use by the service providers. Service providers can develop their own material for promoting the program but this material must be reviewed by program staff. In April, the program mailed letters announcing the program to customers with qualifying accounts. For this mailing, some qualifying accounts were eliminated because of very low electricity usage and could be for operations outside of the scope of the program (e.g., traffic signals). Additionally, schools and churches were not included in the campaign. Although these customers are eligible for the program, staff indicated that their low hours of use make them not a great fit for the program, and as such were not included in the program s outreach effort. The letter provided a description of the program and outlined the simplicity of the participation process. Importantly, it provided the business with the name and telephone number of the firm that provides the energy assessment and installation services for that customer. A targeted campaign was also conducted later in the program year. The targeting of this campaign was based on a propensity model designed to identify small business customers who would be most likely to implement a project. The campaign was considered unsuccessful as few businesses opened the and service providers reported that they did not get any customers in response to the campaign. The program developed a flyer to promote the program (shown in Figure 3-12). The program emphasizes the rebates available, qualification requirements and how customers can verify if they qualify a description of the program steps, and the nonenergy benefits. Overall, this flyer is consistent with best practice in marketing design in that it provides a clear statement of benefits, who qualifies, and a short description of the process. One element that is lacking is a clear statement of the first step to participate. The flyer would be enhanced if it included a clear statement under How do I get started? that indicates the first step the customer needs to take, that is, to schedule and onsite consultation with your business energy consultant. A copy of the address to the page that lists the assigned consultants instead of the main page of the PSO website would also be an enhancement. Energy Efficiency Programs 89

93 PSO High Performance Business Program Figure 3-12 SBES Program Flyer The program website is another tool used to market the program. The program website provides a description of the program and the benefits of participating as well as a tool to look up the contractor assigned to them based on zip code. Participation and Quality Control Processes Figure 3-13 displays the program participation and quality control processes. As shown, participation is initiated when an energy assessment is completed at the customer s business. From the energy assessment, a proposal is developed for the customer to review. The program intends for service providers to follow up with customers that have received proposals. Energy Efficiency Programs 90

94 PSO High Performance Business Program Figure 3-13 Description of Participation and Quality Control Processes Once a customer has elected to proceed with a project, the customer signs the participation agreement. This agreement includes the customer contact person, business information, project cost, and the customer and PSO contributions to the cost. The service provider agreement states that the work should be completed within 30 business days of the customers signing the agreement, however, staff reported that most projects are completed in a few days. Service providers may sub-contract the installation of the measures to other firms. Ensuring the quality of the work that is completed by sub-contractors is the responsibility of the subcontractor. Upon completion of the work, the subcontractor submits invoicing, the customer participation agreement, and supporting documentation such as specification sheets. All submitted projects undergo review by implementation staff. During the review, staff verifies that: The project cost, rebate amount, and customer share is the same on the participation agreement, the customer and utility invoice, and the customer report; The participation agreement is signed by the customer; The customer report has been uploaded; Energy Efficiency Programs 91

95 PSO High Performance Business Program The project completion form is signed by the customer; and The company name and account number on the participation agreement match the account number. The program guidelines state that 10% of all projects receive onsite inspections. Staff reported that onsite verifications were performed for most projects during the first few months of operation. During the inspection, staff documents the business hours of operation and other characteristics, and the quantities and measure specifications of the installed equipment. This information is checked against the customer proposal. The PSO program manager has attended several these verification visits and expressed a high level of confidence in implementation staff s procedures. Staff also reported that the installing contractor is invited to attend the verification visit. Overall, the quality control procedures as described reasonably assure that projects completed under the program will adhere to program standards. Design Benchmarking Table 3-41 provides information on key aspects of other programs operating in the Midwest and South. As shown, the size qualifications vary significantly, with four of the six programs limited sized based on demand, while PSO s program and Consumers Energy Program base qualification on annual energy use. Incentive amounts and structures varied significantly and include covering a percentage of measure costs, prescribing an incentive amount per unit of the measure, offering packages of measures, and paying incentives on a per kwh saved based. Additionally, three of the six programs offered no cost DI. All programs included an energy assessment as part of the program design. Energy Efficiency Programs 92

96 Table 3-41 Summary of Program Designs Program Elements PSO Size qualifications Annual usage less than or equal to 220,000 kwh SWEPCO Arkansas Peak demand of 50 kw or less Focus on Energy Less than 100kW ComEd Consumers Equal or less than 100 kw Less than 400,000 kwh and 6,000 Mcf annually I&M Indiana and Michigan Less than 150 kw Incentive structure Program covers 70% of prescribed measure cost. Dollar per kwh saved targets of $0.295 per kwh saved for lighting and $0.40 per kwh saved for refrigeration Free DI measures $0.12 /kwh for strip curtains and door gaskets and $0.16/kWh saved for all other measures Cap at 90% of cost. Offers one of three packages (Silver, Platinum, Gold), and 'a la carte' discounted products up to $7500 in products No cost DI during assessment, and 30-75% discount on select additional measures recommended by trade ally Prescribed by measure. Up to $7500 (Up to 100% of installation cost) on recommended measures Prescribed by measure. $3,000 per site or $21,000 per company across multiple sites. Assessment? Yes Yes Yes Yes Yes Yes Free DI? No Yes No Yes Yes No Energy Efficiency Programs 93

97 PSO High Performance Business Program Measure Comprehensiveness Benchmarking To assess PSO s program measure comprehensiveness, the evaluators benchmarked the program offerings against other small business programs. As shown, the PSO covers most of the primary lighting and refrigeration measures that likely account for most program savings. It does not include low-flow measures and various other types of equipment such as strip curtains and vending machine controls. Small businesses can also receive incentives for other measures such as HVAC equipment through PSO s standard and custom incentive offerings. Energy Efficiency Programs 94

98 PSO High Performance Business Program Table 3-42 Measures Covered Measure Type PSO SWEPCO Arkansas Focus on Energy ComEd Consumers I&M Indiana I&M Michigan Lighting LED Bulbs &Fixtures Lighting Sensor (Occupancy/Daylight) Linear and other space lighting Exterior Lighting Exit Sign Refrigerator space/case lighting Low Flow Showerhead Aeration Pre-rinse sprayers Refrigeration Strip Curtains Motor Control Anti-sweat heater Other Refrigeration Other Insulation HVAC Vending Machine Controls Pipe Wrap Review of Best Practices This section presents best practices for energy efficiency programs targeting small businesses. The information on best practices is based on a review of existing best practice summaries, papers presented at energy efficiency conferences, and publicly available evaluation reports. Energy Efficiency Programs 95

99 PSO High Performance Business Program Table 3-43 summarizes the findings of the review of small business program best practices. Additional detail on the best practices identified is summarized in Appendix A. As shown in the table, PSO s small business program is largely aligned with best practices. Table 3-43: Summary of Best Practices Review Best Practice PSO Alignment with Practice Use a standard survey instrument during energy assessments. Aligned Develop database tools for managing program projects and customer contacts. Require in-depth assessments and provide tools for incorporate diverse measures (e.g., non-lighting) in assessments to lead to deeper retrofits. Aligned Mostly Aligned: PSO covers refrigeration in addition to lighting. It does not cover low-flow devices and vending misers. Provide payback information to the customer to help them understand longer term benefits. Provide high incentives to reduce first cost barrier Aligned Aligned Cap Incentives but Include opportunities for Standard Incentive Costs Aligned Design eligibility criteria to prevent participation from corporate chain stores. Aligned Provide incentives in the form of a customer discount. Offer 0% financing to overcome first cost barriers. Define product specifications. Aligned Not Aligned Aligned Conduct verifications of project installations to mitigate evaluation risk. Verifications should include assessment of customer satisfaction with the contractor and any missed savings opportunities. Aligned Mostly Aligned: Assessing customer satisfaction is documented as part of the QA/QC but identification of additional opportunities is not included as formalized procedure. Energy Efficiency Programs 96

100 PSO High Performance Business Program Best Practice PSO Alignment with Practice Provide verification inspection results to contractors. Aligned Develop strong relationships with qualified contractors. Aligned Provide ongoing training and development of contractors. Follow-up with customers that receive energy assessments Aligned Aligned: This is a defined responsibility of service providers Use electronic forms and tools to facilitate energy assessments and project proposals. Do not over rely on short-term sales or promotions. Leverage utility credibility in program marketing. Aligned Aligned Aligned Use door-to-door marketing as a strategy for recruiting customers. Leverage community partnerships. Aligned Aligned: Staff has presented at chambers of commerce meetings Design marketing materials to demonstrate the program s legitimacy through use of the utility sponsor brand. Aligned Service Provider Feedback ADM completed interviews with representatives of all six firms that perform program outreach, project sales, and energy assessments. Five of the subcontractor firms provide lighting services while the remaining firm provides refrigeration measures. The interviews took between 20 and 30 minutes, during which time respondents discussed the following topics: Service providers views of program design; Strategies used to promote the program; Project completion process; and Interactions with program staff and program training. Energy Efficiency Programs 97

101 PSO High Performance Business Program Background Information All service provider firms market the program, sell projects, perform energy assessments, and develop customer proposals. The refrigeration service provider and two of the lighting firms also install the lighting measures. The other three lighting service providers work with subcontractors to install the lighting equipment. Interviewees for each of the three firms stated that they worked with different subcontractors rather than a single firm, although one said they primarily work with one subcontractor. In selecting a subcontracting firm to complete the installations, two of the respondents noted that they will use a subcontractor that the customer has previously worked with, if possible, and one stated that they try to select a firm that is located near the customer s business. Consistent with their service agreement, all three service providers that utilized subcontractors for the lighting installation stated that they perform quality checks on the work performed. These checks focus on verifying that the equipment was installed and operating as expected and that customer was satisfied with the installation. Program Design and Barriers to Participation Service providers discussed aspects of the program design including the assignment of service territories, the set costs for equipment and labor and the dollars per kwh ($/kwh) saved budget target. 23 All the lighting service providers held a favorable view of the assignment of a specific territory as opposed to having an open participation model in which a contractor may call on any participant. The primary reason for their favorable view is that contractors believe this approach works well for the customer. Service providers noted that under this model, customers are not called on by multiple service providers, nor do they have to select from multiple offers made by service providers if they are interested in completing a project. As such, the model provides customers a simpler means of participating in the program. Another benefit noted was that the model fosters collaboration among service providers because they are not competing for the same customers. However, service providers also noted some drawbacks stemming from the assignment of service territories. Two service providers, both of whom install the lighting equipment for customers, noted that the assignment of areas means that you may not be able to 23 Lighting projects have a target of $0.295 per kwh saved. The refrigeration target is $0.40 per kwh saved. Energy Efficiency Programs 98

102 PSO High Performance Business Program work with existing customers. However, these service providers also noted that the other service providers are generally open to trades that would allow them to call on their existing customer base. Two service providers also noted that their assigned service territories were fairly spread out and that this increases the cost of providing outreach and program services. Lighting service providers discussed the sufficiency of the costs set by the program for labor and materials and the effect of meeting the program s $/kwh saved requirement. Their responses are summarized in Table As shown, most service providers noted that material or labor costs were low for one or more equipment replacements, but differed on which equipment this was. The differences could reflect differences in the costs for sourcing materials and labor across the different service providers, or in some cases, different experiences with different project types. Table 3-44 Lighting Service Provider Assessment of Program Cost Aspects Service provider Are material and labor costs appropriate? Does $/kwh target limit projects? C1 Material costs for 480 V high bay too low Yes, LED retrofits Labor costs for T5 lighting too low Avoid buildings with lower operating hours C2 Hard to find LEDs that fit in material costs Yes, LED retrofits Avoid buildings with lower operating hours C3 No issues noted Yes, LED retrofits C4 Linear FL material and labor costs are tight No C5 All prices are a little too low Yes, LED retrofits Avoid buildings with lower operating hours However, there was greater consistency regarding the impact of the $/kwh saved target on limiting LED projects. Most of the service providers noted that the target prevented some customers from installing LED lighting, and some stated that there are some customers who are not interested in installing efficient fluorescent lighting as an alternative to LED lighting. In other cases, service providers described scenarios under which they installed LED lamps in the fixtures with the highest hours of operation and fluorescent fixtures in areas with lower hours of operation. In response to this concern, program staff adjusted the incentives for LED lighting to a 50/50 cost sharing split. This split allows a greater number of LED projects that stay within the $/kwh saved target. Another impact of the $/kwh requirement is that it limits the types of participants that are eligible for the program. Three service providers noted that while they will serve all customers, they tend to try to avoid buildings with lower operating hours because this reduces the savings potential and the ability to meet the $/kwh target. One service provider remarked that they have had leads on churches, for example, but noted that they are difficult to complete projects at because of the limited operating hours. Energy Efficiency Programs 99

103 PSO High Performance Business Program The refrigeration service provider did not have any concerns with meeting the program $/kwh target or adhering to the material and labor costs. Service providers discussed the comprehensiveness of the approved measures. The lighting service providers noted a few measures that should be added. Most commonly, TLED lamps were identified as a measure worth including in the program. Especially, both the plug and play type and the type that bypass the existing ballast were noted as worth adding to the program. It was noted that customers do not understand why they cannot get the rebate for these types of lamps and one service provider noted that they are eligible for prescriptive rebates from PSO. Program staff has since added these types of lamps to the program. Additionally, two service providers noted that lower wattage efficient 8-foot linear fluorescents should be added. Currently, the lowest wattage available is 60W but two service providers have installed 44W and 42W lamps. The loss of watts constrains their ability to meet the program s $/kwh target. The comments made by service providers on the program design suggest some potential barriers to program participation. Customers interest in LED tubes that replace linear fluorescents and work with existing fixtures may prevent some customers from participating because they are interested in this type of retrofit. Lack of lower end wattage for 8-foot linear fluorescent replacements may inflate the overall cost per kwh saved for projects, which may prevent deeper retrofits that involve replacements of lower use fixtures. The incentive amounts and $/kwh saved limits the types of businesses that fit well with the program. Specifically, service providers reported avoiding lower hour of use building types. Outreach Approach PSO provides service providers a list of qualified customers to facilitate their outreach efforts. A common theme among lighting service providers outreach approaches was a focus on buildings with longer operating hours, although most interviewees also noted that they do not exclude any customers interested in the program. Service providers described a mix of approaches used to first contact customers. Most often, service providers described a door-to-door sales approach where they canvas an area. However, two service providers noted that they typically first attempt to contact customers by telephone to set up an appointment time. One of these service providers noted that once they have an appointment arranged with customers, they will also do door-to-door sales in the same area. Additionally, in describing their firm s sales Energy Efficiency Programs 100

104 PSO High Performance Business Program approach, one service provider emphasized the importance of identifying a decision maker when first contacting the customer. Consistent with program design and intent, the service providers reported using the marketing materials provided by the program. Service providers stated that they used flyers and program branded shirts when speaking with customers. These service providers had favorable opinions of the material and several noted that the co-branded materials are helpful because the affiliation with PSO helps with making the sale. However, one service provider noted that the sales process is more important than the use of co-branded materials. One service provider also stated that they were developing their own marketing collateral including case studies. The program discount was most commonly mentioned as the key selling point for customers. Secondary selling points that were mentioned include the energy savings from the project, the relationship with PSO, and familiarity with the firm providing the energy assessment. Additionally, one service provider discussed the importance of the third-party financing that their firm offers to customers who may not have the cash flow to fully fund the project co-pay. Two service providers also mentioned that attendance at public events, such as trade shows, and chambers of commerce meetings, often co-attended by PSO, were also used as venues to market the program. Project Completion Process Service providers generally indicated that the participation process was straightforward. All service providers provided favorable assessments of the ADI software. This software was described as slick by one service provider and it was generally regarded as easy to use. One service provider commented on the ease of subdividing a project and being able to save the original project in the system and two service providers noted that it was easy to run reports in the system. An issue raised by one service provider was that if a job gets rejected for payment, the system does not notify him either by or by updating the status of the project in ADI. Two suggestions were made to improve the ADI software that were primarily related to program design choices rather than software functional limitations or usability issues. One was to allow the option of running discount scenarios for 50% copay in addition to 30% copay, which allows for more LED projects. The second was to increase the range of efficient lamp wattages, namely for 8-foot T-lamps. Although the ADI system is designed to allow for the presentation of a project proposal to the customer at the time of assessment, a common theme among the service Energy Efficiency Programs 101

105 PSO High Performance Business Program provider s descriptions of the participation process was to present the proposal sometime after the assessment was completed. Respondents indicated that the assessment information and draft proposals were reviewed by staff for quality control purposes before providing a proposal to the customer. Proposals were typically presented in person either the day of the assessment or within a few days of completing the assessment. Interactions with Program Staff Service providers generally spoke favorably of ICF and PSO staffs support. Respondents commented on the high level of responsiveness from staff and the value of attending verification visits alongside program staff. Additionally, service providers provided favorable assessments of the ADI training provided. None indicated a need for additional program provided training. Collaboration with Other Service Providers The interviews suggest that service providers are collaborating and making referrals to each other to help meet customers needs. Most respondents reported making and or receiving referrals from other participation lighting service providers. These referrals include lighting service provider referrals to other lighting service providers for customers outside of their assigned territory and referrals from lighting service providers to the refrigeration service provider. Two service providers raised concerns with the referral process. One lighting service provider stated that he did not know who to refer customers to for refrigeration measures and suggested that implementation arrange an introduction between the lighting service provider and refrigeration representative. Another lighting service provider stated that they had made referrals but thought that they had not been followed up on because the customers were continuing to contact him about completing a project. Service Provider Recommendations At various points in the interviews, some service providers offered recommendations for improving the program. In several instances, the service providers noted that they had also discussed these recommendations with program staff, suggesting that there are ongoing discussions between service providers and program staff to identify ways to improve the program design and operations. The following summarizes the suggestions provided. Energy Efficiency Programs 102

106 PSO High Performance Business Program Allow multiple incentive scenarios such as 50% and 70% incentive amounts to be run for projects. This flexibility will allow for customers interested in LED retrofits to complete projects. Include LED tubes as a program measure. Ensure that service providers receive notification if a project is rejected for payment. Provide on-bill financing. On-bill financing addresses customer cash flow issues that may prevent projects and strengthens the connection between the service provider program offer and the utility. Include lower wattage eight foot (e.g., 44W) lamps or allow for custom wattage inputs for efficient lamps. Allow for zip code reassignment if the assigned service provider is not calling on customers in that area. Participant Feedback An online survey was administered to program participants during January and February A total of 44 customer decision makers completed a survey about their experience with the Small Business Energy Solutions Program in Table 3-45 summarizes the types of business and building types represented among survey respondents. Energy Efficiency Programs 103

107 PSO High Performance Business Program Table 3-45: Survey Respondent Business and Building Types Building Type Percent of Respondents (n = 44) Office 25% Retail 18% Industrial/Manufacturing 16% Service 16% School 11% Warehouse 5% Religious Facility 5% Grocery / convenience 2% Source of Program Awareness Consistent with the program s design, the program energy consultant was the most common way that people first learned of the program (cited by 61% of respondents). Friends or colleagues was cited as a source of awareness by 16% of respondents. Outreach by a program representative was cited as a source of awareness by 7% of respondents. Advertisements (likely for the full portfolio of PSO programs), chamber of commerce meetings, and vendors/contractors were also cited as a source of awareness for several respondents, each cited by 2-5% of respondents. None of the respondents reported learning about the program through the direct mail or campaigns completed during the year. Energy Efficiency Programs 104

108 PSO High Performance Business Program Table 3-46: Sources of Awareness of Small Business Solutions Rebates How Participants Learned of Program Rebates Percent of Respondents (n = 44) Program energy consultant 61% Friends or colleagues 16% From a PSO program representative 7% TV / radio ads sponsored by PSO 5% From an equipment vendor or building contractor 5% Received an informational brochure or newsletter 2% Chamber of commerce meeting 2% Other 2% Use of My Energy Advisor PSO offers customers an online service called My Energy Advisor that helps customers identify energy saving opportunities and rebates that they are eligible. Customers can also use it to verify that they qualify for the small business energy solutions incentives. Twenty-five percent of respondents were aware of the service and one customer reported using the service and had not done so in the last six months. Table 3-47 Use and Awareness of My Energy Advisor Awareness and Use of My Energy Advisor (n = 44) Aware of My Energy Advisor 25% Use of My Energy Advisor 2% Decision to Participate Twenty-five percent of respondents reported that they had concerns about participating in the program when they first learned of it. The concerns mentioned are summarized in Table Project cost, disruption to the business, and concern about realizing value or saving money were the most frequently noted concerns each was mentioned by 27% of customers that had concerns about participating. Energy Efficiency Programs 105

109 PSO High Performance Business Program Table 3-48: Concerns About Participating in Small Business Solutions Concern Percent of Respondents (n = 11) Project cost 27% Disruption to the business 27% Realizing value / saving money 27% Not sure if staying in the location 9% More trouble than it was worth 9% Type of fixtures used 9% These customers gave a variety of explanations for why they decided to participate despite their concerns. Customers that were concerned about the costs of participation stated they decided to participate because they were given information on what their cost would be or that they eventually saw the value in the project. The verbatim comments were: We were given what our cost would be and we were able to handle Going green, costs, and value Felt like the costs were worth the savings Customers that were concerned about the disruption to their business decided to participate because they saw the value in it or learned that the project would not disrupt their operations significantly. The comments made were: It was a very good investment Contractor assured me they could work around our work flow The construction was minimal and was only one day Those customers that were concerned about the value of the program said that they decided to participate because the numbers ultimately made sense to them. Fifty-two percent of customers viewed program marketing materials during the process of learning about the program. Most of these customers (70%) viewed a program brochure. Additionally, 17% viewed the program website. As shown in Table 3-49, 85% of customers found the marketing material to be somewhat or very influential in their decision to participate. Energy Efficiency Programs 106

110 PSO High Performance Business Program Table 3-49 Influence of Program Marketing Materials on Decision to Participate How influential were the program marketing materials you viewed? Percentage of Respondents Very influential 25% Somewhat influential 60% Only slightly influential 5% Not at all influential 10% Don t know 0% These findings highlight three key programmatic factors that can mitigate barriers to participation: Providing the energy assessment with customer expected savings and payback can convince customers of the value of the project. Contractor assurances about the minimal impact on the customer s business can alleviate concerns about project business impacts. The program flyer and website influence customer decisions to participate. Participation Process In their role as energy consultants, service providers are to discuss with customers that PSO is rebating the cost of their projects, the electricity cost savings from a proposed project, and the project payback. As shown in Table 3-50, nearly all respondents reported that these key topics were discussed with them. Table 3-50 Discussion of Financial Benefits from Participation Did your contractor discuss the following with you? Amount of rebate you would receive from PSO Yes No Don't know 95% 0% 5% Annual electricity cost savings 98% 2% 0% The project payback 98% 2% 0% Participants rated their satisfaction with various aspects of the service provided by the program energy consultant. As shown in Figure 3-14, most participants were satisfied with the experience with the energy consultant. Two customers were dissatisfied with Energy Efficiency Programs 107

111 PSO High Performance Business Program the experience with the energy consultant. One of these customers stated that materials were left behind at the location and had not yet been picked up. The other customer was dissatisfied with the appearance of the light fixtures and stated that he or she would have preferred to have seen a picture of the fixtures before they were installed. Figure 3-14 Satisfaction with Service Provided by Energy Consultant Most customers thought that the equipment covered by the program fit their needs completely, or nearly completely (i.e., rated the fit with their needs as a 4 or 5). The 5% of customers who s responded to this question with a rating of a 3 provided somewhat ambiguous responses about what needs they had that were unmet. One customer stated, heat and air and the other stated store lighting. Table 3-51 Fit of Equipment Offerings with Participant Needs How well did the range of equipment covered by the program meet your needs? Percentage of Respondents (n = 44) 1 - Not at All 0% 2 0% 3 5% 4 18% 5 - Completely 77% Nearly all customers were satisfied with the equipment installation process, as shown in Figure The customer that was somewhat dissatisfied with the installer did not elaborate on the reason for dissatisfaction. Energy Efficiency Programs 108

112 PSO High Performance Business Program Figure 3-15 Satisfaction with Equipment Installation Fifty percent of respondents reported that a PSO program represented inspected the work completed through the program. These respondents were very satisfied with the inspection process. Overall Satisfaction Six customers reported that they contacted a PSO program staff person about the program during the process of completing the program project. These six respondents were very satisfied with the timeliness and thoroughness of the staff person s response to their inquiry. Overall, participant satisfaction with the program was high. All respondents were either very or somewhat satisfied with the program. One participant was somewhat dissatisfied with the equipment that qualifies for the program, however, the respondent did not elaborate on the specific aspect of the equipment offered they were dissatisfied with. Figure 3-16 Overall Program Satisfaction Energy Efficiency Programs 109

113 PSO High Performance Business Program Conclusions and Recommendations Custom and Standard Components The following summarizes the key findings from the review of changes made to the custom and standard components of the High Performance Business Program. Staff continue to focus on achieving regional diversity of projects. Staff made additional changes to the business programs to ensure participation from customers throughout the service territory. A key change that should assist with regional diversity was the introduction of Small Business Solutions. Small Business Solutions assigns service providers to portions of the service territory and requires that the service providers meet savings targets. That said, staff noted that the high demand for program incentives from businesses in the Tulsa region is a challenge to challenge to meeting regional diversity objectives. Program staff has taken key steps toward increasing non-lighting uptake. These steps include mechanical training for service providers, decreasing lighting incentives, and developing marketing materials that focus on non-lighting measures. The new Fulcrum tool has streamlined the verification processes. By electronically capturing information and sending it to Vision, the fulcrum tool improves the efficiency of the program by reducing data entry requirements. There is potential to grow the use of the online application tool. Approximately one-third of interviewed service providers and one-quarter of participant survey respondents used the online application tool to submit their application. Staff should continue to promote this option given the potential to reduce data entry time. Most participants were satisfied with the program. Ninety-two percent of customers reported that they were satisfied with the program and only 2% were dissatisfied. Small Business Component The following summarizes the key findings of the process evaluation of the small business energy solutions component of the High Performance Business Program. Program Design Key aspects of the program design are summarized below. The program is well designed to mitigate barriers to energy efficiency in most small business types. The program provides energy assessments along with incentives that cover 70% of the measure costs. The 70% incentive is consistent with best practice design for small business programs. The program also targets Energy Efficiency Programs 110

114 PSO High Performance Business Program refrigeration measures in addition to lighting and participation of small businesses throughout the PSO service territory. Two potential barriers to participation identified by participant survey respondents include an unwillingness to participate if business owners think the retrofit will be disruptive or if they think it will not financially pay off. Currently, the program primarily relies upon one-on-one interactions with service providers to address these concerns. Assignment of service areas to service provider provides multiple benefits but success is dependent on the assigned service providers meeting their savings goals. The program assigns lighting service providers to regions of PSO s service territory to perform outreach, complete assessments, and install efficient equipment (or subcontract installation). Each contractor is assigned a budget and savings target. The intent of this design is to ensure that an effort is made to enroll customers throughout the service territory and to keep pricing of the efficiency improvements consistent. The success of achieving the objective of reaching customers throughout the service territory is dependent on the success of service providers meeting their saving goals. Service providers attainment of their savings goals varied in 2016 but increased activity at the end of the year suggests that all may attain their savings goals in Other noted benefits of the design were that it prevents customers receiving multiples sales calls and simplifies the participation process by removing the need for customers to choose between multiple offers. All interviewed service providers viewed the assignment of service areas favorably. However, two service providers stated that their assigned area was spread out and required additional travel cost to service. The comprehensiveness of the measures available through PSO s program compares well with other small business programs. The program covers the types of lighting and refrigeration measures that are likely to account for most of the energy saving opportunities in small businesses. It does not, however, cover water heating measures (e.g., low-flow pre-rinse spray valves, hot water pipe wrap), vending misers, or strip curtains. Additionally, other measure comprehensiveness issues identified by service providers include the following: o Service providers noted that linear LEDs that work with existing fluorescent ballasts were not included in the program and that these are often a good fit for customers. The program now includes these lamps in 2017 and requires that the ballasts are replaced when the LEDs are installed. o Service providers noted that 44W and 42W 8 foot linear fluorescents were not included in the program. These lamps can be installed if described as Energy Efficiency Programs 111

115 PSO High Performance Business Program a 60W lamp but the energy use difference between the 60W lamp and the actual wattage negatively affects program savings. An area for future program development that was noted by one of the interviewed service providers was to add on-bill financing of projects through the program. The service provider noted that this service helps small businesses cover the first cost of the equipment, but also enhances the credibility of the offer by linking the program with the utility. Similarly, zero-percent financing was identified as a best practice and can be combined with lower incentive levels to effectively encourage efficiency projects in small businesses. Program Administration Key findings regarding program administration are: The paperless process used in the program contributes to the efficiency of the process by reducing data entry needs and transcription errors. The Audit Direct Install (ADI) software is used by service providers to complete assessments, develop proposals, and submit project information. The ADI system allows for standardized audits and includes measure cost information for calculation of customer costs, incentives, as well as project savings and payback. The system also utilizes data on eligible customers provided by PSO to enable contractors to qualify and target customers. Service providers generally thought that ADI was easy to use though one service provider commented that they do not receive a notification through ADI or an if program staff reject a project proposal. Fulcrum is used to capture information during verification visits, including discrepancies from the reported project, notes, and images. Both ADI and Fulcrum interface with the Vision system to track projects completed through the program. Program staffing is sufficient. The key program staff are the PSO program manager responsible for overall oversite of the implementation contractor and acts as the main point of contact for the implementation contractor, and the implementer s program manager responsible for day-to-day oversight of the program and management of the program service providers, although both the PSO and the implementer managers interact with service providers. Service providers are subcontracted to the implementer and play a key role in the delivery of the program. Service providers are assigned budgets, energy savings goals, and cost targets. Three of the lighting service providers subcontract with contractors to complete the measure installations. These services providers are responsible for overseeing the installation work completed. Program Implementation The following summarizes key findings regarding program implementation and delivery. Energy Efficiency Programs 112

116 PSO High Performance Business Program The program marketing and outreach approach is appropriate for the target market. Direct outreach by service providers is the approach that is most likely to be effective in reaching small business customers. Service providers reported engaging in direct outreach either through door-to-door canvassing or through scheduling appointments in advance of site visits. Program sponsored discounts were the primary selling point noted by service providers; secondary selling points included project energy savings, PSO s sponsorship, and the expertise of the firm completing the assessment. Sixty-one percent of participant survey respondents reported that they learned of the program from a program energy consultant (i.e., service provider). Additionally, program staff support contractors marketing efforts through the development of marketing collateral for use with their customer contacts. Service providers reported using these materials and 70% of program participants reported using the materials. Eighty-five percent of customers who viewed the materials stated that they were somewhat or very influential in their decision to participate. Program staff have presented to chambers of commerce and completed direct postal and campaigns to inform small businesses of the opportunity to further increase program awareness. Staff reported that the campaign, which was targeted towards customers that were most likely to complete a project, was unsuccessful and that few s were opened and that it did not generate any projects. None of the survey respondents reported that they first learned of the program by or from the direct marketing campaign. One customer reported learning of the program at a chamber of commerce meeting. Service provider training has been sufficient to support program processes. All service providers reported that the program provided training on the use of the ADI software all thought this training was sufficient and none identified additional training needs. Additionally, service providers provided a favorable assessment of the software. The ADI software supports providing proposals to customers during the site visits. However, service providers generally review the assessment before providing customers with results and a proposal. Service providers reported delivery times for the results and proposal ranged from the same day of the assessment to within a few days. Most program participants (95% +) reported that the service provider discussed the amount of rebate they would receive, the annual electricity cost savings, and the project payback with them. Interviews with service providers suggest that they are generally collaborating with one another and making referrals as appropriate. However, a few issues were noted: Energy Efficiency Programs 113

117 PSO High Performance Business Program o One lighting service provider reported that they did not know who the refrigeration contact was. This statement likely reflected the withdrawal of the refrigeration service provider firm s contact during the program year, which has since been corrected. o Another lighting service provider stated that they had made several referrals to other lighting contractors that were not followed up on. o Program staff also noted that in some cases service providers were providing services to customers outside of their assigned service territory instead of making referrals. Overall, staff report that these issues were generally resolved by the end of the program year. Verification and quality control processes are sufficient to mitigate evaluation risk. Ten percent of projects completed by program service providers are subject to quality control verification visits. Staff reported that during the first few months of program operations, most projects were inspected. Additionally, PSO staff attended inspection visits and were satisfied with the procedures used by implementation staff. Overall, the rate of verification visits and the data collected during visits are sufficient to support quality program delivery. Additionally, 50% of customers reported that program staff inspected the work performed and all were very satisfied with the inspection. All surveyed program participants were satisfied or very satisfied with the program overall. Most participants were also satisfied with the services provided by the energy consultant, although two participants stated that they were dissatisfied with the energy consultant. One participant was dissatisfied because materials were left on site and the other was dissatisfied because they did not like the appearance of the lighting fixtures. Nearly all customers were satisfied with the equipment installation process and the one customer was somewhat dissatisfied with the experience did not elaborate on why. Participant survey responses regarding concerns about participating suggest that small businesses may not want to participate if they think the retrofit will be disruptive or if they think it will not financially pay off. The following recommendations are offered for continued improvement of the High Performance Business Program: Custom and Standard Components Include instructions in the Retrofit Lighting Rebate Application to submit a completed Retrofit Lighting Worksheet and list it under the application checklist. The application does not state that this is required and one customer Energy Efficiency Programs 114

118 PSO High Performance Business Program noted that it was not clear that this needed to be submitted along with applications for rebates of more than $1,500. Small Business Component Emphasize the Small Impact on Business Operations in Outreach and Marketing Materials. Several program participants stated that they were initially concerned about participating because it would be disruptive to their business. Staff should discuss project timelines and time commitments at outreach events, in marketing materials, and sales pitches made by service providers. Additionally, case studies featuring participant testimonials of the minimal business disruptions may also be useful in alleviating this potential barrier to participation. Use Propensity Analysis to Target Service Provider Outreach. Staff reported that the campaign based on the propensity to participate analysis was ineffective because few customers read the . However, this information may be of benefit to service provider outreach efforts. Consider Modifications to ADI to Notify Service Providers of Project Rejections. One service provider noted that they had had projects rejected by staff but were not aware of this. Staff should consider modifying ADI to notify contractors when a status change occurs for a project. Continue to Develop Collaboration between Service Providers. A few issues with the referral process were noted during interviews with service providers and program staff, namely, that referrals were not happening consistently and that service providers receiving referrals were reportedly not following up on them. While many of these issues have been resolved, staff should continue to encourage collaboration among service providers. Add Lower Wattage 8ft Lamps. Two service providers noted that 42W and 44W 8 foot fluorescent lamps were not included in the measure list. When installing these lamps, contractors selected 60W lamps, resulting in a loss of energy savings and making it more difficult to adhere to the $0.295 per kwh saved target. Consider Adding Vending Machine Controls and Water Heating Measures. While the program provides incentives for the refrigeration and lighting equipment likely to account for most program savings, additional measures to consider adding that are covered in other programs are vending machine controls and water heater measures (low-flow devices and hot water pipe wrap). Consider Adding On-Bill Financing or Other Interest-Free Financing for Future Program Development. Providing interest-free financing is considered a best practice for small business programs and one of the interviewed service providers stated that on-bill financing can strengthen the association with the Energy Efficiency Programs 115

119 PSO High Performance Business Program utility in a way that encourages customers to complete projects. Providing financing options may also enable the program to efficient fund projects at organizations with fewer hours of operations such as churches Program Level Energy and Demand Impacts The High Performance Business program components combined resulted in verified gross annual energy savings are estimated at 53,460,025 kwh (a realization rate of 103% when compared to reported savings). Verified gross peak demand reductions are estimated at 11,021 kw representing a realization rate of 107%. Program level reported and verified energy savings are shown in Table Program level reported and verified peak demand reductions are shown in Table Table 3-52: Summary of Program Level Energy Impacts Program Reported Gross kwh Savings Verified Gross kwh Savings Realization Rate Net-to-Gross Ratio Verified Net kwh Savings Custom and Prescriptive 47,878,154 49,036, % 95.1% 46,609,598 Small Business Energy Solutions 4,054,170 4,423, % 99.5% 4,399,600 Total 51,932,324 53,460, % 95.4% 51,009,197 Table 3-53: Summary of Program Level Demand Impacts Program Reported Gross kwh Savings Verified Gross kwh Savings Realization Rate Net-to-Gross Ratio Verified Net kwh Savings Custom and Prescriptive 9,259 9, % 94.5% 9,403 Small Business Energy Solutions 1,091 1,074 98% 99.5% 1,068 Total 10,350 11, % 95.0% 10, Planned Program Changes The following changes are planned or implemented for the 2017 program year. Energy Efficiency Programs 116

120 PSO High Performance Business Program LED measure costs may be dropped to reflect changes in market prices. TLED lamps were added to SBES. Incentives for LED lamps are now 50% of the measure cost. LED lamps that work with existing fluorescent ballasts are now allowed. For these retrofits, the contractor must replace the ballast as well. Energy Efficiency Programs 117

121 PSO Home Weatherization Program 3.2 Home Weatherization Program Program Overview The Home Weatherization program targets limited income residential customers with total annual household income at or below 200% of federal poverty level. The federal poverty level as defined by the Federal Register increases as family size increases. However, households with annual income of less than $45,000 are eligible for the program regardless of family size. Only single-family homes are serviced through the program. Qualifying homeowners are eligible for a free energy assessment through the program. Once the energy assessment is complete, PSO provides contractor recommended improvements at no cost to the participant. Measures installed through the program include: Infiltration reduction measures, Attic insulation Duct sealing Water heater jackets Water heater pipe insulation LED light bulbs Health and safety measures PSO worked with Titan ES and Rebuilding Tulsa Together to implement the Home Weatherization program. Overall, 2,157 households were serviced in The vast majority of these households were serviced by Titan ES (2,093). Actual program spending in 2016 was nearly identical to projected spending, reflecting the fact that the program was ramped down toward the end of the year as the allocated budget was expended. Verified energy savings were nearly identical to reported values. PY2016 performance metrics are summarized in Table Energy Efficiency Programs 118

122 PSO Home Weatherization Program Table 3-54: Performance Metrics Home Weatherization Program Metric PY2016 Number of Customers 2,157 Budgeted Expenditures $3,876,187 Actual Expenditures $3,805,765 Energy Impacts (kwh) Projected Energy Savings 3,794,338 Reported Energy Savings 5,726,357 Gross Verified Energy Savings 5,537,367 Net Verified Energy Savings 5,537,367 Peak Demand Impacts (kw) Projected Peak Demand Savings 1,202 Reported Peak Demand Savings 1,542 Gross Verified Peak Demand Savings 1,510 Net Verified Peak Demand Savings 1,510 Benefit / Cost Ratios Total Resource Cost Test Ratio 2.36 Utility Cost Test Ratio EM&V Methodologies This section provides a brief overview of the data collection activities, gross and net impact calculation methodologies, and process evaluation activities that ADM employed in the evaluation of the Home Weatherization program. Data Collection The primary data collection for the evaluation consisted of a participant telephone survey to verify program participation and in-home inspections to verify installation of the claimed energy efficiency measures. A total of 47 homes were visited independently to verify all measures were installed to program standards. In total, 183 participants were surveyed by telephone. In-depth interviews with PSO program staff were also conducted to gain insight for the process evaluation. ADM completed Energy Efficiency Programs 119

123 PSO Home Weatherization Program interviews with the PSO Program Coordinator who manages the program during August of 2016 and again in January of Interviews were also completed in September 2016 with staff at Titan ES, a firm that completes most program assessments and installation work, and with the Chief Executive Officer of Rebuilding Together Tulsa (RTT), a nonprofit that PSO partners with to improve the efficiency of homes in Tulsa. Table 3-55 below summarizes the data collection activities and sample sizes. Table 3-55: Sample Sizes for Data Collection Efforts Data Collection Activity Achieved Sample Size On-site Verification Visits 47 Customer Survey 183 In-Depth Interviews with Program Staff 2 In-Depth Interviews with Implementation Staff 2 In addition to the primary data collection efforts, ADM reviewed detailed program tracking data and documentation maintained in the VisionDSM database and accompanying SSRS reporting system. The tracking data was reviewed and screened for duplicate entries, correct climate zone characterization, and potential inconsistencies. Gross Impact Methodologies The methodology used to calculate energy and demand impacts consisted of verifying measure installation for a sample of program participants, reviewing deemed savings estimates for each measure as described in the Oklahoma Deemed Savings Documents, and reviewing the program tracking data to assure that deemed savings were applied appropriately. The application of deemed savings in the tracking data was reviewed for a census of homes and measures. Verification of measure installation was conducted through telephone interviews with program participants and on-site visits to participating homes. The telephone survey included questions to verify: Whether the participant indeed had their home serviced through the program, The measures claimed to be installed in the tracking database matched participant responses, and Energy Efficiency Programs 120

124 PSO Home Weatherization Program The heating, cooling, and water heating systems for respondents matched records in the tracking database. During the site visits, ADM field staff verified that the claimed energy efficiency measures were installed, and recorded key inputs to savings calculations such as R- value of installed insulation and quantity of LED bulbs replaced. Data collected through these activities was used to develop measure level verification rates, which were then used to adjust the deemed savings estimates where necessary. This approach to evaluating the reported savings values provides verification of measure installation and correct application of the deemed savings values. The Oklahoma Deemed Savings Documents have been previously approved by the Oklahoma Corporation Commission for the estimation of energy and peak demand savings resulting from measures installed through the Home Weatherization Program. As a result, ADM s evaluation does not include a specific assessment of the deemed savings validity for PY2016 participants. Net-to-Gross Estimation The Home Weatherization program specifically targets customers with limited income, providing all services at no cost to the customer. It is likely that participating customers would not have funded the installed energy efficiency measures on their own. As a result, ADM assumed a net-to-gross ratio of 100%. Process Evaluation Activities The purpose of the process evaluation was to examine program operations and results throughout the program operating year and to identify potential areas that may need to be addressed to improve the effectiveness or efficiency of the program. The process evaluation focused on a review of the program design, administration, and implementation. During the evaluation, data and information from several sources are analyzed to achieve the stated research objectives. The Evaluators completed reviews of program documentation, interviews with the PSO program coordinator and staff at Titan ES and Rebuilding Tulsa Together, and surveys of participating customers to develop findings pertaining to the program s design and operational procedures Impact Evaluation Findings As described in Section 3.2.2, the gross impact analysis consisted of verifying measure installation and checking the program tracking data to ensure that deemed savings algorithms were appropriately applied. In-Service Rates (ISR) for each measure type were developed based on the findings from the participant telephone survey and on-site visits. Findings from these activities are summarized below for each measure type. Energy Efficiency Programs 121

125 PSO Home Weatherization Program Infiltration Reduction: The Oklahoma Deemed Savings Documents specify the following formula for use in calculating energy and demand impacts of infiltration reduction measures. The air infiltration reduction estimate in CFM is obtained through blower door testing performed by the program contractor for each home serviced. Only homes with electric cooling systems are eligible for the measure (central AC or room AC). Where: Deemed Savings (kwh/kw) = CFM 50 x V CFM 50 = Air infiltration reduction in cubic feet per minute at 50 Pascal V = the value in the following table that corresponds to the climate zone and heating and cooling type. Table 3-56: Deemed Savings Table Infiltration Reduction kwh Savings Impact per CFM50 Reduction Zone AC/Gas Heat kwh Gas Heat (no AC) kwh Gas Heat (no AC) Therms AC/Electric Resistance kwh Heat Pump kwh AC Peak Savings kw Zone Zone 8A Zone 8B Zone Zone Of the 183 respondents to the participant survey, 148 represented homes where infiltration reduction measures were installed. 146 of these respondents indicated that they had indeed had air sealing measures installed. The on-site verification visits included 41 homes where infiltration savings had been claimed. Additionally, ADM s onsite verification work found evidence of air sealing in all the visited homes where infiltration reduction savings had been claimed. Based on these findings, an ISR of 100% was applied. ADM reviewed the deemed savings values used for each home and determined they were applied correctly resulting in a realization rate of 100% for energy savings (kwh) and 100% for peak demand reduction (kw). Energy Efficiency Programs 122

126 PSO Home Weatherization Program Duct Sealing: The algorithms used for estimating reported savings for duct sealing measures come from the 2013 update to the Oklahoma Deemed Savings Documents. These algorithms are shown below for heating and cooling kwh savings. Cooling Savings: Where: DL pre = DL post = EFLH c = h = ρ out = ρ in = Pre-improvement duct leakage at 25 Pa (ft3/min) Post-improvement duct leakage at 25 Pa (ft3/min) Equivalent full load cooling hours, from table Outdoor/Indoor seasonal specific enthalpy (Btu/lb), from table Density of outdoor air (lb/ft3), from table Savings Density of conditioned air at 75 F = (lb/ft3) (default) 60 = Constant to convert from minutes to hours 1,000 = Constant to convert from W to kw SEER = Seasonal Energy Efficiency Ratio of existing system (Btu/W hr) = 13 (default) Heating Savings (Heat Pump): Where: DL pre = DL post = Pre-improvement duct leakage at 25 Pa (ft3/min); assume 35% of fan flow if duct system cannot be pressurized Post-improvement duct leakage at 25 Pa (ft3/min) 60 = Constant to convert from minutes to hours 0.77= Factor to correlate design load hours to EFLH under actual working conditions (to account for the fact that people do not always operate their heating system when the outside temperature is less than 65 F) HDD= Heating Degree Days, from table in Deemed Savings Document 1,000 = Constant to convert from W to kw Energy Efficiency Programs 123

127 PSO Home Weatherization Program HSPF = Heating Seasonal Performance Factor of existing system (Btu/W hr) = 7.7 (default) Heating Savings (Electric Resistance): Where: DL pre = DL post = Pre-improvement duct leakage at 25 Pa (ft3/min); assume 35% of fan flow if duct system cannot be pressurized Post-improvement duct leakage at 25 Pa (ft3/min) 60 = Constant to convert from minutes to hours 0.77= Factor to correlate design load hours to EFLH under actual working conditions (to account for the fact that people do not always operate their heating system when the outside temperature is less than 65 F) HDD= Heating Degree Days, from table in Deemed Savings Document 24 = Constant to convert from days to hours = Volumetric heat capacity of air (Btu/ft3 F) = Constant to convert from Btu to kwh A total of 103 survey respondents represented homes where duct sealing was reported to have occurred. 11, of these respondents, indicated that they had not had duct sealing measures installed. ADM verified all 11 homes were serviced on Vision DSM. This measure is difficult for the customer to confirm occurrence. The duct work is typically in the attic covered by blown in insulation. The on-site verification visits included 35 homes where duct sealing savings had been claimed. One on-site verification visit did not confirm the occurrence of duct sealing. An ISR of 99% was applied for duct sealing. The algorithms for energy savings and peak demand were applied correctly based on the improvements in duct leakage reported for each home. As a result, verified savings for duct sealing measures were almost identical to reported savings giving a realization rate of 99% for both energy savings (kwh) and peak demand reduction (kw). Ceiling Insulation: Reported energy and demand impacts from ceiling insulation are calculated deemed savings values per square foot installed. The deemed savings are also dependent on climate zone, heating and cooling equipment and existing level of insulation. To qualify for the deemed savings, the home must have existing insulation with an R-22 or less and must be brought up to the level of R-38 or greater. The Oklahoma Deemed Savings Documents provides tables for each climate zone that Energy Efficiency Programs 124

128 PSO Home Weatherization Program present the kw and kwh savings per square foot of insulation improved. Table 3-57 below is an example for climate zone 8B. Table 3-57: Example Deemed Savings Table - Ceiling Insulation Zone 8B Ceiling Insulation Deemed Savings Ceiling Insulation Base R-value AC/Gas Heat kwh Gas Heat (no AC) kwh Gas Heat Therms AC/Electric Resistance kwh Heat Pump kwh Summer Peak kw Savings (per sq. ft.) (per sq. ft.) (per sq. ft.) (per sq. ft.) (per sq. ft.) (per sq. ft.) R-0 to R R-5 to R R-9 to R R-15 to R Respondents to the participant survey represented 156 homes where ceiling insulation was claimed to have been installed. All respondents but one verified insulation was installed. ADM verified the home was serviced on Vision DSM. Similarly, evidence of additional ceiling insulation was verified for all projects where on-site verification took place. 24 During the on-site verification visits, the pre-existing and new insulation levels along with square feet installed were measured, where possible. There were no instances where the insulation measurements did not meet the pre- and post-r-value requirements of the Deemed Savings Documents. Approximate square footage measurements also closely matched reported values. As a result, an ISR of 100% was applied for attic insulation. The tracking database review showed that for the vast majority of homes deemed savings values for insulation were applied correctly. However, ADM discovered that in a few specific scenarios the deemed savings values were incorrectly applied. These instances typically involved the application of deemed savings values from mismatched climate zone/existing R-value/HVAC system combinations. Overall, these mismatches were minor and do not point to systematic errors in the application of deemed savings for the program. 24 There were 42 projects that included attic insulation that received on-site verification. For all 42 of these sites, ADM could access the attic and measure pre- and post- attic insulation levels. Energy Efficiency Programs 125

129 PSO Home Weatherization Program The issues identified above had little or no impact on verified kwh savings resulting in a realization rate of 100%. For the reasons described above, the peak demand kw savings was slightly higher than expected giving a realization rate of 101%. Pipe Insulation and Water Heater Jackets: Installation of pipe insulation and/or water heater jackets required electric water heating in serviced homes. As such, the number of recipients was significantly smaller than other program measures. ADM completed 16 verification surveys and five on-site inspections for customers that had one or both measures installed in their homes. All 16 survey respondents indicated that the measures were indeed installed. On-site verifications provided evidence of pipe insulation and water heater jacket installation, with electric water heating confirmed. It is possible that survey respondents may have had trouble correctly identifying their water heating fuel source. Requiring picture evidence of electric water heating for all homes that receive pipe insulation and water heater jackets may be advisable for the weatherization crews. In all sampled cases, the water heater pipe insulation was verified as installed resulting in an ISR 100%. For water heater jackets, a review of the tracking system showed that conservative assumptions were used to inform the use of the deemed savings. Savings values corresponding to 2 thick jackets on 40 gallon tanks were used for all sites. The deemed saving for this measure depend on 1) water heating fuel source, 2) insulation thickness and 3) water heater tank size. There could be additional savings for homes that had larger water heaters, but this information is not recorded during installation. Table 3-58 below shows the deemed savings for water heater jackets installed on electric water heaters. Table 3-58: Deemed Savings Electric Water Heater Jacket Electric Approximate Tank Size Energy Savings (kwh) Peak Savings (kw) " WHJ savings kwh " WHJ savings kwh The verified savings for water heater jackets resulted in a realization rate equal to 100%, for both energy savings (kwh) and peak demand reduction (kw). The verified savings for pipe insulation resulted in a realization rate equal to 100%, for both energy savings (kwh) and peak demand reduction (kw). Energy Efficiency Programs 126

130 PSO Home Weatherization Program LED Light Bulbs: The algorithms used for estimating reported savings for Omni- Directional LEDs partly come from the 2013 update to the Oklahoma Deemed Savings Documents. These algorithms are shown below. kwh = ((Wbase Wpost)/1000) x Hours x ISR Where: W base = W post = Hours = ISR = 60W, Based on wattage equivalent of the lumen output of the purchased LED and pre EISA 2007 Baseline 12W, Based on wattage equivalent of the lumen output of the purchased LED and pre EISA 2007 Baseline 1023, Average hours of use per year 0.97, In service rate The algorithms used for calculating the verified savings for Omni-Directional LEDs are sourced from the 2013 update to the Oklahoma Deemed Savings Documents. The algorithms are shown below. kwh = ((Wbase Wpost)/1000) x Hours x ISR x IEFE Where: W base = W post = Hours = ISR = IEF E = 43W, Based on wattage equivalent of the lumen output of the purchased LED and EISA July 2014 Baseline 9.2W, Wattage of LED installed 1023, Average hours of use per year 0.91, In service rate Interactive Effects Factor to account for cooling energy savings and heating energy penalties, see the following table Table 3-59:: Omni-Directional LEDs Interactive Effects Factor for Cooling Energy Savings & Heating Energy Penalties Heating Type Interactive Effects Factor (IEFE) Gas 1.15 Electric Resistance 0.44 Heat Pump 0.84 Unknown Fuel Type 0.96 Energy Efficiency Programs 127

131 PSO Home Weatherization Program A total of 99 survey respondents represented homes where LEDs installation was reported to have occurred. Homes received varying quantities of LEDs, and represent a total of 615 LEDs. 52, of these LEDs, either failed or were not installed according to respondents. The on-site verification visits included 30 homes where LEDs savings had been claimed and represented a total of 172 LEDs claimed. 157, of these LEDs, were verified in service. Based on these findings, an ISR of 91% was applied. The algorithms for reported energy savings and peak demand were applied incorrectly based on the 2013 update to the Oklahoma Deemed Savings Documents. The algorithms for reported energy savings did not utilize Interactive Effects Factor (IEFE) and utilized an outdated baseline (pre-2007 EISA) of 60W. This coupled with an ISR of 91%, resulted in realization rate of 62% for energy savings (kwh) and 58% peak demand reduction (kw). Program Level Findings Applying the adjustments detailed by measure type above results in a small decrease in estimated energy savings and a small decrease in peak demand reductions overall, as shown in Table Overall, ADM s evaluation findings suggest that measure installations are correctly recorded using the program tracking system and that the approved deemed savings values/algorithms are correctly applied in most instances. Table 3-60: Reported and Verified kwh and Peak kw Measure Reported Energy Savings (kwh) Reported Peak Demand Savings (kw) Verified Gross Energy Savings (kwh) Verified Gross Peak Demand Savings (kw) kwh Realization Rate kw Realization Rate Attic Insulation 1,926, ,934, % 101% Duct Sealing 1,865, ,847, % 99% Air Infiltration 1,452, ,450, % 100% Water Heater Jacket 13, , % 100% Pipe Insulation 6, , % 100% LED Light Bulbs 462, , % 58% Total 5,726,357 1,542 5,537,367 1,510 97% 98% Energy Efficiency Programs 128

132 PSO Home Weatherization Program The savings estimates in the above table result in a gross annual kwh savings realization rate of 97% and a peak kw reduction realization rate of 98%. The verified gross savings estimates reflect confidence that the energy and demand impacts reported for the program were largely consistent with the deemed savings values for each measure type. The deemed savings algorithms are designed to reflect average energy savings for installed measures, and may not reflect actual savings at any given participant meter. For homes where multiple measures affecting the same end use are installed, the deemed savings may not capture the interactive effects between the measures. The Oklahoma Deemed Savings Documents have been previously approved by the Oklahoma Corporation Commission for the estimation of energy and peak demand savings attributable to the measures installed through the Home Weatherization Program. As a result, ADM s evaluation does not include a specific assessment of the deemed savings validity for PY2016 participants Process Evaluation Findings This section presents the findings from the review of program design and operational processes. Participant Survey Results A telephone survey was conducted to collect data about participant experiences with the PSO Home Weatherization program. The surveys were conducted in November of In total, 183 program participants responded to the telephone survey. The following sections discuss key findings from the analysis of survey responses, drawing comparisons between the evaluation results of the current and prior program years when appropriate. As with the prior program year, respondents were very satisfied with their experience with the Home Weatherization program overall, and only a small portion of participants were dissatisfied with any aspect of the program. Participant Characteristics and Demographics This section presents the results of a series of survey questions related to participants space and water heating fuel type and the number of residents in participating households. Figure 3-17 displays the household heating type and as shown, a third of customers have electric heating and represent the households with the greatest electricity saving potential. The share of households demonstrates the value in continuing the partnership with Oklahoma Natural Gas to provide a joint fuel-type program. Energy Efficiency Programs 129

133 PSO Home Weatherization Program Figure 3-17: Survey Respondent Residence Heating Type Similarly, Figure 3-18 displays the share of households with electric and natural gas water heating. Twenty-eight percent of households have electric water heating, which suggests some potential for savings from water heating measures such as low-flow devices that are not currently included in the program. This share of homes with heating fuel type is comparable the share reported in program tracking data (25%). Figure 3-18: Survey Respondent Residence Water Heating Type Energy Efficiency Programs 130

134 PSO Home Weatherization Program The number of residents in the household that received services is displayed in Table 3-61 as shown, the homes served skewed towards smaller households with one or two residents in them. Table 3-61: Number of Residents in Respondent Homes Number of Residents Percentage of Respondents (n = 148) One person 44% Two people 29% Three people 14% Four people 7% More than 4 people 6% Nearly all respondents (93%) reported that they own the home that received services through the program. Figure 3-19 Home Ownership Participant Motivations and Awareness As shown in Table 3-62, respondents most commonly reported that they learned of the program from the informational brochure they received in the mail, with 42% of respondents reporting Energy Efficiency Programs 131

135 PSO Home Weatherization Program learning of the program through this source. This differs from the results of the prior two years of program evaluation, where most respondents reported hearing about the program through indirect word-of-mouth marketing. The informational brochure that customers referred to may have been the bill inserts that were sent in two campaigns by program staff. Consistent with prior years, however, word-of-mouth continued to be an important source of program awareness, with 36% of respondents reporting hearing about the program this way. Fourteen percent of respondents said that they became aware of the program through a PSO representative, and 3% first learned of the program through the PSO website. Table 3-62: How Participants Learned of the Program Response Percentage of Respondents (n = 183) How were you first informed about the Home Weatherization Program sponsored by PSO? Received informational brochure in the mail Heard about it from friends and/or colleagues Approached by a program representative 42% 36% 14% Utility website 3% Other 8% Don t know 2% Table 3-63 Most respondents (92%) stated that they participated in the program to save money on their electric energy bills. Additionally, 39% of respondents reported participating because the services were provided free of charge and 20% reported participating due to environmental concerns. Other stated reasons for participating included specific concerns about the home such as a need for additional air sealing, insulation, or home repairs. The priorities identified by respondents differed from 2015, specifically, the share of respondents who stated that they were motivated to save money on their energy bills increased from 64% in 2015 to 92% in Energy Efficiency Programs 132

136 PSO Home Weatherization Program Table 3-63: Motivations for Participating in the Program Response Percentage of Respondents (n = 179) What motivated you to participate in this program? To save money on your electric energy bill 92% Environmental concerns 20% Services are free 39% Don't know 1% Other 11% *Respondents could provide multiple responses. The percentages shown are percentages of respondents rather than percentages of responses. Thus, the total exceeds 100%. Participant Satisfaction Survey respondents rated their satisfaction with selected aspects of the 2016 Home Weatherization program on a scale of 1 to 5, with 1 representing very dissatisfied and 5 representing very satisfied. Measure Satisfaction Respondents rated their satisfaction with each measure installed through the program in terms of the effect of the measure on increasing home comfort and reducing home energy use. As shown in Figure 3-20, 92% of respondents reported being very satisfied or satisfied with the energy savings resulting from their new insulation. One percent of respondents reported being dissatisfied with the insulation energy savings. With regard to home comfort, 90% of respondents reported being very satisfied or satisfied with the insulation and one percent of respondents indicated that they were dissatisfied. Energy Efficiency Programs 133

137 PSO Home Weatherization Program Figure 3-20: Customer Satisfaction with Insulation (n=155) As shown in Figure 3-21, 89% of respondents reported being very satisfied or satisfied with the energy savings resulting from the duct sealing work. One percent of respondents reported being dissatisfied with the duct sealing energy savings. Ninetythree percent of respondents reported being satisfied or very satisfied with the duct sealing s effect on home comfort. Only 1% of respondents stated that they were very dissatisfied with the effect on home comfort. Figure 3-21: Customer Satisfaction with Duct Sealing (n=92) As shown in Figure 3-22, 89% of respondents were satisfied or very satisfied with the energy savings resulting from the air sealing, and only two percent of respondents reported being dissatisfied or very dissatisfied. Additionally, 92% of respondents reported being satisfied or very satisfied with the comfort improvement gained from the Energy Efficiency Programs 134

138 PSO Home Weatherization Program air sealing work, and only two percent of respondents stated that they were dissatisfied or very dissatisfied. Figure 3-22: Customer Satisfaction with Air Sealing (n=146) As shown in Figure % of respondents reported being very satisfied or satisfied with the energy savings resulting from the installation of LEDs. Two percent of respondents reported being dissatisfied or very dissatisfied with their LED energy savings. Eighty-two percent of respondents reported being satisfied or very satisfied with the LED s effect on home comfort. Three percent of respondents stated that they were very dissatisfied with the effect on home comfort. The reasons given for dissatisfaction with the LEDs were that they were too bright, that they did not work well for reading, that the bulbs stopped working, and that the work crew removed more bulbs than they replaced. Energy Efficiency Programs 135

139 PSO Home Weatherization Program Figure 3-23: Customer Satisfaction with LED Fixtures (n=99) Sixteen survey respondents received water heater jackets or pipe wrap through the Home Weatherization Program. Fifteen respondents said that they were satisfied or very satisfied with the effect that the heat jacket or pipe wrap had on their energy usage. The remaining respondent was neutral. Thirteen respondents were satisfied or very satisfied with the effect that this measure had on the comfort of their home; two respondents were neutral and one respondent did not provide a satisfaction level. No respondent reported being dissatisfied or very dissatisfied with the measure. Because almost all customers received multiple measures through the program, isolating the energy savings or comfort improvement resulting from an individual measure is likely difficult and observing the satisfaction ratings for all measures in aggregate may be more reflective of overall participant sentiment. Generally, respondents reported being highly satisfied with both the energy savings and comfort effects that have occurred since they received the insulation, duct sealing, air sealing, and water heater measures. In addition to satisfaction ratings for individual measure types, respondents rated their satisfaction with various elements of the program experience. The distribution of participant satisfaction findings for these selected program elements is displayed below in Figure As with the prior program year, satisfaction ratings for each program element were high, and there were few instances of dissatisfaction in total. Compared to the 2015 program evaluation, the current satisfaction ratings are slightly higher; dissatisfaction has remained consistently low. Energy Efficiency Programs 136

140 PSO Home Weatherization Program Figure 3-24: Customer Satisfaction with Selected Program Elements (n=180) Ninety-four percent of respondents reported being satisfied or very satisfied with the overall program; a slight increase from the 2015 program evaluation. Similarly, 93% were satisfied or very satisfied with the quality of work conducted by the energy auditor. Eighty-eight percent were satisfied or very satisfied with the information provided by the energy auditor. Savings on the monthly energy bill received the lowest level of satisfaction, with 76% saying that they were satisfied or very satisfied with their savings. However, this satisfaction level is still quite high and is higher than in A few customers who expressed dissatisfaction with one or more program element explained that something had been done incorrectly during the audit or installation process. Many respondents provided open-ended commentary indicating that they highly valued the services provided by the Home Weatherization Program and that their program experience had been very positive. Commentary of this nature included: Very good service, really nice to have something like that, especially for a low-income person. It helped tremendously. I was super impressed with the service, with the people, with whom I came [into] contact, and I couldn t believe it was all for free. I have been very pleased with it and I can really tell the difference from this summer. It was a great thing that saved me money on my electric bill. Additionally, four respondents noted that they did not receive a $20 gift certificate after participating in the referral program. Although the responses were not entirely clear, it appears that some may not have notified Titan ES about the referral. Energy Efficiency Programs 137

141 PSO Home Weatherization Program As with prior program years, the results from the participant satisfaction portion of the survey indicate that participants are very satisfied with their program experiences and that they highly value the information and services offered by the program. Program Design and Operations The following section provides an overview of the Home Weatherization Program design and operations. This review was developed through in-depth discussions with program staff and a review of documents used to guide implementation. ADM completed interviews with the PSO Program Coordinator who manages the program during August of 2016 and again in January of Interviews were also completed in September 2016 with staff at Titan ES, a firm that completes most program assessments and installation work, and with the Chief Executive Officer of Rebuilding Together Tulsa (RTT), a nonprofit that PSO partners with to improve the efficiency of homes in Tulsa. During the interviews, respondents discussed their perspective on the program design and operations to enable the evaluation team to assess how the program is meeting its objectives and if operational improvements can be made to better serve customers while delivering cost-effective energy savings. Program Design The Weatherization Program provides no-cost energy efficiency improvements to PSO customers with household incomes of $45,000 or less a year. As in the prior program cycle, PSO partners with two organizations to deliver the efficiency improvements, Titan ES and RTT. Titan ES provides energy audits, customer education, and installation of efficient equipment. Titan ES delivers services to single family, duplex, triplex quadplex homes that are either leased or owner occupied. RTT is a Tulsa based non-profit organization that provides a variety of home improvement services for low-income homeowners. The services provided by RTT include program sponsored energy efficiency improvements as well as other repairs such as roof repairs. PSO staff reported that they would like to develop additional partnerships with community agencies providing services to low income customers but that the agencies approached have expressed little interest. The program also partners with Oklahoma National Gas (ONG) to co-fund gas and electric improvements to a small number of homes that receive energy services from both utilities. Incentive costs are split 50/50 between PSO and ONG for these jointfunded projects. Through the Home Weatherization program, participants receive diagnostic energy assessments, weatherization measures such as air sealing, insulation, and duct sealing, Energy Efficiency Programs 138

142 PSO Home Weatherization Program and direct installation of up to eight LED light bulbs in higher use areas. The project must have a savings-to-investment ratio (SIR) of 1.0 or higher, although individual measures may fall below this ratio. The Weatherization Program design is generally consistent with low-income weatherization programs. The program provides efficiency improvements at no cost to the customer and manages the efficient use of funds by qualifying projects using the SIR. One area where it differs from other programs is the use of a fixed income threshold that does not vary by household size. Program Management and Staffing The Weatherization Program is managed by a PSO Program Coordinator who also has responsibilities for oversite of the Energy Education Program. Titan ES provides program services to most homes that receive incentives through the program. Titan ES and PSO are responsible for marketing the program to customers, scheduling and completing energy assessments, and installing efficiency measures. Titan ES utilizes four work crews to complete the energy assessments and measure installations. Titan ES implements all projects joint-funded with ONG. RTT provides services for a limited number of homes in the Tulsa area. RTT works with contractors to complete the energy efficiency improvements and conduct diagnostic testing (e.g., blower door testing), but also utilizes volunteers to complete lower skilled work such as applying caulking to windows. Marketing and Outreach Most of the program marketing activity focuses on the homes serviced by Titan ES. The primary means of marketing these services is through a direct mail campaign to customers, through which customers in a specific area are sent a postcard providing information about the program. The marketing approach is to focus on a single geographic area to minimize travel time between home sites. Additionally, PSO promoted the program through bill inserts in March and the June/July period. That campaign generated some incoming calls from outside the area were Titan ES was currently delivering program services. These customers were typically added to a waitlist until the program could target services for the customer s area. However, staff said if they received a large response in an area, they would provide services to customers shortly after the request was made. As in prior years, Titan has provided a $20 gift card to any customer that referred a qualifying customer who completed a program project. RTT stated that most customers learn of the services they provide by word-of-mouth, although they also speak at neighborhood associations and get referrals from other agencies. RTT stated that generally there was strong demand for their services. Energy Efficiency Programs 139

143 PSO Home Weatherization Program Program Implementation and Participation Process The implementation and participation processes differ for Titan ES and Rebuilding Tulsa and are described separately below. Titan ES: Customers initiate participation by contacting program staff. If a renter schedules an assessment, a Titan ES crew will initially visit the site and verify that it qualifies for services. To initially qualify a home for program services, the occupant must sign an attestation that they have a household income of $45,000 or less and that the home was built before If qualified and the home is rented, written permission is also required from the property owner and can be submitted by fax machine or . Staff reported that property owners seldom refuse, but that they can be difficult to get a hold of. Once a customer is cleared to receive services, an energy assessment of the home is scheduled and completed. Titan ES utilizes four work crews based in three regions of the state but noted that these work crews typically move around and lodge overnight to perform the assessment and installation work. The energy assessment is guided by a twelve-page paper-based form that covers basic information about the premise (e.g., size), health and safety issues such as carbon monoxide testing and if the home needs attic ventilation equipment installed, and blower door and duct blasting test results. Additionally, the form includes sections for identifying if the home needs specific measures. For example, the assessor can indicate the square feet of attic insulation required or if door weather stripping is needed. Staff indicated that they have increasingly focused on cost-effective measures over the years the program has operated and noted that average project costs have decreased over the years. Additionally, a home may be disqualified if the project expenses will be very high, for example, if the blower door pre-test is very leaky. Once the assessment is completed installation work is performed was the first year that direct installation of LED light bulbs was added to the program. Titan ES procured a bulk purchase of 9.2W dimmable LED bulbs. Up to eight light bulbs may be installed in a premise in one of the following locations: Kitchen Dining room if connected to kitchen Living room Den/family room Bathrooms Work crews record the location, number installed, and wattage replaced. Up to eight incandescent light bulbs are replaced, regardless of wattage, in the qualified locations. Crews are instructed to only replace bulbs in permanently mounted fixtures. Energy Efficiency Programs 140

144 PSO Home Weatherization Program Once work is completed, post installation blower door and duct blasting testing are completed. RTT: RTT typically completes around 60 homes a year through the program and homes typically receive about $2,000 in funding for efficiency improvements. Projects are initiated by RTT when a resident applies for services. Applications are prioritized for work based on the condition of the home. If a home needs roof repair, for example, these repairs are made before the home would be eligible for program funded efficiency improvements. The project is initiated with a site review. Site reviews are typically done twice a week and 5 to 10 are typically completed. Homes are typically qualified for receiving program incentives based on the blower door test results. Implementation of the efficiency measures is done by a PSO approved contractor, although some lower skilled aspects of the work, such as caulking windows, is done with volunteer labor. Once the work is done, RTT inspects the works, collects a signed statement of completion from the homeowner, and documents the work with photographs. Reporting and Quality Assurance and Control The following describes quality assurance and control procedures used by Titan ES and RTT. Titan ES: All Titan ES crew leaders, supervisors, and senior staff receive Building Performance Institute (BPI) training. This training is provided at an Oklahoma Community Action Group facility that has mocked up homes for training on blower door testing. Once, work is completed, two staff members go through the homes and review paperwork for discrepancies between assessments and the work performed. Additionally, 25 homes per quarter receive a third-party inspection. The third-party inspector reports any issues to PSO and Titan ES and these are resolved. PSO staff noted that there have been very few issues identified through these inspections. Once projects are completed, Titan ES enters project information into the Vision data system that PSO uses and invoicing for the project is submitted to PSO. RTT: RTT utilizes PSO approved contractors to implement the energy efficiency improvements. Additionally, program staff review the work performed, take photographs of the home and have the customer attest that the work was completed. Project data is entered into the Vision database by RTT staff. The staff at RTT commented that learning the Vision system was challenging, but noted that the system is relatively easy to work with having learned how to use it. Energy Efficiency Programs 141

145 PSO Home Weatherization Program Overall, the quality control and assurance procedures seem sufficient. Titan ES staff utilized BPI trained crew leaders to guide and review work performed and a sample of homes are reviewed by a third-party on a monthly basis. While there is not a third-party inspector for RTT, the number of homes completed through this component is small in comparison to the full program activity. Additionally, RTT employs PSO approved contractors to complete project work and documents the work performed with photographs and customer attestations. Home Weatherization Conclusions and Recommendations Program Design Key aspects of the program design are summarized below. Two changes were made to the program design in The household income threshold requirement increased to $45,000 or less per year (from $35,000) and direct install LED light bulbs were included. The program is inclusive of both rental and owner occupied homes. There is a focus on health and safety issues including the installation of carbon monoxide detectors, assessments of attic ventilation requirements, and notification of lead base paint hazards. The measures implemented through the program are consistent with the types of measures offered through this type of program, although Home Weatherization does not offer low-flow devices. Staff stated that low-flow devices are problematic because residents dislike them and often call back work crews to remove them. At midyear, the program also began providing leave behind materials in the form of a 10 energy saving tips sheet. Program Administration Key findings regarding program administration are: Staffing and program administration procedures have remained consistent with prior years. The primary change for 2016 was an increase in the program budget that corresponded to the raised income qualification threshold. As in 2015, the program partnered with Oklahoma Natural Gas, which expanded the program's budget and allowed it to provide services to more homes. Program projects are tracked through PSO s Vision data system and project information is entered by Titan ES and RTT. Both Titan ES and RTT reported little current difficulty in using the data system. Titan ES assessments are paper based. There may be opportunities for enhancing program efficiency by using electronic forms. Energy Efficiency Programs 142

146 PSO Home Weatherization Program Program Implementation and Delivery The following summarizes key findings regarding program implementation and delivery. Program implementation and delivery procedures remained largely unchanged from The primary change was the addition of direct installation of 9.2 W (60W equivalent) 2700k (warm light) dimmable LED light bulbs. The program limits the installation to eight light bulbs installed in higher use areas of the home and Titan ES work crews are instructed to install the bulbs in permanent fixtures. Staff record the location where the bulb is installed and the wattage of the replaced bulb. The program will replace incandescent light bulbs of varying wattages with the 60W equivalent bulbs. This may lead to some dissatisfaction among participants who dislike the amount of light after the replacement. However, very few participants reported dissatisfaction with the replacements. Marketing and outreach remained consistent with prior years. Targeted direct mail campaigns were the primary means of reaching customers. Additionally, two bill inserts were sent to customers. Forty-two percent of respondents reported learning about the program from information provided in the mail, suggesting that the bill inserts were effective at encouraging program participation. Quality control and assurance procedures for work performed by Titan ES remained consistent with prior years and include BPI training of work crews, staff review of work performed, and third-party inspections of a sample of homes. While RTT does not include third-party inspections, the procedures in place for ensuring proper installation are sufficient given the number of homes completed by this organization. Nearly all customers reported that they were satisfied with the measures installed and the work performed. Ninety-four percent of respondents reported being satisfied or very satisfied with the overall program and 93% were satisfied or very satisfied with the work completed through the program. The following recommendations are offered for continued improvement of the program. Add additional LED types. In 2016, the program only offered direct installation of 9.2 W LED lamps (60W incandescent equivalents). While that wattage would cover most household lighting use, staff should consider adding other lamp wattages so that replacements can be matched to the existing lamp wattages found in homes. The experience of some customers may be improved if the amount of light provided by the efficient light bulbs was matched to the amount of light from the replaced light bulbs. Energy Efficiency Programs 143

147 PSO Home Weatherization Program Additionally, all LEDs are currently installed inside homes. Staff should consider installation of lamps in exterior locations such as porch lights. In particular, exterior lighting may provide an opportunity to identify opportunities to install directional LEDs that are exempt from EISA (e.g., 65W equivalent BR30 and BR 40 lamps) Planned Program Changes Overall, the Weatherization Program design and implementation remains unchanged in One development noted by staff is that the program is working with a Spanishspeaking individual to improve the programs reach with Spanish speaking households. This person will be onsite with Titan ES crews at Spanish speaking households to facilitate communication between the work crews and the resident. Energy Efficiency Programs 144

148 PSO Energy Saving Product Program 3.3 Energy Saving Products Program Program Overview PSO s Energy Saving Products (ESP) program seeks to generate energy and demand savings for residential customers through the promotion of CFLs, LEDs and room air purifiers. The purpose of this program is to provide PSO residential customers inducements for purchasing products that meet high efficiency standards. The ESP program consists of retail price discounts for qualifying CFL and LED light bulbs and room air purifiers in This component of the program uses a price mark down mechanism where participating retailers advertise and offer discounted pricing for program sponsored products. The retailers/manufacturers are then reimbursed by PSO for the difference between the discounted price and the normal retail price. This program component also included distribution of free CFLs in partnership with food banks and local food pantries within the PSO service territory during PY2016. Discounted CFL and LED bulbs, including the free CFLs distributed through local food pantries, make up the vast majority (~99.8%) of the reported energy savings for the PY2016 ESP program. The actual number of participants in the ESP lighting component of the program is unknown, as purchaser information is not tracked by participating retailers. In total, 504,758 packages of CFLs/LEDs (1,638,212 individual bulbs) were discounted through participating retailers or distributed in partnership with local food pantries. The total number of room air purifiers sold in the ESP program was 228. A total of 1,638,440 light bulbs and room air purifiers were discounted through participating retailers or distributed in partnership with local food pantries (CFLs only). Table 3-64 provides a summary of program metrics for the 2016 program year. Program costs were $3,894,963.25, while reported kwh savings exceeded program projections. Gross verified energy savings developed through ADM s impact evaluation were slightly higher than reported savings, representing a gross realization rate of 101%. Verified peak demand reduction represents 121% of reported values. Energy Efficiency Programs 145

149 PSO Energy Saving Product Program Table 3-64: Performance Metrics Energy Saving Products Program Metric PY2016 Number of Products 1,638,440 Budgeted Expenditures $4,154,950 Actual Expenditures $3,894,963 Energy Impacts (kwh) Projected Energy Savings 32,043,304 Reported Energy Savings 51,122,622 Gross Verified Energy Savings 51,793,329 Net Verified Energy Savings 32,829,760 Peak Demand Impacts (kw) Projected Peak Demand Savings 3,709 Reported Peak Demand Savings 7,161 Gross Verified Peak Demand Savings 8,653 Net Verified Peak Demand Savings 5,467 Benefit / Cost Ratios Total Resource Cost Test Ratio Utility Cost Test Ratio 4.51 The remainder of this section details the EM&V methodologies and findings for the Energy Saving Products (ESP) program EM&V Methodologies The following section details the methodologies that ADM used to verify retail sales, estimate energy and peak demand impacts, and assess the performance for the Energy Saving Products program. Data Collection Several primary and secondary data sources were used for the evaluation. Tracking data and supporting documentation for the program was obtained from the Energy Federation, Inc. (EFI) database. This tracking data was used as the basis for quantifying participation and assessing program impacts. Supplemental tracking data was provided Energy Efficiency Programs 146

150 PSO Energy Saving Product Program by CLEAResult which included the following information for each combination of retailer, model number, and discount level: Package sales per week (program sales only) Original retail price Manufacturer/Retailer sponsored discounts (if any) PSO sponsored discounts Retail price, including all discounts Number of bulbs per package Rated wattage Rated lumens Rated lifetime in hours Additional documentation including retailer agreements, retailer/manufacturer invoices, promotional event documentation and general program materials were reviewed as part of the evaluation. Primary data collection activities included a general population telephone survey, interviews with program staff members and interviews with retailers. For the general population telephone survey, a newly acquired sample was collected in September The final sampling size for each primary data collection activity is presented in Table 3-65 below. Table 3-65: ESP Data Collection Activities Data Collection Activities N CFL/LED RDD Survey 137 Program Staff Interviews 2 Retailer Interviews 22 The only survey effort was conducted using a Random Digit Dialing (RDD) technique, where residential customers within Oklahoma were contacted and interviewed about recent CFL and/or LED purchases. The RDD procedure was organized by zip codes, targeting locations within specific areas of Oklahoma. Because customer contact information is not tracked for the CFL and LED markdowns, the RDD methodology provided a cost-effective way of reaching many potential program participants or representative consumers. In addition, both landlines and cellular phones were targeted in a breakdown of 75 to 25%, respectively. Interviewers used screening questions to determine whether respondents were (a) a PSO or other electric utility customer, (b) Energy Efficiency Programs 147

151 PSO Energy Saving Product Program recently purchased light bulbs and (c) that the respondent had a general understanding of different light bulb technologies. 25 In total, the RDD survey was completed by 137 PSO customers. Of the RDD respondents, 122 felt they could correctly identify different light bulb technologies. Onehundred and thirty-seven (137) of these respondents had purchased light bulbs during the eight months prior to the survey. All 137 of the respondents indicated that they had purchased CFLs, LEDs or both. To inform the process evaluation, ADM conducted in-depth interviews with program staff at PSO and implementation contractor CLEAResult. These interviews provide insight into various aspects of the program and its organization. The interviews focused on changes to the program that occurred during Interview respondents also discussed aspects of the program operations and results that they considered to be successful as well as challenges faced. In addition, trade ally interviews were conducted with retailers participating in the ESP program to address their feedback on to the types of bulbs and/or room air purifiers incentivized, the marketing materials used and their overall satisfaction with PSO s ESP program. Gross Impact Estimation Methodology: Lighting Only Reported energy and peak demand impacts for the program were calculated using deemed per-unit impacts from the Oklahoma Deemed Savings Documents. For CFLs and LEDs, the deemed savings algorithms came from the 2013 updated Deemed Savings Documents, which reflect baseline bulb wattage changes resulting from the Energy Independence and Security Act of 2007 (EISA). ADM s evaluation consisted of (1) verifying the quantity of program eligible measures that were discounted in-store, (2) reviewing the assumptions and inputs associated with the deemed savings values (3) verifying that the deemed per-unit impacts were applied appropriately and (4) making appropriate adjustments for in-service rates, leakage, and cross sector sales. Verification For CFL and LED markdowns, ADM reviewed the program tracking database consisting of retailer transaction data. Important fields included: item description, number and type of package sold, bulbs per package, bulb lumens, bulb wattage, program and original retail pricing, retail location, and transaction period. This tracking data was compared to participating retailer/manufacturer invoices to verify the quantity of units sold and discounted through the program. The retailer/manufacturer invoices submitted to the program rebate processing center are based on actual sales transaction data from each 25 Customers were asked if they felt they could correctly identify a typical incandescent bulb, a CFL, and an LED if all three were place in front of them. Energy Efficiency Programs 148

152 PSO Energy Saving Product Program Retailer. Manufacturer invoices were also reviewed for the bulbs distributed through local food pantries. Calculation of Gross Annual kwh Savings Gross annual energy savings for discounted CFLs and LEDs were calculated using the algorithm from the Oklahoma Deemed Savings Documents shown below with the exception of the Hours of Use (HOU). For program year 2016, ADM had to determine the most appropriate method for calculating energy savings for residential lighting and the ESP program. Ideally, a metering study conducted in PSO territory would provide the best estimate for HOU, but the results of a metering study would not be available until In the absence of meter data, it is difficult to discredit or uniquely support the 2.8 HOU from the 2005 KEMA study in California or the 2.17 HOU from the 2014 Cadmus study in Arkansas. As an alternative, ADM determined the best approach was to review well-regarded and recent metering studies and calculate an unweighted average across these studies to reduce any possibility of bias. ADM calculated an unweighted average of 2.63 hours using the recent metering studies reported in The Uniform Methods Project (UMP) combined with four additional studies (2014 Cadmus study in Arkansas, the 2005 KEMA study in California, the 2014 GDS/Nexant metering study in Pennsylvania, and 2015 ADM LED study in Nevada) to calculate a robust HOU for use in PSO and OG&E territories for the 2016 program year. Where: W base = W post = Baseline wattage equivalent for the lumen output of purchased bulb Wattage of purchased bulb Hours = Average hours of use per year (960.61) ISR = In Service Rate, or percentage of discounted bulbs that get installed (97%) IEF E = Interactive Effects Factor to account for cooling energy savings and heating energy penalties (0.96 for unknown heating fuel type). Calculation of Gross Peak Demand Reduction Peak demand savings for CFLs and LEDs discounted through the program were also calculated using the algorithm from the Deemed Savings Documents, shown below: Where Energy Efficiency Programs 149

153 PSO Energy Saving Product Program W base = W post = ISR = Baseline wattage equivalent for the lumen output of purchased bulb Wattage of purchased bulb In Service Rate, or percentage of discounted bulbs that get installed (97%) CF = Summer Peak Coincidence Factor (9%) IEF D = Interactive Effects Factor (1.53) In-Service Rate Adjustments The cost-effectiveness testing for the program requires calculating lifetime energy savings for purchased CFLs and LEDs. Less efficient incandescent and EISA compliant halogen bulbs typically have rated lifetimes considerably lower than CFLs and especially LEDs. Additionally, calculating lifetime energy savings requires an estimate of when the newly purchased bulbs are installed. The Deemed Savings Documents stipulate an In-Service Rate (ISR) of 97%, but this reflects the percentage of bulbs estimated to be installed eventually. Previous studies have found that immediate or firstyear installation rates are generally lower, as some bulbs are shelved for later use. To estimate a first-year ISR, ADM asked RDD survey respondents to estimate the number of purchased light bulbs they plan to install within one week and within one year. It was then assumed that the full ISR of 97% is achieved within three years. 26 The second-year ISR is assumed to be the average of the first-year ISR and the full ISR, reflecting an assumed linear rate of installation. The ISR only affects first and second year savings as well as the discounting of energy and demand impacts for costeffectiveness testing purposes. Annual savings estimates are unaffected. Leakage Adjustments Leakage refers to cross-territory sales that occur when program discounted bulbs are installed outside of PSO s service territory. When this occurs, the energy and demand impacts from the discounted bulbs are not being realized within the territory that paid for and claimed the savings. An estimate of leakage was calculated in PY2015 at 3.6% and this was used for PY2016. Cross Sector Sales Adjustments ADM used estimated annual hours of use (HOU) of (as described in Calculation of Gross Annual kwh Savings). This reflects an average daily HOU of 2.63 times days per year. While this is within the range of HOU estimates from previous 26 This three-year period for achieving the full ISR is recommended by the DOE Uniform Methods Project Residential Lighting Evaluation Protocol. Energy Efficiency Programs 150

154 PSO Energy Saving Product Program studies 27 of residential lighting use, it likely underestimates HOU for bulbs that are installed in non-residential buildings. The higher annual HOU for bulbs in non-residential savings implies a shorter expected useful life for the bulbs (in years). The time period in which the savings occur affects the applicable baseline wattage and discount factor for cost-effectiveness savings. ADM used responses from the RDD survey to estimate the percentage of purchased bulbs that are installed in non-residential facilities. For these bulbs, HOU were estimated to be 3,253 based on EUL stipulations from the Arkansas TRM. 28 A corresponding CF of 0.55 is assumed. This has the effect of increasing annual energy savings and peak demand reduction for the percentage of bulbs estimated to be installed in non-residential settings. Gross Impact Estimation Methodology: Room Air Purifiers Only Deemed kwh and peak demand kw savings values for room air purifiers were unavailable in the OKDSD; however, the Illinois TRM v5.0 has established deemed kwh savings and peak kw demand values that will be used for this analysis 29,30. ADM s evaluation consisted of (1) verifying the quantity of program eligible measures that were discounted in-store, (2) reviewing the assumptions and inputs associated with the deemed savings values and (3) verifying that the deemed per-unit impacts were applied appropriately. Verification For room air purifiers, ADM reviewed the program tracking database consisting of retailer transaction data. Important fields included: item description, number and type of room air purifier sold, Dust CADR, program and original retail pricing, retail location, and transaction period. This tracking data was compared to participating retailer/manufacturer invoices to verify the quantity of units sold and discounted through the program. The retailer/manufacturer invoices submitted to the program rebate processing center are based on actual sales transaction data from each retailer. 27 The DOE Uniform Methods Project Residential Lighting Evaluation Protocol summarizes nine recent studies with HOU estimates ranging from 1.5 to 2.98 hours per day. See: 28 Arkansas Technical Reference Manual, Version 4.0. Arkansas Public Service Commission. Table Illinois TRM version 5.0, June 1, Calculation for kwh savings and peak kw demand are based on the Mid-Atlantic TRM version 4.0. This specifies baseline kwh/year consumption and ENERGY STAR kwh/year consumption based on the Clean Air Delivery Rate (CADR) for ENERGY STAR room air purifier. Energy Efficiency Programs 151

155 PSO Energy Saving Product Program Calculation of Gross Annual kwh Savings Gross annual energy savings for discounted room air purifiers were calculated using the algorithm from the IL TRM v5.0 shown below: Where, kwh Base = Baseline kwh consumption per year; based on Table 3-66 kwh ESTAR = ENERGY STAR kwh consumption per year; based on Table 3-66 Table 3-66: kwh per year usage based on Clear Air Delivery Rate Clean Air Delivery Rate (CADR) CADR used in calculation Baseline Unit Energy Consumption (kwh/year) ENERGY STAR Unit Energy Consumption (kwh/year) ΔkWH CADR CADR CADR CADR CADR Over Calculation of Gross Peak Demand Reduction The peak demand kw savings for room air purifiers were calculated using the algorithm from the IL TRM v5.0, shown below: Where, ΔkWh = Gross customer annual kwh savings for the measure Hours = Average hours of use per year, CF = Summer Peak Coincidence Factor for measure, Consistent with ENERGY STAR Qualified Room Air Clean Calculator; 16 hours a day, days a year. 32 Assumes appliance use is equally likely at any hour of the day or night. Energy Efficiency Programs 152

156 PSO Energy Saving Product Program Table 3-67: Peak kw demand based on Clear Air Delivery Rate Clean Air Delivery Rate ΔkW CADR CADR CADR CADR CADR Over Net-to-Gross Estimation Methodology: Lighting Only Program lighting measures were separated into two categories for net-to-gross estimation. For CFLs distributed through local food pantries, the net-to-gross ratio is assumed to be 100%. For the 25,000 CFL packages (100,000 bulbs) distributed through local food banks, the 100% net-to-gross ratio is assumed because customers do not shop for the lighting products but rather are simply offered CFLs without prompting. Individuals who received CFLs through the food banks are also more likely to represent low income customers, potentially limiting their ability or willingness to purchase high efficiency lighting products. Overall, the CFL giveaways represent less than 5.5% of reported energy savings from the ESP program lighting component. For CFLs and LEDs discounted at participating retail stores, ADM estimated freeridership as described throughout the rest of this section. Determining the net effects of the in-store retail discounts requires estimating the percentage of energy savings from efficient lighting purchases that would have occurred without program intervention. Ideally, participating retailers could provide light bulb sales data for non-program time periods and/or from similar non-program retail locations. This data would provide adequate information from which to calculate the lift in CFL and LED sales attributable to the program price markdowns. However, retailers are reluctant to release sales data for this purpose and non-program sales data was not made available to ADM. As a result, evaluating the net effects of the price discounts requires estimating free ridership without non-program sales data. Several methodologies have been used in similar evaluations across the country, all of which have certain advantages and disadvantages. For this evaluation of the PY2016 ESP program lighting component, ADM developed two separate estimates of free ridership, each using a different methodology. Table 3-68 provides a summary of the methodologies and their relative advantages and disadvantages. Details regarding each methodology follow below. Energy Efficiency Programs 153

157 PSO Energy Saving Product Program Table 3-68: Free Ridership Estimation Methodologies Advantages and Disadvantages Methodology Advantages Disadvantages RDD General Population Survey Consumer Demand Model Allows for a more truly random sample than intercept surveys. Allows for discussion of bulbs postinstallation. Large sample size more costeffective than intercept surveys. Estimate is developed from actual sales data, eliminating potential biases that customer self-report data can exhibit. Survey Based Methodology Relies on customer self-reporting of purchase decision making. Potential for recall bias is higher than intercept surveys (discuss purchases over past six months). This may also affect whether the respondent purchased program bulbs. Potential for bias in scoring algorithm. The model is estimated using program sales data only. While the model may fit program sales data well, it is possible that it does not predict sales levels at non-program prices well. The first methodology is based on self-report surveys with a sample of customers aimed at understanding decision making for light bulb purchases. The goal of these surveys is to elicit information from which to estimate the number of bulbs that the customers would have purchased in the counterfactual scenario where CFLs and/or LEDs were not discounted. Self-report survey methods for determining free ridership are generally recognized as susceptible to certain biases and error. This may be especially true for upstream price markdown programs, where the counterfactual scenario of regular retail prices may be difficult to explain or grasp. The self-report methodologies also rely on specific scoring algorithms, which may bias the free ridership estimates if they do not accurately reflect the customer decision making process. This evaluation relies on selfreport survey data from two surveying efforts: The survey-based effort for calculating free ridership was conducted using a Random Digit Dial (RDD) technique. The strength of this approach is the ability to obtain a random and relatively large sample size cost-effectively. It also allows for further questioning regarding the fate of recently purchased bulbs (e.g., installed immediately, stored for future use, location of installation, etc.). The biggest drawback to the approach is the potential for respondent recall bias. It may be difficult to get accurate responses to questions about the number of bulbs the respondent recently purchased and whether they were discounted through the program, for example. Survey respondents were asked a series of questions to elicit feedback regarding influences on their light bulb purchasing decisions. Each respondent was then assigned a free ridership score based on a consistent free ridership scoring algorithm. The free ridership scoring algorithm for the RDD surveys is shown in Figure Energy Efficiency Programs 154

158 PSO Energy Saving Product Program The behavior without discount scoring is the primary determinate of respondents free ridership scores. This section asked whether the respondent would have purchased the same light bulbs if they had cost the regular retail price. This may be a question that is particularly prone to social desirability bias the tendency to respond in a manner that might be viewed favorably by others. For this reason, a consistency check was performed. For the RDD survey, a consistency check was performed by asking each respondent to state light bulb characteristics that are important to them when choosing between available options. If a respondent listed price as the most important characteristic, but then went on to indicate that they would have still purchased efficient options at full retail price, a 50% reduction to the behavior without discount score was applied. For the RDD survey, responses were not weighted. That is, each response had equal weight in estimating the average free ridership level for the program. Energy Efficiency Programs 155

159 PSO Energy Saving Product Program Figure 3-25: Free Ridership Scoring for RDD Survey Respondents Energy Efficiency Programs 156

160 PSO Energy Saving Products Program Non-Survey Based Methodology The second estimate of free ridership was developed through the estimation of a price response model which was used to predict sales levels in the absence of the program. The program tracking data included package and bulb sales for each retailer, by model number and week. 33 For each retailer and model number combination, original retail price and program price data were available. As program price discounts and/or retailer original pricing changed throughout the year, the tracking data was updated, allowing for the comparison of same-bulb sales under slightly different pricing conditions. Price effects are the main program tool for encouraging the purchase of high efficiency lighting choices. However, there are also regular promotional events sponsored by PSO within participating retail locations. The dates, location, and duration of in-store promotional events were also tracked, allowing for estimation of their effects on sales levels as well. The final price response model is used to estimate a free ridership as described in the equation below: Where: = the expected number of bulbs of type, i, purchased given original retail pricing (as predicted by the model). = the expected number of bulbs of type, i, given program discounted pricing (as predicted by the model). kwhi = the average gross kwh savings for bulb type, i. The price response modeling approach is advantageous in that it is built upon actual sales data from participating retailers (as opposed to relying on consumer self-report surveys). There are; however, several limitations for the approach. Most importantly, non-program sales data is unavailable for inclusion in the model. As a result, the modeling of price impacts may fit program sales data well, but it is uncertain whether those price effects apply well to prices outside of program ranges. Additionally, the lack of non-program sales data means that for many bulb types and time ranges, the available sales data lists zero sales. These zeroes in most cases do not actually 33 Most bulb sales were recorded on a weekly basis. However, some retailer/manufacturer partners reported bulb sales bi-weekly or monthly. To produce weekly sales estimates for these bulbs, the biweekly sales were divided by two and monthly sales were divided by four. While this may not be entirely accurate over a given timespan, it is a reasonable assumption in the absence of weekly data. Energy Efficiency Programs 157

161 PSO Energy Saving Products Program represent zero sales, but rather a lack of information because program pricing was not in effect for a given bulb during a given week. This presents a challenge in modeling the sales data using typical time-series or panel data methods. Additionally, during the sales period analyzed there was only pricing variation for a subset of bulb models, limiting the ability of the model to predict price response effects in a robust manner. Finally, there are likely variables that affect sales levels for CFLs and LEDs that are not captured by the program tracking data; thus, there is a risk of omitted variable bias in addition to the inherent amount of error from statistical modeling. Spillover and Market Effects It is worth noting that none of the methodologies used to estimate program free ridership include estimates of spillover or market effects. Spillover refers to savings that occur because of program influences on customers but for which an incentive or rebate is not given. In the context of a program for CFL and LED price markdowns, the following examples illustrate potential sources of spillover: Participant spillover: a customer who purchases program discounted bulbs is influenced to install additional (non-rebated) energy efficiency measures or change their energy usage behavior because of their program experience. Nonparticipant spillover: a customer notices PSO sponsored discounts or receives educational resources from an in-store promotional event. While they do not ultimately purchase program discounted bulbs, their interaction with the program encourages them to install other (non-rebated) energy efficiency measures or change their energy usage behavior. Market effects refer to changes in market structure or market actor behavior due to program influence that results in non-incented adoption of energy efficiency measures. In the context of a program for CFL and LED price markdowns, the following examples illustrate potential sources of market effects: Market pricing related effects: it is possible that the program sponsored discounts for certain lighting products cause downward pressure on prices for competing products (non-program bulbs). The competing products could potentially be CFLs and LEDs at participating retailers or non-participating retailers. If pricing for these competing products is lowered in response to program discounts and a corresponding increase in purchases (and installations) occurs, then there may be additional savings attributable to program influences. Market manufacturing/stocking effects: it is possible that the program sponsored inducements cause bulb manufacturers and retailers to adjust their lighting product offerings. To the extent that the program causes lesser efficiency bulbs to be displaced with higher efficiency bulbs at the manufacturer/retailer level, there may be additional savings attributable to program influences. Energy Efficiency Programs 158

162 PSO Energy Saving Products Program It is likely that some combination of these effects increase the savings attributable to the ESP lighting portion of the program. However, there is also reason to believe these effects may be small overall. Participant and non-participant spillover typically occurs through customer education. The ESP program component does include regular instore promotional/educational event, but the number of customers reached relative to overall program sales is likely small. Additionally, the promotional events usually provide information designed to encourage customers to participate in other PSO energy efficiency programs, which would not constitute spillover if these customers ultimately did participate and receive a rebate. Market effects may exist to some extent, but disaggregating the PSO program influences from other influences such as technological advances and other lighting discount programs across the country is difficult. The current ESP program component covers a substantial share of the bulbs sold in the PSO service territory, with no immediate plans for discontinuing the price markdowns. Overall, it should be noted that spillover and market effects likely remain a minor factor, and the net-to-gross estimate developed in this evaluation should be considered with these omitted effects in mind. Net-to-Gross Estimation Methodology: Room Air Purifiers Only The RDD survey approach was not a viable means of estimating net program savings impacts for the air purifiers. The frequency at which these products were purchased was much lower than light bulb purchases and obtaining a sufficient number of responses from recent product purchasers would likely require very large sample sizes. As an alternative, ADM utilized upstream market actor interviews to assess free ridership for these products. Specifically, ADM completed interviews with retailers to assess the program impact on the sales of the discounted air purifiers. Free ridership was estimated for air purifiers by asking upstream market actors to estimate the effect of the discounts on program sales. Specifically, they were asked: FR1: According to our records, you sold [PRODUCT] that received a discount ranging from [DISC_LOW] to [DISC_HIGH]. If these discounts and the program promotional materials had not been available during the 2016 year, do you think your sales of [PRODCUT] would have been about the same, lower, or higher? The net-to-gross (NTG) ratio was scored as 0 for respondents that indicated that sales would remain the same or would have been higher without the discounts. Respondents that indicated that sales would have been lower will be asked the following question: FR2: By what percentage do you estimate your sales of [PRODUCT] would be lower during this 2016 period if the discounts and program promotional materials for [PRODUCT] had not been available? Energy Efficiency Programs 159

163 PSO Energy Saving Products Program For these respondents, the NTG ratio was equal to the 100 the percent response given to FR2. The NTG ratio questions can be seen in Figure 3-26 below. Figure 3-26: Free Ridership Scoring for Room Air Purifiers based on Retailer Interviews Process Evaluation Activities The purpose of the process evaluation for the Retail Discounts component was to assess how the program operated in PY2016 and to identify any changes made during the 2016 program year. Key research questions to be addressed by the process evaluation of the 2016 Retail Discounts component include: How well did PSO staff, implementation staff, and participating customers/retailers work together? Are there data tracking and/or communication efficiencies that can be gained? How aware are PSO customers of the CFL and LED, and room air purifiers markdowns? Are there sufficient marketing materials in the retail stores? Are retail sales associates aware of the markdowns? Were there any significant changes or new obstacles during the 2017 program year? Energy Efficiency Programs 160

164 PSO Energy Saving Products Program Looking forward, what are key barriers and drivers to program success within PSO s market? During the evaluation, data and information from several sources are analyzed to achieve the stated research objectives. Insight into the customer perspective as it pertains to the lighting component of the ESP the program is analyzed using the Random Digit Dialing (RDD) general population telephone survey. Changes made to the internal organization and operational efficiency of program delivery is examined through analysis of interviews conducted with PSO program staff and CLEAResult staff Impact Evaluation Findings Gross Energy and Peak Demand Impacts: Lighting Only The Energy Federation Inc. (EFI) tracking data for the ESP program lighting component identified a total of 479,758 packages of CFLs/LEDs discounted through participating retail stores. An additional 25,000 packages of CFLs were distributed free-of-charge through local food pantries. Table 3-69 shows the reported quantities and impacts of measures discounted or distributed free-of-charge through the ESP program during PY2016. Distribution Type Retail Discounts Table 3-69: Reported Measure Quantities and Impacts Lighting Only Measure Type Package Bulb Reported Reported Quantity Quantity kwh kw Standard CFL 196, ,595 23,618,453 3, Omni-directional LED 163, ,493 15,088,875 2, Directional LED 99, ,956 7,133,956 1, Discount1 Standard CFL 20,792 83,168 2,376, Food Pantries Standard CFL 25, ,000 2,791, Verification Total 504,758 1,638,212 51,009,418 7, To verify the types and quantities of distributed measures, ADM performed a census review of all retailer/manufacturer invoices for CFL and LED sales. This review verified that the reported quantity of light bulbs sold through retail stores and distributed free-ofcharge through local food pantries matched exactly with the invoices that PSO paid. ADM also reviewed the program tracking database to determine if energy and demand impacts were correctly calculated according to the Oklahoma Deemed Savings Document algorithms for each CFL and LED type. For PY 2016, ADM calculated verified energy and demand impacts based on OKDSD, but used an adjusted HOU (960.61). Energy and demand impacts from the program tracking data were calculated using values from the OKDSD without an adjusted HOU as it would have been too expensive and time consuming to update the program tracking data with the adjusted Energy Efficiency Programs 161

165 PSO Energy Saving Products Program HOU in the middle of the program year (as per CLEAResult). To cross-check reported with verified energy and demand impacts; however, ADM used OKDSD with no adjusted HOU. ADM found that for the vast majority of light bulbs, reported impacts were calculated in accordance with the deemed savings algorithms. Each program eligible bulb was checked to determine the correct bulb wattage and ensure the correct lumen output and baseline wattage was applied. The discrepancies identified through the database review required adjustment for the actual wattages and/or baseline wattages used in the calculation of energy and demand impacts for some bulbs. In addition, there were some incorrect calculations of kwh and kw savings in the program tracking data. Table 3-70 shows each of these adjustments, which cumulatively decreased gross energy savings from program discounted light bulbs by 50,246 kwh. Table 3-70: Gross kwh Savings Adjustments Lighting Only Measure Type 13.9W Standard CFL 6.1W Directional LED 7.2W Directional LED 8.5W Directional LED 6W Omnidirectional LED 6.5W Omnidirectional LED 8.5W Omnidirectional LED 11W Omnidirectional LED 11.8W Omnidirectional LED 15.5W Omnidirectional LED 15.6W Omnidirectional Retailer Quantity of Bulbs Reported kwh Adjustment to Verified kwh DIY3 80 2, DIY3 36 1,334 3 DIY ,543-2,584 DIY , DIY , DIY ,350 4,113 DIY , DIY , DIY , DIY , DIY , Reason for Adjustment The actual wattage for this bulb was 13W not 13.9W. The actual wattage for this bulb was 6W not 6.1W. The actual wattage for this bulb was 7W not 7.2W. Additionally, this is a directional LED with a lumen output of 300, corresponding to a baseline wattage of 45W not 65W. The actual wattage for this bulb was 8W not 8.5W. The actual wattage for this bulb was 5.6W not 6W. The actual wattage for this bulb was 5.6W not 6.5W. This represents two different model numbers that were inaccurately classified as 8.5W bulbs. The actual wattage for the two bulbs were 8W and 7W. The actual wattage for this bulb was 12.5W not 11W. The actual wattage for this bulb was 11W not 11.8W. The actual wattage for this bulb was 15W not 15.5W. The actual wattage for this bulb was 15W not 15.6W. Energy Efficiency Programs 162

166 PSO Energy Saving Products Program Measure Type LED 9.5W Directional LED 9.6W Directional LED 4W Directional LED 11W Omnidirectional LED 4.5W Directional LED 7W Directional LED 7.5W Directional LED 12W Directional LED Retailer Quantity of Bulbs Reported kwh Adjustment to Verified kwh DIY , DIY ,781 4,157 Mass Merchant2 Mass Merchant1 Mass Merchant1 Mass Merchant1 Mass Merchant1 Mass Merchant ,218-52, , , ,774-4, , , Reason for Adjustment The actual wattage for this bulb was 8W not 9.5W. The actual wattage for this bulb was 12.5W not 9.6W. Additionally, this is a directional LED with a lumen output of 1120, corresponding to a baseline wattage of 72W not 65W. The actual wattage for this bulb was 4.2W not 4W. Additionally, this is a directional LED with a lumen output of 300, corresponding to a baseline wattage of 25W not 45W. This represents five different model numbers that were inaccurately classified as 11W bulbs when they were 10W bulbs. This represents two different model numbers that were inaccurately classified as 4.5W bulbs. The actual wattage for the two bulbs were 4.2W and 2.8W. This is a directional LED with a lumen output of 500, corresponding to a baseline wattage of 45W not 72W. The actual wattage for this bulb was 7W not 7.5W. This was a directional LED with a lumen output of 1000, corresponding to a baseline wattage of 72W not 65W. Total 29, ,036-50,246 Cumulative Effect The same discrepancies identified in Table 3-70 also affected peak demand reduction for light bulbs distributed through the program, resulting in a decrease of 7.12 kw. After verifying that energy and demand impacts were calculated as stipulated in the OKDSD without an adjusted HOU, ADM calculated verified kwh and peak demand kw savings using the OKDSD with the adjusted HOU (960.61), and this is what is presented from this point forward. In-Service Rate Adjustments The 137 survey respondents purchased a total of 1,334 bulbs. These respondents indicated that 63.6% of the bulbs would be installed within a week, and ~95.4% of bulbs would be installed within a year. For the purpose of calculating program cost effectiveness, a first-year ISR of 85% was assumed. This does not affect annual kwh Energy Efficiency Programs 163

167 PSO Energy Saving Products Program savings estimates, as is assumed that 97% of the bulbs are installed within three years based on the stipulations in the deemed savings documents. 34 Leakage Adjustments ADM estimated leakage of light bulbs outside of PSO territory in PY2015. A total of 3.6% of bulbs leaked out of PSO territory into the Oklahoma and the surrounding states. The leakage estimate of 3.6% was used in PY2016. The verified gross savings developed by ADM include an adjustment for leakage using 3.6% as calculated in PY2015. This results in a decrease of 1,630,752 kwh for annual energy savings and kw for peak demand reduction. Cross Sector Sales Adjustments An adjustment to gross impacts was made to account for the proportion of program bulbs estimated to be installed in non-residential settings, where HOU and CF are typically higher than residential sockets. The general population RDD survey included a question related to cross sector sales. Respondents who indicated they had purchased CFLs or LEDs in the past eight months were asked: We re any of the CFLs or LEDs you purchased in the past eight months installed in a business or commercial setting? Of the 137 CFL and/or LED purchasers, just two indicated that they installed bulbs in a non-residential setting. 35 The resulting non-residential allocation is therefore 1.5%. The result from the RDD survey is within the range of values that previous evaluations of residential lighting mark down programs have estimated. A recent meta-analysis of 23 recent evaluations of a similar nature found estimates ranging from 0.0% to 18.7%, with various methodologies used. 36 The average non-residential allocation estimate from these studies was 6.7%. Only two respondents installed light bulbs in a commercial setting, which is too small of a sample size to accurately calculate crosssector sales estimates. ADM utilized the 5.0% non-residential allocation estimate from the average of the intercept and RDD surveys from PY Calculating cost-effectiveness requires an estimation of when the bulbs are installed to correctly discount future year savings. The cost-effectiveness estimates for the ESP program presented in this report assume that 85% of the bulbs are installed within the first year. By the third year, it is assumed that 97% of bulbs are installed, based on the deemed savings document. For the second year, 91% are assumed to be installed (a linear interpolation of years one and two). 35 These two respondents were further asked how many of the CFLs and/or LEDs they purchased were installed in non-residential settings. A sample size of two is too small to accurately calculate a cross sector sales estimate based on the number of bulbs Memo.pdf Energy Efficiency Programs 164

168 PSO Energy Saving Products Program The verified gross savings developed by ADM included an adjustment for cross sector sales by using 3,253 HOU and a 0.55 CF for 5.0% of the bulbs (rather than the adjusted HOU of and 0.09 CF for residential installations). This resulted in an increase of 5,479,001 kwh for annual energy savings and 1, kw for peak demand reduction. Final Verified Gross Savings Estimates After considering leakage and cross-sector sales adjustments annual energy savings for the ESP program were estimated to be 51,679,155 kwh. Verified peak demand savings were 8, kw. Table 3-71 compares reported and verified impact estimates for this program component. Table 3-71: ESP Program Impact Findings Lighting Only Distribution Measure Verified Reported Type Type Quantity kwh Verified kwh Reported kw Verified kw Standard CFL 813,595 23,618,453 24,063,810 3, , Omnidirectional Retail Discounts LED 476,493 15,088,875 15,356,812 2, , Directional LED 164,956 7,133,956 7,215,751 1, , Discount1 Standard CFL 83,168 2,376,933 2,421, Food Standard Pantries CFL 100,000 2,791,200 2,620, Total 1,638,212 51,009,418 51,679,155 7, , Gross Energy and Peak Demand Impacts: Room Air Purifiers Only The Energy Federation Inc. (EFI) tracking data for the ESP program room air purifier component identified a total of 228 room air purifiers sold at participating retail stores. Table 3-72 shows the reported quantities and impacts of room air purifiers through the ESP program during PY2016. Table 3-72: Reported Measure Quantities and Impacts Room Air Purifiers Only Distribution Type Measure Type Total Quantity Reported kwh Reported kw Retail Discounts Room Air Purifiers , Verification Total , To verify the types and quantities of distributed measures, ADM performed a census review of all retailer/manufacturer invoices for room air purifier sales. This review verified that the reported quantity of room air purifiers sold through retail stores matched exactly with the invoices that PSO paid. The invoice did, however, find several small discrepancies relating to Dust CADR and associated kwh and peak demand kw Energy Efficiency Programs 165

169 PSO Energy Saving Products Program savings. Energy savings for 300 Dust CADR room air purifiers were incorrectly calculated in the program tracking as seen below in Table The adjustment to verified kwh savings was an increase of 970 kwh. Table 3-73: Gross kwh Savings Adjustments Room Air Purifiers Only Measure Type 300 Dust CADR Retailer Quantity Reported kwh Adjustment to Verified kwh DIY , Reason for Adjustment The kwh savings value specified by the IL TRM v5.0 for a room air purifier with Dust CADR over 250 is 1,169, whereas these were incorrectly assigned a kwh savings of 1,072. Total 10 10, Cumulative Effect The same discrepancies identified in Table 3-73 also affected peak demand reduction for light bulbs distributed through the program, resulting in an increase of 1.33 kw. There were no other adjustments specific to gross peak demand reduction estimation. Final Verified Gross Savings Estimates Table 3-74 compares reported and verified impact estimates for this program component. Table 3-74: ESP Program Impact Findings Room Air Purifiers Only Distribution Verified Reported Verified Reported Verified Measure Type Type Quantity kwh kwh kw kw Retail Room Air Discounts Purifiers , , Total , , Net-to-Gross Estimation Results Lighting Only The net-to-gross analysis for the ESP program was conducted using the methodologies outlined in Section The results of this analysis are summarized below. Random Digit Dialing Free Ridership Estimate The second self-report survey used to estimate free ridership was conducted using Random Digit Dialing (RDD) to reach the general population of Oklahoma residents (PSO and non-pso service territory). A total of 151 respondents completed the RDD survey in September 2016 (100% of whom reported being PSO electric utility customers). Of those 151 survey respondents, 137 had purchased CFLs and/or LEDs in the prior 8 months and these respondents were used to calculate free ridership as shown in Figure 3-25 and Table Zip codes were cross-checked to confirm that respondents reported utility company was accurate based on PSO service territories. Energy Efficiency Programs 166

170 PSO Energy Saving Products Program Table 3-75 below shows the results of this free ridership calculation by the type of bulbs respondents claimed to have recently purchased. Respondent Type Table 3-75: RDD Survey Free Ridership Estimate N Prior Experience Score Behavior w/o Program Score Free Ridership Estimate Mitigating Factor CFL Purchasers LED Purchasers Both Purchasers Total The average free ridership score for all 137 respondents was 39.5%. This is 10.1% higher than the free ridership level estimated from the RDD survey in PY2015. Several possible explanations exist for the difference in free ridership estimates between PY2015 and PY2016. While the questions used to assess free ridership in PY2015 and PY2016 were identical, there were some differences between the survey effort in PY2015 versus PY2016. In PY2015, exclusively landlines were called for the RDD survey. In PY2016, both landlines and cell phones were surveyed at a breakdown of 75%-25%, respectively. Cell phones were added to the sample as prior research has shown that 51.4% of adults in the Midwest live in wireless-only households 37. To reach a larger proportion of these wireless-only customers in PSO territory, 25% of customers surveyed were cellular numbers. ADM believes that adding cellular phones to the sample size in PY2016 reaches a broader audience of PSO customers that were not surveyed in PY2015. In addition, the sample size was almost twice as large in PY2016 in comparison to PY2015 (137 versus 73, respectively). The larger sample size reduces the overall margin of error of the sample and reduces the uncertainty of the free ridership of the estimate. One component of the RDD survey free ridership scoring algorithm that is somewhat subjective has to do with the behavior without discount category. Survey respondents were asked whether they would have still purchased the same program bulbs if they had been at regular retail price (the retail price was given to the respondent in the form of $X more ). For PY2016, an additional category was added to these questions to assign equal free ridership to each category using the principle of equal distribution of ignorance. The idea behind this principle is that we do not apriori know what percentage of free ridership should be assigned to any given category; therefore, we equally divide free ridership amongst all categories ( Figure 3-25). Those who 37 Energy Efficiency Programs 167

171 PSO Energy Saving Products Program responded probably were assigned a free ridership score of 0.75 for that category, unless they also said they would have purchased fewer bulbs. Those who responded, not sure were assigned a free ridership score of 0.5 for that category, unless they also said they would have purchased fewer bulbs. Those who responded, probably not were assigned a free ridership score of 0.25 for that category. For CFLs, 58.9% of customers would have definitely or probably still purchased CFLs, 9.6% of customers were not sure if they would have still purchased CFLs and 31.5% definitely would not or probably would not still have purchased CFLs. In contrast, for LEDs, 38.6% of customers would have definitely or probably still purchased CFLs, 6.0% of customers were not sure if they would have still purchased LEDs and 55.4% of customers definitely would not or probably would not still have purchased LEDs. This would suggest that customers are still more likely to purchase CFLs over traditional incandescent or halogen bulbs than LEDs. Although the price of LED bulbs continues to drop, cost plays a crucial role in customer preference for CFLs over LEDs. Table 3-76 shows the responses in the RDD surveys collected in PY2016. Table 3-76: RDD survey responses to Behavior without discount category questions If the particular CFL light bulbs you purchased in the past eight months had cost $1 more per bulb than they did, would you still have bought CFLs over other available light bulb types? Definitely would have still purchased CFLs Probably would have still purchased CFLs Not sure if I would have purchased CFLs Probably would not have still purchased CFLs Definitely would not have still purchased CFLs Responses If the particular LEDs you purchased in the past eight months had cost $5 more per bulb than they did, would you still have bought LEDs over other available light bulb types? Definitely would have still purchased LEDs Probably would have still purchased LEDs Not sure if I would have purchased LEDs Probably would not have still purchased LEDs Definitely would not have still purchased LEDs Responses Total 73 Total 83 In addition to assessing free ridership, ADM also included a question on the survey to determine if the prior purchase of discounted bulbs affected the purchase of discounted bulbs in PY2016. Most customers surveyed in the RDD survey did not recall purchasing PSO discounted bulbs previously (64.7%). Customers that did respond that they had previously purchased light bulbs that were discounted by PSO said that their prior purchase of a PSO program discounted bulb was very important in the decision they made to purchase a PSO program discounted bulb in PY2015 (8.5 out of a score of 10). Table 3-77 shows the responses to these questions for the RDD survey collected in PY Energy Efficiency Programs 168

172 PSO Energy Saving Products Program Table 3-77: Purchasing behavior in PY2016 based on purchase of previously discounted bulbs Do you recall purchasing any light bulbs that you knew were discounted by PSO? Responses On a scale of 1 to 10, where one is not important at all and ten is very important, how important was the PSO lighting discount to your decision to purchase those specific light bulbs? Yes No 13 NA Don't Know 0 NA Refused 1 NA Total 16 To address spillover, ADM included a couple of follow up questions on the survey to determine if customers that had previously purchased light bulbs that they knew were discounted by PSO purchased any additional CFLs or LEDs that were not discounted. Only two respondents (1.5%) had previously purchased PSO discounted bulbs and these two respondents did not purchase any additional non-discounted CFLs or LEDs. Price Response Model Free Ridership Estimate Free ridership was also estimated using an econometric price response model that estimates the effect of program discounts and promotional events on bulb sales. Coefficients from the model were used to predict sales quantities at regular retail pricing and with an absence of program promotional events. The difference in model predictions for sales quantities under program and non-program conditions produces an estimate of free-rider (or naturally occurring) bulb sales. Multiplying the free-rider bulb sales quantities by SKU specific deemed gross savings estimates results in the final estimate of free-ridership. The analysis resulted in a program level free ridership estimate of 41.6%, which closely aligned with the RDD estimation methodology (39.5%). The price response model also allows for estimating free ridership by bulb type. Consistent with other evaluations that have used this type of model, the estimated free ridership for standard and specialty LEDs is higher than for standard CFLs, as shown in Figure However, in this instance, the model likely overestimates the free ridership level for LED bulbs relative to CFLs due to low variability in program pricing for these measures during PY2016. Energy Efficiency Programs 169

173 PSO Energy Saving Products Program Figure 3-27: Price Response Model Free Ridership Estimates by Bulb Type Price discounts for LEDs are larger in absolute terms, but relative to overall product price they represent a smaller fraction than the CFL price discounts. This is one possible explanation for why LED free ridership might be higher than CFL free ridership. Net-to-Gross Estimation Results Room Air Purifiers Only ADM completed interviews with retailer staff at seven retailer locations that offered the air purifier discounts. Of these seven retailer locations, staff at four of them were unable to provide responses to the questions about the impact of program sales on discounted purifiers. The retailers that did respond to the free ridership questions accounted for 41% of the air purifiers discounted through the program. The weighted net-to-gross ratio for these responses is 33% (i.e., free ridership was 67%). The limited sample size calls into question the appropriateness of generalizing the selfreported free ridership results to the broader population of retailers that provided air purifier discounts. Consequently, ADM completed a benchmarking analysis of free Energy Efficiency Programs 170

174 PSO Energy Saving Products Program ridership results for air purifiers. The results of this analysis are summarized below in Table ADM identified four recent and publicly available evaluations of air purifier net savings. None of the evaluated programs provided upstream rebates for air purifiers each offered a downstream rebate. However, three of the four programs matched the incentive amount offered by PSO ($50). Table 3-78: Free Ridership Benchmarking Findings Program Incentive Amount Free Ridership Ameren Missouri $ ComEd $ Efficiency Maine Appliance Rebate Program $ * Ameren Illinois $ Average 0.31 * Included room ACs and freezers in the estimate Three of the four free ridership estimates, and the average across the studies were significantly lower than that based-on retailer interviews. The difference between the retailer interview results and the benchmarking results supports the conclusion that the retailer interview results are not generalizable to the broader population of retailers. Consequently, ADM estimated air purifier net savings by applying a blended free ridership estimate based on retailer interview results and the benchmarking findings. Specifically, the interview results were applied to the respective stores for which interviews were completed and the average net savings from the benchmarking results were applied to the remaining retailer sales. The results are summarized in Table The net-to-gross ratio for air purifiers is Table 3-79: Net-to-Gross Room Air Purifiers FR Estimate Source NTGR Percent of Sales Retailer interview % Benchmarking review % Program NTGR 0.54 Final Net-to-Gross Ratio ESP Program The discussion above outlines the results of two efforts to understand the level of attribution appropriate for the energy savings resulting from the lighting bulb sales through the ESP program. The one self-report RDD survey methodology resulted in an estimate of free-ridership of 39.5%. The price response modeling resulted in a free ridership estimate of 41.6%. Energy Efficiency Programs 171

175 PSO Energy Saving Products Program Ultimately, the two methods for calculating free ridership are within 2.1% of each other and as such, ADM decided to use the simple average of the free ridership estimate from the RDD survey and the price response model. The final free ridership ratio applied to retail discounted bulbs in this evaluation is therefore 40.5%. There is conflicting evidence regarding how free ridership differs for LEDs as compared to CFLs. The free ridership estimate for CFLs and LEDS differs only slightly based on the RDD surveys by bulb type. The price response model; however, suggest that free ridership could be higher for LEDs. However, this is likely due in part to limited price variation in the program sales data for LEDs preventing a more robust estimation of price effects. Because of the conflicting evidence, the program level estimate of free ridership was applied to both CFL and LED sales. Ultimately, both a survey-based and non-survey based methodology resulted in a similar estimate for free ridership that was higher than the estimate in PY2015 (29%). An explanation for the difference between free ridership estimates between PY2015 and PY2016 using the RDD survey has been described in detail above in section Random Digit Dialing Free Ridership Estimate. In addition, the overall cost of CFLs and LEDs has continued to fall in the market, making them more accessible to everyone. As both CFLs and LEDs are more cost-effective, it is likely that the overall price of non-program CFLs and LEDs drops and more customers are likely to implement this measure in absence of the ESP program (thus increasing free ridership). The measure level net-to-gross ratios are calculated as 1- estimated free ridership. 38 The final net-to-gross ratios and associated net savings for each measure in the ESP program are shown in Table Similarly, CFLs distributed through the food bank giveaways are assumed to have a net-to-gross ratio of 1.0. Table 3-80: Verified Gross and Net Impacts Retail Discounts Distribution Verified Net Peak Measure Type Verified kw NTGR Net kwh Type kwh kw Standard CFL 24,063,810 4, ,305,935 2, Omni-directional Retail 15,356,812 2, ,129,625 1, LED Discounts Directional LED 7,215,751 1, ,289, Room Air Purifiers 114, , Discount1 Standard CFL 2,421, ,421, Food Pantries Standard CFL 2,620, ,620, Total 51,793,329 8, ,829,760 5, This is sometimes referred to as a net-of-free-ridership ratio, as it excludes any estimation of spillover or market effects. Energy Efficiency Programs 172

176 PSO Energy Saving Products Program Process Evaluation Findings This section presents the core process related findings derived from the following data collection activities: Random Digit Dial surveys conducted with Oklahoma residents; Interviews with retailer staff; Store visits; and In-depth interviews conducted with PSO and third party implementation staff members involved in the Retail Discounts program component. Program Sales and Product Discounts As shown in Table 3-81, CFLs accounted for 61% of the lamp sales and distributions during The share of sales and distributions that were LEDs increased significantly over last year, during which LEDs accounted for 7% of program sales and distributions. CFL sales and distributions accounted for a smaller share of savings in part because of the rapid state of change the retail lighting market was in during During the year, various manufacturers stopped producing CFLs and some retailers stopped offering them. The cost of reported energy savings for standard LED bulbs fell by approximately 50% in With the broadening of eligible lamps under the ENERGY STAR 2.0 guidelines, the cost of LEDs and associated discounts will likely fall further in Table 3-81: 2015 Lamp Sales and Distributions and Discount Amounts Product Type Total Units Percent of Sales Average Discount Discount Amount per Reported kwh Saved Standard CFL 996,763 61% $3.91 $0.03 Standard LED 476,493 29% $4.32 $0.04 Specialty LED 164,956 10% $5.48 $0.06 Total 1,638, % $4.62 $0.04 Figure 3-28 displays the share of the total number of CFLs sold through the participating retailers. As shown, Mass Merchant 1 accounted for the largest share of CFLs sold through the program followed by DIY 1 and Mass Merchant 2. The remaining retailers accounted for 16% of the CFLs sold. Figure 3-29 displays LED sales by retailer. Mass Merchant 1 accounted for 38% of sales, DIY 1 accounted for 28%, Mass Merchant 2 accounted for 15%, Mass Merchant 5 accounted for 12% and DIY 2 accounted for 7%. The remaining retailers accounted for less than 3%. Energy Efficiency Programs 173

177 PSO Energy Saving Products Program Figure 3-28: Percent of CFLs Sold by Retailer Figure 3-29: Percent of LEDs Sold by Retailer In addition to the retailer sales, the program distributed 100,000 CFLs through regional food banks. The program offered discounts on ENERGY STAR air purifiers for the first time in The air purifier discounts were offered at two DIY chains beginning in June and October, respectively. The program targeted sales of 175 discounted units and Energy Efficiency Programs 174

178 PSO Energy Saving Products Program exceeded this goal with 228 units sold as shown in Table Most sales were made at one of the two DIY retailers, which accounted for 93% of sales. Product Type Table 3-82: Room Air Purifier Summary Data Total Units Discount Discount Amount per Reported kwh Saved Air Purifiers 228 $20 -$50 $0.10 Promotional Activities An important component of the program is the outreach events and in-store demonstrations that promote awareness of the discounts and provide information about the energy efficient lamps and the available discounts. ADM summarized information provided by CLEAResult on the 2016 promotional activity. As shown in Table 3-83, a total of 74 promotional events were held during the year. The number of events were evenly distributed across the months of Table 3-83: Promotional Events by Month Month Number of Presentations January 4 February 5 March 5 April 10 May 4 June 6 July 6 August 6 September 4 October 8 November 7 December 9 Total 74 The promotional events were held at a variety of locations, but as shown in Table 3-84, most events were held at the most active retailers. Energy Efficiency Programs 175

179 PSO Energy Saving Products Program Table 3-84: Number of Presentations by Location Retailer / Location Number of Presentations Mass Merchant 1 29 Mass Merchant 2 18 DIY 1 14 DIY 2 7 Mass Merchant 5 3 Other 3 Total 74 Random Digit Dial (RDD) Survey Findings ADM conducted a general population survey with PSO customers using random digit dialing. These responses are discussed in the sections below. In total, 137 responses were collected from PSO customers who had purchased light bulbs in the past eight months. Respondent Demographics Table 3-85 displays demographics of respondents who completed the survey in Overall, the demographics of the program participant population was similar to the population of Participants overall generally lived in single family detached homes (74%), owned their homes (83%), and had a low household size. Energy Efficiency Programs 176

180 PSO Energy Saving Products Program Table 3-85: 2015 and 2016 Survey Respondent Demographics Demographic Characteristic Percent of Responses (n = 137) Description of home/residence Single-family detached home 74% Single-family, factory manufactured/modular 5% Condominium 7% Single-family mobile home 2% Apartment 6% Don't know 2% Refused 4% Own or rent home Own 83% Rent 12% Don't know 1% Refused 4% Household income Less than $10,000 5% $10,000 to $29,999 15% $30,000 to $49,999 13% $50,000 to $69,999 10% $70,000 to $89,999 7% $90,000 to $99,999 4% $100,000 to $149,999 6% $150,000 or more 7% Don't know 6% Refused 27% Square feet of home Less than 1,000 9% 1,000-1,500 16% 1,501-2,000 20% 2,001-2,500 13% Greater than 2,500 23% Don't know 15% Refused 4% Household Size 1 23% 2 41% 3 14% 4 10% 5 4% More than 5 1% Don't know 2% Refused 4% Energy Efficiency Programs 177

181 PSO Energy Saving Products Program Customer Awareness of Energy Efficient Light Bulbs Customers were asked a series of questions related to their understanding of energy efficient light bulb technologies. Respondents rated their general level of familiarity with household light bulb technologies. As shown in Table 3-86, 89% of respondents reported that they were very or somewhat familiar with light bulb technologies, up 18 percentage points from last year. Table 3-86: Familiarity with Light Bulb Technology Would you say you are very familiar, somewhat familiar, not too familiar, or not at all familiar with currently available household light bulb technologies? Very familiar Somewhat familiar Responses Not too familiar Not at all familiar Don't know 47% 42% 7% 4% 0% 137 N When asked to rate their awareness of compact fluorescent light bulbs (CFLs) and light emitting diode (LED) light bulbs is displayed in Table Sixty-five percent of respondents reported that they had heard of CFLs and 93% reported that they had heard of LED light bulbs. Awareness of LEDs increased by seven percentage points over last year, and awareness of CFLs dropped slightly by two percentage points. These changes may reflect broader changes occurring as LED prices fall and CFLs become less available. Table 3-87: Awareness of CFLs and LED Light Bulbs Bulb Type Yes n Previously heard of CFLs 65% 137 Previously heard of LEDs 93% 137 Customers were asked if they had purchased energy efficient light bulbs prior to % stated that they had purchased LED light bulbs, and 67% stated that they had purchased CFL light bulbs prior to the current year. This is a significant increase of customers reporting the purchase of LEDs as compared to last year, with a 16 percentage-point increase in respondents reporting LED purchase. While this increase may be due to decline of LED prices, there was also 16 percentage point increase in customers reporting purchasing CFL light bulbs. Respondents were also asked if they had purchased CFL or LED bulbs in the past eight months. The results are shown below in Table The results show that most respondents purchased at least one type of energy efficient bulb in the last eight months. Energy Efficiency Programs 178

182 PSO Energy Saving Products Program Table 3-88: Reported purchase of efficient bulbs in the last eight months Bulb Type Yes n Purchased CFLs in the last 8 months 58% 137 Purchased LEDs in the last 8 months 67% 137 Together, these responses suggest that most customers understand energy efficient light bulb technologies, and this understanding has increased significantly from last year. Nearly all customers reported familiarity with LEDs and approximately two-thirds are familiar with CFLs Important Lighting Characteristics Respondents were asked to list characteristics of light bulbs that are important when making purchasing decisions. The responses can be seen below in Figure Brightness of the bulb was cited most often, followed by cost, and energy savings. Bulb price is a barrier program discounts are designed to overcome. Because energy efficient bulbs tend to be more expensive than traditional bulbs, offering a financial incentive to purchase energy efficient bulbs helps remove the barrier to purchasing these bulbs. Figure 3-30: Important Bulb Characteristics for Purchase Decision (n=137) These factors suggest some areas where program educational efforts could focus to help customers choose discounted bulbs that are most appropriate for their homes. First, education on how to select bulbs for brightness may help customers make appropriate selections, particularly in a market where lamp wattages are changing Energy Efficiency Programs 179

183 PSO Energy Saving Products Program significantly as technologies have shifted from incandescent lamps to CFLs, and now LEDs. Second, the importance of bulb costs suggests that continued education of customers on the lifetime cost savings of LEDs may encourage customers to select LEDs. Reasons for Replacing Bulbs Figure 3-31 summarizes the reasons for purchasing the efficient bulbs given by survey respondents. The most common reasons given by customers who purchased both CFLs and LEDs was for replacing burned out bulbs and to save energy. Figure 3-31: Reasons for Purchasing Energy Efficient Bulbs Purchase Locations Table 3-89 displays the locations where these respondents reported purchasing light bulbs. Walmart and Lowes were the most frequently mentioned locations followed by Home Depot. Only 4% of respondents state that they did not purchase light bulbs at any of these locations. This suggests that discounts are available in retail locations that customers generally purchase light bulbs from. Energy Efficiency Programs 180

184 PSO Energy Saving Products Program Table 3-89: Where Respondents Purchased Light Bulbs in the Last Eight Months Of the light bulbs you purchased in the past eight months, were any of them purchased through any of the following retail stores Program Awareness Percentage Response of Respondents (n=136) Walmart 53% Lowes Home Improvement 43% The Home Depot 21% Sam's Club 9% Ace Hardware 8% None of these retailers 4% Target 3% Dollar Tree 2% Family Dollar 2% Walgreens 1% Batteries Plus 0% Don't know 0% Refused 0% The program design includes targeted marketing materials and in-store promotional events and point of sale signage to inform customers of the discounts and to educate customers on the benefits of choosing efficient lighting technology. These activities help drive awareness of program incentives, and ultimately, influence customer purchasing decisions. In addition to these in-store efforts, PSO also advertises the availability of the discounts on its Power Forward website and its efficiency program more broadly through customer mailings, television, radio, and print advertisements. To gauge the effectiveness of these efforts, the survey included a series of questions about discount awareness. Customers were asked if they were aware that PSO offered in-store price discounts for efficient light bulbs. Eleven percent of PSO customers stated they are aware of the price markdowns, as shown in Table Energy Efficiency Programs 181

185 PSO Energy Saving Products Program Table 3-90: Awareness of PSO Sponsored Discounts In 2016, PSO discounted certain energy saving light bulbs through participating retailers. Were you aware of these PSO lighting discounts? Percentage Response of Respondents (n=137) Yes 11% No 79% Don't recall 3% Refused 7% The 15 customers who stated that they were aware of the lighting discounts were asked to identify how they learned of the discounts. The most frequently mentioned source of awareness, mentioned by 47% of respondents, was direct mailings including bill inserts, newsletters, or other direct mailing materials. In-store promotional events were cited by 3 (20%) of respondents, and two cited in store signage. Other sources of awareness included the PSO website and other sources. One respondent was not sure where they heard of the discounts. These details are shown in Figure 3-32 below. Figure 3-32: Source of Awareness for PSO Discounts (n = 15) Only two of the sixteen respondents who stated that they knew about the lighting discounts stated that they recalled purchasing light bulbs that they knew were discounted. Energy Efficiency Programs 182

186 PSO Energy Saving Products Program Store Visits ADM visited stores offering discounted to review the availability of discounted products, the in-store marketing materials, and their placement and display in the store. In total 19 visits were completed. At the two discount locations visited; however, ADM could not locate any discounted products or marketing materials. Both locations were stores at one chain that staff noted has difficulty maintaining the discounted product in stock. Table 3-91 displays the number of visits completed by program retailer type for the 19 locations that participated in the discounts program. Table 3-91: Summary of Store Visits Store Type Number Visited Number Participating Mass Merchant DIY 6 21 Discount 3 23 Of the six DIY stores visited, two participating were offering discounted ENERGY STAR air purifiers at the time of the visits. However, during the visits, ADM did not locate any signs denoting the air purifiers as discounted at one of the two locations. Additionally, at the other location, ADM located four models of Honeywell ENERGY STAR certified air purifiers but only one was marked as discounted. Of the three models without discounts, ADM confirmed that two was listed in the MOU. Use of Marketing Materials Table 3-92 displays the placement of signs in the stores offering discounted lighting products. As shown, signs on shelves and walls were the predominant form of discounted product advertising. At one Mass Merchant retailer that had discounted light bulbs in stock, staff were unable to identify any signs promoting the discounts. Table 3-92: Location of Lighting Discount Signage Sign Location and Type Percent of Store Locations (n = 17) Sign on shelf/wall 88% Free standing in store entrance 6% No signs 6% The store location that offered discounted air purifiers had the discounts clearly marked using a discount tag next to the price tag, although, as previously noted, only one of four ENERGY STAR models was marked as discounted. Energy Efficiency Programs 183

187 PSO Energy Saving Products Program Product Placement ADM assessed the placement of discounted products at the participating retailers visited because retailers that place discounted products at eye level are likely to generate higher sales rates for the products than those that place them on upper or lower shelves. 39 Seventy-six percent of the stores offering discounted LEDs displayed most of the discounted lamps at eye-level and one of the three stores that offered CFLs did so. The two stores that did not display discounted CFLs at eye-level did display the discounted LEDs at eye-level, which suggests that these stores may have focused on promotion of LEDs over CFLs. The third store offering CFLs displayed both the CFLs and the LEDs at eye-level. The breakdown of lighting product placement is shown below in Table Table 3-93: Lighting Product Placement Percent of Discounted Products Displayed at Eye Level CFLs (n = 3) > 0% - 25% 67% > 25% - 50% 0% > 50% - 75% 0% > 75% - 100% 33% LEDs (n= 17) > 0% - 25% 0% > 25% - 50% 6% > 50% - 75% 18% > 75% - 100% 76% The store offering discounted air purifiers displayed the discounted units at eye-level. Competing Non-Discounted LEDs At one of the DIY store locations, ADM identified two models of LEDs that did not receive program sponsored discounts and were priced at a similar amount on a per unit basis ($1.12 and $1.99 per bulb). The availability of non-program LEDs offered at a similar price of discounted bulbs is a function of the declining costs of LEDs. Typically, these products are not ENERGY STAR certified and may not perform as well as 39 TRC Solutions (2014). Literature review of customer lighting preferences. CALMAC Study ID: PGE Energy Efficiency Programs 184

188 PSO Energy Saving Products Program program qualified products in terms of lifetime, efficacy, color rendering, and other factors. Retailer Survey The Evaluators surveyed 22 retail store representatives from 20 different retailer locations. Survey participants included managers, assistant managers, supervisors, and electric specialists at retail locations that participated in the Energy Efficient Products Program. The surveyed managers represented 24% of total participating locations. Most retailers reported that their location had been participating in the program for at least a year, and several said that they had been participating for 3 4 years. Despite these several years of experience with the program, only one respondent could recall the name of the program representative that they work with. Interviewees who described their role in the program said that they talk to program representatives, make sure that the customer purchase limit is being enforced, educate customers about the program, and generally making sure that the program is running smoothly. Markdown and Rebates Most respondents said that they thought that the program had helped stimulate sales but several were unable to quantify the extent to which sales had increased because of the program discounts. Four of the interviewed retailers stated that they offer discounts on the rebated items. Three of these respondents stated that they offer corporate discounts and one said that items may go on clearance if they aren t selling well or are about to be replaced by a new model. Eight respondents stated that when program rebates are available, they display the products that qualify for a rebate differently. Three respondents were not sure how the rebated products had been displayed differently, only that products were rearranged periodically. One respondent said that discounted products were sometimes put closer to the front of the store, one said that they made the products more visible to consumers, one said that discounted products are placed on additional side-stacks, one said that they now stock more of the discounted products and one said that discounted products were briefly featured by the cash registers. Most respondents (65%) said that the configuration of their lighting displays is decided by the store s corporate office. Other factors that contribute to the setup of product displays included price, sale prices, popularity of the product, the patterns of foot traffic within the store, the quantity of the product available and the types of products in stock. Sixty-two percent of the respondents either did not know that there was a limit or did not know that that the limit was 12 light bulbs. Of those who knew there was a limit, the Energy Efficiency Programs 185

189 PSO Energy Saving Products Program range of the per-customer reported fell between 2 and 12 bulbs, with 45% of these retailers reporting a limit size of bulbs. When asked how the limit is enforced, three retailers said that the cash register will not scan more than the allocated number of products or will automatically charge full price for any discounted item over the program rebate, three said that employees intervene to remind customers about the program limit and five said that they either did not know how their store enforces the limit or did not think that it was being enforced consistently. Retailer representatives who said that they rely on employees to enforce the limit also noted that the limit is not always enforced consistently. No retailer reported having a limit on the number of discounted air purifiers that a customer can buy. Product Line Responses suggest that the program has increased the availability of energy efficient lighting products. Thirty percent of responding retailers said that they have expanded their line of at least one of their energy-efficient products in response to the program discounts. The remaining respondents indicate that had either carried the product line or the specific discounted products. When asked about air purifiers, four retailers said that they had carried the discounted products prior to program participation, and six said that they did not know if their store had carried the same product line before joining the program. Program Marketing When asked what types of marketing materials and outreach efforts offered by PSO, almost all retailers identified some sort of in aisle signage or tags designating the products that are discounted. When asked how effective these marketing materials were, 80% of respondents stated that they are at least somewhat effective. The reasons given for their effectiveness were that the materials are eye-catching and help educate customers about the benefits of the program. When asked what they thought the most effective marketing strategy was, the most common response was that it helps to have an associate in place to discuss the program with potential customers. Two respondents said that the tags and stickers are most effective because they can be attached to the specific product that is being discounted, making them easier for customers to identify. One respondent said that signs are most effective because they attract attention. One respondent said that no marketing effort is as good at promoting the program as the price of the discounted item itself. Apart from program provided marketing materials, three retailers stated that they promote the program through different means. These include making displays and encouraging associates to discuss the program with customers. One retailer said that Energy Efficiency Programs 186

190 PSO Energy Saving Products Program they felt that the strategy of promoting discussion between the associate and customer compliments PSO s marketing efforts well because it makes the program personable and helps customers understand how the discount works. Most respondents did not have specific suggestions for improving the program s marketing materials or did not feel that the materials had to be improved. One respondent suggested that it would be helpful if their store could advertise the program through the flyers and newsletters that it distributes. Several respondents suggested providing more information about how the program works as part of program marketing. Eleven respondents said that the program had held in-store promotion days at their store. Nine thought that the event had been at least somewhat effective at promoting the discounted lighting. Those who said that the event had been effective said that it helped answer customer s questions about the discounts, showed them the variety of products available, and generally made customers more aware of the discounts. The reasons given for why the table events were not effective were that the table was not actively manned by program representatives and/or that it caused confusion for customers (the respondent did not elaborate on why it caused confusion). One respondent also noted that, while they felt that the event had been successful, the organization of the event felt rushed and the event likely would have been more successful if there had been more advanced notice. Respondents were also asked if their store did anything to market the program. Four said that their store has taken additional steps to market the program discounts. One respondent said that they use their digital sign located outside the store to advertise the program, and they felt that this was an effective marketing tactic because the outdoor sign is seen by a lot of people. The remaining three respondents said that have special light bulb sales or truckload events. All three of these respondents said that they thought their special store events were good complements to the PSO marketing efforts and that they worked well together. Store Visits Seventy-five percent of respondents said that representatives from the program have visited their store. The frequency of visits varied, with visits occurring as frequently as once a week to once every two to three months. Respondents said that during these visits, program staff teach them about the program, talk to customers about the program, check pricing, and deliver signs and stickers. Reported visit length ranged from fifteen minutes to half a day. The differences in the frequency of visits reported largely corresponds to the number of products sold (i.e., visits were more frequent at stores that sold more products). This relationship suggests that the program is appropriately allocating greater staff time to stores with larger sales volumes. Energy Efficiency Programs 187

191 PSO Energy Saving Products Program Training and Support Four respondents reported receiving training or support from PSO aside from the periodic visits from program staff. Respondents did not provide a lot of detail on what these training sessions entailed many of them were not personally familiar with the training materials. Respondents said that PSO representatives have talked to their staff about the program. It is important to note that much of the training delivered through the program occurs during the site visits and consequently the small number of respondents who reported receiving training outside of these visits should not be taken as indicative of insufficient training. Overall Experience with the Program Table 3-94 summarizes the retailer contact satisfaction with the program overall. As shown, most respondents (88%) were satisfied with the program and rated it as four or higher on a five-point scale where five meant very satisfied. The one dissatisfied respondent said that they hadn t had a lot of contact with the program staff and did not know a lot about the program. An additional respondent rated their experience with the program as a three and said that this because they were dissatisfied with the signage provided by the program. For the rest of the respondents, when asked why they are satisfied with the program, respondents listed a wide variety of positive aspects of the program, saying that the bulbs sell well, that the program staff are friendly, that the program discount helps the customer and helps promote energy efficiency and the wide variety of products available for discount. Specific commentary includes: [The program] helps customers get what they need. The program helps [the store] sell more products and helps customers get on board with energy efficiency. Table 3-94: Retailer Contact Satisfaction Overall Satisfaction with the (n = 17) Program 5 - Very satisfied 59% 4 29% 3 6% 2 6% 1- Very dissatisfied 0% The most frequently noted strength of the program was the affordability of the discounted products. Other noted strengths were the: Promotion of energy efficiency; Energy Efficiency Programs 188

192 PSO Energy Saving Products Program Educational value of the program; Presence of the representative who can speak directly with customers about the lighting; Program signage; and Instant discount as opposed to a mail-in rebate. The limitations or challenges of the program identified by respondents were as follows: Two respondents mentioned that their store does not receive enough bulbs through the program. Two respondents expressed dissatisfaction with the purchase limits on the discount products. Two respondents said that there is not enough customer awareness either of the benefits of LED lighting or of the program itself. One respondent said that the program does not last long enough and one said that the program does not cover a wide enough variety of products. One respondent said that their store does not have enough opportunities to market the program themselves and respondent said that their store does not have enough contact with program staff. Respondents were also given an opportunity to provide feedback on how they would like to see the program improve in the future. One respondent said that they would like more training opportunities from PSO staff. Program Staff Interviews ADM conducted interviews with program management staff from PSO and CLEAResult. These interviews provide insight into various aspects of the program and its organization. The interviews focused on changes to the program and the lighting market that occurred during Interview respondents also discussed aspects of the program operations and results that they considered to be successful as well as challenges faced. Changes in Program Design and Management A few changes were made to the program design in First, CLEAResult was directly contracted with PSO to provide the discounts, rather than providing services as a subcontractor to ICF. Staff noted that this change did not have a significant impact on the management of the program but that the frequency of scheduled meetings had increased from a bi-weekly basis to a weekly basis. In 2016, the program focused more heavily on discounted LEDs in response to the rapidly changing lighting market (discussed below). Additionally, Costco was added as Energy Efficiency Programs 189

193 PSO Energy Saving Products Program a participating retailer to the program. The participating Costco location is the only retail location for that company in the PSO service territory. Another change made to the program was the addition of discounts for ENERGY STAR air purifiers. The discounted air purifiers were offered at two DIY chains on models manufactured by Honeywell, Febreze, and Idylis. The launch of the discounts was delayed by the process of signing Memorandum of Understandings (MOUs) and were not available until mid-year. Discounts of $20 for Febreze units and $50 for units manufactured by Honeywell and Idylis were offered. Staff noted that a challenge to promoting the ENERGY STAR air purifiers is that the units are difficult to locate in the store. However, the impact of this barrier may be minimal since it is more likely that the program would influence the decision about which air purifier to purchase, rather than influence the decision to purchase an air purifier when the customer had not already planned on making that purchase. As such, the key to promoting efficient air purifiers is to ensure that the discounted units are clearly marked and that the discounts and efficiency benefits are understood and valued by the customer. Moreover, the program sales of air purifiers exceeded the goal of 175 units by 53 units. The program had planned on offering discounts on ENERGY STAR ceiling fans but due to difficulty in identifying the type of light bulb in the package, no longer plans to offer this measure. In 2017, the program is looking at offering additional measures as well as an online channel for product sales. Changes in the Lighting Market Program staff responded to multiple developments in the lighting market during Perhaps most notably is that the transition of the residential efficient lighting market from CFLs to LEDs accelerated with manufacturers such as GE deciding to no longer supply CFLs and major retailers such as Walmart committing to decreasing the CFLs offered in their stores. 40 Staff reported that CFLs were removed throughout the year and not many models were discounted by the end of the program year. A related change was the continued fall in LED costs throughout the year that has increased customer interest in the technology as the bulbs have become more affordable. Additionally, staff prepared for the adoption of ENERGY STAR Lamp Specifications v.2.1, which went into effect in January An important change in the v 2.1 guidelines is a reduction of longevity standards for LED lamps. This change allows 40 New York Times (2016). G.E. to Phase out CFL Bulbs. Energy Efficiency Programs 190

194 PSO Energy Saving Products Program several lower cost LEDs to qualify for the program, which should continue to drive down costs. The program will discontinue discounts on CFLs in Marketing and Promotion of Discounts The strategies for promoting and marketing the discounted products remained like those used in PY2015. The primary means for promoting the discounts were in-store signage and promotional events. The in-store materials for discounted lighting focused on the discounts provided. Additionally, information about the discounts is available on the PSO website as well as educational materials on the efficiency benefits of LEDs and CFLs. CLEAResult staff noted that they revised the light meter display that shows the amount of electricity used by three different types of light bulbs. This change was made to make the display more user-friendly. The program offered a light bulb trade-in promotion during 2016 that focused on participating retailer locations in rural communities. The events were held in rural locations to maximize the program s reach with those customers that have typically participated less in PSO s programs. During the event, customers could exchange an incandescent light bulb for an LED. Initially, staff limited the number of bulbs that could be exchanged to five, but increased this limit to eight. Staff indicated that there were not as many customers interested in exchanging bulbs as they had hoped, but that they did have some success with the event. The program ran the Shine-a-Light campaign again in As in prior years, PSO donated a CFL for each discounted light bulb purchased during a portion of the program year and then distributed these light bulbs through three food banks. The program gave away 100,000 light bulbs through this channel in ESP Program Conclusions and Recommendations Based on the findings from the PY2016 evaluation of the in-store lighting program, ADM has developed the following conclusions. Program Design Two primary design changes were the addition of ENERGY STAR air purifiers to the program and an increasing focus on LED lighting. Sales of ENERGY STAR air purifiers exceeded program expectations, despite the discounts introduction at midyear. The focus on LED lighting was largely driven by changes in the lighting market and the reduced availability of LEDs. Additionally, a Costco location was added to the program and added significantly to the total program sales. Energy Efficiency Programs 191

195 PSO Energy Saving Products Program Staff will only offer discounts on LEDs in The new ENERGY STAR guidelines should lower the cost of qualified lighting and allow the program to set lower discounts rates. The incentive cost per expected kwh saved for standard LEDs continued to decline in 2016 and was about 12% higher than for standard CFLs. Program Administration 2016 was the first year that CLEAResult was directly contracted with PSO to administer the lighting discounts. Staff reported that this change had minimal effect on program management. Currently, savings for the program are tracked outside of the Vision system used to track the remainder of PSO s program savings. Program Implementation During site visits, ADM found that most retailers displayed most discounted LEDs at eye level and that 76% of the stores displayed between % of the LEDs at eye level. This finding suggests that products are effectively placed to maximize sales by lighting retailers. Most of the stores visited displayed information on lighting discounts in the lighting aisle. However, ADM found one Mass Merchant Retailer that did not have any signage displayed for lighting discounts. Two stores that offered discounts on air purifiers were visited. At one location, ADM did not identify any signs noting that the air purifiers were discounted. At the other location, ADM found three ENERGY STAR models of the same manufacturer without a sign noting the discount. Two of the three models were confirmed as included in the MOU. Interviews with retailer contacts found that most retailers are satisfied with the program and confirmed CLEAResult s in store quality control activities (e.g., program staff teach them about the program, talking to customers about the program, check pricing and delivering signs and stickers). However, many interview respondents either did not know that there was a limit to the number of bulbs purchased or did not know the correct number of bulbs that could be purchased. Declining LED pricing has resulted in some non-discounted LEDs competing with discounted ENERGY STAR LEDs on price. During store visits, ADM identified one location that offered non-program LEDs at a price comparable to the discounted bulbs. Customers may need additional education on understanding the advantages of ENERGY STAR LED qualified bulbs over non-qualified bulbs. An advertising campaign was launched in 2017 to educate customers on the lighting color, lumens, and quality benefits of ENERGY STAR lamps. Staff noted that an education challenge for the program was to help customers understand lumens to make appropriate selections for their homes given that changes in lamp wattages have made it more difficult to select an appropriate bulb. Energy Efficiency Programs 192

196 PSO Energy Saving Products Program Corroborating the importance of this aspect of lighting were survey responses that indicated the brightness of the bulb was the most frequently endorsed reason for purchasing an energy efficient light bulb. Based on the findings from the PY2016 evaluation of the in-store lighting program, ADM offers the following recommendations: Review protocols to verify that program discounts are consistently displayed. CLEAResult currently performs onsite visits to verify that signs are on display during site visits but an additional review may be needed given the identification of stores that did not display signs. Consider additional lighting types. As staff considers expanding the EPS product line, staff should consider offering LED Retrofit Trim Kits when reviewing measures to offer through the program. Additionally, consider ER30 and ER40 directional LEDs with 65W equivalencies or less than 50W equivalency. Update lighting educational materials on the website. The educational lighting materials on the website should be updated to focus on LEDs rather than CFLs. Additionally, materials should be added to help customers understand the benefit of choosing a discounted LED over a non-energy STAR lamp in the same price range. In addition to the development of PSO branded materials, the program website could add a link to the ENERGY STAR page titled Why Choose ENERGY STAR Qualified LED Lighting? Staff also noted that customers likely need education on how to choose appropriate lighting for their home. An advertising campaign was launched in 2017 that focuses on the benefits of ENERGY STAR lamps Planned Program Changes The primary changes made or planned for 2017 are as follows: The program will only offer LEDs that are certified under ENERGY STAR v.2.1 in The program is reviewing adding new measures to the program including: Mail-in rebates for ENERGY STAR water coolers, bathroom ventilation fans, dryers and heat pumps water heaters. Point of Sale discounts for advanced (smart) power strips. A small residential pilot for connected lighting. Energy Efficiency Programs 193

197 PSO High Performance Homes 3.4 High Performance Homes Program Program Overview PSO s High Performance Homes program seeks to generate energy and demand savings for residential customers through the promotion of comprehensive efficiency upgrades to building envelope measures and HVAC equipment for both new construction homes and retrofits to existing homes. Offering PSO customers direct inducements for higher efficiency measures offsets the first cost obstacle, encouraging customers to choose the upgraded products. The program can essentially be divided into two components: New Homes, Multiple Upgrades, and Single Upgrades. The New Homes component provides home builders with inducements for increasing building envelope, HVAC system, and lighting efficiency for homes built in the PSO service territory. A key design change for 2016 was changing the program incentive structure from performance incentives based on kwh savings to prescriptive incentives based on meeting program design specifications and for making efficiency improvements beyond the base requirements. The 2016 incentives are summarized in Figure Energy Efficiency Programs 194

198 PSO High Performance Homes Figure 3-33: Rebate Guidelines and Amounts The Multiple Upgrades component of the program focuses on energy efficiency upgrades to existing residential homes. To qualify for the program in 2016, customers needed to three or more eligible equipment upgrades. Eligible measures include: Central air-conditioning systems (CAC) SEER 15 or higher Air source heat pumps (ASHP) SEER 15 or higher Ground source heat pumps (GSHP) Duct system sealing (or replacement) Air Infiltration reduction measures Attic insulation Exterior wall, knee wall, and floor/crawl space insulation ENERGY STAR windows and glass doors Solar screens Energy Efficiency Programs 195

199 PSO High Performance Homes Radiant barriers Electronically Commutated Furnace Fan Motor (ECM) High-Efficiency Electric Water Heater The Single Upgrades component of the program focuses on energy efficiency upgrades to existing residential homes. To qualify for the program in 2016, customers needed to one or two eligible equipment upgrades. Eligible measures include: Central air-conditioning systems (CAC) SEER 16 or higher Air source heat pumps (ASHP) SEER 16 or higher Ground source heat pumps (GSHP) Attic insulation ENERGY STAR windows and glass doors Solar screens Electronically Commutated Furnace Fan Motor (ECM) High-Efficiency Electric Water Heater Variable Speed Pool Pumps The Multiple Upgrades component includes a walk-through assessment from a PSO approved contractor to help identify energy efficiency measures that could improve customers comfort level while reducing energy costs. Once the initial audit is performed, a PSO/ICF contracted employee, also referred to as PSO Home Energy Rater, will perform a diagnostic test on the home before and after installation of upgrades are made. This process is in place to measure and document efficiency gains from infiltration reduction and duct sealing measures. In 2016, the program had a total of 4,895 measures installed as part of 1,258 unique projects through the Multiple Upgrades component of the program and 8,510 measures installed as part of 2,598 unique projects through the Single Upgrades component of the program. Additionally, participating builders completed 677 rebated homes through the New Homes component. PY2016 performance metrics are summarized in Table. Overall, reported energy savings exceeded projected values considerably. Verified gross energy savings of 8,486,679 kwh represent 101% of reported values. The level of interest in the HPH program allowed for additional customer incentives while holding implementation expense unchanged. At year end, actual expenditures were $128,681 under budget (1.47% under budget). The kwh energy savings for the year was 11% over projected savings. Energy Efficiency Programs 196

200 PSO High Performance Homes Table 3-95: Performance Metrics High Performance Homes Program Metric PY2016 Number of Customers 5,542 Budgeted Expenditures $8,746,440 Actual Expenditures $8,617,759 Energy Impacts (kwh) Projected Energy Savings 7,540,775 Reported Energy Savings 8,405,641 Gross Verified Energy Savings 8,486,679 Net Verified Energy Savings 6,710,752 Peak Demand Impacts (kw) Projected Peak Demand Savings 3,985 Reported Peak Demand Savings 3,286 Gross Verified Peak Demand Savings 3,353 Net Verified Peak Demand Savings 2,652 Benefit / Cost Ratios Total Resource Cost Test Ratio 1.45 Utility Cost Test Ratio 1.19 The remainder of this section details the EM&V methodologies and findings for the High Performance Homes program. The New Homes component is reported first in Section 3.4.2, with the Multiple Upgrades component following in Section New Homes Component EM&V Methodologies This section provides a brief overview of the data collection activities, gross and net impact estimation methodologies, and process evaluation activities that ADM employed in the evaluation of the New Homes component of the High Performance Homes program. Energy Efficiency Programs 197

201 PSO High Performance Homes Data Collection For the New Homes component, the main impact evaluation activity was an engineering review of building simulation models and project documentation for a sample of program rebated homes. Data collection activities that supported the evaluation include builder interviews, verification site visits, and in-depth interviews with program staff at PSO and implementation. The sample for the engineering review of building simulation models was designed to achieve ±10% relative precision or better at the 90% confidence interval. Table 3-96 below shows the achieved sample design. Table 3-96: Sample Design High Performance New Homes Stratum Number Reported kwh Savings Strata Boundaries (kwh) Population Size CV 41 Sample Size 1 228,129 < 1, ,558 1,250-2, ,388 2,250-14, ,879 > 14, Total 1,291,954 N/A 677 N/A 42 For each sampled home, ADM requested and received project documentation and modeling files from the program implementer. The provided materials included REM/Rate simulation files, HERS rating certificates, QA/QC field data forms, Beacon PST Input/Output Summaries, DOE2 Input files, and DOE2 simulation files. The documentation reviews were supplemented by ride-along field visits with the implementation contractor and interviews with builders. The ride along visits allowed ADM to observe the program data collection protocols and access the inputs to the simulation models. The builder interviews were used for the program attribution analysis and to obtain builder feedback about the program. Furthermore, ADM conducted 25 independent site visits. Lastly, ADM performed six desk reviews of homes that were not visited by either the implementer or ADM. Table 3-97 summarizes the data collection activities and sample size for the New Homes component of the program. The achieved 41 The coefficient of variation (CV) used for sample design purposes was the based on the variation in reported kwh savings within each stratum: CV = σ (kwh) / µ (kwh). Energy Efficiency Programs 198

202 PSO High Performance Homes sample design resulted in verified gross kwh estimates with ±9.8% relative precision at the 90% confidence interval. Table 3-97: Samples Sizes for Data Collection Efforts New Homes Data Collection Activity Achieved Sample Size New Homes: Engineering Reviews 42 New Homes: On-Site M&V 25 In-depth Interviews with Program Staff 3 Interviews with Participants (Builders or Home Owners) 15 Gross Impact Methodologies The first step in conducting measurement of the New Homes component activity was to review program tracking data and identify the population of homes and plan types built and rebated through the program in The data tracking system was reviewed to ensure that the proper data fields required to support this evaluation as well as future evaluations were included. Furthermore, the tracking data was screened to ensure there were no duplicate entries or other inconsistencies. After reviewing the program tracking data, establishing a program population, and pulling a sample of rebated homes for further engineering review, ADM performed the following verification steps for the 2016 analysis: ADM Reviewed ex ante Model (Beacon PST TM ) The building characteristics that form the basis of the simulation modeling for each rebated home are initially submitted by participating builders/hers raters in the form of REM/Rate modeling files. The implementer s proprietary Beacon Predictive Savings Tool then converts the REM/Rate home model into a DOE2.1 simulation file. The DOE2.1 input and output files were reviewed for each sampled prototype home. Specifically, the differences between input files were compared to identify the modeling approach take for above code improvements. This also included a thorough review of DOE2.1E parameters, their values, and their applications. These models were then updated as described in the bullet points below and used to derive the reported ex post verified savings values. Energy Efficiency Programs 199

203 PSO High Performance Homes ADM Developed Baseline 42 Values for Key Simulation Inputs The ex post reference home modeling assumptions are outlined in Table Values for the listed parameters were taken from either the 2009 building energy codes (IECC or IRC) or the Mortgage Industry National Home Energy Rating Systems Standards (RESNET). ADM also reviewed the values used for these parameters in the Beacon PST reference home models. ADM found most of the ex ante values to be consistent with the ex post values. ADM Developed Key Simulation Inputs for ex post Design Models ADM verified the modeling inputs for each sampled home in several ways. First, in our engineering desk review of the ex ante simulation files, we found several modeling methods that needed updating. These included recalculation of the floor heat transfer coefficient term to be consistent with use of custom weighting factors, updating heating/cooling set-point schedules to be consistent across reference and rated home simulations (and with RESNET), and application of TMY3 weather data. In addition to the modeling changes, ADM also updated the key model parameters (the same parameters listed in Table 3-98) with values consistent with our on-site findings. 43 Finally, some values were further researched using satellite imaging and data provided by the county assessor s office (for example, home orientations were confirmed using satellite images and pictures taken on-site). 42 Note that the term baseline in this instance refers to the "reference" home model. 43 ADM observed/verified as-built conditions through on-site data collection activities. Energy Efficiency Programs 200

204 PSO High Performance Homes Table 3-98 PY2016 Baseline Home Assumptions Input Ex post Reference Home Source Roof Solar Absorptivity IECC Reference home, Table (1). Attic Insulation: R IRC Table N values. Cathedral Ceiling Insulation: R-19 Ceiling Insulation Grade: IRC section N requirements of ceilings without attic spaces. The overall U-factor of the ceiling assembly is calculated based on RESNET standards. Wall Construction: 2x4 (16-inch O.C.) Wall Insulation: R IRC Table N values. Wall Insulation Grade: 1 RESNET Standard Wall Sheathing: Plywood 2009 IRC Table N values. Door R: R IRC Table N fenestration requirements. Window U IRC Table N values. Window SHGC IRC Table N values. Infiltration F-L-A 2009 IECC Reference home, Table (1). Slab Edge Insulation None 2009 IRC Table N values. HVAC Equipment Type As proposed 2009 IECC Reference home, Table (1). Cooling Efficiency (SEER) 13 NAECA minimum values. Heating Efficiency (AFUE) 80 NAECA minimum values. DSE (Heating / Cooling) 0.8 ANSI-RESNET 301 Table 4.2.2(1) Percent Fluorescent Lighting 50% IRC 2009 Prescriptive Requirements ADM analyzed 42 homes, 35 of which were engineering reviews. The remaining 7 were desk reviews. Of those 42 homes, ADM visited homes 1 through 25, meanwhile, implementation performed QC on homes 26 through 35. ADM also performed some Energy Efficiency Programs 201

205 PSO High Performance Homes 'post-processing' on the ex post simulation outputs to account for components that were not modeled in DOE2.1E. These components included impacts from the domestic hot water system and the duct system efficiency (listed in Table 1-4 as DSE). Verified energy and demand impacts for each home were calculated by taking the difference between the reference and rated home energy simulations. The verified gross peak demand reduction was calculated using the DOE2.1E output - taking the maximum difference concurrent demand from June through September. Realization rates for gross annual kwh savings and gross peak kw reduction were calculated for all sampled homes. Program results were derived by extrapolating the results from sampled homes to the population of participating homes according to the sample design. Net-to-Gross Estimation As the program primarily consists of the same builders from year to year, ADM conducted builder interviews for 14 builders, representing just over 70% of the total program ex ante kwh savings to estimate a Net-to-Gross ratio for the New Homes component of the High Performance Homes program. The interview methodology has been included below, which shows the calculation of Net-to-Gross. Spillover was not considered for the New Homes component of the program based on feedback from participating builders and program staff. Builders reported that they include energy efficiency measures designed specifically to qualify for the program for all the homes they build within the PSO service territory. Additionally, based on conversations with program staff, those builders who do not participate in the program often choose not to because they are not interested in adopting the required efficiency measures or undergoing the HERS rating requirements of the program. As such, anecdotal evidence suggests that non-participant spillover is likely limited or non-existent as well. The program likely has some market transformation effects through builder education that could result in future non-incented savings. These effects are not quantified for this analysis. Net-to-Gross scores were developed for each interviewed builder by analyzing responses to three lines of questioning: program influence, building practices in the absence of the program, and co-participation in other rebate programs. Each line of questioning was used to account for 1/3 of the overall free ridership score for each respondent. That is: Total Free Ridership = 1/3 x Program Influence FR + 1/3 x Building Practices in the Absence of the Program FR + 1/3 x Co-Participation FR. The scoring for each line of questioning is detailed below. Program Influence: Builders were asked to rate the influence of the program on their decision to build an energy efficient home. The ranking was recorded on a scale of one to five with one representing not at all influential and five representing very influential. Free ridership percentages were applied to the answer as follows; 5 = 100%, 4 = 75%, 3 = 50%, 2 = 25% and 1=0%. The Energy Efficiency Programs 202

206 PSO High Performance Homes builders were then asked to list all factors influencing their decision to build an above code energy saving home. In cases where builders reported the program as having very little or no influence, but also reported consideration of rebate reductions to building costs, guidance from raters or program staff, or competition with other program builders as being a contributing factor, the initial Net-to-Gross score was increased by 10 percent. Building Practices in the Absence of the Program: Builders were then asked about the percentage of homes they would have built to an above code energy standard if the PSO New Homes Program were not available during They were also asked to report the percentage of homes they would build to an above code standard if the program had never existed (to account for prior year program influence). The reported percentages from the two questions were averaged to determine a free ridership score for this line of questioning. Co-participation in other Rebate Programs: Builders were then asked about participation in the Federal Energy Star New Homes Tax Credit (which expired on December 31, 2016). If they did not participate in the Federal Energy Star New Homes Tax Credit a Net-to-Gross score of 100% was applied for this line of questioning. If they did participate in the Federal Energy Star New Homes Tax Credit program and the average square footage of the homes they built in the New Homes program was greater than 2000 feet, a Net-to-Gross score of 100% was applied. If they did participate in the Federal Energy Star New Homes Tax Credit program and the average square footage of the homes they built in the New Homes program was less than 2000 feet, a Net-to-Gross score of 75% was applied. Energy Efficiency Programs 203

207 PSO High Performance Homes Figure 3-34: New Homes Net-to-Gross Logic Chart Process Evaluation Activities The purpose of the process evaluation of the New Homes component was to examine program operations and results, to assess how the program has developed since the prior program cycle, and to identify any key areas that may need to be addressed in future program years. Energy Efficiency Programs 204

208 PSO High Performance Homes In addition to a review of the program design and operations, the process evaluation sought to address the following questions. Has the modification of the performance incentive amounts, incentive limits, and the addition of prescriptive incentives increased builder interest in the program? Are builders that have not participated in recent years participating again? What has been builder s response to the ENERGY STAR certification bonus? Do they see ENERGY STAR certification as adding marketable value to the home? How does the design assistance incentive address barriers to construction of ENERGY STAR certified homes? Impact Evaluation Findings Overall, verified gross energy savings are estimated at 1,350,358 kwh, as shown in Table 3-99, the difference in the reported and verified energy savings results can be attributed to some differing model inputs for the sample homes. This represents a gross realization rate of 105%. Verified gross peak demand reduction is estimated at 462 kw, with a corresponding realization rate of 91%. Table 3-99: Reported and Verified Gross Impacts - New Homes Component Reported Energy Savings (kwh) Reported Peak Demand Savings (kw) Verified Gross Energy Savings (kwh) Verified Gross Peak Demand Savings (kw) New Homes 1,291, ,350, The impact analysis found that, for the sampled homes, the ex ante simulation models generally reflected the building characteristics verified during implementation QA/QC field verification visits. While some exceptions to this were found, these led to minor and mixed variances across the sampled homes as illustrated in Table Note that each + and (-) signifies a difference between the reported model input files and the verified model input files that either increased or decreased the savings. Finally, all above code improvements documented during the ADM field visits were confirmed with builders when interviewed. Energy Efficiency Programs 205

209 PSO High Performance Homes Table 3-100: Changes to Reported Model Inputs by Sample Home Building Model Inputs Modeling Approach Site Ceiling Insulation Infiltration Lighting EER Vs. SEER TMY3 U EFF Site RR 1 (+) ( ) (+) (+) 118% 2 (+) ( ) (+) (+) 110% 3 (+) ( ) (+) (+) 110% 4 (+) ( ) (+) (+) 110% 5 (+) ( ) (+) (+) 117% 6 (+) ( ) (+) (+) 100% 7 (+) ( ) (+) (+) 114% 8 (+) ( ) (+) (+) 114% 9 (+) (+) ( ) (+) (+) 124% 10 ( ) ( ) (+) (+) 98% 11 (+) ( ) (+) (+) 137% 12 (+) ( ) (+) (+) 103% 13 (+) ( ) (+) (+) 117% 14 (+) ( ) (+) (+) 113% 15 ( ) (+) (+) 106% 16 ( ) (+) (+) 101% 17 (+) (+) ( ) (+) (+) 120% 18 (+) ( ) (+) (+) 95% 19 (+) ( ) (+) (+) 119% 20 ( ) (+) (+) 85% 21 (+) ( ) (+) (+) 105% 22 (+) ( ) (+) (+) 107% 23 (+) ( ) (+) (+) 135% 24 ( ) ( ) (+) (+) 79% 25 (+) ( ) (+) (+) 122% 26 (+) (+) ( ) (+) (+) 117% 27 ( ) (+) (+) 117% 28 ( ) ( ) (+) ( ) (+) (+) 88% 29 ( ) (+) ( ) (+) (+) 73% 30 (+) (+) ( ) (+) (+) 121% 31 (+) ( ) ( ) (+) (+) 113% 32 (+) ( ) (+) (+) 125% 33 ( ) ( ) (+) (+) 101% 34 ( ) (+) (+) 97% 35 (+) (+) ( ) (+) (+) 115% 36 ( ) (+) ( ) 80% 37 ( ) (+) (+) 88% 38 ( ) (+) ( ) 82% 39 ( ) (+) (+) 102% 40 ( ) (+) (+) 83% 41 ( ) (+) (+) 103% 42 ( ) (+) ( ) 103% Energy Efficiency Programs 206

210 PSO High Performance Homes Program Net-to-Gross Estimation Results Previous builder interviews were used to estimate net-to-gross ratios for the New Homes component of the High Performance Homes program. The methodology used for the net-to-gross analysis is described in section 0. For the New Homes program component, Net-to-Gross ratios (ranging from zero for complete free ridership to one for no free ridership) were determined for 14 participating home builders representing over 70% of the total program ex ante kwh savings. Net-to- Gross ratios were weighted by the builder s kwh savings contributions to the program. The final component level Net-to-Gross ratio was 84.7% (commiserate free ridership score of 15.3%). Anecdotal evidence suggests that the magnitude of both participant and non-participant spillover is negligible. The program may have some market transformation effects, but no attempt was made to quantify these effects in terms of additional energy and demand impacts. Results from the builder interviews suggest that the new program design for the New Homes component has had a positive impact on free ridership levels. Table below summarizes gross and net impacts for the New Homes component. Table 3-101: Verified Gross and Net Impacts New Homes Program Component Verified Gross kwh Verified Gross Peak kw NTGR Net kwh Net Peak kw New Homes 1,350, % 1,143, Process Evaluation Findings This section presents the key findings derived from interviews conducted with program management and implementation staff from PSO and ICF, as well as from interviews with participant HERS raters in PSO s New Homes component of the High Performance Homes program. Program Participation During 2016, 677 homes were completed and received incentives through the New Homes Program. The average cash-inducement paid to participating builders in 2016 was $1,924 per home, and a total of $1,308,817 in cash inducements was paid during the program year. Fifty-three builders completed homes through the program during the 2016 program year. Builders completed an average of 13 homes during the program year, however, the distribution of the number of homes built had a strong positive skew and the median Energy Efficiency Programs 207

211 PSO High Performance Home Program number of homes completed was 1. The distribution in the number of homes completed is shown in Figure Figure 3-35 Number of Homes Completed by Builders Similarly, the expected kwh savings was also positively skewed and the nine most active builders accounted for 78% of program expected kwh savings. Figure 3-36 Percent of Expected kwh Savings by Builder Energy Efficiency Programs 208

212 PSO High Performance Home Program Table summarizes 2016 builder participation during the last four years. As shown, 30% of the 2016 builders had participated in the program continuously since Forty-five percent of builders that completed projects in 2016 did not participate in the program during the last cycle. These new builders accounted for a small share of homes during 2016 but represent an expansion of the programs reach to builders in Oklahoma. Staff indicated that the new prescriptive incentives had led some builders who had not been active in recent years to complete program homes in Analysis of the program data provides some support for this. Eight percent of the 2016 participants had not completed a home since 2014 and two percent had not completed one since Although multiple factors may have impacted the decision of these builders to complete homes through the program in 2016 after a one or two-year hiatus, it is possible that the new incentive structure contributed to their decision to participate in Table Builder Participation during the Participation During the Period Percent of 2016 Participants Percent of 2016 Homes Completed Participated All Four Years 30% 22% Did not participate during % 1% 2016 Participant that Last Completed a Home in % <1% 2016 Participant that Last Completed a Home in % <1% Builder Interviews The Evaluator completed in-depth interviews with 13 builders and two individual homeowners that built homes through the program during the 2016 program year. Interview topics addressed 2016 program design changes, program influence, the value of partnering with HERS raters, program provided training and participation processes. Service Providers also provided feedback on their overall satisfaction with the New Homes program component and made recommendations for improvements. Organizational Background The New Homes program component is comprised of large, mid-size and small builders, and defined by the number of program homes they completed, as well as single homeowners that are not in the business of building homes, but rather seek program incentives for the construction of their own home. Energy Efficiency Programs 209

213 PSO High Performance Home Program To better understand the participant builder population, the evaluation team asked builders to approximate how many homes they plan to finish in As shown in Table 3-103, we spoke with five large home builders that completed 100 or more homes through the program, one mid-size builder and seven small home builders. Table Respondent Sample Size Relative to Distribution of Program Savings Builder Size # of Builder Respondents Large Home Builder ( homes) 5 Mid-Size Home Builders (16 99 homes) 1 Small Home Builders (2-15 homes) 7 Single Homeowners (1 home) 2 Total 15 Interviewed builders indicated that most of the homes their firms built during the year were constructed in PSO s service territory. All but two of the respondents indicated that 100% of the homes built in the service territory would receive program incentives. Builders also provided feedback regarding what percent of the homes they built in 2016 were custom-built to the owner s specification and what percent were production homes. The majority (47%) of large home builders indicated they build both custom and production homes. The small to mid-size home builders were more likely to indicate 100% custom or 100% production. Table Builder Types Production verse Custom Builder Type Count 100% Production Builder 4 100% Custom Builder 2 Both Custom and Production 7 Homeowners that Completed Program Projects 2 Energy Efficiency Programs 210

214 PSO High Performance Home Program Of the 13 builders interviewed, four indicated they had not participated in the New Construction component of the HPH Program in These four builders stated they were not aware of the program prior to their 2016 participation. The feedback suggests that program awareness is still growing among the builders operating in the PSO service territory Program Changes The New Homes program component implemented two key design changes in 2016, the introduction of incentives and design assistance for ENERGY STAR certification and providing prescriptive incentives instead of performance based incentives. The following section summarizes builders feedback regarding those changes. ENERGY STAR Certification Bonus Incentive and Design Assistance. In 2016, the program added a $1000 bonus incentive for homes that met the ENERGY STAR v3.1 requirements. To assist builders in developing plans that met the ENERGY STAR certification standards, the program provided design assistance to builders to modify architectural and HVAC plans at no cost to the builder. Thirteen (87%) builders indicated they were aware of the bonus incentive for the certification, but only 5 (33%) were aware that design assistance is available. Three respondents indicated a portion of their homes will receive the ENERGY STAR certification. Builders also provided feedback regarding whether the ENERGY STAR certification provides additional value to home buyers or if it helps builders sell their efficient homes. Most builders indicated that homebuyers are interested in the efficiency of the home, including tight thermal barriers and high-efficiency appliances, but are not more inclined to buy a home with an ENERGY STAR certification. Examples of the comments made include: We are not concerned with the ENERGY STAR logo because I can meet with customers to talk to them about energy efficiency - discuss the things that make their home a better-quality home. The cost of upgrading to ENERGY STAR v3.1 was expensive, three to four more hours on every home to develop a special system, it was not cost effective. It doesn't make it easier to sell a home, but it's because we re one of the few companies actually doing it. [The ENERGY STAR certification] does not add much value to homebuyers. Also, banks don't look at it differently either. Appraisers add $500, so that's not worth it. 44 Two interview respondents were home owners that completed projects through the program. Energy Efficiency Programs 211

215 PSO High Performance Home Program It [the ENERGY STAR certification] does not add value. Home buyers value HERS ratings. The ENERGY STAR rating adds costs and delays, which reduces my ability to be competitive. So I go part way there. I like that [in the program] I can choose the level of efficiency and different tiers. Prescriptive Incentives. In 2016, the incentive structure changed and rebate amounts were no longer based on a dollar per kwh saved rate; rather the dollar amount was based on a prescriptive set of measures, that were considered the minimum requirements, with bonus incentives for equipment types that exceeded the minimum requirements. Builder feedback on the new prescriptive incentives was largely positive with the primary benefit being the increased certainty that prescribed incentive levels provide over performance based incentives. That increased certainty enables builders to anticipate the rebate amounts and to know in advance of home completion what portion of the cost of the efficiency improvements will be covered by the program. If it's [the rebate amount] based off savings, it's a gray area for me. If you do it based on ratings, it's a lot more clear cut. There seemed to be more rebate dollars in 2016 in the structuring verse how it was structured in We appreciate that. The performance approach is always a trick; we hope that the upgrade performs well enough to do what it is supposed to do. The prescriptive approach helps us make decisions for the home. The prescriptive measures make the program more enticing. When our dollars are based on testing there a lot of other factors at play, there s just no guarantee how much were going to get. Experience with HERS raters Participating builders are required to partner with a HERS rater to qualify for program incentives. Builders provided feedback on the benefits of this partnership and the points in the design/construction process when they get the greatest value from consulting with a HERS rater. The most common benefits mentioned by builders are summarized below. HERS raters know the program well; they can assist with all the paperwork and interpret the program guidelines when necessary. HERS raters also have information on other utility rebate programs and federal tax incentives. They can perform load calculations prior to construction, offer advice on insulation and duct sealing, complete the necessary diagnostic testing required for program incentives, and even provide suggestions on framing strategies for energy efficiency. Energy Efficiency Programs 212

216 PSO High Performance Home Program Several builders indicated that a HERS rating can be a selling point, although it takes an informed buyer. Builders indicated that they consult with HERS raters during the design as well as the construction phases. Although HERS raters seemed to be of most value once the framing was complete and insulation was ready to be installed, several builders noted that they valued the HERS rater s opinion during the design process, especially when they have new plans being developed or existing plans are modified to account for codes changes. One builder said that he involved the HERS rater in the design phase of his last project, but found the assistance provided lacking because the rater could not estimate the cost of the recommendations made. Another builder commented on the cost of hiring a HERS rater, stating he was charged $900 which accounted for approximately 20% of his rebate. The builder suggested that the program provide an incentive to offset the cost associated with hiring a HERS Rater. Program Influence on Efficient Construction PSO s New Homes program component seeks to inform builders about the latest technologies and strategies for building beyond code. About one-half of the interviewed builders (53%) indicated the program was very effective in increasing their company s knowledge of energy efficiency. Some of the insights gained included increased knowledge of insulation, air sealing, framing, HVAC systems, and the steps necessary to not only install these measures but also to ensure it s done correctly. Another key benefit of participating in the program noted by builders was the knowledge gained through working with the program HERS raters. Many builders indicated that through program participation they learned from, and valued, the input HERS raters provided at various points in the design and construction process. Only two builders said the program did not impact their knowledge of energy efficiency construction. One respondent noted that the HERS raters really drive the program, and as a result, they were not too involved in the participation process. Builders discussed the long-term impacts of the New Homes program component has had on their companies building practices. Several interviewees noted that they would still be building homes that exceed the building code because high-efficiency home building strategies have been integrated into their business models. A common belief espoused by the builders was that home buyers associate their homes with premium quality construction and that efficient construction is part of quality construction. One builder described the home building industry as one that moves in a forward direction, buyers want the latest and greatest, which is why they purchase a newly constructed home. The builder emphasized the point that over the years PSO's program, in combination with other regional and federal programs, have been very influential in developing the market for high-efficiency home building and that they would not be building to the same efficiency standards without these long-standing Energy Efficiency Programs 213

217 PSO High Performance Home Program programs. Several other builders shared this sentiment. One home builder emphasized the importance of the ENERGY STAR bonus offered in Specifically, the interviewee said they may have exceeded the code requirements without the program, but that they would not have achieved the ENERGY STAR Certification had the new bonus incentive not been offered. These comments suggest that for several builders, the program s continued impact on efficient construction is likely to be the result of increasing its efficiency standards either in response to code changes or as part of general program development. Program developments such as the addition of incentives and design assistance for ENERGY STAR certifications are important for the program to continue to affect the efficiency of new construction. Program Provided Training PSO s High Performance Homes Program offers various outreach and training events for both Service Providers and builders throughout the program year. One of the 15 builders interviewed indicated they received training through the program and many were surprised to hear that training was available. Program staff also meet with builders on an individual basis and these comments may not refer to this form of training and outreach The builder that received training was the most active and largest builder in the program. They indicated their sales staff had positive feedback regarding the training and said they learned about how to emphasize the benefits of the energy efficiency to potential homebuyers. Program Satisfaction and Suggestions for Program Improvements Builders provided feedback on the participation process and their overall satisfaction with the New Homes program component. This feedback is summarized below. Most respondents noted that their HERS raters do most of the paperwork and therefore they could not comment directly on the participation processes, paperwork or documentation requirements. All but one respondent rated their overall satisfaction with the program a 4 or 5, on a 5-point scale. The one builder who provided a rating of 2 indicated that it was difficult to get enrolled in the program and indicated it took more than 10 phone calls and s for someone to get back to him. This was the only critical feedback provided on the enrollment process. The only other points of dissatisfaction expressed by builders were related to the timing of payment, several builders indicated that the rebate payment timeline is unclear. Staff should consider providing information on the rebate payment timeline in the Homebuilder Agreement. Energy Efficiency Programs 214

218 PSO High Performance Home Program Suggestions for Improvements Builders offered the following suggestions as ways to improve the New Homes program component. Several builders commented on the difficulty of reconciling rebate checks with individual homes. In the future, they would like more project-specific information provided with the rebates. Additionally, they would like to know when to expect payment. Many of the builders also suggested increasing incentive amounts to cover more of the costs associated with meeting program requirements. This includes higher incentives for not only energy savings measures but also a rebate to cover the cost of the HERS rater. Education was a common theme discussed by builders. They believe that not only builders but also the public needs to be educated about the benefits of building and living in more efficient homes. Many buyers are focused on amenities such as granite counter tops. Four builders, we spoke with mentioned the Home Builders Association as a good place to reach out to other area builders regarding the program offerings. Program Design and Operations The following section provides an overview of the High Performance Homes New Construction Program design and operations developed from in-depth discussions with program staff and a review of program documents used to guide implementation and inform program partners. Program Design A key design change for 2016 was changing the program incentive structure from performance incentives based on kwh savings to prescriptive incentives based on meeting program design specifications and for making efficiency improvements beyond the base requirements. Projects permitted on or after February 1 st, 2016 fell under the prescriptive program incentives. The 2016 incentives are summarized in Figure Energy Efficiency Programs 215

219 PSO High Performance Home Program Figure 3-37 Rebate Guidelines and Amounts Staff introduced the prescriptive approach in response to concerns raised by builders that there was uncertainty about what their incentive payment would be at the time construction decisions were made and costs were incurred. Another key change was the introduction of the ENERGY STAR certification bonus. The program offered a $1000 bonus for new homes that met the ENERGY STAR v3.1 requirements. Because the ENERGY STAR certification requires additional administrative work for HERS raters, the program also offered HERS raters a $150 bonus for certified homes. To further support builders in meeting the ENERGY STAR requirements, the program offered incentives to reduce the additional costs of architectural and HVAC design fees associated with meeting the program requirements. Staff indicated that very few ENERGY STAR homes were completed in Oklahoma during the period when PSO incentives were not offered for meeting ENERGY STAR Energy Efficiency Programs 216

220 PSO High Performance Home Program guidelines. Data provided by the U.S. Department of Energy s website corroborates this. In 2015, four homes received ENERGY STAR certification. 45 Staff noted that 2016 budgets were set for new homes projects with allocations made for production builders, single family home owners building homes through the program, and new builders participating in the program. These budgets are reviewed in mid- September and reallocations are made as necessary. Staff noted that they did not have to turn down any projects in 2016 but incentive dollars are available on a first come, first served basis. Program Management and Staffing The program team is comprised of both PSO and ICF staff members, who are collectively responsible for program implementation and oversight. PSO program staff addresses issues with implementation as they arise, which include but are not limited to resolving issues with program builders, interpreting program guidelines, and overseeing the budget. The ICF program team manages the day-to-day operations of the HPH Program including application review and approval, payment processing, on-sight home assessments and testing, as well as service provider recruitment and outreach. PSO and ICF hosts a bi-weekly conference call to discuss topics related program activity and progress towards goals, guideline interpretation, and project specific issues. Staff also and speak on the phone as needed and in-between the regularly scheduled meetings. All staff interviewed indicated the relationship and level of communication between PSO and ICF remains strong and adequately supports the administration needs of the program. Builder and HERS raters enter project information into an online tool. Through this tool, incentive dollars are reserved for specific projects. Builder information is entered once into the tool and then accessed on subsequent occasions as new construction projects are added. For completed homes, builders and their HERS rater partners enter all data required in the online system, attach a confirmed REM Rate file, and provide the AHRI certificate for HVAC equipment. To receive the ENERGY STAR bonus a completed ENERGY STAR certificate or sticker is required and additional supporting documentation is to be provided upon request. The program does not require any additional documentation such as invoicing or spec sheets for the efficient equipment installed in the homes. The program requires that builders adhere to a set of performance milestones, outlined in Table 3-105, for completing homes. The milestones are intended to ensure that builders are progressing on completing program homes and to allow the program 45 U.S. Department of Energy ENERGY STAR Certified New Homes Market Share. Energy Efficiency Programs 217

221 PSO High Performance Home Program control the reservation of incentive funds. The program may withdraw incentives for homes if the builder does not meet the milestone requirements. Table Builder Agreement Milestones Date April 30th July 30th September 30th December 15th Performance Milestone Date Requirement At least 20% of total committed homes must be entered on The Program s online system. Homebuilder must have selected at least one preferred HERS Rater on the online system. At least 50% of total committed homes must be entered on The Program s online system. Homebuilder must have selected at least one preferred HERS Rater on the online system. At least 50% of total committed homes must be invoiced on The Program s online system. The ENERGY STAR Certificate and checklists (if applicable) for each invoiced home must be delivered to ICF. All (100% of) total committed homes must be invoiced on The Program s online system. All ENERGY STAR Certificates and checklists (if applicable) for the invoiced homes must be delivered to ICF. Program Implementation Staff indicated that much of the HPH marketing effort was directed towards the New Homes component. The marketing effort was undertaken to inform builders of the new ENERGY STAR bonus as well as targeted, direct to customer marketing. Regarding the direct customer marketing, staff reported that they developed a profile of likely new home purchasers and conducted an marketing campaign directed at this group. Builders are also provided with materials for use in marketing the efficiency of program homes, such as flyers and yard signs. Staff indicated that contractors are informed of the availability of materials through individual meetings during the process of getting signed builder agreements. Staff noted that builders like the signs promoting the efficient duct work. Another outreach strategy offered was to provide a bonus incentive for ENERGY STAR homes that were included in the 2016 Parade of Homes TM event. A condition of receiving the bonus was to include a PSO sponsored information booth. The Parade of Homes is attended by home shoppers and building industry representatives seeking Energy Efficiency Programs 218

222 PSO High Performance Home Program design ideas and provided an opportunity to inform both potential buyers and other builders of the incentives and benefits of efficiency construction. Staff indicated that in 2016 they were successful in recruiting smaller builders, Tribal Nations, and builders from more diverse geographies. With the launch of the new program year, the program required builders sign new builder agreements and to re-register with the program and provide updated W-9s and documents verifying their insurance coverage. Additionally, builders were required to receive training on new construction standards as well as marketing and communication guidelines and materials. Enrollment is completed through an online portal. Builder information is entered once and then accessed electronically for registration of subsequent homes that are planned to receive incentives for the program year. HERS raters play a key role in the implementation of the program. They provide outreach and consultation to builders and complete much of the necessary steps for registering homes and supplying required program documentation. In exchange for this support, they receive a $50 payment and an additional $150 if the home receives ENERGY STAR certification. Staff emphasized the importance of HERS raters to the success of the program and noted that they are continually looking for ways to streamline and improve the processes for them. Staff confirms HERS ratings on all houses that receive program incentives. As part of that process, they verify that the AHRI certificate submitted matches with the installed HVAC unit. High Performance Homes Conclusions and Recommendations New Homes The following presents the key conclusions from the 2016 program process evaluation: Program Design The program changed the incentives from performance based to prescriptive incentives in The prescriptive incentives applied to any home permitted after February 1 st, The intent of the prescriptive incentives was to enable builders to know exactly what incentive they will receive. Under the performancebased incentive approach, builders were faced with uncertainty if the finished home s efficiency was not as high as intended. Feedback provided by builders suggest that they prefer the prescriptive incentives and that it provides greater certainty when making decisions about equipment and design features. Additionally, the number of builders that participated in the program in 2016 was 53, up from 37 in 2015 and 7% of the 2016 builders had not completed a home since 2014 and 2% had not completed a home since The program offered a $1,000 incentive for receiving ENERGY STAR v3.1 certification. In total, 119 homes, or 11% of the completed homes, received Energy Efficiency Programs 219

223 PSO High Performance Home Program ENERGY STAR certification. While a minority of homes completed through the program received the ENERGY STAR certification, the 119 homes compare very favorably to data reported by ENERGY STAR that indicates only 4 homes built in Oklahoma during 2015 received ENERGY STAR certification. An incentive of $2,000 was provided for any ENERGY STAR certified home that was included in the Parade of Homes event and included a PSO sponsored information booth. Staff provided this incentive to increase awareness of the program. The Parade of Homes is attended by home shoppers and building industry representatives seeking design ideas. Both program staff and interviewed builders spoke highly of the key role that HERS raters play in the program and the assistance they provide to help builders incorporate efficiency into the design and construction process and in managing the administrative aspects of the program such as documentation submissions. Administration The key program staff is the PSO Program coordinator, the ICF program manager, and two ICF account managers. PSO program staff addresses issues with implementation as they arise, which include but are not limited to resolving issues with program builders, interpreting program guidelines, and overseeing the budget. The ICF program team manages the day-to-day operations of the HPH Program including application review and approval, payment processing, on-sight home assessments and testing, as well as service provider recruitment and outreach. All interviewed staff indicated that there were strong working relationships between PSO and ICF and that all communication needs were fully met. The program allocates budgets for different segments of the new construction market, namely, production builders and single family home owners building homes through the program. Additionally, some funds are reserved for new builders participating in the program. Staff indicated that budgets are reviewed in September and adjusted accordingly. Builder and project information is entered and stored in an online system. The electronic tools allow builder information to be linked to subsequent projects without requiring repetitive data entry. Implementation and Delivery Program marketing and outreach is directed at potential home buyers, builders, and is also intended to support builder marketing efforts. In 2016, the program launched an campaign targeted at customers that fit a profile of likely new Energy Efficiency Programs 220

224 PSO High Performance Home Program home purchasers. This outreach strategy is consistent with builders suggestion that an area for further development was increasing awareness of the incentives and efficient construction benefits with homebuyers, which builders characterized as often focused on other types of amenities (e.g., granite countertops). Builder marketing of the efficiency improvements I supported through marketing materials provided to the builders, including flyers and yard signs. Ongoing outreach to builders continues to attract new builders to the program. Forty-four percent of builders that completed projects in 2016 had not completed any projects during the period. Additionally, four of the thirteen builders that were interviewed were new to the program in 2016 and were not aware of it previously. This finding suggests that there remains an opportunity for increasing awareness among builders. Interview responses suggest that builder awareness of the new ENERGY STAR Certification bonus is high, while awareness of the program design assistance is low. Some builders noted that the cost to meet the ENERGY STAR v3.1 requirements is greater than the perceived benefits, that there is not much market demand for ENERGY STAR Certification in the PSO service territory, and that lenders do not appraise an ENERGY STAR Certified home at a much higher value. One builder indicated they may add up to $500 to the appraisal value to an ENERGY STAR certified home. Most builders thought the HPH Program has significantly increased their company s knowledge of ways to build homes beyond code. They noted insulation techniques, air sealing, framing, and HVAC as specific areas where they have gained the most knowledge. Some also noted the value of working with and learning from their HERS raters. However, findings from the interviews with builders suggest that once efficient construction is integrated into their building processes, builders are unlikely to change that practice if program support is no longer available. Only one of the 13 interviewed builders reported receiving training from the program in They said the training was influential in how their sales teams present the benefits of homes built beyond code. Interviewed builders stated that the rebate payment timeline was unclear and that that they would like to see improved communication about rebate payments. ADM offers the following recommendations for the program: To maximize net savings opportunities among production builders, the program should focus on recruiting new builders into the program and on increasing the efficiency of the homes built by legacy builders. Interview Energy Efficiency Programs 221

225 PSO High Performance Home Program results suggest that these builders may continue to incorporate efficient designs and equipment in the absence of program support. Promote ENERGY STAR design services as a means of increasing the number of homes that receive ENERGY STAR certification. Although completion of 119 ENERGY STAR certified homes is a noteworthy achievement given that only four were completed in the state of Oklahoma in 2015, there may be opportunities to grow this number. Several builders indicated that they were not aware of the design services available and increased awareness of these services may improve builders perceptions of the feasibility of completing ENERGY STAR homes. Improve communications about rebate payment timelines. Interviewed builders indicated that there was a lack of clarity regarding when they would receive the rebates. It is recommended that language be added to the New Homes Rebate Guidelines that outlines a schedule for when rebates are to be paid upon completion of a home, HERS ratings, and verification activities. The online portal may also be used to communicate rebate status. The tool has a field for project submission and construction status and may be expanded to provide information on post construction testing, verification, and payment status. Planned Program Changes The only change planned for the New Homes component of the HPH Program is to receive incentives from the program, homes are required to have 75% CFL lighting, up from 50% in Multiple Upgrades (MU) Component EM&V Methodologies This section provides an overview of the data collection activities, gross and net impact estimation methodologies, and process evaluation activities that ADM employed in the evaluation of the Multiple Upgrades component of the High Performance Homes program. Data Collection The primary data collection activities for the Multiple Upgrades component of the program consisted of a participant telephone survey, a separate sample of on-site verification visits, in-depth interviews with program staff, and discussions with project auditors. Additional data reviewed included a census of program tracking data from SSRS and, when necessary, project documentation obtained from VisionDSM. Table summarizes the sample size for each primary data collection activity. The random Energy Efficiency Programs 222

226 PSO High Performance Home Program sample for verification was designed to achieve ±10% relative precision or better at the 90% confidence interval. Table 3-106: Sample Sizes for Data Collection Efforts Multiple Upgrades Data Collection Activity Achieved Sample Size Participant Survey 150 On-Site Verification Visits 28 In-Depth Interviews with Program Staff 4 Service Provider Survey 27 Program tracking data for the MU component of the program included customer contact information and descriptions of the measures installed. Each MU participant was assigned a random number, and the list of customers was sorted by the random number to create a prioritized call list. Ultimately, 150 surveys were completed. On-site verification visits were conducted for 23 participants. During the visits, ADM field staff verified measure installation and operational characteristics. For HVAC equipment, the type, capacity, efficiency ratings, and model numbers were recorded. For attic insulation, installation was visually verified and pre- and post- insulation levels were measured where possible. Heating and cooling system types were also recorded along with an approximate measurement of square feet of insulation installed. For highefficiency windows/doors, the total count of new units was recorded along with heating and cooling system type. Where possible, pre-existing window type (dual/single pane) was recorded. Pictures were taken during all on-site visits to document measure installation and characteristics. The findings from the on-site verification visits were compared to information in the program tracking database to verify that input to savings calculations were correctly recorded. ADM completed a survey of 27 participating MU and SU service providers to elicit their feedback regarding various aspects of the program. 46 Finally, program staff members from PSO and implementation contractor were interviewed to elicit the program administrator perspective on program processes and operations. 46 These 27 service providers were interviewed about their participation in the Single Upgrades program component and the Multiple Upgrades component of the High Performance Homes program. Many service providers submitted projects through both program components, while some submitted projects through only one or the other. Energy Efficiency Programs 223

227 PSO High Performance Home Program Gross Impact Methodologies The methodology used to calculate energy and demand impacts consisted of verifying measure installation for a sample of program participants, reviewing deemed savings estimates for each measure as described in the Oklahoma Deemed Savings Documents, and reviewing the program tracking data to ensure that deemed savings were applied appropriately. ADM s review of the tracking data was performed to ensure correct deemed savings application for a census of projects and measures rebated through the program during PY2016. Verification of measure installation was conducted through telephone surveys with program participants and on-site visits to participating homes. The telephone surveys included questions aimed at verifying program participation and the number and type of measures installed by program participants. During the site visits, ADM field staff verified that the claimed energy efficiency measures were installed, and recorded key inputs to savings calculations such as capacity and efficiency of HVAC equipment, and R-value and square footage of installed insulation. Data collected through these activities was used to develop measure level verification rates. The measure verification rates were then applied to the deemed savings estimates to determine verified savings. This approach to evaluating the reported savings values provides verification of measure installation and correct application of the deemed savings values. The Oklahoma Deemed Savings Documents have been previously approved by the Oklahoma Corporation Commission for the estimation of energy and peak demand savings resulting from measures installed through the Multiple Upgrades program component. As a result, ADM s evaluation does not include a specific assessment of the deemed savings validity for PY2016 participants. Net-to-Gross Estimation This section provides a summary of the methodology ADM used to score survey responses for free ridership and spillover. PSO customers who received rebates through the Multiple Upgrades component of the High Performance Homes program were surveyed and asked a series of questions aimed at estimating program attribution. The attribution scoring system is broken down into three components: measure level free ridership score, project level free ridership score, and spillover score. Each component is described individually below, followed by a section discussing how the scores will be weighted to extrapolate the survey results to the program component level (all Multiple Upgrades projects). Measure Level Free Ridership Scoring: The survey was pre-populated with information about each respondent. Specifically, the list of energy efficiency measures the respondent installed through the program was known to the surveyor. The survey was programmed to ask a series of questions about the program s effect on deciding to purchase and install each measure. For example, if a respondent installed a new heat pump, ceiling insulation, and had their ducts sealed, they were asked a series of Energy Efficiency Programs 224

228 PSO High Performance Home Program questions about each of those measures (but not other program eligible measures which they did not install). The measure level free ridership questions were arranged as follows: 1. Indicator one: prior planning 2. Indicator two: stated likelihood in absence of program recommendations 3. Indicator three: stated likelihood in absence of 50% bonus incentives 4. Indicator four: stated likelihood in absence of all program incentives 5. Mitigating factor one: reported efficiency/quantity increase 6. Mitigating factor two: timeliness of upgrade How these questions work together to determine a measure level free ridership score is displayed in Figure 3-38 on the following page. Note that the scoring algorithm requires the respondent to indicate a burden of proof that they are a free rider. They must have stated that either 1) they had prior plans to install the measure or 2) they would have likely installed the measure in the absence of the program. To be considered a full free rider for the specific measure, they must have gone on to say the program did not affect the level of efficiency (or quantity, where appropriate) they chose for their upgrade nor did it affect the timeliness of the upgrade. Energy Efficiency Programs 225

229 PSO High Performance Home Program Figure 3-38: Measure Level Free Ridership Scoring Multiple Upgrades Energy Efficiency Programs 226

230 PSO High Performance Home Program Project Level Free Ridership Scoring: The measure level free ridership scores for each respondent were weighted by the ex ante kwh savings by measure. For Multiple Upgrades, the measure level free-ridership score for each measure a respondent installed was first multiplied by the ex ante kwh savings that measure represents. Total measure level free-ridership kwh was then summed over the incentivized measures and divided by the total project ex ante kwh savings to get the respondent s freeridership score. This means that if a respondent indicated free ridership for a low kwh impact measure, but no free ridership for a high kwh impact measure, the overall free ridership score will take this into account. Spillover Scoring: Spillover, as addressed in this survey, was restricted to energy efficiency measures that respondents report installing in their home without receiving additional incentives. Additionally, respondents must have indicated that the additional non-incentivized measures were installed based on program influence. Potential spillover respondents were identified using the question below: Following your participation in the PSO s Multiple Upgrades Service, did you install any additional energy efficiency upgrades in your home for which you did not receive a rebate or financial incentive? Customers who responded no to this question were determined to not be potential spillover candidates. If a respondent indicated that they have installed additional energy efficiency measures, they were then asked: Did your participation in PSO s Multiple Upgrades program influence your decision to install the additional energy efficient upgrades in your home? Customers who responded yes, strongly influenced or yes, somewhat influenced were considered potential spillover candidates, and were asked to identify the additional measures they have installed. The measures identified were then checked to ensure they had not received a corresponding PSO rebate through any program and that they represent measures with the potential to save energy. All measures that passed through this screening process were assigned a savings value based on the deemed savings documents. Verified spillover savings for a given customer were then divided by the project level ex ante savings for that customer. The resulting percentage is the total spillover score. The average spillover score per sampled customer was applied to the program level savings to determine program level spillover. Weighting of Respondent Scores and Determination of Program Level NTGR: The project level free ridership scores for each respondent was weighted by the ex ante kwh savings per project to determine the final weighted average free-ridership estimate per customer in the sample. This estimate along with the spillover estimate was used as the program level verified gross savings to determine net savings. Energy Efficiency Programs 227

231 PSO High Performance Home Program Process Evaluation Activities The purpose of the process evaluation of the Multiple Upgrade component was to examine program operations and results, to assess how the program has developed since the prior program cycle, and to identify any key areas that may need to be addressed in future program years. A key focus was to identify any program changes or developments and their impacts on the program. Key research questions addressed by the process evaluation of the 2016 MU component include: How do participants hear about the program? What percentage is contacted directly by PSO or implementation staff? What percentage hears about the program through service providers? Were the program participants satisfied with their experience? What are the perceived energy and non-energy benefits associated with the program? What steps were taken to engage service providers and were they successful? Were there any significant changes or new obstacles during the 2016 program year? During the evaluation, data and information from several sources are analyzed to achieve the stated research objectives. The internal organization and operational efficiency of program delivery are examined through analysis of interviews conducted with PSO program staff and implementation staff. Participating Multiple Upgrades customers and service providers were surveyed to obtain their feedback regarding the program as well Impact Evaluation Findings Multiple Upgrades Verified Gross Impacts As described in Section , the gross impact analysis consisted of verifying measure installation and checking the program tracking data to ensure that deemed savings algorithms were appropriately applied. In-Service Rates (ISR) for each measure type were developed based on the findings from the participant telephone survey and on-site visits. Findings from these activities are summarized below for each measure type. Energy Efficiency Programs 228

232 PSO High Performance Home Program Infiltration Reduction: The OK DSD specifies the following formula for use in calculating energy and demand impacts for infiltration reduction measures. The air infiltration reduction estimate in CFM is obtained through blower door testing performed by the program contractor for each home serviced. Only homes with electric cooling systems are eligible for the measure (central AC or room AC). Where: Deemed Savings = Air infiltration reduction in Cubic Feet per Minute at 50 Pascal V = Corresponding Value in the following Table Table 3-107: Infiltration Control Deemed Savings Values Infiltration Control Deemed Savings kwh Impact per CFM50 Reduction Weather Zone ac_gas_heat Gas Heat (no AC) kwh Gas Heat (no AC) Therms ac_electric heat_pump AC Peak Savings kw A B One hundred and one of the 150 participants surveyed were identified in the program tracking database as having infiltration reduction measures installed and blower-door testing performed. Fifteen of the 23 site verification visits that ADM performed were homes that had infiltration reduction measures installed. Five customers did not recall that this measure was installed as part of their upgrade. However, a review of the program tracking data for those sites produced evidence of participation including measure invoices. It is unclear why these respondents did not recall having this measure installed, but when done correctly, this is a largely invisible measure. In all cases reviewed, evidence of air-sealing measures was identified either on site or via documentation review. As a result, an In-Service Rate (ISR) of 100% was applied for air sealing measures. Energy Efficiency Programs 229

233 PSO High Performance Home Program ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all but five projects. Adjustments were made to four of the five based on climate zone. ADM looked up the zip code and zip code database to determine the county in which the project took place. This county was then used to look up the climate zone in the Oklahoma Deemed Savings Document. The remaining project did not meet the 10% reduction of the pre-installation reading which is required to qualify for an incentive. The magnitude of these variances is minor, resulting in a realization rate of 99.8% for energy savings (kwh) and 99.6% for peak demand reduction (kw). Energy Efficiency Programs 230

234 PSO High Performance Home Program Air Conditioner and Heat Pump Retrofits: The OK DSD specifies annual energy savings and peak demand to be calculated as shown in the formulas below. The algorithms are dependent upon climate zone, capacity, and efficiency ratings of the installed units. Cooling Annual Energy Savings 1, 1, Heating Annual Energy Savings 1, Peak Demand Savings 1 Where: /, 1 1, / 1 1, , ,000 = Rated equipment cooling capacity of the existing unit (BTU/hr) = Rated equipment cooling/heating capacity of the new unit (BTU/hr) 47 = Season Energy Efficiency Ratio of existing cooling equipment = Season Energy Efficiency Ratio of installed cooling equipment = Energy Efficiency Ratio of the existing equipment = Energy Efficiency Ratio of the installed equipment = Equivalent full load hours for cooling = Equivalent full load hours for heating = Heating Seasonal Performance Factor for existing heating equipment = Heating Seasonal Performance Factor for installed heating equipment = Coincidence Factor = 0.87 (default) 47 Rated capacity of the new unit shall no exceed capacity of the existing unit; if completing this with other measures, use existing unit capacity Energy Efficiency Programs 231

235 PSO High Performance Home Program ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all but twelve projects. Adjustments were made to four Central Air Conditioner projects based on climate zone. One Window Air Conditioner project claimed negative savings due to calculating savings with a SEER post of 0.4 even though the SEER post claimed in the project is Of the remaining seven Heat Pump projects, two had adjustments made based on climate zone. One project reported negative savings, which were recreated with a HSPF post of 1 while the reported HSPF post was 0. The remaining four projects had discrepancies is peak demand reduction (kw). These variances result in a realization rate for all HVAC measures to be 101.9% for energy savings (kwh) and 153.1% for peak demand reduction (kw). Energy Efficiency Programs 232

236 PSO High Performance Home Program Duct Sealing and Duct Insulation: The algorithms for cooling and energy savings listed in the OK DSD are: Where: DLpre DLpost EFLHc h ρout Cooling Savings:, = Pre-improvement duct leakage at 25 Pa (ft3/min) = Post-improvement duct leakage at 25 Pa (ft3/min) = Equivalent full load cooling hours, from table = Outdoor/Indoor seasonal specific enthalpy (Btu/lb), from table = Density of outdoor air (lb/ft3) ρin = Density of conditioned air at 75 F (lb/ft 3 ) = = Constant to convert from minutes to hours 1,000 = Constant to convert from W to kw SEER Where: DLpr DLpost = Seasonal Energy Efficiency Ratio of existing system (Btu/W hr) = 13 (default) Heating Savings (Heat Pump), = Pre-improvement duct leakage at 25 Pa (ft3/min = Post-improvement duct leakage at 25 Pa (ft3/min) 60 = Constant to convert from minutes to hours.77 = Factor to correlate design load hours to EFLH under actual working conditions = Volumetric heat capacity of air (Btu/ft3 F) HDD = Heating Degree Days 1,000 = Constant to convert from W to kw HSPF = Heating Seasonal Performance Factor of existing system (Btu/W hr) = 7.30 (default) Energy Efficiency Programs 233

237 PSO High Performance Home Program Where: DLpre DLpost Heating Savings (Electric Resistance), ,412 = Pre-improvement duct leakage at 25 Pa (ft3/min = Post-improvement duct leakage at 25 Pa (ft3/min) 60 = Constant to convert from minutes to hours.77 = Factor to correlate design load hours to EFLH under actual working conditions = Volumetric heat capacity of air (Btu/ft3 F) HDD = Heating Degree Days 3,412 = Constant to convert from Btu to kwh Where: kwhsavings,c EFLHc SEER Cooling Demand Savings (Electric),, = Calculated kwh savings for cooling = Equivalent full load cooling hours, from table = Seasonal Energy Efficiency Ratio of existing system (Btu/W hr) = 13 (default) EER = Energy Efficiency Ratio of existing system (Btu/W hr) 48 CF = Coincidence Factor = 0.87 (default) One hundred and twenty-four of the 150 participants surveyed were identified in program tracking database as having duct sealing measures. Twenty-one of 23 site visits that ADM performed were homes that had duct sealing measures installed. Twenty-eight customers did not recall that this measure was installed as part of their upgrade. However, a review of the program tracking data for those sites produced evidence of participation including measure invoices. It is unclear why these 48 EER = -.02 X SEER X SEER Energy Efficiency Programs 234

238 PSO High Performance Home Program respondents did not recall having this measure installed. In all cases reviewed, evidence of duct sealing measures was identified either on site or via documentation review. As a result, an In-Service Rate (ISR) of 100% was applied for air sealing measures. ADM s review of the program tracking data found that deemed energy savings (kwh) were applied and reported correctly for all but one duct replacement project and 28 duct sealing projects. Adjustments were made the duct replacement project and 7 duct sealing projects based on climate zone. ADM looked up the zip code and zip code database as described in the Air Infiltration description above. Twenty of the duct sealing projects were adjusted for the HSPF values. Deemed values used an HSPF value of 7.7, as per the Oklahoma Deemed Savings Document, while the HSPF of an installed Heat Pump when applicable. One thousand one hundred and twenty-two duct sealing projects had adjustments for peak demand reduction (kw) calculations. All deemed savings values used an EER post of 11.8 and a SEER post of 13. As per the Oklahoma Deemed Savings Document, the SEER post to be used should only be 13 if a SEER post is not available. If an EER post is not available, it is to be calculated by: The magnitude of the variances are minor, resulting in a realization rate of 99.7% for energy savings (kwh) and 94.8% for peak demand reduction (kw). Energy Efficiency Programs 235

239 PSO High Performance Home Program High-Efficiency Windows: Deemed savings are calculated per square foot of window, inclusive of frame and sash. The following tables apply to qualified Energy Star Windows replacing single- or double-paned clear gas. Table 3-108: Deemed Values for Energy Star Windows Window and Glass Door Improvements Deemed Savings Climate Zone Existing Window Pane Type ac_gas_heat Gas Heat (no AC) kwh Gas Heat (no AC) Therms ac_electric heat_pump AC Peak Savings kw 9 Single Pane Double Pane a Single Pane a Double Pane b Single Pane b Double Pane Single Pane Double Pane Single Pane Double Pane Two of the 150 participants surveyed were identified in the program tracking database as having windows and glass door improvements installed and blower-door testing performed. ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all projects. This results in a 100% realization rate for energy savings (kwh) and peak demand reduction (kw). Energy Efficiency Programs 236

240 PSO High Performance Home Program Electronically Commutated Motors: Deemed savings values were taken from the Oklahoma Deemed Savings Document. The savings are based on weather zone, savings mode, and square footage. Table 3-109: ECM Deemed Savings Values Electronically Commutated Furnace Fan Motor Impact per SqFt Weather Zone Energy Savings in Heating Mode (kwh) Energy Savings in Cooling Mode (kwh) Peak Demand Savings (kw) a b ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all but eighteen projects. Adjustments were made to one project based on climate zone. Adjustments were made to 11 eleven projects as they claimed cooling savings while an Air Conditioner was installed. This would mean no cooling savings should be applied. Adjustments were made to five projects that claimed savings even though a Heat Pump was installed, which would not qualify the project for savings. One project was adjusted to 0 savings due to not having any other measures installed besides the ECM Fan. Of the 23 sites visited, 13 had claimed an ECM Fan to be installed, which were all verified. Attic/Ceiling Insulation: Deemed savings values are calculated, in accordance to the OK DSD, for each weather zone. Deemed savings are based on the R-value of the baseline insulation, weather zone, and location of the insulation installed. Retrofit insulation must meet a minimum R-value of R-38 to be considered for savings. Savings are then calculated by multiplying the corresponding savings value by the square footage. Energy Efficiency Programs 237

241 PSO High Performance Home Program Baseline Insulation R-Value Table 3-110: Deemed Savings Ceiling Insulation AC/Gas Heat kwh Gas Heat (No AC) kwh Ceiling Insulation Impact per SqFt Gas Heat Therms Zone 9 AC/Electric Resistance kwh Heat Pump kwh AC Peak Savings kw R-0 to R R-5 to R R-9 to R R-15 to R Zone 8a R-0 to R R-5 to R R-9 to R R-15 to R Zone 8b R-0 to R R-5 to R R-9 to R R-15 to R Zone 7 R-0 to R R-5 to R R-9 to R R-15 to R Zone 6 R-0 to R R-5 to R R-9 to R R-15 to R Energy Efficiency Programs 238

242 PSO High Performance Home Program Baseline Insulation R-Value Table 3-111: Deemed Savings for Attic Encapsulation AC/Gas Heat kwh Gas Heat (No AC) kwh Attic Encapsulation Impact per SqFt Gas Heat Therms Zone 9 AC/Electric Resistance kwh Heat Pump kwh AC Peak Savings kw R-0 to R R-5 to R R-9 to R R-15 to R Zone 8a R-0 to R R-5 to R R-9 to R R-15 to R Zone 8b R-0 to R R-5 to R R-9 to R R-15 to R Zone 7 R-0 to R R-5 to R R-9 to R R-15 to R Zone 6 R-0 to R R-5 to R R-9 to R R-15 to R Eighty-eight of the 150 participants surveyed were identified in the program tracking database as having attic insulation measures. Seventeen of the 23 site verification visits Energy Efficiency Programs 239

243 PSO High Performance Home Program that ADM performed were homes that had attic insulation measures installed. Eight customers did not recall that this measure was installed as part of their upgrade. However, a review of the program tracking data for those sites produced evidence of participation including measure invoices. It is unclear why these respondents did not recall having this measure installed. In all cases reviewed, evidence of attic insulation measures was identified either on site or via documentation review. As a result, an In- Service Rate (ISR) of 100% was applied for air sealing measures. ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all but fifteen projects. These projects installed ceiling insulation and fall in Climate Zone 6, which has deemed kw savings of 0 for all preinstallation R-Values greater than 4. Claimed savings were used with a deemed kw savings value of per sq. ft., which is the deemed savings value for pre-installed R-Values of R-0 R-4. These discrepancies are minor, resulting in a realization rate of 100% for energy savings (kwh) and 98.9% for peak demand reduction (kw). Solar Screens: Deemed savings are calculated, in accordance with the Oklahoma Deemed Savings Document, by the square footage of the solar screens are added, weather zone, and cooling type as seen in the tables below. Table 3-112: Deemed Savings for Solar Screens Solar Screens Impact per SqFt Weather Zone AC/Gas Heat kwh Gas Heat (no AC) kwh Gas Heat Terms AC/Electric Resistance kwh Heat Pump kwh AC Peak Savings kw A B One of the 150 participants surveyed were identified in the program tracking database as having solar screen measures. One of the 23 site verification visits that ADM performed were homes that had solar screen insulation measures installed. Energy Efficiency Programs 240

244 PSO High Performance Home Program ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all projects resulting in a 100% realization rate for both energy savings (kwh) and peak demand reduction (kw). Knee Wall Insulation: Deemed savings values are taken from the OK DSD and are dependent upon weather zone and the heating/cooling type. Table 3-113: Deemed Savings Value for Knee Wall Insulation Attic Knee Wall kwh Impact per SqFt ZONE ac_gas_heat Gas Heat (no AC) kwh Gas Heat Therms ac_electric heat_pump AC Peak Savings kw A B Seventeen of the 150 participants surveyed were identified in the program tracking database as having attic knee wall insulation measures. Four of the 23 site verification visits that ADM performed were homes that had attic insulation measures installed. Five customers did not recall that this measure was installed as part of their upgrade. However, a review of the program tracking data for those sites produced evidence of participation including measure invoices. It is unclear why these respondents did not recall having this measure installed. In all cases reviewed, evidence of attic knee wall insulation measures was identified either on site or via documentation review. As a result, an In-Service Rate (ISR) of 100% was applied for air sealing measures. ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all but four projects. Adjustments were made this project based on climate zone. ADM looked up the zip code and zip code database as described in the Air Infiltration description above. The magnitude of this variance is minor, resulting in a realization rate of 99.9% for energy savings (kwh) and 99.5% for peak demand reduction (kw). Energy Efficiency Programs 241

245 PSO High Performance Home Program Wall Insulation: Deemed savings values, taken from the OK DSD, are the savings values below, dependent upon the weather zone and heating/cooling type, multiplied by the square footage of the net wall area insulated, not including window or door area. Table 3-114: Deemed Savings Values for Wall Insulation Wall Insulation Deemed Savings Impact per SqFt Weather Zone ac_gas_heat Gas Heat (no AC) kwh Gas Heat (no AC) Therms ac_electric heat_pump AC Peak Savings kw AC/Electric Resistance & Heat Pump Summer Peak kw A B Three of the 150 participants surveyed were identified in the program tracking database as having wall insulation measures. These participants verified the installation of the wall insulation measures. ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all projects. This results in realization rates of 100% for energy savings (kwh) and peak demand reduction (kw). Energy Efficiency Programs 242

246 PSO High Performance Home Program Floor Insulation: Deemed savings values, taken from the OK DSD, are dependent upon weather zone and equipment type. This savings value is multiplied by the square footage of the first level floor area above the crawl space. Table 3-115: Deemed Savings Values for Floor Insulation Floor Insulation Impact per SqFt Equipment Type kwh Savings kw Savings Therm Savings Zone 9 Electric AC with Gas Heat Electric AC with Electric Resistance Heat n/a Heat Pump n/a Zone 8a Electric AC with Gas Heat Electric AC with Electric Resistance Heat n/a Heat Pump n/a Zone 8b Electric AC with Gas Heat Electric AC with Electric Resistance Heat n/a Heat Pump n/a Zone 7 Electric AC with Gas Heat Electric AC with Electric Resistance Heat n/a Heat Pump n/a Zone 6 Electric AC with Gas Heat Electric AC with Electric Resistance Heat n/a Heat Pump n/a Zero of the 150 participants surveyed were identified in the program tracking database as having floor insulation measures. Zero of the 23 site visits had floor insulation measures installed. ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all projects. This results in realization rates of 100% for energy savings (kwh) and peak demand reduction (kw). Energy Efficiency Programs 243

247 PSO High Performance Home Program Radiant Barrier: Deemed savings values, taken from the OK DSD, are dependent upon weather zone, cooling/heating type, and the R-Value of the additional Radiant Barrier. The savings value is multiplied by the square footage of the ceiling are over the conditioned space, which the radiant barrier was applied. Addition of Radiant Barrier with existing attic insulation level Table 3-116: Deemed Saving Values for Radiant Barrier AC/Gas Heat kwh Radiant Barrier Impact per SqFt Gas Heat (No AC) kwh Zone 9 Gas Heat Therms AC/Electric Resistance kwh Heat Pump kwh AC Peak Savings kw Attic insulation R Attic insulation >R Zone 8a Attic insulation R Attic insulation >R Zone 8b Attic insulation R Attic insulation >R Zone 7 Attic insulation R Attic insulation >R Zone 6 Attic insulation R Attic insulation >R Zero of the 150 participants surveyed were identified in the program tracking database as having radiant barrier measures. Zero of the 23 site visits had radiant barrier measures installed. ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all projects. This results in realization rates of 100% for energy savings (kwh) and peak demand reduction (kw). Energy Efficiency Programs 244

248 PSO High Performance Home Program Direct Install Measures Faucet Aerators: Deemed savings values, taken from the OK DSD, and are for residential, retrofit-only installations and apply to weather regions. Table 3-117: Deemed Savings Values for Faucet Aerators Faucet Aerators - Elec. DHW kwh Savings per home Peak kw Savings per home ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all projects. This results in realization rates of 100% for energy savings (kwh) and peak demand reduction (kw). Low Flow Shower Heads: Deemed savings values, taken from the OK DSD, are dependent upon the gallons per minute, t number of shower heads retrofitted, and the number of showerheads per household and are for residential, retrofit-only installations and apply to weather regions. Table 3-118: Example of Deemed Savings Values for Low Flow Shower Heads kwh Savings for 1.5 GPM # shower heads retrofitted Low Flow Shower Heads Showerheads per household One Two Three Four One Showerhead Two Showerheads Three Showerheads Four Showerheads Energy Efficiency Programs 245

249 PSO High Performance Home Program Energy Star Standard Compact Fluorescent Lamps (CFLs): The algorithms for calculating savings for CFLs are: Where: = Based on wattage equivalent of the lumen output of the purchased CFL bulb and the program year purchased/installed = Wattage of CFL purchased/installed Hours = Average hours of use per year = 1,023 ISR = In-Service Rate = 0.97 = Interactive Effects Factor to account for cooling energy savings and heating energy penalties. = Interactive Effects Factor to account for cooling demand savings and heating demand penalties = Summer Peak Coincidence Factor ADM s review of the program tracking data found that deemed energy savings (kwh) were applied and reported incorrectly for the one CFL. Adjustments were made to the Heating/Cooling Type for the project. The deemed savings used a Heating/Cooling Type of Unknown, resulting in an of The project has a heat pump Heating/Cooling Type, resulting in an of 084. These variances result in a realization rate of 87.5% for energy savings (kwh) and 100% for peak demand savings (kw). Energy Efficiency Programs 246

250 PSO High Performance Home Program Omni-Directional LEDs: The algorithms for calculating savings for LED s are: Where: = Based on wattage equivalent of the lumen output of the purchased CFL bulb and the program year purchased/installed = Wattage of CFL purchased/installed Hours = Average hours of use per year = 1,023 ISR = In-Service Rate = 0.97 = Interactive Effects Factor to account for cooling energy savings and heating energy penalties. = Interactive Effects Factor to account for cooling demand savings and heating demand penalties = Summer Peak Coincidence Factor ADM s review of the program tracking data found that deemed energy savings (kwh) were applied and reported correctly for all projects. This results in 100% realization rates of 100% for energy savings (kwh) and peak demand savings (kw). Multiple Upgrades Component Level Gross Impacts Combining the measure level adjustments detailed above results in the component level verified gross energy and demand impacts detailed in Table Energy Efficiency Programs 247

251 PSO High Performance Home Program Table 3-119: Reported and Verified Gross Impacts Multiple Upgrades Measure Reported Energy Savings (kwh) Reported Peak Demand Savings (kw) Verified Gross Energy Savings (kwh) Verified Gross Peak Demand Savings (kw) Infiltration Reduction 354, , AC and Heat Pumps 960, , Furnace Fan ECM 106, ,567 8 Insulation & Radiant Barrier 785, , Omni-Directional LEDs 10, ,543 2 Faucet Aerators 19, ,460 2 Low Flow Shower Heads 17, ,578 2 CFLs Duct Sealing & Replacement 1,551, ,546, Duct Insulation 38, ,090 7 High Efficiency Windows/Doors 14, ,749 9 Window Screens Total 3,858,886 1,509 3,868,015 1,603 Overall, verified gross energy savings are estimated at 3,868,015 kwh, representing a realization rate of 100% for the Multiple Upgrades component of the program. Verified gross peak demand reduction is estimated at 1,509 kw, a gross realization rate of 106%. The verified gross savings estimates reflect confidence that the energy and demand impacts reported for the program were largely consistent with the deemed savings values for most measure categories. The deemed savings algorithms are designed to reflect average energy savings for installed measures, and may not reflect actual savings at any given participant meter. For homes where multiple measures affecting the same end use are installed, the deemed savings may not capture the interactive effects between the measures. The Oklahoma Deemed Savings Documents have been previously approved by the Oklahoma Corporation Commission for the estimation of energy and peak demand savings attributable to the measures installed through the Multiple Upgrades program component. As a result, ADM s evaluation does not include a specific assessment of the deemed savings validity for PY2016 participants. In-Service Rates As described in Section , the gross impact analysis consisted of verifying measure installation and checking the program tracking data to ensure that deemed savings algorithms were appropriately applied. In-Service Rates (ISR) for each measure type were developed based on the findings from the participant telephone survey and Energy Efficiency Programs 248

252 PSO High Performance Home Program on-site visits. Findings from these activities are summarized below for each measure type. Net-to-Gross Estimation Results ADM surveyed 150 Multiple Upgrades participants to determine the net-to-gross ratio for this program. Survey respondents were asked a series of questions aimed at determining the program influence on the purchase and installation decisions for each installed measure. Each respondent was assigned a free ridership score (ranging from 0 for no free ridership to 1 for complete free ridership) based on their responses for each measure they installed. The scoring methodology is described section The free ridership scores for all survey respondents were then weighted by kwh savings and averaged to determine the program level free ridership rate. The resulting free ridership estimate is 12.75%. Survey respondents were also asked a series of questions to determine if they had installed any additional, non-rebated, energy efficiency measures because of their participation in the program (spillover). Nineteen respondents said they had installed additional measures, and that their participation in the program was influential in their decision to do so. However, only eight of these respondents reported measures that were not rebated and had potential energy savings. The final spillover estimate was 0.4% for the MU program. The final Multiple Upgrades net-to-gross ratio is calculated as 1 free ridership + participant spillover and is estimated at 87.7%. This estimate is consistent with prior year estimates for the program (PY2012 = 84%, PY2013 = 82%, PY2014 = 85%). It is slightly higher than the estimated net-to-gross ratio for the Single Upgrade component of the ESPS program. This suggests that the 50% cash-inducement bonus for homes completing three or more measures likely does encourage some participants to install additional measures beyond their initial plans when entering the program. Table below summarizes gross and net impacts for the Multiple Upgrades component. Table 3-120: Verified Gross and Net Impacts Multiple Upgrades Program Component Verified Gross kwh Verified Gross Peak kw NTGR Net kwh Net Peak kw Multiple Upgrades 3,868,015 1,603 85% 3,304,627 1,370 Energy Efficiency Programs 249

253 PSO High Performance Home Program Process Evaluation Findings This section presents the findings from the review of program design and operational processes, interviews with program staff and service providers, and surveys of participating customers. Program Participation During 2016, there were 1,247 projects incentivized through the Multiple Upgrades program component. The average number of upgrades per customer was approximately 3.5, and the average cash-inducement amount per project was approximately $2,384. The average reported kwh savings of the projects was 3,105. During 2016, a total of 35 service providers participated in the program by completing at least one Multiple Upgrades project. As shown in Figure 3-39, three service provider firms accounted for most reported kwh savings (56%). Aside from the 10 firms individually displayed in the figure, 24 other service providers each accounted for no more than 2% of reported kwh savings in Figure 3-39: Distribution of ex ante kwh Savings by Service Provider Company Energy Efficiency Programs 250

254 PSO High Performance Home Program Table summarizes program activity by measure type. As shown, duct sealing accounted for the largest share of program reported savings. The cost per expected kwh appeared high in comparison to the evaluator's experience with this measure in other jurisdictions and similar to the cost of central air conditioner replacements. Review of the tracking data found that 49% of the duct sealing projects received incentives of more than $500 and that 18% received incentives of more than $1,000. Moreover, as shown in Table there is not a strong relationship between reported savings and incentive payments. As shown in Table 3-123, multiple programs impose a cap on duct sealing incentives. Table 3-121: Summary of Measures Implemented during the Program Year Measure Percent of Reported kwh Savings Dollars per Reported kwh Saved Duct System Sealing 40% $0.49 Central AC 22% $0.52 Insulation - Attic 16% $0.84 Air Sealing Package 9% $0.82 Insulation - Kneewalls/Vertical Attic Wall 3% $0.59 Heat Pump 3% $0.03 Heating System ECM-type Blower Fan 3% $0.70 Duct Replacement 1% $9.66 Low Flow Shower Heads Electric WH <1% $0.00 Window and Glass Door Improvement <1% $0.58 Faucet Aerators Electric WH - Bathroom <1% $ watt Omnidirectional LED <1% $0.00 Insulation - Exterior Wall <1% $0.34 Faucet Aerators Electric WH - Kitchen <1% $0.00 Insulation - Basement/Enclosed Crawlspace <1% $0.87 Radiant Barrier <1% $0.81 Window Screens <1% $1.14 Replacement Door <1% $ Watt Integral CFL <1% $0.00 * Costs of direct install measures were not provided in the project tracking data Energy Efficiency Programs 251

255 PSO High Performance Home Program Table 3-122: Relationship between Duct Sealing Incentive Payment Reported kwh Savings Incentive Payment Share of Duct Sealing Projects Average Reported kwh Savings $500 51% 1,335 $500- $1,000 30% 1,015 $1,000 - $1,500 13% 1,295 $1,500 - $2,000 4% 1,136 > $2,000 1% 695 Table Summary of Duct Sealing Incentive Caps Program Incentive Caps Source Tennessee Valley Authority escore Program Up to $200 x ComEd Up to $300 Arizona Public Service Company Home Performance with Energy Star Salt River Project Home Performance with Energy Star Up to $400 Up to $250 Entergy New Orleans Up to $200 urhome/pages/weatherizationrebates.aspx ng_rebates2.html ades.aspx Table displays the share of savings by district. As shown, the program achieved 47% of expected savings outside of the Tulsa area. Table 3-124: District Share of Reported kwh Savings District Percent of Reported kwh Savings Participant Survey Lawton 29% McAlester 15% Tulsa 53% Tulsa Northern 4% A telephone survey was conducted to collect data on participants experience with PSO High Performance Homes program. The surveys were conducted in September and Energy Efficiency Programs 252

256 PSO High Performance Home Program December of In total, 150 program participants responded to the telephone survey in two waves. Figure 3-40 displays respondent household type. As shown, most respondents occupy single-family homes with detached construction. Figure 3-40: Survey Respondent Residence Type Eighty-nine percent of respondents reported owning their own homes. Table displays the year of construction for respondents homes. Table 3-125: Age of Respondent Homes Year Home was Built Percentage of Respondents (n = 150) Before % % % % % % 2006 or later 4% Don t know 9% Energy Efficiency Programs 253

257 PSO High Performance Home Program Most respondents (71%) reported living in homes between 1,000 and 3,000 square feet, as shown in Table Table 3-126: Size of Respondent Homes Size of Living Space Within Home Percentage of Respondents (n = 150) Less than 1,000 square feet 7% 1,000 2,000 square feet 51% 2,000 3,000 square feet 20% 3,000 4,000 square feet 6% 4,000 5,000 square feet 1% Greater than 5,000 square feet 1% Don t know 11% Source of Program Awareness and Reason for Completing Project As shown in Table 3-127, respondents most commonly reported that they learned of the program through their service providers. Other common sources of program awareness included word-of-mouth, contractors, and TV. Other less common responses included bill inserts, the PSO website, print ads, and PSO s monthly newsletter. Table 3-127: How Participants Learned of the Program Response Percentage of Respondents (n = 150) How did you hear about PSO s energy efficiency upgrade rebates available through the Energy Efficiency Program? From my service provider 59% Word-of-mouth 15% Contractors 9% TV 4% Other 13% Fifty-five percent of respondent said that they had had a specific concern in mind when contacting their service provider as opposed to a general interest in energy efficiency. The most common concern, mentioned by 72% of respondents, was issues related to broken equipment (Table 3-128). Respondents who provided other responses Energy Efficiency Programs 254

258 PSO High Performance Home Program frequently noted that their existing equipment was old or otherwise insufficient. These replies suggest that the desire to improve a piece of equipment is a common motivator for program participation. Measures Not Installed Ten percent of respondents reported that they did not install some measures recommended by their service provider. The measures that were most likely to be reported as not installed were air infiltration measures, attic and ceiling insulation, wall insulation, and new windows. When asked why they did not install these recommended measures, almost all respondents replied that they did not complete the upgrades because they were prohibitively expensive. Overall, these findings suggest that most participants are installing the measures recommended by service providers and that rejecting recommended measures is not significantly limiting the scope of efficiency projects. Table 3-128: Measures that were Not Installed Response Percentage of Respondents (n = 16) What energy saving upgrades did you not install? ENERGY STAR qualified windows/doors 37% Attic/Ceiling Insulation 13% Air Infiltration (Air Sealing, House Tightening, Weather Sealing) 13% Wall Insulation 13% Air Conditioner Replacement 6% Other 19% Non-Energy Benefits Eighty-three percent of respondents said that the energy upgrades have made their home more comfortable. The most commonly-reported financial and non-energy benefits noticed by participants were lower utility bills, improved home comfort, and improved appliance reliability. Additionally, several participants noted other benefits such as reduced noise, health improvements, and that safety concerns had been addressed. Energy Efficiency Programs 255

259 PSO High Performance Home Program Table 3-129: Non-Energy Benefits Response Percentage of Respondents (n = 150) What are the biggest benefits you have noticed since the work was done on your home? Utility bills are lower 47% The home feels more comfortable 47% The appliances and heating equipment are more reliable 33% There is less noise from the appliances 18% There is less noise from the outside 17% There have been health improvements 11% Some or all of my home s safety concerns have been addressed 11% Other 11% No benefits 4% Don t know 5% *Respondents were able to provide multiple responses. The percentages shown are percentages of respondents rather than percentages of responses. Thus, the total exceeds 100%. Seventy-six percent of respondents said that the program had made them more aware of the benefits of energy efficiency. A remaining twenty-two percent of respondents said that their awareness level had not changed. Most respondents (90%) did not identify anything that they disliked about the efficiency improvements. Those respondents who did provide comments on aspects of the improvements they did not like primarily remarked that they felt that the energy upgrades were too expensive. Use of Program Website Most respondents surveyed (77%) indicated that they had not visited the PSO website. Of those who had visited the website, most (58%) said that they visited the website to learn about what rebates were available. Another 39% visited the website to find energy saving tips. Respondents who had visited the website were also asked if they experienced any difficulties looking for the information that they were seeking. Ninetyfour percent of respondents said that it was easy or very easy to find the information that they were looking for, with another 6% saying that they did not know. Participant Satisfaction Survey respondents rated their satisfaction with selected aspects of the 2016 Multiple Upgrades Program. Results were provided on a scale of 1 to 5, with 1 representing very dissatisfied and 5 representing very satisfied. Energy Efficiency Programs 256

260 PSO High Performance Home Program Overall, satisfaction with all components of the program was quite high. The components of the program with the highest satisfaction scores were satisfaction with the Service provider and the overall program, which received scores of somewhat satisfied or very satisfied from 96% and 97% of respondents, respectively. Figure 3-41: Customer Satisfaction Few respondents expressed dissatisfaction with any component of the program. Those who did indicate dissatisfaction with some program element were given an opportunity to elaborate on their reasons for dissatisfaction. In total eight customers provided reasons for their dissatisfaction. Most of the issues related to issues with Service providers and completion of the testing requirements. The Service provider issues noted were the cost was higher than estimated, the project included oversized HVAC equipment, and that a mess was left. One respondent noted that it took two months to complete the testing. Two respondents indicated dissatisfaction with program staff and commented that staff had entered the customer s apartment without knocking and that testing paperwork was lost and needed to be repeated. A sample of respondent comments is provided below. They called after the installation and wanted to do the inspection right away, they did the inspection but I got a call and they wanted to do another inspection. They lost the original inspection and had to do it again. It held up the rebate credit because it showed up as incomplete. Poor management. Energy Efficiency Programs 257

261 PSO High Performance Home Program We had [two] different people [,] it was hard to know who to communicate with. We would call and the other one would call us back, there was not good communication. Gave me an estimate that was 12 something. Told me I would get more discount and it ended up being 14 thousand. Bad estimates. Many respondents provided open-ended commentary indicating that they highly valued the services provided by the High Performance Homes Program and that their program experience had been very positive. Commentary of this nature included: This is definitely a program that I find important, and I wouldn t have chosen such energy efficient equipment without the rebate. I m very pleased with what they put in and very pleased with everything going on. It s wonderful to see the difference in my monthly bill. It was very eye-opening to find out we were losing so much energy because of the broken ceiling. The work that was done was so helpful, even if it was expensive to fix. I was very happy, they were professional and worked well together and did an outstanding job. Thermostat Use Some respondents reported that they had modified the setting of the household heating and cooling equipment since making the efficiency improvements. Specifically, 17% reported that they keep their summer thermostat settings at a cooler setting and 14% reported that they keep it at a warmer setting. Similarly, 7% said they now set their thermostat at a cooler setting in the winter, and 11% said they now set it to a warmer setting. Service provider Interviews The Evaluators conducted in-depth interviews with 27 of the High Performance Homes (HPH) Service provider who completed Multiple and Single Upgrade projects in Interview topics covered: The company s history with the HPH Program; Service offerings and outreach strategies; Perspectives on the programs strengths and challenges; and Suggestions for program improvement. Organizational Background Service providers provided background information about their organizations including what types of energy efficiency equipment they specialize in, the number of staff they Energy Efficiency Programs 258

262 PSO High Performance Home Program employ in the PSO service territory and how many years they have participating in the High Performance Homes Program. Most respondents indicated HVAC equipment and insulation as their primary service offering, followed by insulation and windows and doors. Table below summarizes their responses. Table 3-130: Service provider Specializations Equipment Type Count (n=27) HVAC 11 HVAC, Insulation 4 Insulation 3 Insulation, Windows, and Doors 3 Windows and Doors 3 HVAC, Insulation, Windows and Doors 1 Pool Pumps 1 All EE Equipment 1 Most respondents (68%) indicated their company employs less than 20 staff members in the PSO service territory. Approximately 40% indicated they have worked with the program for 3 years or less, the other 60% indicated they are long standing Service providers who have worked the program for 4 or more years. Program Design Changes Service providers discussed two program changes introduced in 2016: static pressure testing for duct sealing and HVAC replacements and bonus incentives for contractors that install efficient HVAC units. Seven service providers who had completed projects that required static pressure testing discussed the value of this requirement. Most respondents indicated the test takes approximately 10 minutes to complete and it was something they did prior to the program requiring it. Only one Service provider indicated the testing process was too time-consuming; this company also did not perform the testing prior to the program change. Specific comments are provided below: [Static pressure testing] ensures quality service and installation; holds contractors accountable and allows us to verify the benefits to the homeowner. We appreciate it and like it. If you're trying to get a true home performance value, you can't do it without this measurement. I think it's an added benefit to the homeowner and added benefit that contractor that has to stand behind the work. It s time-consuming and is better for identifying a problem. In 2016, the Service providers were eligible to receive a bonus incentive for HVAC replacements. Previously, these incentives were paid to the equipment distributor. Ten (67%) out of the 15 respondents that specialize in HVAC services were not aware of the bonus rebate. The other 5 (33%) indicated they were aware of the bonus. Energy Efficiency Programs 259

263 PSO High Performance Home Program Specifically, they mentioned that the rebate helps cover administrative costs associated with processing the applications. Service providers if they have feedback regarding the rebate amounts or suggestions for other energy saving equipment or home improvements that should be included in the list of eligible measures. They suggested the following: Increase rebate for ENERGY STAR certified windows. One Service provider suggested indicated that their price for ENERGY STAR windows is approximately $550 - $600 per window and a $30 rebate is just not enough to encourage many homeowners. Include additional varieties of high-efficiency doors besides ENERGY STAR certified doors. The Service provider stated that ENERGY STAR doors can be difficult to source and the style options are limited. The respondent suggested that the program should consider including high-efficacy doors that are constructed with a certain percent of glass or less. Increase options for efficient pool pumps. A Service provider that specializes is pool installation and maintenance indicated that many elaborate pools designs often have main pumps and support pumps, he would like to see incentives for more than one pump. He also suggested adding rebates for multiple speed pumps. Include rebates for HVAC maintenance. Two program Service providers referenced the Oklahoma Natural Gas rebates for gas furnace maintenance. They emphasized the importance measuring refrigerant levels and cleaning coils as ways to improve the efficiency of an electric unit during the cooling season. Include rebates for air barriers other than spray foam insulation. The respondent indicated that there are high-efficiency insulation solutions besides spray foam. He indicated that even with the rebate the cost of spay foam is significantly higher than other types of insulation. He suggests adding tiered incentives that pay for some of the less expensive types of air barriers. Include Energy Recovery Ventilators (ERVs). Two HVAC Service providers mentioned the growing interest in this type of blower. Include Variable Refrigerant Flow (VRF) HVAC systems. One respondent noted that these units are the biggest growing trend within the HVAC market today. Marketing and Participation Barriers Service providers discussed their approach to marketing program incentives and barriers to customer participation. Twenty Service providers (74%) indicated they actively promote the program to their customers, while the other 7 Service providers Energy Efficiency Programs 260

264 PSO High Performance Home Program (26%) do not. The promotional efforts fell into 11 general promotional channels, as summarized in Table below. Table 3-131: Promotional Channels used by Program Service providers Strategies and Channels Used by Service providers Count (n=20) Discussions with Homeowners 7 Website 4 Radio 4 Printed Materials 3 Social Media 3 TV 2 Newspaper 2 Subcontractors 1 Trade Shows 1 Realtor Associations 1 Co-Branded Materials 1 Discussions with homeowners during the sales process was the most commonly mentioned means by which Service providers promoted the program. Many of the Service providers indicated that they are reluctant to include the program rebates in marketing materials because program funds are limited and not available year-round. Several Service providers mentioned they promote the rebates via their website, radio ads, social media, on TV, and in the newspaper. However, these promotions generally reference the availability of rebates rather than specifically referencing the PSO rebates. Service providers provided feedback regarding ways the program could help support their efforts promoting the program. Seven Service providers (26%) mentioned the need to improve program continuity and avoid stopping the program mid-year. Several window and door service providers suggested increasing the rebate amount, while a Service provider specializing in pool pumps suggested in-store brochures that provide details about program qualifying pumps. One Service provider suggested more case studies and/or testimonials from homeowners that provide information on their experience, which would help legitimize the program to homeowners who are skeptical of the rebate offer. While Service providers had suggestions to better support their promotional efforts the majority are aware that program funds are limited and expressed concern over too much promotion. Although most Service providers (78%) stated that they always recommend efficient equipment, the remainder identified some factors that may act as barriers to participation related to the perception of program incentives, requirements, and the market. Energy Efficiency Programs 261

265 PSO High Performance Home Program Cost of Foam Kneewall Insulation Too High to Recommend: A Service provider that specializes in insulation indicated that if the home requires only knee wall insulation then they typically will not suggest the program qualifying spray foam because the marginal cost is not worth it, even with the rebate. They strongly believe the less expensive options are just as efficient for the knee wall insulation. Focus on Lowest Cost Equipment Hinders Multifamily Efficiency: Several HVAC contractors do not suggest program qualifying equipment in rental homes. These property manager or owner that is paying for the equipment are focused on the lowest cost solution and are not as concerned with efficiency. Belief that Upsizing is Necessary for Some Projects: One HVAC Service provider indicated that system replacements often require upsizing, which is not currently eligible for program incentives. The specific scenario the respondent referred to was the need to upsize when the homeowner has added living space to their home. However, program guidelines recommend ACCA Manual J calculations to properly size the unit, which would not prevent increasing the size of the unit if necessary for the conditioned space. Service providers also provided information on the types of reasons they hear for customers not completing projects. These reasons are summarized in Table 3-132Table Table 3-132: Reasons Homeowners Give for Not Installing Program Qualifying Equipment Reasons Count (n=27) Cost 18 Misconceptions of EE or Program 6 Rent or Selling Soon 3 Don't Need Full Replacement 2 The primary reason for not installing the program qualifying equipment noted by customers is cost, followed by misconceptions about the program, namely the homeowners opinion that the program is either not legitimate or the equipment is inferior. Several other Service providers mentioned that often the decision maker is a renter or will be selling the home soon, or in the case for HVAC equipment they may not need a full replacement, just a tune-up. Service providers also discussed barrier to participation specific to multifamily properties. Cost of Efficiency Improvements and Split Incentives. These properties are typically rental units and therefore property owners are not typically responsible Energy Efficiency Programs 262

266 PSO High Performance Home Program for paying utilities. They are most interested in installing low-cost equipment upon fail and less concerned with high-efficiency upgrades that offer cost savings in the long run. Long Project Timelines. One Service provider indicated that energy efficient windows seem to be a measure multifamily property owners are interested in. However, they want to complete the work on individual units when they are vacant, therefore these types of projects tend to be drawn out over several years and therefore may not always be a good project for an incentive program. Difficult to Identify the Decision Makers. Two Service providers who have experience working with these property types indicated that the person makes a service request or interested in equipment are not the financial decision maker, sometimes it s a tenant or facilities person. They stated that getting in front of the right person and then ultimately getting approval can be challenging, especially when there are owned by management companies or investors with many different properties. Respondents also provided feedback on the recommendations they would make if the program rebates were not available. Ninety-three percent of Service providers indicated they would still make the same recommendations, while just a 7% said they would not. Program Communication and Training The program implementation contractor is responsible for most of the outreach, support, and training received by program Service providers. We asked respondents to comment on their experiences with staff and program sponsored training and events. Of the 27 Service providers interviewed, 22 (81%) indicated they had interactions with program staff and were, therefore, able to provide feedback. Overall, the feedback was positive and Service providers praised the timeliness and responsiveness of program staff. Some respondents did provide more feedback which is summarized below. One respondent said internal communication could be improved, while the staff is responsive and helpful he sometimes receives different information from different staff members which have caused delays in rebate processing. Several Service providers mentioned the program team s preference of using communication over direct phone calls. They said that some issues could be resolved faster if staff were more inclined to initiate a direct phone call rather than lengthy s. Phone calls seemed to be preferred by many Service providers we spoke with. Two Service providers also mentioned the issues with the electronic transmission of supporting documentation. Both Service providers indicated they had to resubmit documentation several times which was burdensome and inefficient. Energy Efficiency Programs 263

267 PSO High Performance Home Program We asked Service providers to rate their level of satisfaction with staff communication and with HPH Program overall. For both aspects, more than 80% of respondents provided a rating of 4 or 5, meaning Service providers are generally very satisfied with both staff communication and the program overall. Figure 3-42: Service provider Satisfaction One of the changes made to the program was providing budgets for the first and second months of the program year, and for regions and measure types for, Multiple and Single Upgrades, respectively. Thirteen of the service providers interviewed were asked to comment on the communication from program staff on the budget. Five Service providers indicated they received status updates on a weekly basis, while the remaining seven indicated they did not receive weekly updates but did receive notice just a day or two prior to the program funds running out. Overall, there was inconsistent feedback among Service providers regarding the frequency with which they were notified about the status of the budget. The HPH Program hosted training events throughout the program year, 25 (93%) of Service providers indicated they attended at least one program-sponsored event. Most respondents stated that they attended the program kick-off meeting, during which, program staff provided an overview of the program guidelines, new application requirements, and tools. Several respondents stated that program staff provided on-site training regarding the online intake tools and mobile applications. One Service provider said they received training on how to present the program to customers. Service providers made the following suggestions regarding ways to improve future training events: More in-depth sessions, at the kick-off meeting, that is measure specific and optional for the various attendees. Several respondents indicated that it s a diverse group and contractors need to be able to spend their time hearing information relevant to their services/equipment offerings. Energy Efficiency Programs 264

268 PSO High Performance Home Program One Service provider suggested that program staff administer a short quiz after new information is presented, to test comprehension. He indicated that some attendees seemed confused by the technical information presented. Another respondent suggested a separate session for Single and Multiple Upgrades. The respondent stated that there was confusion about the different application requirements and participation processes. One contractor suggested a hands-on training as opposed to lecture-style meetings. Generally, the feedback suggests that some Service providers would prefer an approach that is more tailored to their specific businesses. Almost all the respondents that made this suggestion also indicated that the kickoff meeting was the event they attended in Service provider Suggestions for Improvements The evaluation team asked Service providers for suggestions on how the participation processes or program designed could be improved. Most respondents praised the progress of the program with regards to the application tools and ease of participation. They emphasized that the mobile tools and electronic systems have simplified application submission. Some Service providers did have suggestions for improvements, below is a summary of their collective feedback. Several Service providers commented on the lack of detail provided with the rebate checks. They indicated that the checks do not include project references, such as a project number or address. As a result, it has been difficult to reconcile which projects they are getting paid for. They suggested including the home address with each rebate payment. Staff implemented new procedures in November and now provide additional project information. Once a Service provider applies for the Multiple Upgrades Program it can take several days to schedule the test-in visit. Several Service providers indicated that this lag is scheduling creates delays. Several Service providers expressed dissatisfaction over the electronic document submission process. They indicated that it often took multiple tries to get the documents successfully submitted and received by program staff. They understood that the system was new but said there are more glitches than they expected. Service providers made the following suggestions to Program Design. Pool Pumps: One pool pump contractor suggested adding the following questions to the application. Energy Efficiency Programs 265

269 PSO High Performance Home Program o How many gallons of water does the pool hold? o Does the pool operate seasonally or year around? According to this respondent, there are energy savings opportunities for variable speed pumps, seasonal pumps, and water feature pumps. While multiple factors may affect the energy saved by replacing single-speed pool pumps with variable speed pumps, the Arkansas TRM v. 6.0 deems savings based on unit horsepower and type (multi vs. variable speed). Windows: A Service provider who focuses on windows and doors suggested changing the application to account for projects that install windows from various manufacturers with varying U-Factors and solar gains. This respondent stated that the application only allows for one make and model. However, the program guidelines only require that windows have a U-factor less than or equal to 0.30 and SHGC of less than or equal to Duct Sealing: The program requirement for a 10% reduction in duct leakage based on the condenser tonnage. One Service provider indicated the reduction should be based on the design CFM, not the condenser tonnage. However, this comment reflects a misunderstanding of program guidelines. The duct sealing requirements are based on tonnage and CFM pre- and post-values. Program Continuity: Eighteen percent of Service provider interviewed commented on the lack of program continuity during the 2016 program year. They stated that the program stops were disruptive to their businesses during their busiest time of the year (late Spring/early Summer). Some complained of the lack of notice they were given regarding the program cessation, stating they had provided bids that included the rebates and the program ended days later. As a result, they still had to honor those discounts which were harmful to their bottom line. Program Design and Operations The following section provides a detailed overview of the High Performance Homes - Single and Multiple Upgrades Program design and operations developed through indepth discussions with program staff and a review of the various documents that are used to guide implementation, inform program partners, and reach out to prospective customers. ADM completed interviews with the PSO Program Coordinator who manages the program, the Program Manager and an Account Manager during the months of June August of 2016 and during the December 2016 January During the interviews, staff discussed their perspective on the program design and operations to enable the evaluation team can assess how the program is meeting its objectives and if operational improvements can be made to better serve customers while delivering cost-effective energy savings. This section will summarize key aspects of program design, program Energy Efficiency Programs 266

270 PSO High Performance Home Program management, marketing and outreach, project implementation, and quality control and verification procedures. The section concludes with a highlighting of key findings will highlight the most salient themes from the program areas and research activities described above. Program Design The evaluation team asked program staff to provide feedback on 2016 program design as well as energy and non-energy goals. The evaluation team interviewed program staff during the months of June - August of 2016 and in the December 2016 January 2017 period, they indicated the program is well designed to meet its energy savings goals with the designated budget and thus far the level of program activity indicates that all goals will be met or exceeded by the end of the program year. Staff attributed the program s strong performance to continuous improvement during the last seven years of operation, which has resulted inexperienced contractors and a well-designed program that has a significant level of awareness in the market. The High Performance Homes Program has a savings target of 6,160 MWh and a peak demand reduction target of 3.4 MW. These expected savings targets include savings for New Construction component in addition to Multiple and Single Upgrades. The program also seeks to achieve a distribution of savings across regions of the service territory as summarized in Table Staff indicated they also strive to achieve non-energy goals such as customer satisfaction ratings and customer service processing times. To achieve these objectives, ICF s compensation is tied to performance on key performance indicators that define the performance standards for these non-energy objectives. Staff noted that the program has performed well over the last few years and that they attributed this performance to their operation of a well-designed program that has been improved continuously since it was first offered. Table 3-133: High Performance Homes 2016 Regional Diversity Targets Region Goal Eastern > 1% Western > 6% Tulsa <90% Northern >3% Overall, the design of the 2016 program was consistent with that of the 2015 program. Figure 3-43 summarizes the program incentives for single-family homes by project. As was the case in prior years, incentive rates are higher for multiple upgrade projects. This strategy is used to encourage customers to make deeper retrofits in the house and achieve greater savings. Certain measures, such as duct sealing and building envelope improvements. Staff stated that these are not offered as single upgrades offered Energy Efficiency Programs 267

271 PSO High Performance Home Program through the single upgrades component are not offered through multiple upgrades because of the pre- and post-testing requirements that make the measures not cost effective when offered as single upgrades. Figure 3-43: 2016 Single and Multiple Upgrade Incentives A third-party test-in and test-out requirement are required for multiple upgrade projects. The cost of this testing is $500, $250 of which is covered by the contractor and $250 is covered by the program. The program excludes new construction and mobile / manufactured home. While the guidelines do not explicitly define eligible building types, the program offers a separate incentive schedule for multifamily properties. The measures offered for multifamily properties are comparable to the single/multiple upgrade offers but include direct install CFLs/LEDs, and low-flow aerators and showerheads at no cost. Incentives for the other measures are approximately one-third to one-half of the single or multiple upgrade incentives. Multifamily projects require a pre-installation energy assessments and duct Energy Efficiency Programs 268

272 PSO High Performance Home Program sealing is required for any HVAC replacement projects. Staff indicated that most multifamily projects come to the program through PSO outreach efforts on communicating directly with city officials and Service providers. The program budgeting process was modified for For 2016, the incentive budgets for both single and multiple upgrades were broken into separate allocations for the January through June and the July through December periods. Additionally, separate allocations were made for multiple upgrades for the four primary regions served by PSO and for rental and non-rental properties. The budget for the single upgrade component is broken out by measure type. These budgets are summarized in Table These budgeting allocations were made principally to manage the incentive budget so that funds remain available through the program year. Additionally, the strategy supports the program s regional savings objectives. Staff reported that the new budgeting approach had been communicated to Service providers and that their feedback was generally positive. Service providers had previously expressed frustrations regarding the inequitable distribution of funds between the more populated Tulsa region and the rest of the service territory, as well as early program cessation due to budget exhaustion. A challenge staff noted in implementing the new incentive budgeting procedures is maintain sufficient communication with Service providers regarding the status of the budget. Overall, staff thought this budgeting approach was a significant program improvement. Staff noted that the regional allocations ensured that the program met its regional objectives and that measure budgets enabled the program to allocate funds to produce the desired mix of measures. One aspect of the 2016 budgeting approach that will not be continued is the split of the incentive budget into two six-month portions. Staff reported that this created some challenges, including slower program activity during the early part of the second half of the year, and issues with contractors who provided discounts to customers when the budget was out of funds. Energy Efficiency Programs 269

273 PSO High Performance Home Program Table 3-134: 2016 HPH Program Budgetary Allocations- Single & Multiple Upgrades 49 Multiple Upgrades Region Total 2016 Rebate Budget Rebate Budget (Jan-Jun) Rebate Budget (Jul-Dec) Tulsa - Non-Rental $1,488,326 $744,163 $705,017 Tulsa - Rental $449,770 $224,885 $224,885 Multifamily $351,902 $175,951 $351,902 Northern - Non-Rental $85,046 $42,523 $40,387 Northern - Rental $108,086 $108,086 $0 Eastern - Non-Rental $226,792 $113,396 $104,912 Eastern - Rental $0 $0 $0 Western - Non-Rental $396,888 $198,444 $177,323 Western - Rental $80,000 $80,000 $86,200 Total $3,186,810 $1,687,448 $1,690,626 Single Upgrade Upgrade (End-use Category) Total 2016 Rebate Budget Rebate Budget (Jan-Jun) Rebate Budget (Jul-Dec) HVAC Replacement $586,525 $293,263 $290,826 ENERGY STAR Windows/Doors $129,328 $64,664 $45,833 Attic Insulation $342,000 $171,000 $134,600 Electronically Commutated Motor (ECM) $80,000 $40,000 $32,100 High Efficiency Water Heater $1,500 $750 $1,500 VFD Pool Pump $32,000 $16,000 $18,800 Total $1,171,353 $585,677 $523,659 In addition to the new budget allocations, program staff indicated that the overall HPH Program budget decreased in 2016, primarily due to the additional of a behavioral program, as well as the need to balance the commercial and residential budgets. In 2016, the program also made a change regarding the HVAC bonus incentive. Prior to 2016, HVAC distributors, that completed HVAC replacements for the Single Upgrades component, received a bonus incentive for each unit rebated through the program. In 2016, the bonus incentive was paid directly to contractors. 50 Program staff indicated the change was made in response to distributor feedback, suggesting that the incentives may be more effective if they target the contractors. 49 PSO. PSO Multiple Upgrades and Single Upgrade Rebate Budget Update Report xlsx 50 The bonus incentive structure is tiered, based on the unit s efficiency level: SEER 16 - $75, SEER 17 - $100, SEER 18 or greater- $125. Energy Efficiency Programs 270

274 PSO High Performance Home Program Another change for 2016 was the introduction of static duct pressure testing on multiple upgrade projects that involve HVAC replacements or duct sealing. 51 The guideline defines the static pressure threshold that the systems must test within. The static pressure is the standard used to ensure proper ductwork sizing, design airflow delivery and electrical power consumption by the blower motor. 52 Program staff indicated that in 2016 Service providers received training on how to perform the diagnostic testing required fulfill the new program requirements. Staff indicated that some Service providers were more comfortable with the testing requirements than others. Staff will continue to monitor contractor feedback regarding this requirement in future program years. ICF staff provided additional insight into the ongoing relationship the program has maintained with HVAC distributors. The program requires eligible systems to meet minimum efficiency standards. Staff noted that they work closely with distributors throughout the program year to ensure they have adequate supplies of program qualifying systems available for contractors to purchase. Staff indicated that strong relationships and consistent communication with distributors are necessary to ensure distributors are notified of program changes to eligible equipment and deadlines. Program Management and Staffing The program team is comprised of both PSO and ICF staff members, who are collectively responsible for program implementation and oversight. PSO program staff addresses issues with implementation as they arise, which include but are not limited to resolving issues with program Service providers, interpreting program guidelines, and overseeing the budget. The ICF program team manages the day-to-day operations of the HPH Program including application review and approval, payment processing, onsight home assessments and testing, as well as service provider recruitment and outreach. The team hosts a bi-weekly conference call to discuss topics related program activity and progress towards goals, guideline interpretation, and project specific issues. Staff also and speak on the phone as needed and in-between the regularly scheduled meetings. All staff interviewed indicated the relationship and level of communication between PSO and ICF remains strong and adequately supports the administration needs of the program. 51 Multiple Upgrades projects with HVAC equipment replacement must include duct sealing or duct replacement on the system that was replaced. 52 PSO 2016 Home Rebates Program Guidelines v1.5. Energy Efficiency Programs 271

275 PSO High Performance Home Program Marketing and Outreach Program marketing strategy is largely set by PSO and its consulting marketing firm, VI Marketing. ICF s role in marketing is to provide subject matter content for use in marketing materials and data analytics. This role includes writing contact for the program website and communications. Additionally, ICF adheres to the program brand standard for communications such as thank you letters to customers and certifications. To facilitate cohesion in the marketing of the program ICF attends a monthly check-in call with VI marketing and PSO. ICF also noted that they are using micro-segmentation data to inform the marketing approach with detailed information about the service territory. However, ICF indicated that they do have access to energy usage information, but not enough to target customers for outreach based on usage. In 2016 PSO s marketing strategy focused only on the New Homes Program component. Marketing and outreach for the Multiple and Single Upgrade components were stopped due to the high level of demand in 2015 and the reduction in program budget from Staff noted that the PSO does send an electronic newsletter to several hundred thousand customers that references to the program website and provide tips for home energy conservation. This ongoing customer education piece is the only marketing performed for the Single and Multiple Upgrade Program components. With regards to outreach for the multi-family properties, staff indicated that most of these projects come to the program through PSO internally or through HVAC Service providers that hear about potential projects through communication with city officials, housing authorities, or property managers. Staff said that it is difficult to find buildings that are all electric (which are targeted by the program), but they manage to get a few good projects in every year. Service Provider Management and Training The HPH Program hosted a series of kick-off meetings for Service providers early in the program year. The events lasted approximately two hours and covered general topics regarding program changes, application updates, and the new mobile app for Single Upgrades. Staff also covered more technical topics for Service providers interested is discussing HVAC upgrades and the new diagnostic testing requirements. As in prior years, the program also provides sales training to help Service providers sell efficiency projects. Table below provides a summary of 2016 Service provider outreach and training events. Energy Efficiency Programs 272

276 PSO High Performance Home Program Table 3-135: High Performance Homes 2016 Service provider Outreach Events Event Type Quantity Specialized Service Provider Training 31 Program Kick-Off Events or Updates 3 Lighting Event 1 Total Training Events 35 The program implementation contractor is responsible for coordinating outreach efforts to existing Service providers and onboarding new Service providers. Staff indicated that outreach to educate and recruit program Service providers is ongoing. Companies may have changes in staffing or leadership that could change their level of interest in the program, or homeowners might want to participate in outlying areas which would require the program to find a Service provider that could serve that customer. Long term, staff hopes to assist companies in evolving their business models to offer more program qualifying equipment and energy efficiency solutions. Constant Contact is used to maintain communications with Service providers and provide program updates and any changes to the rebate schedule. Additionally, a new challenge this year was to provide updates on the status of the budget. These are provided on a weekly basis in Excel and PDF format and includes information on total budget, reserved budget, and remaining budget by region for Multiple Upgrades and by measure for Single Upgrades. To improve the service provider experience this year, staff implemented Automated Clearing House (ACH) payments to accelerate Service provider payment. This system has reduced the days until payment is provided by five days. Additionally, Service providers are sent notifications of payment that itemize payments by project and service provided so that Service providers can more easily reconcile the payments with their records. Project Implementation and Participation Process In 2016, the program integrated more electronic communication and technology into the participation process, including a mobile application custom designed to facilitate application submission at the project site by the Service provider. Figure 3-44 below provides a screenshot of the mobile application page that intakes the business information. The Service provider, utilizing a smartphone, enters the homeowner s information and project details, including pictures of the pre- and post-installation conditions, into the mobile application form. The Service provider submits the application directly from their smartphone while at the project site; the project specific information is electronically transcribed into the program database and reviewed by staff for completeness. Program staff said they have received positive feedback from Service providers regarding the new mobile application. Energy Efficiency Programs 273

277 PSO High Performance Home Program Figure 3-44: Screenshot of PowerRebate Mobile Application In addition to the mobile application, Service providers can submit Single Upgrades applications using the online intake tool, 53 hosted by the implementation contractor. Staff indicated that the objective of the online intake tool, as opposed to a paper-based application, is to reduce applications with missing information, which reduces processing efficiency for both the application and ultimately the rebate payment. The Multiple Upgrades program component does not have an online intake tool like those described for Single Upgrade projects. Multiple Upgrade project applications must be submitted by . The format allows for the collection of additional information that the Single Upgrade application tool is not currently able to capture. Additionally, the notifications are received more quickly than submissions made through the application tool. The quick receipt of the applications allows for prompt scheduling of pre-testing that is required by the program. Monthly, the project data is the uploaded to PSO s database, referred to internally as Vision. It is from the Vision database that utility program staff can access projectspecific tracking data including measures installed, savings, and incentives paid. Staff does not depend solely on the Vision database for energy savings calculations, they 53 Web address for SU intake tool: Energy Efficiency Programs 274

278 PSO High Performance Home Program utilize an internal excel based calculator to calculate savings for each measure and project on an individual basis. Overall, program staff indicated that the databases used to implement and track program activity are sufficient for the administering the High Performance Homes Program. Next, program staff was asked to provide feedback regarding the project approval process for multi-family properties and how it differs from those completed with single family properties. Feedback indicates multi-family projects typically require earlier involvement on behalf of program staff. Staff will often accompany the Service provider to the project location and suggest a package of qualifying measures that will be cost effective. Typically, program funds can be reserved for these types of larger projects. Quality Control and Verification Quality control and verifications procedures have not changed in recent program years. For the Single Upgrades program component, 100% of projects receive a desk review. During a desk review, a program staff member will review the application for completeness and verify that invoices meet the supporting documentation requirements. The program requires that rebate applications include an itemized invoice and the PSO rebate amount provided to the customer. Staff reported that five to ten percent of Single Upgrade projects receive a site inspection, during which time implementation staff collect model numbers for installed equipment, measure and record existing insulation levels, and record windows and doors counts and types to ensure the specifications match the submitted invoices. Staff indicated that 100% of Multiple Upgrade projects receive pre- and post-installation verification visits and testing, performed by the third-party verification (TPV) firm which is contracted through the implementation contractor. Implementation staff also attend 5% of the post-installation visits. The program staff and TPVs use the third party mobile application can be accessed via a smartphone and collects the pre and post data necessary to verify the energy savings associated with program projects. Once the information is entered into the mobile application, each project is quality checked and assigned a pass/fail status. If the project fails, the verifier will make notes on what needs to be done to resolve the issue(s). Staff stated that the primary cause for fails in 2016 were HVAC systems that did not meet the minimum efficiency requirements (SEER 15). Program Strengths and Challenges A key asset of the program is there is a strong team in place whose members have significant institutional knowledge developed or consecutive years of delivering and managing the program. Additionally, the team has good communication that fosters efficient identification and resolution of issues that arise. Energy Efficiency Programs 275

279 PSO High Performance Home Program The Single Upgrades application tools have evolved due to continuous improvement efforts from staff who not only integrate feedback from program Service providers but are also committed to improving ease of use and reliability. The mobile application has been successful and staff hopes that its uptake will increase in Similarly, the Multiple Upgrades participation processes have been streamlined. Staff believes these program efficiencies have reduced the administrative burden for Services Providers while adding accountability and transparency for the homeowner. As with most energy efficiency programs, there is always room for improvement. Implementation staff indicated that there are opportunities to improve the scheduling of TPVs for the Multiple Upgrades projects. For Single Upgrades, implementation staff said they are considering bringing on additional account managers to help with customer services and Service provider support. Additionally, implementation staff indicated there is always more opportunities to reach out to the more rural customers in the PSO service territory. High Performance Homes Conclusions and Recommendations Multiple and Single Upgrades Program Design Overall the Multiple and Single Upgrade components of the High Performance Home programs are well designed to reach the residential market. Key aspects of the design are summarized below. The Multiple and Single Upgrade components are mature, well-established offerings. Through these components, PSO customers have access to rebates for a comprehensive range of energy efficiency improvements. Higher incentives are offered for implementing multiple measures to encourage customers to implement deeper efficiency improvements. Few service providers suggested additional measures to include in the programs. Recommended measures included incentives for multiple speed pool pumps, HVAC tune-ups, energy recovery ventilators, and variable refrigerant flow HVAC systems. Incentive structures for multifamily units are approximately one-third of the multiple upgrade incentives. The program targets all electric buildings because of the greater savings generated for the incentive costs, however, buildings with natural gas service are eligible to participate. Single family and duplex rental units managed by property managers receive efficiency improvements at no cost. Staff reported that no-cost upgrades encourage participation from this segment of the market where the owners generally do not cover the utility costs. Energy Efficiency Programs 276

280 PSO High Performance Home Program The program primarily utilizes Service providers to perform outreach and sell efficiency improvements to the customers. Development of multifamily projects more directly involves outreach effort by program staffs. The High Performance Homes Program has an overall savings goal, but also regional targets to ensure that program services and incentives are provided throughout the service territory. Payments for duct sealing projects exceeded $500 for 49% of projects and more than $1000 for 19% of the project. Duct sealing accounted for 40% of expected savings for Multiple Upgrades. The cost per kwh saved was $0.49 per expected kwh saved. Although this amount was comparable to other program measures such as central air conditioner replacements and door replacements and less than most insulation measures, it is a higher per kwh saved rate than found in other programs. For example, the cost per kwh of duct sealing for Entergy New Orleans was $ While regional differences may account for some of this difference, the difference is sufficiently large to warrant review of the current incentive structure. Moreover, it is not uncommon for programs to cap duct sealing incentives in the range of $200 to $400. Program Administration Key findings regarding program administration procedures are: To prevent the incentive budget from running out early in the program year, the program split the budget into two portions, one for the January June period and the other for the July December period. The program funds lasted until May and then restarted in mid-july. While this approach ensured that funds were still available at the end of the program year, it also introduced some new challenges. Staff reported that restarting the program in the second period slowed initial activity more than anticipated and that there were some issues with contractors completing projects after the budget for the first six months expired. Contractors indicated that this budgeting approach caused confusion and loss of momentum due to the need to re-engage in the program in midyear. The staff does not plan on using this approach in the future. Budget allocations for Multiple Upgrades were also made for the four regions of the service territory (Tulsa, Tulsa-North, Lawton, and McAlester). This strategy was effective at ensuring the program met its regional savings objectives. The Single Upgrade component incentive budget was allocated across measures. Staff indicated that this was a useful modification because it allowed them to effectively manage the measures incentivized through the program. 54 ADM Associates (2016). Evaluation of PY5 Energy Efficiency Programs Portfolio. Energy Efficiency Programs 277

281 PSO High Performance Home Program The program has enhanced its use of electronic tools. The program now uses an application that records site inspection information and transfers this information to the program database. The use of the application. Additionally, Single Upgrade applications can be submitted using an application. Service providers noted that they were some issues with using the application, namely that multiple submissions were often required to get documents successfully transmitted. Program staff believes these some of the greatest strengths of the HPH Program and will continue to enhance these tools in future program years. Program Implementation and Delivery The following summarizes key findings regarding program implementation and delivery. Staff has made changes to the program to improve the Service provider experience with it. Key changes include the use of Automated Clearing House (ACH) payments that have reduced payment time by several days. Staff reported that the accelerated payment processing is particularly appreciated if they can pay the Service provider before the Service provider is invoiced for equipment and materials. Additionally, the program now provides additional information along with payment to enable Service providers to match payments to specific projects. The program marketing was limited during the year. This is a sensible strategy given past and current demand. The program has strong quality control and verification procedures. Five to ten percent of Single Upgrade projects receive site inspections and all Multiple Upgrade projects receive third party verifications. While the third-party verification procedure increases costs, which prevents duct sealing and air sealing from being offered as single measures, they also help to mitigate risks from erroneous or fraudulent pre-testing values that could occur. Staff reported that through their verification procedures, some issues were noted with contractors during the program year. One of the programs active Service providers received disciplinary action during the year. Staff reported that the combination of increasing standards for work (e.g., the introduction of static pressure testing for HVAC projects) and changes in the firms work crews, led to the lax performance. Staff noted that the incident emphasizes the need for continued training and education with Service providers to effectively communicate the program's technical requirements and the importance of adhering to them. Additionally, some of the language in the program guidelines was clarified to reduce confusion. Energy Efficiency Programs 278

282 PSO High Performance Home Program Service provider training continued in 2016 with three kickoff events providing program updates and thirty-one specialized training sessions. The program again offered sales training to help Service providers sell efficiency improvements to customers. Service providers also receive an after attending training with links to program documents. Despite the educational efforts made by staff, interviews with service providers suggested that there may be some misunderstandings about the program requirements among Service providers including duct sealing performance requirements and HVAC sizing requirements. Participant satisfaction remained high. 97% percent of survey respondents reported that they were satisfied with the program overall. Most respondents (90%) did not identify anything that they disliked about the efficiency improvements. Those that did primarily cited the higher cost. The following recommendations are offered for continued improvement of the program. Conduct diagnostic testing of the online application tool to identify submission errors. Service providers reported that multiple submissions were often required to transmit documents. Review marketing approach for in consideration of the insulation incentive reduction. Staff has limited program marketing due to the high level of program demand. However, staff reported that a reduction of program incentives for insulation lead to a Service provider that accounted for a significant share of Single Upgrade savings discontinuing participation. Concern was expressed that this change may hinder the program s ability to meet its savings goal. Staff should consider targeted campaigns emphasizing lower cost improvements like attic and ceiling insulation as a way of encouraging homeowners not interested in larger upgrades to improve the efficiency of their home. Explore introducing air sealing and duct sealing as single or bundled upgrades. Currently, the program only offers air sealing and duct sealing as part of a Multiple Upgrades project because the cost of third party verification makes single upgrade with these measures to costly relative to the benefit. The thirdparty verification is important because it prevents erroneous or fraudulent pretesting results. An alternative approach that may allow these projects to be offered as either single improvements or combined improvements is to conduct pre-testing on a limited sample of participating homes. Additional Training and Education on Program Requirements. Staff noted that some quality issues were identified during the program year and our interviews with Service providers suggested some misconceptions of the program requirements. While staff currently provides training and documents to Energy Efficiency Programs 279

283 PSO High Performance Home Program Service providers, they also noted the need for continually providing this training and that changes in work crews can lead to a lapse in adhering to standards. Staff should consider strategies for ensuring that new crew members are trained on program standards. For example, working closely with Service provider firms and requiring notification of new staff and the development of online videos demonstrating technical skills and adherence to program requirements. Add Language to Guidelines that Distinguishing Single Family from Multifamily homes. The Home Rebate Guidelines currently state that Mobile Homes, Manufactured Homes, and New Construction Homes do not qualify but lack corresponding language on multifamily properties, which have a different rebate scheduled. Add Language to Guidelines that Distinguishing Single Family from Multifamily homes. The Home Rebate Guidelines currently state that Mobile Homes, Manufactured Homes, and New Construction Homes do not qualify but lack corresponding language on multifamily properties, which have a different rebate scheduled. Adjust Duct Sealing Incentives. The current incentive structure allows for relatively large incentive payments for this measure and larger incentive payments do not necessarily result in greater program savings. Staff should consider either basing incentives on leakage reduction rather than the number of supply and return registers or instituting a cap on incentives for the measure Planned Program Changes The following changes are planned or currently implemented for The following incentive changes were made for 2017: o The cap for efficient windows was increased from $500 to $600 per project; o High efficiency water heaters dropped from single upgrades; o Duct return added increased from $120 to $150; o Air infiltration up from $300 to $500 for a 20% reduction in leakage. Incentives are $150 for between 10% and less than 20% reduction in leakage. o Staff reported that insulation incentives decreased. Staff clarified language in the guidelines on ducts in slab foundations to make them easier to understand. Energy Efficiency Programs 280

284 PSO High Performance Home Program The program will no longer split the incentive budget into two portions corresponding to the first and second half of the program year. One annual budget will be provided. The program plans to further expand the features of the Single Upgrades Online Intake to incorporate the data fields available in Vision. Doing so will enhance the user experience and provide information to Service providers on incentive payments and energy savings Single Upgrades (SU) Component EM&V Methodologies This section provides an overview of the data collection activities, gross and net impact estimation methodologies, and process evaluation activities that ADM employed in the evaluation of the Single Upgrades component of the High Performance Homes program. Data Collection The primary data collection activities for the Single Upgrades component of the program consisted of a participant telephone survey, a separate sample of on-site verification visits, in-depth interviews with program staff, and discussions with project auditors. Additional data reviewed included a census of program tracking data from SSRS and, when necessary, project documentation obtained from VisionDSM. Table summarizes the sample size for each primary data collection activity. The random sample for verification was designed to achieve ±10% relative precision or better at the 90% confidence interval. Table 3-136: Sample Sizes for Data Collection Efforts Multiple Upgrades Data Collection Activity Achieved Sample Size Participant Survey 150 On-Site Verification Visits 28 In-Depth Interviews with Program Staff 4 Service Provider Survey 27 Program tracking data for the SU component of the program included customer contact information and descriptions of the measures installed. Each SU participant was assigned a random number, and the list of customers was sorted by the random number to create a prioritized call list. Ultimately, 150 surveys were completed. On-site verification visits were conducted for 28 participants. During the visits, ADM field staff verified measure installation and operational characteristics. For HVAC equipment, Energy Efficiency Programs 281

285 PSO High Performance Home Program the type, capacity, efficiency ratings, and model numbers were recorded. For attic insulation, installation was visually verified and pre- and post- insulation levels were measured where possible. Heating and cooling system types were also recorded along with an approximate measurement of square feet of insulation installed. For highefficiency windows/doors, the total count of new units was recorded along with heating and cooling system type. Where possible, pre-existing window type (dual/single pane) was recorded. Pictures were taken during all on-site visits to document measure installation and characteristics. The findings from the on-site verification visits were compared to information in the program tracking database to verify that input to savings calculations was correctly recorded. ADM completed a survey of 27 participating MU service providers to elicit their feedback regarding various aspects of the program. 55 Finally, program staff members from PSO and the implementation contractor were interviewed to elicit the program administrator perspective on program processes and operations. Gross Impact Methodologies The methodology used to calculate energy and demand impacts consisted of verifying measure installation for a sample of program participants, reviewing deemed savings estimates for each measure as described in the Oklahoma Deemed Savings Documents, and reviewing the program tracking data to ensure that deemed savings were applied appropriately. ADM s review of the tracking data was performed to ensure correct deemed savings application for a census of projects and measures rebated through the program during PY2016. Verification of measure installation was conducted through telephone surveys with program participants and on-site visits to participating homes. The telephone surveys included questions aimed at verifying program participation and the number and type of measures installed by program participants. During the site visits, ADM field staff verified that the claimed energy efficiency measures were installed, and recorded key inputs to savings calculations such as capacity and efficiency of HVAC equipment, and R-value and square footage of installed insulation. Data collected through these activities was used to develop measure level verification rates. The measure verification rates were then applied to the deemed savings estimates to determine verified savings. This approach to evaluating the reported savings values provides verification of measure installation and correct application of the deemed savings values. The Oklahoma Deemed Savings Documents have been previously approved by the Oklahoma Corporation Commission for the estimation of energy and peak demand 55 These 28 service providers were interviewed about their participation in the Single Upgrades program component and the Multiple Upgrades component of the High Performance Homes program. Many service providers submitted projects through both program components, while some submitted projects through only one or the other. Energy Efficiency Programs 282

286 PSO High Performance Home Program savings resulting from measures installed through the Multiple Upgrades program component. As a result, ADM s evaluation does not include a specific assessment of the deemed savings validity for PY2016 participants. Net-to-Gross Estimation This section provides a summary of the methodology ADM used to score survey responses for free ridership and spillover. PSO customers who received rebates through the Single Upgrades component of the High Performance Homes program were surveyed and asked a series of questions aimed at estimating program attribution. The attribution scoring system is broken down into two components: free-ridership score and spillover score. Each component is described individually below, followed by a section discussing how the scores will be weighted to extrapolate the survey results to the program component level (all Multiple Upgrades projects). Free Ridership Scoring: The survey will be pre-populated with information about each respondent. Specifically, the list of energy efficiency measures the respondent installed through the program is known to the surveyor. The survey will be programmed to ask a series of questions about the program s effect on deciding to purchase and install each measure. For example, if a respondent installed a new heat pump they would be asked a series of questions specifically about the heat pump (but not other program eligible measures, which they did not install). The free ridership questions are arranged as follows: 1. Indicator one: prior planning 2. Indicator two: stated likelihood in absence of program incentives 3. Mitigating factor one: reported efficiency/quantity increase 4. Mitigating factor two: timeliness of upgrade How these questions work together to determine a measure-level free ridership score is displayed in Figure 3-45 on the following page. Note that the scoring algorithm requires the respondent to indicate a burden of proof that they are a free rider. They must state that either 1) they had prior plans to install the measure or 2) they would have likely installed the measure in the absence of the program. To be considered a full free rider for the specific measure, they must go on to say the program did not affect the level of efficiency (or quantity, where appropriate) they chose for their upgrade nor did it affect the timeliness of the upgrade. Energy Efficiency Programs 283

287 PSO High Performance Home Program Figure 3-45: Measure Level Free Ridership Scoring Single Upgrades Energy Efficiency Programs 284

288 PSO High Performance Home Program Spillover Scoring Spillover, as addressed in this survey, is restricted to energy efficiency measures that respondents report installing in their home without receiving additional incentives. Additionally, respondents must indicate that the additional non-incentivized measures were installed based on program influence. Potential spillover respondents are identified using the question below: 1. Following your participation in the PSO s Energy Efficiency Program, did you install any additional energy efficiency upgrades in your home for which you did not receive a rebate or financial incentive? Customers who respond no to this question will be determined to not be potential spillover candidates. If a respondent indicates that they have installed additional energy efficiency measures, they will then be asked: 2. Did your participation in PSO s Energy Efficiency Program influence your decision to install the additional energy efficient upgrades in your home? Customers who respond yes, strongly influenced or yes, somewhat influenced will be considered potential spillover candidates, and will be asked to identify the additional measures they have installed. The measures identified will then be checked to ensure they had not received a corresponding PSO rebate through any program and that they represent measures with the potential to save energy. A spillover ratio is calculated for the program that is equal to the sum of spillover savings identified by survey respondents divided by the sum of gross ex post savings for all survey respondents. Process Evaluation Activities The purpose of the process evaluation of the Single Upgrades component was to examine program operations and results, to assess how the program has developed since the prior program cycle, and to identify any key areas that may need to be addressed in future program years. A key focus was to identify any program changes or developments and their impacts on the program. Key research questions addressed by the process evaluation of the 2016 SU component include: How do participants hear about the program? What percentage is contacted directly by PSO or implementation staff? What percentage hears about the program through service providers? Were the program participants satisfied with their experience? What are the perceived energy and non-energy benefits associated with the program? What steps were taken to engage service providers and were they successful? Energy Efficiency Programs 285

289 PSO High Performance Home Program Were there any significant changes or new obstacles during the 2016 program year? During the evaluation, data and information from several sources are analyzed to achieve the stated research objectives. The internal organization and operational efficiency of program delivery are examined through analysis of interviews conducted with PSO program staff and ICF staff. Participating Multiple Upgrades customers and service providers were surveyed to obtain their feedback regarding the program as well Impact Evaluation Findings Single Upgrades Verified Gross Impacts As described in Section , the gross impact analysis consisted of verifying measure installation and checking the program tracking data to ensure that deemed savings algorithms were appropriately applied. In-Service Rates (ISR) for each measure type were developed based on the findings from the participant telephone survey and on-site visits. Findings from these activities are summarized below for each measure type. Attic/Ceiling Insulation: Deemed savings values are calculated, in accordance to the OK DSD, as described in the Multiple Upgrades Verified Gross Impacts section. Forty-Seven of the 150 participants surveyed were identified in the program tracking database as having attic insulation measures. Seventeen of the 28 site verification visits that ADM performed were homes that had attic insulation measures installed. One customer did not recall that this measure was installed as part of their upgrade. However, a review of the program tracking data for those sites produced evidence of participation including measure invoices. It is unclear why this respondent did not recall having this measure installed. In the case reviewed, evidence of attic insulation measures was identified either on site or via documentation review. As a result, an In- Service Rate (ISR) of 100% was applied for attic insulation measures. ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all but thirteen projects. Adjustments were made to these projects based on climate zone. ADM looked up the zip code and zip code database as described in the Air Infiltration description in the Multiple Upgrades section above. These discrepancies are minor, resulting in a realization rate of 100% for energy savings (kwh) and 100.2% for peak demand reduction (kw). Air Conditioner and Heat Pump Retrofits: The OK DSD specifies annual energy savings and peak demand to be calculated as shown in the formulas described in the Multiple Upgrades section. Fifty-six of the 150 participants surveyed were identified in the program tracking database as having Air Conditioner or Heat Pump measures. Seven of the 28 site verification visits that ADM performed were homes that had attic insulation measures Energy Efficiency Programs 286

290 PSO High Performance Home Program installed. All customers did recall that this measure was installed as part of their upgrade. As a result, an In-Service Rate (ISR) of 100% was applied for Air Conditioner and Heat Pump measures. ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all but twenty-six projects. Adjustments were made to these projects based on climate zone. ADM looked up the zip code and zip code database as described in the Air Infiltration description in the Multiple Upgrades section above. These discrepancies are minor, resulting in a realization rate of 100% for energy savings (kwh) and 100% for peak demand reduction (kw). Electronically Commutated Motors: Deemed savings values were taken from the Oklahoma Deemed Savings Document as described in the Multiple Upgrades section. ADM s review of the program tracking data found that deemed energy savings (kwh) were applied and reported correctly for all but sixty projects. Adjustments were made to twenty-six projects based on climate zone. Adjustments were made to 15 projects as they claimed no Air Conditioner was installed when, upon review of their projects, it was. Adjustments were made to 19 projects claiming an Air Conditioner unit and Heat Pump unit was installed when, upon review of their projects, there were none installed. Forty-four projects claimed peak demand savings (kw) incorrectly. These projects claim a new Air Conditioner unit was installed, making the claimed savings 0. Upon review of these projects, an Air Conditioner unit was not installed. These discrepancies result in a realization rate of 103% for energy savings (kwh) and 173.5% for peak demand reduction (kw). High Efficiency Windows and Doors: Deemed savings are calculated per square foot of window, inclusive of frame and sash as described in the Multiple Upgrades section. Forty of the 150 participants surveyed were identified in the program tracking database as having Air Conditioner or Heat Pump measures. One of the 28 site verification visits that ADM performed were homes that had windows or door measures installed. Two customers did not recall that this measure was installed as part of their upgrade. However, a review of the program tracking data for those sites produced evidence of participation including measure invoices. It is unclear why these respondents did not recall having this measure installed. In the case reviewed, evidence of window or door measures were identified either on site or via documentation review. As a result, an In- Service Rate (ISR) of 100% was applied for attic insulation measures. ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all but five projects. Adjustments were made to these projects based on climate zone. ADM looked up the zip code and zip code database as described in the Air Infiltration description in the Multiple Upgrades section above. Energy Efficiency Programs 287

291 PSO High Performance Home Program These discrepancies are minor, resulting in a realization rate of 100.1% for energy savings (kwh) and 100.1% for peak demand reduction (kw). Solar Screens: Deemed savings are calculated, in accordance with the Oklahoma Deemed Savings Document as described in the Multiple Upgrades section above. None of the 23 site verification visits that ADM performed were homes that had solar screen insulation measures installed. ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all projects resulting in a 100% realization rate for energy savings (kwh). Ground Source Heat Pump: The OK DSD specifies annual energy savings and peak demand to be calculated as shown in the formulas below. The algorithms are dependent upon climate zone, capacity, and efficiency ratings of the installed units. Where: CAP EFLHC EFLHH EERbase EERGSHP COPBase COPGSHP,, Energy Savings 1 1, ,412 1 Peak Demand Savings 1 1, , / = Rated equipment cooling capacity of the new unit (Btu/hr) = Equivalent full load hours for cooling = Equivalent full load hours for heating = Energy Efficiency Ration of the baseline cooling equipment = Energy Efficiency Ration of the installed GSHP = Coefficient of Performance for the baseline heating equipment = Coefficient of Performance of the GSHP CF = Coincidence Factor = 0.87 One of the 150 participants surveyed was identified in the program tracking database as having ground source heat pump measures. One of the 28 site verification visits that ADM performed were homes that had attic insulation measures installed. The one customer did recall that this measure was installed as part of their upgrade. As a result, an In-Service Rate (ISR) of 100% was applied for attic insulation measures. Energy Efficiency Programs 288

292 PSO High Performance Home Program ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all but four projects. Adjustments were made to two projects as they did not meet efficiency standards after the installation. These discrepancies result in a realization rate of 109% for energy savings (kwh) and 99% for peak demand reduction (kw). Pool Pumps: The savings calculations for pool pumps, taken from the Oklahoma Deemed Savings Document, are seen below. Deemed values for Energy Factors are dependent upon the Horse Power of the pool pump installed. Annual Energy Savings /1000 Peak Demand Savings Where: = # / # / Capacitygal = Capacity of swimming pool in gallons. = 22,000 gal (default) # Turnovers = Number of pool turnovers per day. = 1 (default) EF = Energy Factor (EF) of the pump, motor, and speed combination in gallons/wh. From table below Hrsbase = Average hours a single-speed pump and motor need to turn over the volume in a pool. = 5.7 hours (default) Hrslo = Average hours motor and pump spent in low speed for filtering. Hrshi = 7.4 hours (default) = Average hours motor and pump spent in high speed for operation for pool cleaners and other additional pool features. = 2hours (default) Days = Days of pool operation per year. Year Round = 365 days Summer only = 167 days Six of the 150 participants surveyed was identified in the program tracking database as having ground source heat pump measures. One of the 28 site verification visits that ADM performed were homes that had attic insulation measures installed. All customers did recall that this measure was installed as part of their upgrade. As a result, an In- Service Rate (ISR) of 100% was applied for attic insulation measures. Energy Efficiency Programs 289

293 PSO High Performance Home Program ADM s review of the program tracking data found that deemed savings were applied and reported correctly for all projects. This results in a realization rate of 10% for energy savings (kwh) and peak demand reduction (kw). Determination of Program Level NTGR The free ridership scores for each respondent will be weighted by the ex ante kwh savings per project to determine the final weighted average free-ridership estimate per customer in the sample. The NTGR including the spillover estimate was multiplied by the program level verified gross savings to to determine net savings Process Evaluation Findings This section presents the core findings developed from a review of program tracking data, surveys conducted with participating customers. Findings pertaining to program operations and design and interviews with program service providers are discussed under the Multiple Upgrades section. There is considerable overlap in program operations and service provider participation between Multiple and Single Upgrade components and consequently, a single set of findings is presented to prevent unnecessary repetition. Program Participation Table summarizes program participation activity by measure type. As shown, HVAC projects accounted for approximately two-thirds of program expected savings. Table 3-137: Single Upgrade Measures and Incentive Amounts Measure Number of Units Percent of Reported Savings Incentives Per Unit Dollars per Reported kwh Saved ASHP - SEER % $500 $0.18 ASHP - SEER % $550 $0.21 ASHP - SEER % $600 $0.17 Attic Insulation % $400 $0.42 CAC - SEER % $500 $0.49 CAC - SEER % $550 $0.50 CAC - SEER % $600 $0.40 Electronically Commutated Motor (ECM) 1,117 9% $100 $0.39 ENERGY STAR Doors 65 0% $30 $0.17 ENERGY STAR Windows 5,291 11% $30 $0.42 Geothermal Heat Pump 9 1% $600 + $350 per ton Solar Screens 1 <1% $20 na $0.37 Variable Speed Drive Pool Pump 77 3% $400 $0.29 Energy Efficiency Programs 290

294 PSO High Performance Home Program The program saw participation from a large number of service provider firms. In total, 113 firms assisted customers with Single Upgrade projects. As shown in Figure 3-46, while energy savings were concentrated in the top 15 firms, which accounted for a combined 70% of program savings, there was generally broad participation from service providers. Unlike in 2015, the most active firm achieved most of its savings from HVAC replacements (in 2015, the most active firm achieved savings through insulation projects). Most of the savings for the second most active firm came from insulation projects. The program has since made a change to the incentive structure of the program that resulted in this firm no longer operating in PSO s service territory in Figure 3-46: Distribution of Reported Savings by Service Provider Company Energy Efficiency Programs 291

295 PSO High Performance Home Program Table displays the share of savings by district. As shown, the program achieved 8% of expected savings outside of the Tulsa area. Table District Share of Reported kwh Savings District Percent of Reported kwh Savings Lawton 7% McAlester 1% Tulsa 87% Tulsa Northern 5% Participant Survey A telephone survey was conducted to collect data about participant decision-making, preferences, and opinions of the PSO High Performance Homes program. The surveys were conducted in September and December of 2016 and 150 combined program participants responded to the telephone survey. The following sections discuss key issues that were discovered through an analysis of survey responses. Source of Program Awareness and Reason for Completing Project As shown in Table 3-139, respondents most commonly reported that they learned of the program through their service providers. Other sources of program awareness included word-of-mouth, contractors, online advertising, and PSO website, and TV. Energy Efficiency Programs 292

296 PSO High Performance Home Program Table 3-139: How Participants Learned of the Program Response Percentage of Respondents (n = 150) From my service provider 75% How did you hear about PSO s energy efficiency upgrade rebates available through the Energy Efficiency Program? Word-of-mouth 7% Contractors 3% Online Ad 2% PSO Website 2% TV 2% Other 6% Fifty-three percent of respondents said that they had had a specific concern in mind when they contacted their service provider, most commonly the issue was broken equipment (Table 3-140). Respondents who provided other responses tended to note that they were seeking ways to lower their energy bills or were generally trying to improve their home. Table 3-140: Participant s Primary Issues or Concerns What was the primary issue or concern you were trying to address? Percentage of Response Respondents (n = 81) Broken equipment 58% Home comfort 16% Health or safety concerns 4% Other 21% Non-Energy Benefits Seventy-five percent of respondents said that the energy upgrades have made their home more comfortable, while 17% said that they did not notice a change in their home s comfort level. One respondent said that the energy efficiency improvements had made their home less comfortable. The most commonly reported benefits from the efficiency improvements were improved home comfort and lower utility bills. Energy Efficiency Programs 293

297 PSO High Performance Home Program Table 3-141: Non-Energy Benefits Response Percentage of Respondents (n = 150) What are the biggest benefits you have noticed since the work was done on your home? The home feels more comfortable 28% Utility bills are lower 21% There is less noise from the outside 16% The appliances and heating or cooling equipment are more reliable 13% There is less noise from the appliances 13% Other 8% Some or all of my home s safety concerns have been addressed There have been health improvements 3% 7% *Respondents were able to provide multiple responses. The percentages shown are percentages of respondents rather than percentages of responses. Thus, the total exceeds 100%. Sixty-four percent of respondents said that the program had made them more aware of the benefits of energy efficiency. A remaining thirty-four percent of respondents said that their awareness level had not changed, and two percent said that they did not know. Most participants (85%) did not note any aspects of the energy efficiency upgrades that they disliked. Those respondents who did provide comments primarily remarked that their energy bills were higher than expected, that they felt that the upgrades were too expensive, or that the new equipment was too loud. Use of Program Website Most respondents surveyed (77%) indicated that they had not visited the PSO website. Of those who had visited the website, most (58%) said that they visited the website to learn about what rebates were available. Another 33% visited the website to find energy saving tips, while 15% said that they visited the website to find a service provider and 15% used the website to find contact information for a program representative. Respondents who had visited the website were also asked if they experienced any difficulties looking for the information that they were seeking. Eighty-five percent of respondents said that it was easy or very easy to find the information that they were looking for, with another 9% saying that they did not know. Energy Efficiency Programs 294

298 PSO High Performance Home Program Participant Satisfaction Survey respondents rated their satisfaction with selected aspects of the program. Results were provided on a scale of 1 to 5, with 1 representing very dissatisfied and 5 representing very satisfied. Participants were asked to rank their satisfaction with their interactions with program staff, their service provider, and their overall program experience. Overall, satisfaction with all components of the program was high 94% of the respondents reported that they were somewhat or very satisfied with the program. The component of the program with the highest satisfaction scores was respondent s interactions with their service provider, which received scores of very satisfied or somewhat satisfied from 95% of respondents. Satisfaction was lowest with interactions with PSO and program staff. However, 83% of respondents were satisfied or somewhat satisfied with these interactions. Figure 3-47: Customer Satisfaction Respondents indicated dissatisfaction with some program element were given an opportunity to elaborate on their reasons for dissatisfaction. Ten customers provided reasons for their dissatisfaction. Of these, two comments related to the rebate and may reflect some misunderstanding that the rebate is providing a discount or disliking that it is provided as a discount: I thought somebody would have done something to get me a check for 600 dollars. Energy Efficiency Programs 295

299 PSO High Performance Home Program They shouldn't send the rebate to the company because they just take the money. It doesn't benefit me or anyone else that applies just the company. Two customers reported issues with the equipment installed. I think they should have sold better windows. They were the cheapest ones in order to get the refund. Other issues noted by customers included that the paperwork was a hassle (stated by one respondent), issues with the service provider (stated by two respondents), or more general comments about the program (e.g., it s not as great as it s supposed to be ). Many respondents provided open-ended commentary indicating that they highly valued the services provided by the program and that their program experience had been very positive. Commentary of this nature included: Very happy with people in the program that were involved. I m just very happy that my PSO bills have gone down because of the upgrades. I was able to buy something that was more efficient than what I would have gotten if it wasn t for the saving s program. I believe it was a very good incentive that they provided. Service Provider Interviews To avoid redundancy, the findings from the service provider interviews are presented in the section for High Performance Homes within this report. The findings presented represent service providers perspectives on both the Single Upgrade and Multiple Upgrade components. Details can be found in section Program Design and Operations Because of the overlap in program operations and design elements between the single and multiple upgrade components, findings on the program design and operations are summarized in section to avoid redundancy. High Performance Homes Conclusions and Recommendations Single Upgrades The Multiple Upgrades section summarizes conclusions and recommendations pertaining to service provider feedback and the program design and operations. This section summarizes conclusion and recommendations pertaining to the Single Upgrades participant survey. Participant satisfaction remained high. 94% percent of survey respondents reported that they were satisfied with the program overall. Most respondents (85%) did not identify anything that they disliked about the efficiency Energy Efficiency Programs 296

300 PSO High Performance Home Program improvements. Those that did primarily cited the higher cost or noise created by the new equipment Planned Program Changes The following changes are planned or currently implemented for The following incentive changes were made for 2017: o The cap for efficient windows was increased from $500 to $600 per project; o High efficiency water heaters dropped from single upgrades; o Staff reported that insulation incentives decreased. The program will no longer split the incentive budget into two portions corresponding to the first and second half of the program year. One annual budget will be provided. The program plans to further expand the features of the Single Upgrades Online Intake to incorporate the data fields available in Vision. Doing so will enhance the user experience and provide information to Service providers on incentive payments and energy savings Program Energy and Demand Impacts The New Homes, Multiple Upgrades, and Single Upgrades components combine to create the High Performance Homes program as shown in PSO s original program filings. Combining the components of the program together, verified gross annual energy savings are estimated at 8,486,679 kwh (a realization rate of 101% as compared to reported savings). Verified peak demand reductions are estimated at 3, kw representing a realization rate of 102%. Program level reported and verified savings by component are shown in Table below. Table 3-142: Program Level Gross Impacts Program Component Reported Energy Savings (kwh) Reported Peak Demand Savings (kw) Verified Gross Energy Savings (kwh) Verified Gross Peak Demand Savings (kw) New Homes 1,291, ,350, Multiple Upgrades 3,858,886 1, ,868,015 1,603 Single Upgrades 3,254,801 1, ,268,306 1, Total 8,405,641 3, ,486,679 3, Energy Efficiency Programs 297

301 PSO High Performance Home Program Table and Table summarize the verified net impacts of the complete High Performance Homes program. Overall, the evaluation estimates a program level net-togross ratio of 79%. Program Component Table 3-143: Verified Gross and Net kwh Savings Free Ridership Participant Spillover NTG Ratio Verified Gross Annual Energy Savings (kwh) Net Savings (kwh) New Homes 15.3% 0% 84.7% 1,350,358 1,143,753 Multiple Upgrades 16% 1.3% 85% 3,868,015 3,304,627 Single Upgrades 36.7% 4.9% 69.2% 3,268,306 2,262,371 Total 79% 8,486,679 6,710,751 Program Component Table 3-144: Verified Gross and Net Peak kw Reduction Free Ridership Participant Spillover NTG Ratio Verified Gross Peak Demand Reduction (kw) Net Reduction (kw) New Homes 15.3% 0% 84.7% Multiple Upgrades 16% 1.3% 85% 1,603 1, Single Upgrades 36.7% 4.9% 69.2% 1, Total 79% 3, , Energy Efficiency Programs 298

302 PSO Education Program 3.5 Education Program Program Overview PSO s Education program seeks to generate energy and demand savings by providing customers with energy conservation kits and tips. During 2016, three types of energy efficiency kits were distributed through PSO s Education Program; School Program Kits, Residential Energy Conservation Kits, and Non-Residential Energy Conservation Kits. Schools Program Kits The program implementer, Resource Action Programs (RAP), provided Energy Efficiency related curriculum and materials to fifth-grade teachers within PSO s service territory. The teachers incorporated the energy efficiency materials into their curriculum. After students had completed classroom discussions and lessons on how to save energy, students were provided with an energy conservation kit which contains the following measures: (4) 9-watt LED light bulbs (1) 7-plug advanced power strip (1) FilterTone alarm (1) LED night light (1) Digital thermometer Residential Energy Conservation Kits The Residential Energy Conservation kits were developed to offer residential customers education and resources on how to save energy in their homes. The kits included easy to install energy efficient measures, educational materials, and information on PSO s Power Hours energy efficiency program as well as information on how to apply. Along with the educational materials, the kits contained the following measures: (3) 9-watt LED light bulbs (2) 16-watt LED light bulbs (1) 7-plug advanced power strip (1) FilterTone alarm (1) LED night light (1) Digital thermometer Demand Response Programs 299

303 PSO Education Program PSO Non-Residential Conservation Kits The Non-Residential Energy Conservation kits were developed to offer PSO s small business customers education and resources on how to save energy in their businesses. The kits included easy to install energy efficient measures, educational materials, and information on PSO s High Performance Business energy efficiency program as well as information on how to apply. Along with the educational materials, the kits contained the following measures: (2) 9-watt LED light bulbs (1) 11-watt BR30 LED light bulb (1) 7-plug advanced power strip Demand Response Programs 300

304 PSO Education Program PY2016 performance metrics for all three kits are summarized in Table Table 3-145: Performance Metrics Education Program Metric PY2016 Number of Customers 28,638 Budgeted Expenditures $2,127,000 Actual Expenditures $1,782,216 Energy Impacts (kwh) Projected Energy Savings 4,949,066 Reported Energy Savings 5,733,915 Gross Verified Energy Savings 7,948,862 Net Verified Energy Savings 7,496,205 Peak Demand Impacts (kw) Projected Peak Demand Savings 700 Reported Peak Demand Savings 1,088 Gross Verified Peak Demand Savings 1,443 Net Verified Peak Demand Savings 1,381 Benefit / Cost Ratios Total Resource Cost Test Ratio 3.95 Utility Cost Test Ratio EM&V Methodologies This section provides a brief overview of the data collection activities, gross and net impact calculation methodologies, and process evaluation activities that ADM employed in the evaluation of the Energy Education program. Data Collection The primary data collection for the evaluation consisted of the following: Participant Surveys: participant telephone survey to verify program participation as well as self-report data collection from return-by-mail surveys included in the kits. Demand Response Programs 301

305 PSO Education Program Staff Interviews: ADM completed two interviews with program staff; the PSO Program Coordinator who manages the program, and the Resource Action Programs (RAP) Program Manager that implemented the program during July and August of 2016 and again in January of Tracking Data Review: ADM reviewed detailed program tracking data. The tracking data was reviewed and screened for duplicate entries, correct climate zone characterization, and potential inconsistencies. In the table below below summarizes the survey data collection activities and sample sizes. Table 3-146: Sample Sizes for Data Collection Efforts Achieved Sample Size Data Collection Activity Data use Schools Residential Non- Residential Student Survey In Service Rate 7, Teacher Survey Process Evaluation Parent Survey Process Evaluation Participant Survey (Return by Mail and Phone) Net-To-Gross In Service Rate - 8,031 1,470 Implementation Staff Interviews Process Evaluation 2 Gross Impact Methodologies The impact evaluation effort of the Education program included the following: Desk Review of Residential Calculations. ADM utilized various sources (explored below) in assessing savings from residential measures found within each kit. ISR Calculation. ADM calculated In Service Rates (ISRs) using data from customer returned surveys, included in the kits. ADM s approach to savings analysis was unique to each measure type installed. Primarily, algorithms and deemed savings from the Arkansas TRM and Oklahoma Deemed Savings Document were used to determine verified gross energy and demand Demand Response Programs 302

306 PSO Education Program impacts; however, when either of the sources was not available, ADM utilized savings algorithms or deemed savings values from the PA TRM. ADM utilized primary data collection in the form of participant surveys to determine inputs into the savings calculations. Care was taken to assure any assumptions were reasonable and current, and that there were no errors in the algorithms. For each measure in the program, total gross energy and demand savings were determined as a product of the number of measures verified as installed and the deemed savings per measure. To determine the quantity of measures delivered and installed, ADM reviewed all entries in the tracking system to ensure there were no duplicate entries. Additionally, through RAP and ADM administered participant surveys, ADM determined the kit level receipt rate, verified that kits were delivered in 2016 and determined measure level ISRs. ADM also examined the Excel workbook utilized by implementation staff (RAP) to assess savings by the project. The workbook utilizes TRM savings algorithms to estimate per kit savings based on input parameters. ADM reviewed the project savings for each measure to ensure the values were appropriate and referred to these numbers as the ex-ante values for PY2016. Process Evaluation Activities The purpose of the process evaluation was to examine program operations and results throughout the program operating year and to identify potential areas that may need to be addressed to improve the effectiveness or efficiency of the program. The process evaluation focused on a review of the program design, administration, and implementation. During the evaluation, data and information from several sources are analyzed to achieve the stated research objectives. The Evaluators completed reviews of program documentation, interviews with program staff, mail-in parent surveys, and surveys of participating teachers to develop findings pertaining to the program s design and operational procedures. Energy Savings Calculations In the following subsections, gross savings calculation methodologies are detailed by measure category. ENERGY STAR LEDs The energy savings for ENERGY STAR LEDs can be calculated by using the following equation. 56 Inputs for lighting calculations will be determined from the data from the 56 Algorithm source: Arkansas TRM Vol Demand Response Programs 303

307 PSO Education Program participant surveys in combination with algorithms and inputs found in the Arkansas TRM. Equation 1: Energy Savings Calculation for LEDs Where: /1,000 = Average delta watts for specified measure. Delta watts for LEDs will be determined by the difference in watts between an EISA compliant baseline bulb and the distributed LED. Baseline wattages will be determined based on the lumen range of the measure and the EISA baseline standards. = In-service rate, the percentage of LEDs distributed that are installed. = Average hours of use per year, assumed to be hours. 57 = Interactive effects factor to account for cooling energy savings and heating energy penalties. The peak demand reduction for ENERGY STAR LEDs can be calculated by using the following equation. Equation 2: Demand Reduction Calculation for LEDs Where: /1,000 = Average delta watts for specified measure. Delta watts for CFLs and LEDs will be determined by the difference in watts between an equivalent incandescent and the watts of the installed LED. Equivalent wattages will be determined based on the lumen range of the measure and the EISA baseline standards. = In-service rate, the percentage of LEDs distributed that are installed. = Coincidence factor for measure. = Interactive effects factor to account for cooling energy savings and heating energy penalties. Advanced Power Strips Deemed energy savings for advanced power strips in residential and commercial applications are determined based on the end-use of the strip either in a home office setting, a home entertainment setting or some other use. The determining factor of the 57 Input source: ADM 2016 HOU Benchmarking Memo Demand Response Programs 304

308 PSO Education Program end-use is what type of appliance is plugged into the master outlet of the advanced power strip. For advanced smart strips installed in residential and commercial applications, ADM utilized deemed savings values from the OK Deemed Savings Document. Deemed savings were applied based on the percentage of participant survey respondents that indicate their advanced power strip s master outlet is occupied by a television, PC, other category of appliance. Table Advanced Power Strips-Demand and Annual Energy Savings System Type kw Demand Reduction kwh Savings Residential Home Entertainment System Home Office Other Devices Commercial Small Business Whole System FilterTone Alarm Neither the Arkansas TRM or the Oklahoma Deemed Savings Document provide an algorithm for FilterTone Alarms; thus, ADM utilized the algorithm for calculating deemed savings for FilterTone Alarms provided in the Pennsylvania TRM. 58 Equation 3: Energy Savings Calculation for FilterTone Alarm Where: Assumed to be 1,337 hours The most recent version of the Pennsylvania TRM may be found here: Demand Response Programs 305

309 PSO Education Program Assumed to be 1,069 hours. Average motor full load electric demand kw, assumed to be 0.5 kw. Efficiency improvement, assumed to be 15% or = In-service rate, or percentage of units that get installed. Equation 4: Demand Reduction Calculation for FilterTone Alarm Where: Average motor full load electric demand kw, assumed to be 0.5 kw. Efficiency improvement, assumed to be 15% or In service rate, or percentage of units that get installed. Coincidence factor, assumed to be LED Night Light The savings for LED Night Lights can be calculated by using the following equation. 61 Equation 5: Energy Savings Calculation for LED Night Light Where: Wbase Assumed to be a 7 watt incandescent nightlight. Wpost Assumed to be a 1 watt LED nightlight. Hours The nightlight is assumed to operate 12 hours per day. ISR In Service Rate, or percentage of delivered units that get installed There is no peak demand reduction associated with this measure Digital Thermometer No kwh savings or kw demand reduction was claimed for the digital thermometer that was distributed in the Schools Program and Residential Energy Conservation kits. Net-to-Gross Estimation 59 and were modeled for PSO s service territory by ADM in support of PSO s PY2016 to PY2018 DSM Portfolio Plan. 60 Coincidence factor for demand reduction associated with air conditioning units provided in Arkansas TRM Vol Algorithm source: 2016 Pennsylvania TRM. Demand Response Programs 306

310 PSO Education Program The objective of the Net-to-Gross analysis is to estimate the share of program activity that would have occurred in the absence of the program. To accomplish this, the Evaluators administered a survey to program participants that contained questions regarding the participants plans to implement the kit items and the likelihood of implementing those measures had they not been provided through the program. Program participants were asked questions regarding: Whether they had plans to purchase and install the kit item; When would they have implemented the kit item in the absence of the program; The likelihood of purchasing and installing the kit item had they not received it for free. Participant responses to these questions were used to calculate two scores corresponding to the presence of prior plans and the likelihood of installing the items in the absence of the program. Prior Plans Score The prior plans score was calculated as follows: Respondents who indicated that they did not have plans to install the kit item were scored as 0. Respondents who indicated that they did have plans to install the kit item were scored as 1. This score was adjusted based on the quantity of the number of items the participant planned to install and the timing of that planned installation. The quantity adjustment was based on the share of items sent that the respondent planned to install. That is if the respondent indicated that they would have installed three of the six LEDs, the score of 1 was multiplied by.5. The timing adjustment was based on when they would have likely installed the items. For respondents that said they would have likely installed the items in the next six months, no timing adjustment was made. Respondents who indicated that they would have installed the item in the next 6 12 months, the plans score was multiplied by.5. For those that would have installed in more than 12 months, the plans score was set to 0. Likelihood of Project Completion Score The score reflecting the likelihood of completing the project in the absence of the program was based on the following question: How likely or unlikely would you have been to purchase and install the kit items if you had not received them for free? A score was assigned to each response for this question as follows: Very likely: 1 Demand Response Programs 307

311 PSO Education Program Somewhat likely:.75 Neither particularly likely or unlikely:.5 Somewhat unlikely:.25 Very unlikely: 0 Final Free Ridership Score The final free ridership score is equal to the following: Free Ridership = Average (Plans Score, Likelihood Score) * Previous experience adjustment The previous experience adjustment was based on a question about whether the respondent had similar items currently installed in the home. The free ridership score for those that answered No to this question was multiplied by Impact Evaluation Findings Table below displays the gross electric savings by kit for the program. Table 3-148: Gross Electric Savings Summary by Kit Type for PY2016 Expected Realized kwh Expected Realized Peak kw Kit kwh kwh Realization Peak kw Peak kw Realization Savings Savings Rate Savings Savings Rate Residential 1,714,552 3,202, % % Schools 3,653,761 4,185, % % Non- Residential 365, , % % Total 5,733,915 7,948, % 1,088 1, % Table outlines the ex ante and verified ex post lifetime energy (kwh) savings by measure for the 2016 Education Program. Demand Response Programs 308

312 PSO Education Program Table Gross Lifetime Savings Summary by Kit Type for PY2016 Kit Ex-ante Lifetime Energy Savings (kwh) Ex post Lifetime Energy Savings (kwh) Ex-ante Lifetime Energy Savings (kw) Ex post Lifetime Energy Savings (kw) Residential 29,218,020 31,143,578 3,988 6,658 Schools 53,087,580 47,851,527 11,696 9,803 Non-Residential 5,147,711 8,030, ,086 Total 87,453,311 87,025,193 16,579 17,547 Table presents the net savings summary, by measure, for the 2016 Education Program. The overall program NTG ratio is 91%. Table Net Savings Summary Kit Net-to- Gross Ratio Net Energy Savings (kwh) Ex Ante Ex Post Net Demand Savings (kw) Ex Ante Ex Post Net RR Net Lifetime Energy Savings (kwh) Residential 88% 1,714,552 2,832, % 27,971,203 Schools 100% 3,653,761 4,185, % 47,851,527 Non-Residential 85% 365, , % 6,868,275 Total 91% 5,733,915 7,496,205 1,101 1, % 82,691,005 Additional measure level details on kit contents are found in the following sections Residential Kit Savings Summary and Findings This section provides calculated energy savings by measure level. The variance in the ex-ante estimates and ex-post verified savings for the lighting measures is attributed to three main differences. The first is the baseline savings assumption. The ex-ante values assumed a pre-eisa baseline while the ex-post baseline considered EISA legislation. This variance was washed out by conservative ex-ante ISR assumptions and a difference in Hours of Use (HOU) estimates. The exante values utilized the HOU directly from the Arkansas TRM. The ex-post values used a slightly higher HOU assumption generated from a benchmarking exercise performed by ADM during the middle part of the program year. Demand Response Programs 309

313 PSO Education Program The difference in ex-ante estimates and ex-post verified savings for the 7-Plug advanced power strip was mostly because of the assumption in usage category. The ex-post calculation utilized survey data to determine usage type and calculated savings in accordance with the savings bin for each reported usage type. In the absence of survey data, the implementation contractor calculated unit savings based on the other use savings bin. Most of the units were reported to be used in the entertainment category. Table presents the verified ex-post energy savings (kwh) results of the 2016 Education Program Residential Energy Conservation Kit, by measure. Table Gross kwh & KW Savings Summary by Measure Measure Number of measure per Kit ISR Ex Ante Energy (kwh) Savings Ex Post Energy (kwh) Savings kwh Realization Rate Ex Ante Peak kw Savings Ex Post Peak kw Savings kw Realization Rate (3) 9-watt LED Light Bulbs 3 87% 586, , % % (2) 16-watt LED 2 73% 621, , % % (1)7-Plug Advanced Power Strip (1) LED Night Light (1) FilterTone Furnace Filter Alarm 1 85% 354, , % % 1 96% 103, , % % 48, , % % Total 1,714,552 3,202, % % Table outlines the verified ex post lifetime energy savings (kwh) by measure for the 2016 Education Program Residential Energy Conservation Kit, by measure. Demand Response Programs 310

314 PSO Education Program Table Gross Lifetime kwh savings Summary by Measure Measure Estimated Useful Lifetime (EUL) Tier One Estimated Useful Lifetime (EUL) Tier two Ex Ante Lifetime Energy Savings (kwh) Ex Post Lifetime Energy Savings (kwh) (3) 9-watt LED Light Bulbs ,739,323 4,866,180 (2) 16-watt LED ,429,872 4,496,601 (1) 7-Plug Advanced Power Strip 10 N/A 3,540,452 7,760,019 (1) LED Night Light 8 N/A 827,049 1,633,382 (1) FilterTone Furnace Filter Alarm 14 N/A 681,324 12,387,397 Total 29,218,020 31,143,578 Table shows the effects of applying NTG ratios to the verified gross impact estimates for each measure. The overall kit NTG ratio for the residential kit is 87.9%. Measure (3) 9-watt LED Light Bulbs Net-to- Gross Ratio Table Residential Kit Net Savings Summary Net Energy Savings (kwh) Net Ex Ex Ante Post Net Demand Savings (kw) Ex Ante Net Ex Post Net RR Net Lifetime Energy Savings (kwh) 83% 586, , % 4,048,662 (2) 16-watt LED 87% 621, , % 3,894,056 (1) 7-Plug Advanced Power Strip (1) LED Night Light (1) FilterTone Furnace Filter Alarm 86% 354, , % 6,669,736 87% 103, , % 1,423,493 96% 48, , % 11,935,257 Total 87.9% 1,714,552 2,832, % 27,971,203 Demand Response Programs 311

315 PSO Education Program Schools Kit Savings Summary and Findings Table presents the verified ex post energy savings (kwh) results of the 2016 Education Program School Kit, by measure. ADM verified measure level savings per TRM guidelines and obtained results that differed from RAP s calculations. Table School Kit Gross kwh and kw Savings Summary by Measure Measure Number of Measure in Kit ISR of Measure Ex Ante Energy (kwh) Savings Ex Post Energy (kwh) Savings kwh Realization Rate Ex Ante Peak kw Savings Ex Post Peak kw Savings kw Realization Rate 7-Plug Advanced Power Strip LED Night Light Furnace Filter Alarm 9-watt LED 1 77% 1,224,077 1,159,435 95% % 1 81% 203, , % % 883,255 1,218, % % 4 75% 1,342,484 1,477, % % Total 3,653,761 4,185, % % Table outlines the verified ex post lifetime energy savings (kwh) by measure for the Schools Kit. Table School Kit Gross Lifetime Savings Summary by Measure for PY2016 Measure Estimated Useful Lifetime (EUL) Tier One Estimated Useful Lifetime (EUL) Tier two Ex Ante Lifetime Energy Savings (kwh) Ex Post Lifetime Energy Savings (kwh) 7-Plug Advanced Power Strip 10 12,240,765 11,594,355 LED Night Light 8 1,631,564 2,641,718 Demand Response Programs 312

316 PSO Education Program Furnace Filter Alarm 14 12,365,573 17,061,301 (4) 9-watt LED ,849,678 16,554,154 Total 53,087,580 47,851,527 The school kits program had a standard NTG of 100%. Table School Kit Net Savings Summary by Measure for PY2016 Measure Net-to- Gross Ratio Net Energy Savings (kwh) Ex Ante Ex Post Net Demand Savings (kw) Ex Ante Ex Post Net RR Net Lifetime Savings 7-Plug Advanced Power Strip 100.0% 1,224,077 1,159, % 11,594,355 LED Night Light 100.0% 203, , % 2,641,718 Furnace Filter Alarm 100.0% 883,255 1,218, % 17,061,301 (4) 9-watt LED 100.0% 1,342,484 1,477, % 16,554,154 Total 100.0% 3,653,761 4,185, % 47,851,527 Non-Residential Kit Summary and Findings Table presents the verified ex post energy savings (kwh) results of the 2016 Education Program Non-Residential Kit, by measure. Table Non-Residential Gross Electric Savings Summary by Measure Measure Number of Measure per Kit ISR Expected kwh Savings Realized kwh Savings kwh Realization Rate Expected Peak kw Savings Realized Peak kw Savings Peak kw Realization Rate 9 Watt LED Light Bulbs BR30 11 Watt Directional LED 7-Plug Advanced Power Strip 2 87% 167, , % % 1 84% 130, , % % 1 81% 67,265 72, % - - Demand Response Programs 313

317 PSO Education Program Total 365, , % % Table outlines the verified ex post lifetime energy savings (kwh) by measure for the non-residential kit. Table Non-Residential Gross Lifetime Savings Summary by Measure Measure Estimated Useful Lifetime (EUL) Tier One Estimated Useful Lifetime (EUL) Tier two Ex Ante Lifetime Energy Savings (kwh) Ex Post Lifetime Energy Savings (kwh) (2) 9 Watt LED Light Bulbs ,514,912 4,553, BR30 11 Watt Directional LED ,960,152 2,714, Plug Advanced Power Strip , ,225 - Total 5,147,711 8,030, Demand Response Programs 314

318 PSO Education Program Table shows the effects of applying these NTG ratios to the evaluated gross impact estimates for each measure. The overall kit NTG ratio is 84.8%. Table Non-Residential Kit Net Savings Summary Measure Net-to- Gross Ratio Net Energy Savings (kwh) Ex Ante Ex Post Net Demand Savings (kw) Ex Ante Ex Post Net RR Net Lifetime Energy Savings (kwh) (2) 9 watt LED Light Bulbs BR30 Dimmable 11 Watt LED 7 Plug Advanced Power Strip 84% 167, , % 3,829,483 89% 130, , % 2,419,865 81% 67,265 61, % 618,927 Total 85% 365, , % 6,868, Process Evaluation Findings The following section provides a detailed overview of the Education Program; School Program Kits, Residential Energy Conservation Kits, and Non-Residential Energy Conservation Kits components design and operations. This review was developed through in-depth discussions with program staff and a review of the various documents that are used to guide implementation and provided to teachers and kit recipients. ADM completed interviews with the PSO Program Coordinator who manages the program and the Resource Action Programs (RAP) Program Manager that implements the program during July and August of 2016 and again in January of During the interviews, staff discussed their perspective on the program design and operations including marketing and outreach efforts, staffing and program management, and the participation process. Program Operations Residential and Non-Residential Kits: The implementation and participation process is similar for the Residential and Non-Residential kits. Initial steps are to configure RAP s Synergy data system to the program specifications and import data with customer lists for contact. Customers are then contacted primarily by direct mail but also by other marketing avenues. Because of the delay in launching the kits component, the process for contacting kits was not particularly strategic, aside from the focus on rural and small business customers, but in future years, staff indicated that they intend Demand Response Programs 315

319 PSO Education Program to work with community affairs staff to identify more specific communities that they want to target. A customer can enroll to receive a kit by returning the mailer, calling the toll-free number, or enrolling online. For online enrollment, customers enter a code provided on the mailing, which links the request to the data. Alternatively, the customer enters their PSO account number for the request. The plan in future years is to ship kits on a monthly schedule, but the intervals between shipments this year were less than one month because of the time constraints caused by the delayed launch. Kits ship with the kit items, a survey for verifying the receipt of the items and reporting the installation of the items, installation instructions, and a program flyer. Kits that are returned undeliverable are entered into a queue and staff follows up with the customer. Kits are resent if the shipping issue is resolved or are otherwise marked as undeliverable. Returned items are tracked and documented. Returned surveys are scanned electronically into the Synergy system and uploaded for use by the client and evaluation purposes. As with the School Kits, program activity data is transferred from RAP to PSO via a flat file provided by RAP to PSO. Additionally, RAP provides any additional reporting or summaries of activities as requested by PSO. Overall, the delivery process of the kits appears to operate well and staff did not note any issues with that aspect of program delivery. Program Design The design of the School Kits and the Residential and Non-Residential Kits components are discussed below. School Kits: The School Kits components have operated continuously since 2010 and have evolved over the years. The only change noted for 2016 was that the year marked the first time the program claimed savings for the program. The School Kits are offered to 5 th -grade students and their teachers. The School Kits component of the Education program provides education on the value of electricity use and energy conservation delivered by participating teachers and the energy savings kits provided to students. In terms of numerical goals for the program, the program targets at least 90% participation from the eligible schools. This level of participation is greater than what RAP typically targets, which is around 80%. Additionally, the program targets distribution of 16,000 kits to students and teachers during the year. Both the school participation and kits distribution goals were met. More broadly, the program seeks to realize energy savings and peak demand reductions through the installation of the kit items and to educate students and teachers on the value of electricity and energy conservation. A third objective is to increase Demand Response Programs 316

320 PSO Education Program awareness of the availability of all PSO incentives, which are promoted through the inclusion of a flyer in the kits. RAP provides teachers and students several materials to support the educational objectives of the program. The materials provided to teachers include a teacher book, a checklist for administering the program, a five-day teaching plan, a chart that links the program content to Oklahoma educational standards, and an evaluation form. Table summarizes the five-day educational curriculum. Each day s lesson is estimated to take between minutes. Table Summary of School Kits Curriculum Lesson Expected Time Primary Content Correlation with State Standards (As Assessed by RAP) Day Minutes Introduce program, learn about sources of energy. Science Process Standard 1 Science Process Standard 3 Language Arts Standards: Language Arts Reading Standard 1, 1.4.b Language Arts Reading Standard 3, 3.3.b Language Arts Reading Standard 5, 5.1.c, 5.1.e Math Process Standard 2, 2.3 Day Minutes Saving electricity, career paths, kit contents. Science Process Standard 3 Language Arts Reading Standard 1, 1.4.b Language Arts Reading Standard 3, 3.3.b Language Arts Reading Standard 5, 5.1.c, 5.1.e Day minutes Using electricity and measure its use. Science Process Standard 1 Language Arts Reading Standard 1, 1.4.b Language Arts Reading Standard 3, 3.3.b Math Process Standard 2, 2.3 Day minutes Peak periods and demand response. Process Standard 1, 1.2 Language Arts Reading Standard 1, 1.4.b Language Arts Reading Standard 3, 3.3.b Day Minutes Learning about kit contents. Science Process Standard 1, 1.2 Language Arts Reading Standard 1, 1.4.b Language Arts Reading Standard 3, 3.3.b Math Process Standard 2, 2.3, 2.a, 2.c Student learning is supported by a reading guide with chapters that correspond to each day s lessons. The program also incorporates in-class exercises to engage students and reinforce learning. In addition to the educational component, the program provides each student with an energy saving kit that contains LED light bulbs, an advanced power strip, a filter tone alarm, a LED nightlight, and a digital thermometer. Students are encouraged to take the Demand Response Programs 317

321 PSO Education Program items home and work with their parents to install them. The kit also includes an evaluation form and a survey for students to return that includes information on the student s home (e.g., type of home, the number of people living in it), behavioral actions such as adjusting the thermostat, and whether the kit items were installed. The kit includes installation guides and website links to online materials. Materials are provided in English and Spanish. To assess the program s impact on student learning, students receive a pre- and postprogram quiz. The quiz includes questions about energy, energy use, and energy costs. Staff report that the number of correct items is generally higher on the post-program quiz. Additionally, staff also referred to a study completed in California that found that program participants had higher test scores than non-participants. Residential and Non-Residential Kits: The Residential and Non-Residential Kits components are intended to provide customers with items to help them save energy and to provide information about all of PSO s services. The programs are specifically targeting harder to reach customer groups; the Residential Kits component targets customers living in rural areas of the service territory and the Non-Residential Kits target smaller businesses with annual consumption of less than 220,000 kwh. 62 Both the Residential and Non-Residential Kits include information on PSO s incentive programs to support the objective of improving program awareness. Materials are provided in English and Spanish. The kit distribution goals for the Residential and Non-Residential kits were 7,000 and 3,000 kits distributed, respectively. The program exceeded its Residential goal but fell short of the Non-Residential goal. The reasons identified for not meeting the goals (discussed in greater detail below) were that the program launched late, there was not a dedicated campaign for the Non-Residential kits and that the response to the direct mail campaign fell well short of expectations. Program Management and Staffing The program team is comprised of both PSO and RAP staff members, who are collectively responsible for program implementation and oversight. PSO s role is primarily to provide oversite but also develops marketing materials about PSO s incentive programs for inclusions in the kits. Additionally, in conjunction with their marketing consultant, PSO develops marketing and outreach materials for the program. RAP s role is to offer a turn-key program and provide full implementation of it including outreach, school and customer recruitment, procurement of kit items, development of teaching materials, delivery and tracking of kit shipments, and collection of survey data for program feedback purposes and to collect information on installation rates used to 62 This is the same consumption threshold used to qualify customers for the Small Business Direct Install Program. Demand Response Programs 318

322 PSO Education Program estimate program savings. The RAP program manager is supported by staff from several functional groups as education design, outreach, procurement, and order fulfillment. The PSO and RAP hold a bi-weekly conference call to discuss topics related to current program activities, progress towards goals, and other issues that need to be addressed during the program year. RAP has little direct interaction with the marketing consultancy, for which PSO primarily serves as a conduit for communication between the two parties. Overall, staff at PSO and RAP indicated that communications between the parties were effective. PSO noted that RAP was very thorough in its implementation of the program. RAP noted that PSO is effective at assigning and utilizing backup staff during periods when the primary contact is unavailable and that this is very valuable for keeping program implementation on track. However, RAP staff indicated that some coordination improvements may be realized if they had direct contact with the marketing consultant. Marketing and Outreach Schools Kits: RAP performs outreach to enroll teachers in the School Kit component. This outreach is limited to a list of qualified schools that is developed in consultation with PSO. RAP provides PSO with a list of schools and the portion of the areas surrounding the school that is in PSO s service territory. From that list, PSO selects the eligible schools. RAP s outreach is primarily directed at teachers at the eligible schools. For teachers that participated in the past, staff sends a renewal . Teachers that do not respond are contacted by phone or fax. Staff noted that teachers that participate in the program are an important asset in recruiting new teachers. RAP noted that a key barrier to recruiting teachers is the time commitment. This is particularly an issue with newer teachers who are more likely to feel somewhat overwhelmed by the job. The program uses multiple tactics to address this barrier. First, staff noted that a key asset to recruiting new teachers is teachers who have previously participated and can assuage potential new participants concerns. Another strategy employed is to contact teachers early in the year to provide them time for planning the incorporation of the materials into their lesson plans. Lastly, the alignment of the program s teaching content with State learning standards is a key selling point with teachers. Residential and Non-Residential Kits: Multiple approaches were used to market the availability of the kits to customers. First, electronic communications were sent to customers promoting the residential and non-residential kits. For the Residential Kits component, RAP developed an campaign that focused exclusively on promoting the kits. For the commercial programs, the kits were promoted as an item in a PSO newsletter. Second, customers may access information about the kits on the Demand Response Programs 319

323 PSO Education Program residential and business program websites. However, the primary marketing approach were direct mail campaigns. Because the program is seeking to reach customers with lower participation rates, the direct mail campaigns targeted residential customers in rural locations and smaller businesses. Although the intent of the program is to reach customers that have not participated in the programs, staff indicated that the mailing lists were not scrubbed for past program participants. However, staff considered it unlikely that many past participants would have received incentives from the program given the lower participation from rural residential customers and small businesses. The residential and non-residential mailers included a picture of the kit contents and emphasized that the items were free. The residential kit mailer differed from the nonresidential mailer in one regard, namely, it noted that the kit contents were personalized for the customer s home. RAP expected that in response to the mailing, approximately 20% of residential customers and 8-14% of non-residential customers would request a kit. The program met the expectation for the residential program, though staff noted that it took eight weeks to hit that target instead of the usual four weeks. The program fell short of the expectation for the non-residential component and that the actual response rate was between one and one-half percent and two percent. RAP noted that while they have less experience providing non-residential kits than residential kits, the low response was a surprise and that it may be due to customers skepticism of a free offer. To offset the impact of the poor response to the non-residential kit offer, staff provided 50 kits to the Service Providers delivering services for PSO s Small Business Efficiency Solutions Program. Staff reported that Service Providers believed they could easily distribute the kits and that they would provide a door opener for the program. Three of the five Service Providers distributed all kits allotted to them, while the remaining two firms distributed very few of the kits Program Implementation and Participation Process School Kits: Staff noted that the School Kits component has operated for several years and several improvements have been made during that time. Because of these continual improvements, staff indicated that the program is operating smoothly, like a well-oiled machine, as one staff member commented. Participation is initiated when teachers agree to participate by sending an response or a response by fax machine (30-35% enroll by fax). Kits are assembled by RAP and delivered to teacher classrooms along with the educational materials. Kits are shipped at two points of the year, fall, and spring. Because of the program s budget cycle, most of the kits are distributed during the fall, with a few additional kits delivered in the spring. The heavy fall activity is somewhat atypical among the program s run by RAP, but staff indicated that 20-30% of the programs operate this way. Demand Response Programs 320

324 PSO Education Program RAP manufactures the furnace whistle included in the kits but LED light bulbs and other items are sourced from different vendors. Large orders for these items are placed in several different programs and, during a program year, all products of a given type are sourced from a single vendor. Kits are quality checked before shipping to verify that all the correct items are included and that that the packaging is correct. Once kits have shipped, outreach staff contact teachers to confirm receipt of the kit and to offer additional support. Outreach staff are assigned to a teacher. The assigned outreach person will contact the teacher for enrollment, confirm kit delivery and provide any needed support. This is done to encourage a one-to-one relationship and provide a single point of contact for teachers. The next step in the participation process is that the teacher incorporates the program content into his or her lesson plans. During the delivery of the program curriculum, the kits are distributed by the teachers to students to take home. The kit includes a pledge form that is signed by the student and parent. The pledge form asks the student and parent to commit to installing the items and to make two additional commitments to save energy and water. The language in the form emphasizes that the pledge is a promise to save energy and water. The intent of this form is to encourage future conservation efforts. Once the student completes the workbook and installs the items, teachers collect the student surveys and pre- and post-quizzes. These items are then collected by RAP staff. Forms are electronically scanned and uploaded to RAPs data system. Program activity data is transferred from RAP to PSO via a flat file provided by RAP to PSO. Additionally, RAP provides any additional reporting or summaries of activities as requested by PSO. School Kits Participating Teacher Survey Results The following presents a summary of teacher survey feedback on the program. The survey was administered by RAP and ADM. In total, 181 teachers provided feedback on the program. Figure 3-48 displays the activities that teachers reported completing. The School Survey (65%), How much do we use? (61%) and the Heat from Light Bulbs (56%) activities were the most commonly completed activities. Demand Response Programs 321

325 PSO Education Program Figure 3-48: Activities Completed As shown in Figure 3-49, teachers provided favorable assessments of the kits and materials with all stating that they strongly agree or agree that the kits were easy for students to use and that the materials were clear and well organized. Figure 3-49 Assessment of Kits and Materials Demand Response Programs 322

326 PSO Education Program Nearly all teachers stated that they would participate again and that they would recommend the program to their colleagues (Table 3-161). Table Teachers Willingness to Participate Again and Recommend the Program to Colleagues Percent Yes stating Would you participate again? (n = 181) 99% Would you recommend the program? (n = 180) 98% The aspect of the program that teachers most often identified as liked by students was the efficiency kits. However, several teachers also noted other aspects of the program that students liked including learning about energy use and how to save energy, the educational materials such as the workbooks, the hands-on activities, and the opportunity to work with their parents on installing the kit items in their homes. Many teachers reported that they liked how well organized the program was or how easy it was to implement. Other aspects that the teachers appreciated were the kits, the opportunity to teach conservation to students, the hands-on activities, and the fit with their curriculum. Most teachers did not have any suggested changes for the program, but some did note some potential improvements. The types of improvements noted were to improve the clarity of the teacher or student materials; incorporate speakers, videos, or field trips; or to make the student guide more visually appealing. School Kits Parent Survey Responses The following presents ADM s summary of the parent feedback survey administered by RAP. In total, data was provided from 107 parent surveys. All parents reported that the program materials were easy for them and their child to use. All but one parent stated that the program should continue in the future. Five parents provided additional comments on the program. These parents expressed appreciation for the opportunity to learn how to save money and energy. Residential and Non-Residential Kit Recipient Surveys A telephone survey was conducted to collect data about participant decision-making, preferences, and opinions of the PSO Education program. A total of 125 participants Demand Response Programs 323

327 PSO Education Program that received either the residential or non-residential kit were interviewed (In total 250 participants were interviewed). The following sections discuss conclusions that were drawn through an analysis of survey responses. Source of Program Awareness As shown in Table there were slight differences between residential and non-residential customers in how respondents heard of the availability of the kits. Non-residential customers were more likely to hear of the program through direct mail (83%) above all other means. Residential customers were also most likely to hear of the program through direct mail (70%), however a significant percent also heard of the program online or from the program website (20%). Table 3-162: How Participants Learned of the Program How did you learn about PSO's the energy conservation kits available through PSO's Education Program? Response Residential Kit (n=119) Non-Residential Kit (n=122) Direct mail from PSO 70% 83% Family and/or friends 9% 9% Online/Program Website 20% 7% Other 1% 1% Educational Benefits Participants were asked questions regarding the kit s effect on their knowledge of energy efficiency in general and on their opinion of energy efficiency technologies. Most participants who received either a residential or non-residential kit stated that their knowledge of energy efficiency had increased, although there were differences between these groups of customers. The residential customers were more likely to have learned about energy efficiency through their experiences with the program, with 85% stating their knowledge had increased somewhat or a lot compared to only 66% of nonresidential customers. In comparison, a third of non-residential customers stated that their knowledge had remained the same. Despite these differences, very few of either population stated that their knowledge had decreased. This indicates that the program is reaching its goal of educating customers on energy efficiency through the Education program. Demand Response Programs 324

328 PSO Education Program Table 3-163: Knowledge of Energy Efficiency Since receiving the kit, would you say that your knowledge of energy efficiency has Response Residential Kit (n=121) Non-Residential Kit (n=122) Increased a lot 28% 16% Increased somewhat 57% 49% Remained the same 13% 32% Decreased somewhat 2% 2% Decreased a lot 0% 0% Most participants stated that their opinion of energy efficient technologies had increased after participating in the program. Residential customers saw a larger increase, with 85% stating that their opinion of energy efficient technologies improved a lot or somewhat, compared to 76% of non-residential customers. Additionally, no participant from either group stated that their opinion of energy efficient technologies worsened. Table 3-164: Opinion of Energy Efficient Technologies Since receiving the kit, would you say that your opinion of energy efficient technologies has: Response Residential Kit (n=124) Non-Residential Kit (n=123) Improved a lot 40% 19% Improved somewhat 46% 58% Remained the same 15% 24% Worsened somewhat 0% 0% Worsened a lot 0% 0% Awareness of other PSO programs Participants were asked about their awareness of other PSO energy efficiency programs. A similar percentage of residential and non-residential kit participants, slightly more than half, stated that they knew PSO offered discounts and incentives. When asked if they became aware of the discounts and incentives because of receiving the energy efficiency kit, slightly less than half of residential (48%) stated they had, and 39% of non-residential customers stated that the efficiency kit made them aware of other programs. Demand Response Programs 325

329 PSO Education Program Table 3-165: Awareness of other Programs Did you become aware of any of these discounts and incentives through receiving the energy efficiency kit? Response Residential Kit (n=62) Non-Residential Kit (n=66) Yes 48% 39% No 52% 61% Participant Satisfaction Survey respondents rated their satisfaction with selected aspects of the program. Results were provided on a scale of 1 to 5, with 1 representing very dissatisfied and 5 representing very satisfied. Participants were asked to rate their satisfaction with aspects of the program as well as their satisfaction with the measures in the kit they received. Non-residential program participant s satisfaction with each program element and the program overall was very high. 98% of respondents stated that they were satisfied or very satisfied with the time it took to receive the kit, and 97% were satisfied or very satisfied with the PSO Education Program overall. Additionally, all three of the program participants who had interactions with program staff stated that they were very satisfied with the interactions. Figure 3-50: Non-Residential Customer Satisfaction Overall, respondents were very satisfied with the kit components. Participants were satisfied with the LED omnidirectional bulbs and the advanced power strip, with 91% of Demand Response Programs 326

330 PSO Education Program participants stating that they were satisfied or very satisfied with these measures. 84% of participants stated that they were satisfied with the directional LED lightbulb. Figure 3-51: Non-Residential Measure Satisfaction One participant elaborated on their dissatisfaction with the two omnidirectional LED bulbs. This participant stated that one of the bulbs included in the kit failed after being used for a short amount of time. Residential participant s satisfaction with the timeline and the overall program was high. Almost all participants (99%) stated that they were either satisfied or very satisfied with the program overall, and most participants (95%) stated that they were satisfied or very satisfied with the time it took to receive the kit. Demand Response Programs 327

331 PSO Education Program Figure 3-52: Residential Customer Satisfaction Most participants were satisfied with the measures that came in the kit. Participants were most satisfied with the LED nightlight (98%), and the omnidirectional LED bulbs (97%). Participants were least satisfied with the filter tone alarm, with 61% of participants stating they were satisfied or very satisfied with the measure. Figure 3-53: Residential Measure Satisfaction Demand Response Programs 328

332 PSO Education Program Respondents who indicated dissatisfaction with a program element or measure were given an opportunity to elaborate on their reasons for dissatisfaction. Eleven customers left comments about dissatisfaction with a measure received through the program. Most comments were left by participants that stated that the measure did not work. Other comments were generally about the customer s preferences not being met with the measure. One customer stated that they did not like the color of the measure, and one stated that they did not find the measure useful Energy Education Conclusions and Recommendations The following summarized the key conclusions for the Energy Education Programs The School Kits component continues to operate well and no issues with program operations and procedures were identified. Teachers provided very favorable feedback on the program and nearly all stated that they would participate again and that they would recommend the program to colleagues. All parents stated that the materials were easy for them and their child to use. The Residential Kits component met the goal, despite a late start. The Non-Residential Kits Component did not meet its distribution goal. The primary reasons for not meeting the goal was that the distribution of kits started late in the program year and the response to the direct mail campaign was less than anticipated. Additionally, electronic marketing to business customers may have been less impactful than the approach used for residential kits. Specifically, business kits were promoted as an item in a newsletter and residential kits were promoted through an blast that focused specifically on the kits. As of late January, the Non-residential Kits component was on track to meets its distribution goal and Residential Kits was ahead of its target goal. The distribution process for kits has sufficient quality assurance and control procedures in place. Additionally, the program tracks issues with kits such as delivery failures and returns. The following recommendations are offered for continued improvement. Track the kits as a source of awareness for participants in the residential and small business programs to monitor the effect of the program in increasing participation in the other PSO offered rebate programs Planned Program Changes There are no changes currently planned for the Energy Education Program. Staff stated they are reviewing strategies to improve uptake of Non-residential kits and that once survey data is reviewed, modifications may be made to the kit contents. Demand Response Programs 329

333 4. Demand Response Programs PSO s demand response (DR) portfolio in 2016 consisted of one program that targeted commercial and industrial customers. Evaluation, measurement, and verification resulted in verified peak demand reduction of MW, as shown below in Table 4-1. This represents a realization rate of 84% percent as compared to reported demand reduction for demand response programs. Table 4-1: Peak Demand Reduction Demand Response Programs Gross Peak Demand Reduction (MW) Net Impacts Program Projected Reported Verified Gross Realization Rate NTG Ratio Net Peak Demand Reductio n (MW) Business Demand Response % Demand Response Totals % As shown in Table 4-2, PSO did report annual energy savings for demand response programs. These programs sole aim is to provide load reduction capabilities during times of high demand. However, because of participants voluntary load reductions during event days, there are energy savings associated with the program. These energy savings are not persistent, in the sense that an energy efficient equipment installation provides energy savings for the life of the equipment, while energy savings from DR programs only occur during event days. The program evaluation findings resulted in verified energy savings of MWh that occurred during the three event days in Table 4-2: Annual Energy Savings Demand Response Programs Gross Annual Energy Savings (MWh) Net Impacts Program Projected Reported Verified Gross Realizatio n Rate NTG Ratio Net Annual Energy Savings (MWh) Business Demand Response NA Demand Response Totals NA Demand Response Programs 330

334 PSO Business Demand Response Program 4.1 Business Demand Response Program Overview The Business Demand Response program, also referred to as Peak Performers 63, is a Demand Response (DR) program for commercial and industrial customers in the PSO service territory. Nonresidential PSO customers enroll in the program and are notified when a load reduction event is initiated. Participants have the option of participating in each event individually, and are paid incentives based on average reduction over the course of all events. Incentives are set at $32 per average kw reduction over all event hours, and participants receive a 5% payment bonus if they opt to participate in all reduction events throughout the year (two events in 2015). There is no direct penalty for opting out of specific event days. The program is active during summer months, when average demand typically approaches designated capacity thresholds. During the summer of 2016, 186 customers (representing a total of 1,179 premise account numbers) participated in three DR events. Two events lasted from 2-6 PM ( and ). The third event lasted from 2-4 PM ( ). Table 4-3 provides an overview of PY2016 program metrics. Overall, reported peak demand reduction exceeded projections. ADM s evaluation developed verified demand reduction estimates that were lower than reported values. Both reported and verified peak demand reduction represent the average kw reduction for each customer over all 10 event hours (three event days, four hours for two events, two hours for one event), summed across participants. 63 While the official program name is Business Demand Response, it is marketed to customers as Peak Performers. Both names are used interchangeably in this report. Demand Response Programs 331

335 PSO Business Demand Response Program Table 4-3: Performance Metrics Business Demand Response Program Metric PY2016 Number of Customers 186 Budgeted Expenditures $3,116,200 Actual Expenditures $2,285,657 Energy Impacts (kwh) Projected Energy Savings 230,000 Reported Energy Savings 0 Gross Verified Energy Savings 853,240 Net Verified Energy Savings 853,240 Peak Demand Impacts (kw) Projected Peak Demand Savings 46,000 Reported Peak Demand Savings 52,541 Gross Verified Peak Demand Savings 44,018 Net Verified Peak Demand Savings 44,018 Benefit / Cost Ratios Total Resource Cost Test Ratio 7.55 Utility Cost Test Ratio EM&V Methodologies Impact Evaluation Methodologies The impact evaluation for the Business Demand Response program involves calculating participants load reduction during event periods in reference to a counterfactual baseline demand estimation. The details regarding how this process was used to estimate verified impacts are provided in the following subsections. Replicating Reported Impacts For the purposes of financial settlement with Peak Performer participants, PSO uses a top 3-of-10 baseline days methodology to estimate participants baseline load, or the demand that participants would have used had no Peak Performer event been called. Demand Response Programs 332

336 PSO Business Demand Response Program Reported program impacts were calculated based on this baseline estimation methodology. For each participant 64, one applies the following algorithm: 1. For an event day, let be the participant s actual electric demand at hour on. 2. Starting with the day before, take the most recent 10 non-weekend, nonholiday, non-peak Event days. These are the eligible baseline days. 3. For each of the eligible baseline days, calculate the average midday electric demand during the hours corresponding to the Peak Event (usually 2 PM 6 PM, but can be any two to four-hour period between 1 PM and 7 PM). Rank the eligible baseline days in descending order of this average peak time demand. 4. Take the top 3 days from the previous step and average their loads hour by hour. This is the unadjusted baseline,. 5. If, on average, the ratio of /, between 10 AM and 12 PM, is less than 1 (that is, the baseline is too low), multiply by the reciprocal of that ratio so that the baseline and event loads match prior to the event. The most can be adjusted upward is 30%; no downward adjustments are made. Reported demand reduction and payments made to Peak Performers participants depend on the difference,. PSO provided hourly interval data for all the facilities involved in the Peak Performers program. PSO staff also provided internal audits for all the events, which are produced by a database script that implements the 3-of-10 baseline. ADM used these audits and interval data to independently verify that the baseline loads reported by PSO were calculated according to the algorithm described above. Calculating Verified Impacts For the purposes of providing an independent and accurate estimate of program demand reductions, ADM used a generalized version of the 3-of-10 algorithm, which is identical to the rules used above except in four respects. Step 5 from the previous section is modified to be additive rather than multiplicative with from 10 AM to 12 PM being added to. Second, the ADM algorithm allows downward baseline adjustments. Third, the magnitude of the adjustment is not limited to 30%. This kind of adjustment, which can increase or decrease the baseline, is known as a symmetric adjustment. By design, the default asymmetric baseline must have a positive bias, since the baseline can only increase, even when having a lower baseline would be more accurate, which is what using a symmetric baseline allows for. This symmetric adjustment was used for most participants that followed a typical usage and event response pattern. The third modification to the rules is that if a participant is missing 64 By participant, we mean an individual premise. Multiple premises may belong to a single customer, but the impacts are calculated on a per-premise basis. Demand Response Programs 333

337 PSO Business Demand Response Program baseline days in its metering data, the nearest day to the event day, either before or after, is used as a baseline day as long as it followed other rules such as being a weekday and non-holiday. Baseline days are added in this manner until there is 10 total for every account premise number. After developing a reduction estimate for all participants using the methodology outlined above, ADM examined the baselines and usage patterns of the largest 30 premise account number by usage during event baseline days. These 30 sites accounted for 45% of usage during event baseline days by participants. If any of these facilities displayed atypical usage and event response patterns the baseline was manually adjusted to fit pre-event data while considering participant information such as business type. Another difference in ADM s evaluation is that a single large industrial customer (which we call Customer X), whose demand reductions are around one-quarter of the reported program impacts, was evaluated separately from the rest of the participants for extra scrutiny. Customer X operates several facilities that interact with each other and the facilities electric demand is process driven. Customer X and the 30 premise account numbers whose baselines were examined individually represent 56% of energy usage during baseline days. Overall, 1,144 premise account numbers were determined to represent facilities with typical usage and event response patterns. These facilities were analyzed using the 3- of-10 algorithm with symmetric adjustment for day-of pre-event usage. Ten account numbers were determined to represent facilities with atypical usage and/or event response patterns. These facilities were analyzed on a case-by-case basis and had their baseline manually adjusted. Finally, five premise account numbers representing Customer X were analyzed separately due to the influential nature of this customer. Seven separate baselines were developed for the customer. Evaluation staff selected the model that performed the best for each account number according to residual rootmean-square error (RRMSE) on high usage days. Net-to-Gross Demand response programs are not likely to have net-to-gross effects because customers are unlikely to curtail load in absence of the program. A net-to-gross ratio of 100% was assumed for this program. Process Evaluation Methodologies This section discusses the methodologies and activities used for the process evaluation of the 2016 Business Demand Response program. The purpose of the process evaluation is to assess the program from a structural and operational perspective to identify program strengths, weaknesses, and opportunities with a focus on program changes and developments since This evaluation is based upon surveys with Business Demand Response participants, interviews with program staff, and analysis of program data and documentation. This section provides a description of the process evaluation objectives, and a summary of the program design, background, and structure. Demand Response Programs 334

338 PSO Business Demand Response Program Evaluation Objectives The purpose of the process evaluation is to examine program operations and results throughout the program operating year and to identify potential program improvements that may prospectively increase program efficiency or effectiveness in terms of participation and satisfaction levels. This process evaluation was designed to document the operations and delivery of the PSO Peak Performers program during the 2016 program year. Key research questions to be addressed by this evaluation of the 2016 program year include: What outreach activities were undertaken during the program year to recruit new participants? How did new participants learn of the program? What factors motivated their decision to participate? Were new program participants satisfied with their experience? What was the level of satisfaction with the event notification process and other aspects of program participation? What actions did participants take to reduce load? What were the key successes and challenges during the 2016 program year? Looking forward, what are the key barriers and drivers to program success within PSO s market? In addition to addressing these general research questions, the evaluation also focused on the program changes noted by program staff during the 2016 evaluation. Summary of Primary Data Collection ADM completed an interview with the PSO program manager. The program staff interview covered the relevant researchable issues identified above. Program staff discussed program changes as well as any key successes and challenges during the program year. Staff also discussed their experience in working directly with AEG in maintaining the program database as well as any changes to event thresholds and the impacts of those changes. ADM completed two types of participant surveys during the program year. For all program participants, ADM administered a brief web-based survey shortly demand reduction events. The survey covered the following topics: Sufficiency of time given for notice of the event; How the participant was notified of the event; and Actions taken, if any, to reduce load during the event. The purpose of administering these surveys was to get a better understanding of the types of actions that customers take to reduce their load and to get quick feedback on Demand Response Programs 335

339 PSO Business Demand Response Program the notification process. By administering these surveys shortly after the time when the event is called, customer responses should be less likely to be adversely affected by poor recall. Due to the timing of the events, ADM was only able to complete a survey after one of the three events. The second survey was administered after the peak period season to customers who were new to the program in The purpose of this survey was to understand: Sources of program awareness; Factors that influenced participation; Awareness of energy use monitoring tools provided through Vision; Likelihood of participating again in 2017; and Program satisfaction. Peak Demand Reduction Impact Evaluation Findings Demand response event impacts are estimated by comparing the event day demand curves with the estimated baseline demand curves; the difference between the two is the estimated peak demand reduction. As described in Section 1.1.2, ADM used hourly interval data to recreate the baseline estimations used to determine reported impacts. The process was then repeated, this time using a symmetric adjustment, represented by ADM Adjusted Baseline in the graphs below. The symmetric adjustment was applied to all facilities with typical usage and event response patterns. The figures below show the results of the baseline estimation effort, along with actual electric demand for typical participants on event days. Thirty facilities received additional scrutiny to determine if they had atypical usage patterns during event days that required a baseline adjustment. Of these 30, 10 premise account numbers received a baseline adjustment. One customer (Customer X) was analyzed separately with multiple baseline methodologies considered. Demand Response Programs 336

340 PSO Business Demand Response Program Figure 4-1: Event 1, (Typical Participants) kw Event Period Default Adjustment Baseline Event Usage ADM Adjusted Baseline Demand Response Programs 337

341 PSO Business Demand Response Program Figure 4-2: Event 2, (Typical Participants) 65 kw Event Period Default Adjustment Baseline Event Usage ADM Adjusted Baseline 65 Event usage trails off sharply at hour 23. This is due to missing data during that hour on the event day. This missing data did not affect program impacts. Demand Response Programs 338

342 PSO Business Demand Response Program Figure 4-3: Event 3, (Typical Participants) kw Event Period Default Adjustment Baseline Event Usage ADM Adjusted Baseline As shown in the figures above, the baseline with symmetric adjustment had less of a positive bias than the default adjustment used by PSO for the first two event days and mirrored it closely on the last event day. To visualize the potential bias in the baseline estimation algorithms, the symmetric and asymmetric (default) baselines were calculated for all non-event weekdays between and compared to the actual usage. This was also done for the three hottest weekdays in the 2016 summer. The graphs below illustrate that the symmetric baseline and actual usage are nearly identical, indicating that the symmetric baseline is a more accurate counterfactual for event day analysis. Demand Response Programs 339

343 PSO Business Demand Response Program kw Figure 4-4: Average Weekday Real and Predicted Loads (Typical Participants) Event Period Average Weekday Usage Default Adjustment Baseline Symmetric Adjustment Baseline kw Figure 4-5: Average Weekday Real and Predicted Loads, three hottest days (Typical Participants) Event Period Average Weekday Usage Default Adjustment Baseline Symmetric Adjustment Baseline Figure 4-5 and Figure 4-6 compare the weekday load for all typical Peak Performer participants with the loads predicted by the 3-of-10 rule with 1) the default adjustment, Demand Response Programs 340

344 PSO Business Demand Response Program and 2) the symmetric adjustment. Of the two, the symmetric adjustment achieves the best overall fit, especially during pre-event hours (by design there will be some residual positive bias during the event hours since the baseline uses the three past days with the highest usage during the event hours). Moreover, this residual bias is primarily driven by a few large industrial customers whose load patterns are both volatile and of a high magnitude; for small- and medium-sized business, as well as large C&I customers with more predictable loads, the symmetric baseline predicts actual usage almost exactly. Figure 4-6 below illustrates this with the predicted and actual loads for one of the large retail chains participating in the program (which has a regular and predictable daily load shape) Figure 4-6: Average Weekday Real and Predicted Loads for One Retail Participant kw Event Period Average Weekday Usage Default Adjustment Baseline Symmetric Adjustment Baseline As previously mentioned, ADM Evaluation Staff examined the load shapes of the 30 premise account numbers with the highest baseline day usage to determine if the baseline with symmetric adjustment was appropriate. Of these 30 (totaling 90 event days), a separate analysis was conducted on 10 (totaling 22 event days) premise account numbers that had atypical usage and/or event response patterns. For these premise account numbers, a separate baseline development process was used. The figures below show the results of the baseline estimation effort, along with actual electric demand for these atypical participants on event days. Demand Response Programs 341

345 PSO Business Demand Response Program Figure 4-7: Event 1, (Atypical Participants) kw Event Period Default Adjustment Baseline Event Usage ADM Adjusted Baseline Demand Response Programs 342

346 PSO Business Demand Response Program Figure 4-8: Event 2, (Atypical Participants) kw Event Period Default Adjustment Baseline Event Usage ADM Adjusted Baseline 66 Event usage trails off sharply at hour 23. This is due to missing data during that hour on the event day. This missing data did not affect program impacts. Demand Response Programs 343

347 PSO Business Demand Response Program Figure 4-9: Event 3, (Atypical Participants) kw Event Period Default Adjustment Baseline Event Usage ADM Adjusted Baseline As shown in the figures above, the baseline ADM developed for atypical participants matched closely with the default adjustment used by PSO for all event days. This is due to most of the manual adjustments being relatively minor. Finally, a separate analysis was conducted for one customer whose demand reductions are around a quarter of the reported program impacts. This customer (Customer X) operates several facilities that interact with each other and the facilities electric demand is process driven. Five of the facilities operated by this customer participated in the Peak Performers Program in The observed loads for these five facilities during the month of the events and on event days are shown in Figure 4-10 and Figure 4-11 Demand Response Programs 344

348 PSO Business Demand Response Program Figure 4-10: Customer X Event Month Demand for Each Site Figure 4-11: Customer X Event Day Demand for Each Site Demand Response Programs 345

349 PSO Business Demand Response Program As the figures above show, the counterfactual event day usage is difficult to determine due to Customer X s unpredictability in day-to-day usage. ADM Evaluation Staff developed seven baseline methodologies for Customer X. For the first two events, only models without day of adjustment were considered as the participants were notified the day ahead of the events. For the third event, all seven baseline methodologies were considered. The model used for impact evaluation was then chosen using residual root mean squared error (RRMSE) to determine which of the models had the best fit during typical event hours (2-6 pm) of the program s five highest usage non-event days. These five days and times were chosen as they were most like event days making them optimal for testing models. Table 4-4 presents which baseline methodology was chosen for each site. Table 4-4: Baseline by Facility and Event Day Facility Event 1 and 2 Model Selected Event 3 Model Selected A 9-of-10 without Day of Adjustment 9-of-10 without Day of Adjustment B 9-of-10 without Day of Adjustment 9-of-10 without Day of Adjustment C 9-of-10 without Day of Adjustment 9-of-10 with Capped Day of Adjustment D 9-of-10 without Day of Adjustment 9-of-10 with Capped Day of Adjustment E 9-of-10 without Day of Adjustment 3-of-10 Uncapped Day of Adjustment Figure 4-12, Figure 4-13, and Figure 4-14 display aggregated usage and aggregated modeled kw for Customer X. Figure 4-12: Customer X Aggregated Load and ADM Model, Event Period Actual ADM Model Demand Response Programs 346

350 PSO Business Demand Response Program Figure 4-13 Customer X Aggregated Load and ADM Model, Event Period Actual ADM Model Figure 4-14 Customer X Aggregated Load and ADM Model, Event Period Actual ADM Model Energy Savings The Business Demand Response Program is designed primarily as a resource for procuring peak demand savings during periods of high demand. As such, the program does not report annual kwh savings. However, the program does generate energy impacts during the period of time during and surrounding called events. These impacts are not lasting, in the sense that kwh savings from a lighting retrofit might last the lifetime of the installed lighting fixtures. When a peak demand event is called usually hours to a full day before the actual event period participants have several options. They might decrease electric energy usage immediately in anticipation of the upcoming Demand Response Programs 347

351 PSO Business Demand Response Program event, or they might increase usage for the remaining pre-event hours in anticipation of future usage reduction. Additionally, the post-event hours are of interest because it may take several hours for facilities to restore electric energy usage to pre-event operation levels. Facilities might also increase electric energy usage immediately after the conclusion of an event to make up for previously reduced usage. In past evaluations, ADM evaluation staff calculated kwh savings for the ramp-in period (one hour before the event), during the event, and the snapback period (two hours following an event). For this evaluation, a survey was conducted to determine if load shifting was occurring amongst participants and if it was appropriate to expand the kwh calculation period from the ramp-in, event, and snapback hours to the whole event day. Respondent s answers did not overwhelming show that load shifting was occurring, but did suggest that the kwh calculation period should be expanded to the full event day. Of respondents, 92% indicated that they reduced AC usage during events with 31% of those saying they precooled during an event. This precooling likely extended beyond the one hour ramp in period pre-event. Evaluating the whole day for kwh savings would capture these impacts as well as any cooling that is completed after the event. In addition, 13% of participants who sent staff home early or reduced their industrial/manufacturing operations stated that they shifted their operations to another time. Again, an action of this nature is likely to occur before the one-hour ramp-in period as the participants are given notice of the event early in the day or the day before an event. Most these savings should be fully credited to the program because the usage is not increasing during other time periods. More detail on participant load reduction strategies discussed with participants during the survey is available in the process evaluation findings. ADM evaluators chose to use the full event day to evaluate kwh savings for Verified kwh savings presented below represent the net difference in energy consumption (between the estimated baseline and the observed usage) summed over the event days (72 total hours). It is possible that some facilites shifted event related load outside of the event day due to their reduction; however, given the post-event survey findings, investigating the entire event day appears to be the most accurate method. Impact Summary Peak demand reduction for the Business Demand Response program is defined as the amount of load reduced among all participants, averaged over the three DR event days (10 event hours). Table 4-5 summarizes the verified program impacts. Results for typical facilities, atypical facilities and Customer X are listed separately in the table because each category was analyzed separately for determining verified impacts. Demand Response Programs 348

352 PSO Business Demand Response Program Table 4-5: Program Impact Summary Customer Reported Peak kw Reduction Verified Average Peak kw Reduction Reported kwh Savings Verified kwh Savings Peak kw Realization Rate Customer X 12,202 9, ,929 80% Typical Facilities 37,038 31, ,006 85% Atypical Facilities 3,300 2, ,164 87% All Customers 52,541 44, ,240 84% Process Evaluation Findings The program design has remained consistent since 2013 The 2013 Annual Report contains a comprehensive discussion of the program design. Program Participation Location of Participant Facilities: Table 4-6 displays the number of participating premise accounts by district. Tulsa and Tulsa Northern had accounted for approximately 60% of the projects. Overall, the distribution of participating accounts was like that of the distribution of businesses across the PSO service territory districts. McAlester was the one district that was slightly under-represented. Table 4-6 Distribution of Premise Accounts Across Districts District Percent of Premise Accounts Percent of Business Establishments 1 Lawton 19% 22% McAlester 21% 11% Tulsa 42% 54% Tulsa Northern 18% 13% 1. Source: U.S. Census, County Business Patterns: 2014, ZIP Code Industry Detail File Figure 4-15 shows the growth in the number of program participants and the year-toyear changes in MW reductions that have been facilitated by the high retention rate and recruitment of new participants. Figure 4-15: Number of Premise Accounts and MW Reductions * Demand Response Programs 349

353 PSO Business Demand Response Program Number of Premise Accounts Program Year New Participant Survey Results An survey was administered to new program participants in October and November ADM identified new participants by comparing project numbers and company names listed in the 2016 program data with data from the program years. The survey was conducted to collect data on how participants learned of the program, motivations for participating, potential barriers to participation, and overall experience with the program. The survey was distributed to 43 new program participant contacts. Contacts were sent an initial invitation and two follow-up s. Table 4-7 summarizes the response to the survey. Table 4-7: New Participant Survey Response Response Metrics Number of Participant Contacts Number of Contact s 43 delivery Failures 1 Completions 18 Completion Rate 42% Table 4-8 summarizes the roles of survey respondents in their organization s participation in the program. Table 4-8: Survey Respondent Program Role Demand Response Programs 350

354 PSO Business Demand Response Program Role in Participation in Program Percent of Respondents (n= 18) Communicate to others in organization that an event will be called 67% Signed up for Peak Performers 50% Manage energy use during event 39% Informing others about Peak Performers 33% Other 6% Don't know 6% Enrollment Although program staff engaged in limited outreach during the 2016 program year, most new participants learned of the program from a PSO representative, a PSO , or the PSO website (Figure 4-16). Trade associations/business groups or friends/colleagues were other sources of program awareness. Figure 4-16: Initial Source of Program Awareness Participants in the Peak Saver opportunity primarily participated for financial motivations. Seventy-eight percent cited the opportunity to reduce energy bills as a motivation for participating and 39% cited the program incentive. Other motivations for participating included sustainability objectives and the lack of demand reduction targets. While the reported motivations point to multiple factors influencing the decision to participate in the program, none of these responses can be construed as indicating that Demand Response Programs 351

355 PSO Business Demand Response Program these participants would have made demand reductions on the event days without the program incentive or the notification of the event. Figure 4-17: Motivations for Participating in Peak Savers Most participants did not have any concerns about participating in the program, although one respondent did. This respondent was concerned about maintaining employee comfort during an event. Participation in Events Twenty-eight percent of respondents reported that they did not participate in some events. These respondents gave a variety of reasons for not participating: Too many customers/clients onsite during event (n = 2); Personal reasons (n = 1); Time constraint (n = 1); and And did not have the correct AMR meter initially (n = 1). These responses provide insight into the variety of factors that may affect an organization s participation in any single event. In general, the reasons given by the new participants did not suggest any design characteristics of the program that limited participants ability to participate in the events by reducing their load. None of the respondents reported that PSO contacted them when they missed an event. Demand Response Programs 352

356 PSO Business Demand Response Program Forty-four percent of respondents reported that the number of events called during the peak season was about what they were expecting. One respondent stated that the number of events was more than expected and that they expected two events (instead of the three events that were called). Thirty-nine percent of respondents expected more events than were called; on average these respondents expected that nine events would be called during the year. One of these respondents stated they expected 16 events even though the program caps the number of events called at 12. Overall, these responses indicated that the number of events called did not exceed most new participants expectations, though several reported that fewer events were called than expected. It is likely the case that most participants reported fewer than expected events because three events were called when up to 12 may be called during a season. Table 4-9: Expected Number of Events Number of Events as Compared to Expected Percent of Respondents (n= 18) Average Number of Expected Events (n= 7) More than expected 6% 2 Fewer than expected 39% 9 About the number expected 44% NA Don't know 11% NA Use of Vision System Participants in Peak Performers may view their usage in the vision system, although a large majority of new participants stated that they were not aware of this service. Additionally, neither of the two respondents that were aware of the system reported using it. Staff has previously noted that customers information needs are likely largely met through post-event s summarizing customers load reductions. Table 4-10: Awareness and Use of Vision System Awareness of Vision Awareness of Vision (n= 18) Use of Vision (n = 2) Yes 11% 0% No 89% 100% Don't know 0% 0% Demand Response Programs 353

357 PSO Business Demand Response Program Satisfaction Most participants (78%) reported that they were very likely to participate in the opportunity during One respondent indicated that they were very unlikely to participate. This respondent did not provide a reason for why they were unlikely to participate in 2017 and did not report any dissatisfaction with the program. Table 4-11: Likelihood of Future Participation Likelihood of future participation Percent of Respondents (n= 18) Very unlikely 6% Somewhat unlikely 0% Neither particularly unlikely nor likely 6% Somewhat likely 0% Very likely 78% Don't know 11% None of the new participants reported that they were dissatisfied with the program and more than 80% reported that they were very satisfied with it (Figure 4-18). Figure 4-18: Satisfaction with Peak Savers A few customers did report dissatisfaction with the event notification process, the availability of energy usage data, and the information initially received about the offer. These respondents were asked to elaborate on the reason(s) for their dissatisfaction. Demand Response Programs 354

358 PSO Business Demand Response Program One customer was dissatisfied with the notification process because he or she did not receive text notifications for all events. A second customer appeared to not be aware that their organization was eligible for participation in the program, despite program data that indicated that they participated in an event and reduced their load. Lastly, a third customer was interested in real-time data on energy usage. Survey respondents were asked to provide any additional suggestions for the program. The suggestions made included more communication throughout peak season and more notice of the events. Peak Event Survey Results An survey was administered to program participants shortly after the August 4 th, 2016 event. The survey was conducted to collect data about participant decisionmaking, preferences, and opinions of the Peak Performers program. The survey was distributed to 169 program participant contacts. Contacts were sent an initial invitation and one follow-up . Table 4-12 summarizes the survey response. Table 4-12: Peak Event Survey Response Response Metrics Number of Participant Contacts Number of Contact s 169 Delivery Failures 10 Completions 59 Completion Rate 35% Reduction Strategies Ninety-one percent of respondents stated that they acted to reduce their load during the August 4 th event, while 5% did not know if they did anything to reduce their usage. Table 4-13 displays the share of respondents reporting various actions to reduce load during the peak event. The action taken that was most commonly reported by participants was to adjust the temperature setting or to shut off the air conditioner. Ninety-two percent of respondents reported that they had taken this action. Several of the actions reported by survey respondents may have resulted in electricity savings in addition to load reductions during the event. Customers that turned off lights (70%) and closed window blinds (16%) likely saved electricity in addition to reducing load. Some actions taken by participants may have resulted in shifting of load from the peak event to another period. As such, the action would not have resulted in energy savings even if peak load was reduced. Respondents that reported these types of actions were Demand Response Programs 355

359 PSO Business Demand Response Program asked follow-up questions to determine if they shifted the load or if their actions may have resulted in energy savings. As summarized below and in Table 4-13, many of the respondents did not report behaviors consistent with load shifting. Thirty-one percent of respondents that adjusted the air conditioner temperature setting or turned off the air conditioner reported that they precooled the building. Ten percent of respondents that shut off non-industrial equipment stated that they increased use of that equipment at another period. Thirteen percent of respondents that reduced their industrial/manufacturing operations increased or extended operations at another time. Thirteen percent of respondents that closed early or sent staff home early reported that they extended business hours at another time. Demand Response Programs 356

360 PSO Business Demand Response Program Table 4-13: Actions Taken to Reduce Electricity Usage During Peak Event Peak Demand Reduction Action Percent of Respondents (n = 50) Load Shifting Follow-Up Question Load Shifting Response Adjusted air conditioner temperature setting / Turned off air conditioner 92% Did you cool the building to a lower temperature than usual in preparation for the peak event? Turned off lights 70% Shut off non-industrial equipment 42% Did you increase use of that non-industrial equipment at another time? Reduced or shut down industrial/manufacturing operations 30% Did you increase or extend manufacturing/industrial operations during another time to make up for the reduced operations or shutdown? Closed window blinds 26% Closed early / sent staff home early 16% Did you extend your business hours / ask staff to make up the hours at another time? Other 8% * Some respondents reported taking more than one action. Event Notifications and Intent to Participate in Future Events Ninety-six percent of respondents reported that they received sufficient notification of the event. Four percent said they did not get notice. One respondent reported that they did not receive notice. The customer s non-receipt of notice may have resulted from the notification problems that resulted from the use of a new notification system an issue that is known to staff and was resolved by the third event. The remaining participant would have preferred 1 to 2 days notice. The program provided notice of the event on the morning of the event for event 3 ( ) and the day before event 1 and 2 ( and ). Demand Response Programs 357

361 PSO Business Demand Response Program Ninety-eight percent of respondents reported that they intended to participate in future events called during the year. One respondent (2%) stated that he or she did not know if they would participate. Program Operations Assessment This section summarizes the findings from the review of the program design and operations. This review focused on program changes and developments during The material presented below was developed from interviews with program staff. The Peak Performers program design remained unchanged in 2016 from However, some administrative processes did change during the program year. The following sections summarize the key findings related to these changes. Energy Monitory Pilot On-Hold Staff reported that a planned pilot program that would provide energy-use monitors to customers was put on hold. AEP is in the process of formulating a company-wide approach to demand control and until that policy is set this project is on hold. Change in Procedures for Managing Program Data Prior to 2016, program data was supplied by Load Research to PSO s implementation contractor, who uploaded the data to the Vision system and reviewed it for completeness. This process changed in 2016 when PSO directly contracted with AEG to manage program data. Under the new process, AEG supplies a daily file of current program participants to PSO s Information Technology department who then provide the meter data for those customers back to AEG. An additional step was added to the process during the program year to solve an issue where historical data for newly enrolled customer accounts was not pulled, which prevented the calculation of load reductions. During this step, PSO staff marks the account with a flag indicating that it is newly enrolled. Accounts marked as new have historical data pulled and the account status automatically changes to active so that historical data is not continually pulled. The historical data is needed to estimate baseline hourly demand to calculate load reductions during the events. Overall, staff reported that the new data management process is streamlined and more highly automated than the data management process in place for the previous cycle. A data issue that was identified during the year was that for less than 1% of accounts, AMI communication errors occurred that resulted in meter data not being sent to AEG. These issues were due to both the stable factors that prevented the meter from transmitting data (e.g., the location of the meters) and transitory sources of interference (e.g., a parked vehicle blocking the data transmission). Staff indicated that screening of customer meters prior to enrollment was not feasible because the frequency of interval Demand Response Programs 358

362 PSO Business Demand Response Program data needed for the program is not collected for billing purposes. As such, the issues with meter communications can only be identified once meters are enrolled in the program. Customers Re-Enrolled for New Cycle Participants that were enrolled in Peak Performers during the cycle were required to re-enroll in 2016 to continue participation during the cycle. To minimize the paperwork burden on customers, program staff completed most of the application form for them based on information associated with the account numbers that the customer wanted to enroll. This assistance potentially mitigated a barrier to participation stemming from customers needing to retrieve information required for the application. Limited Outreach The program did not actively recruit new participants for the program during the program year. Staff limited outreach for two reasons. First, staff did not want to complicate the data management change process with also managing many new enrollments. Second, unlike prior years, the program did not provide funding for installation of AMIs for customers that were interested in participating but did not currently have that meter type. This funding was not provided because PSO was nearing its goal of completing the installation of AMIs, which has since been completed. Despite the limited outreach, staff reported that there were new participants in An enrollment trend that staff noted was that several new schools that participated in 2016 and that the increase in school s participation was driven by declining state funding for schools in Oklahoma. Changes in Notification Procedures Event notifications, which were previously sent by ICF, are now sent out through the AEP s corporate demand response notification system. Staff completed a test of the notification system in May that involved sending participants an that they would receive a test notification and then send out the test message. Despite performing the test, some customers indicated that they did not receive notification of the first two events. The issue was resolved by the third event. The program discontinued telephone notifications for most customers in The primary reason for this was that the current number of program participants was too large for the phone notification system to handle well some customers would receive notification calls after the event began. However, some customers without internet service at their work location still receive telephone notifications. Additionally, the program continues to provide and text notifications of events. Demand Response Programs 359

363 PSO Business Demand Response Program Peak Performers Conclusions and Recommendations The following conclusions were developed from the evaluation findings. Few Changes in Program Design or Operations. Overall, the program has remained relatively unchanged. The most significant change in program operations during 2016 was the change in procedures for managing program data. Under the new data management procedures, PSO is directly contracted with AEG to manage the data. Staff report that the process is streamlined and more highly automated than the process in place during the previous cycle. By the end of the peak season, the process appeared to be working well. In addition to the change in the data management process, the program also modified the notification procedures. One of the notification procedure changes was to utilize the AEP demand response notification system. While there were some initial challenges with ensuring that participants received these notifications, these were resolved by the third event of the year. The other change in notification procedures was to discontinue telephone notifications. Overall, the changes appeared to not have diminished the effectiveness of the notification process. 98% percent of the respondents to the survey following the second event reported that they received the notification and 96% reported that they received sufficient notification. Additionally, few new program participants reported dissatisfaction with the notification process; one of the new participant survey respondents was dissatisfied because they did not receive text message notifications. High Retention Continues. The beginning of the new cycle required reenrollment of customers that participated during Program staff provided a high level of assistance to customers to facilitate completion of the enrollment paperwork. This effort was successful as the number of accounts increased again with continued participation of past participants and the addition of new accounts. The Number of Participating Accounts Increased Substantially Despite Limited Outreach and Marketing. Staff reported that they engaged in limited outreach during the program year. Despite the limited outreach, the growth in the number of participating accounts continued increasing by 55% over the 2015 program year. Moreover, most of the new participants learned of the program through staff s efforts to promote the program either directly with potential participants or from program website. This growth bodes well for the program s capacity to manage peak load demands in the future. Not only may additional growth increase the total demand reductions achieved during the year, the growth diversifies the sources of demand reductions making the program less susceptible to impacts of stemming from individual customer decision making, economic conditions, or other factors. New Participants Satisfied with the Program. None of the program participants reported dissatisfaction with the program overall and more than 90% reported that they were satisfied with it. Two respondents reported dissatisfaction Demand Response Programs 360

364 PSO Business Demand Response Program with the availability of usage data and one noted that they would prefer real-time data on their usage. Providing this type of data is something the program is considering for a pilot project that would provide customers with energy monitors. The pilot program has not been launched because of a difficulty in finding monitors that will allow it to be cost effective. Ninety-one percent of respondents took actions to reduce peak demand during the second event of the season. This high number of customers that participated in the event suggest a high level of engagement in the program. Most of these participants (92%) reported that they cut back on their air conditioner usage. Shutting lights and non-industrial equipment off were other commonly taken actions. PSO fully deployed AMI meters in 2015 and the first half of These communicating interval meters for all customers improved the efficiency of enrolling customers and allowed customers to view their data in near real time with the My Energy Advisor web platform. Previously, if a customer didn t have an interval meter, which most did not, PSO had to install one Planned Program Changes Program outreach. Staff noted that the program will engage in outreach again in New marketing materials may be developed as part of this in collaboration with the firm that PSO is contracted with to provide program marketing support. Demand Response Programs 361

365 Appendix A. Glossary Cash Inducement Costs: Refers to customer and service provider rebate/incentive costs incurred by PSO in the implementation of a program. Coincidence Factor (CF): For energy efficiency measures, the CF represents the fraction of connected load reduction that occurs during the peak demand period. Deemed Savings: A savings estimate for relatively homogeneous measures. Generally, an assumed average savings across many rebated units is applied to each individual unit installed. Effective Useful Life (EUL): The number of years (or hours) that an energy-efficient technology is estimated to function. Also, referred to as measure life. EM&V Administrative Costs: EM&V administrative costs include all costs associated with evaluation, measurement and verification of reported energy and demand impacts resulting from the implementation of a program. Ex Ante: Refers to estimates of energy savings and peak demand reduction developed before program evaluation. Equivalent to reported impacts. Ex Post: Refers to estimates of energy savings and peak demand reductions developed from program evaluation. Equivalent to verified impacts. Free-ridership: Percentage of participants who would have implemented the same energy-efficiency measures in a similar timeframe even in the absence of the program. Gross Impacts: Changes in energy consumption/demand that result directly from program-promoted actions regardless of the extent or nature of program influence on these actions. Impact Evaluation: Impact evaluation is the verification and estimation of gross and net impacts resulting from the implementation of one or more energy-efficiency or demand response programs. Measure: An energy-efficiency measure refers to any action taken to increase energy efficiency, whether through changes in equipment, control strategies, or behavior. Net Savings: The portion of gross savings that is directly attributable to the actions of an energy-efficiency or demand response program. Net-to-Gross Ratio (NTGR): A factor representing net program savings divided by gross program savings that is applied to gross program impacts to convert them into net program impacts. Generally calculated as 1 (free-ridership %) + (Spillover %). Non-Cash Inducement Costs: Non-cash inducement costs include third party implementation costs and advertising costs incurred by PSO in the implementation of a program. PSO earns no incentives on advertising costs. Appendix A: Glossary 362

366 PSO Glossary Non-Energy Benefits: Non-energy benefits refer to any benefits PSO customers may experience due to their participation in PSO programs beyond energy savings. Examples include improved comfort, aesthetic enhancements, better indoor air quality, improved security, better employee productivity, etc. Non-EM&V Administrative Costs: Non-EM&V administrative costs include PSO staff labor costs and overhead costs associated with implementing a program. Oklahoma Deemed Savings Documents (OKDSD): Refers to the Oklahoma Deemed Savings, Installation & Efficiency Standards and associated work papers for small commercial and residential energy efficiency measures. These documents were originally submitted to the OCC as part of Cause # PUD and approved for use as part of Order # In 2013, the documents were update to reflect more recent and applicable baseline conditions. Participant Cost Test (PCT): The PCT examines the cost and benefits from the perspective of the customer installing the energy efficiency measure. Costs include incremental costs of purchasing and installing the efficient equipment, above the cost of standard equipment. Benefits include customer bill savings, incentives received from the utility, and any applicable tax credits. Peak Demand: For the purposes of this report peak demand refers to the average metered demand during the peak period, defined as 2 to 9 PM during the summer months, June through September, excluding weekends and holidays. Note that for the Business Demand Response program, peak demand reduction is calculated as the average reduction during event hours. Process Evaluation: A systematic assessment of an energy efficiency program for documenting program operations at the time of examination and identifying potential improvements that can be made to increase the programs efficacy or effectiveness. Projected, Reported, and Verified Savings: Projected impacts refer to the energy savings and peak demand reduction forecasts submitted to the OCC as part of PSO s initial portfolio filing on July 1, Reported impacts refer to energy savings and peak demand reduction estimates based on actual program participation in PY2016, before program evaluation activities. Finally, verified impacts refer to energy savings and demand reduction estimates for PY2016 developed through independent program evaluation, measurement, and verification (EM&V). Ratepayer Impact Measure (RIM): The RIM examines the impact of energy efficiency programs on utility rates. Reduced energy sales can lower revenues and put upward pressure on retail rates as the remaining fixed costs are spread over fewer kwh. Costs include overhead and incentive payments and the cost of lost revenue due to reduced 67 Cause No. PUD , Direct Testimony of Eric Raines. Appendix A: Glossary 363

367 PSO Glossary sales. Benefits include cost savings associated with not delivering energy to customers. These avoided costs include generation, transmission, and distribution costs. Realization Rate: The ratio of verified (ex post) impacts to reported (ex ante) impacts. Societal Cost Test (SCT): The SCT includes the same costs and benefits as the TRC, but uses a lower discount rate to reflect the overall benefit to society over the long term. Spillover: Energy and/or demand savings caused by a program, but for which the utility did not have to provide cash inducements. Total Resource Cost Test (TRC): The TRC measures the net benefits of the energy efficiency program for the region as a whole. Costs included in the TRC are incremental costs of purchasing and installing the efficient equipment, above the cost of standard equipment and overhead cost associated with implementing the program. Benefits include cost savings associated with not delivering energy to customers. These avoided costs include generation, transmission, and distribution costs. Utility Cost Test (UCT): The UCT examines the costs and benefits of the energy efficiency program from the perspective of the utility company. Costs include overhead (administration, marketing, EM&V) and incentive costs. Benefits include cost savings associated with not delivering energy to customers. These avoided costs include generation, transmission, and distribution costs. This test is also often referred to as the Program Administrator Cost Test (PACT). Appendix A: Glossary 364

368 Appendix B. Portfolio Cost-Effectiveness This appendix provides an overview of each programs participation, verified reduction in peak load, verified kwh savings, annual admin costs, total program costs, as well as a summary of the cost effectiveness analysis. 5.1 Cost Effectiveness Summary This appendix covers all verified electricity and peak demand savings, and associated program costs incurred in the implementation of PSO s 2016 energy efficiency and demand response portfolio from January 1, 2016 through December 31, The cost-effectiveness of PSO s 2016 programs was calculated based on reported total spending, verified energy savings, and verified demand reduction for each of the energy efficiency and demand response programs. All spending estimates were provided by PSO. The methods used to calculate cost-effectiveness are informed by the California Standard Practice Manual. 68 The demand reduction (kw) and energy savings (kwh) presented throughout this appendix represent net savings at the generator by applying program level net-to-gross (NTG) ratios and adjusting for line losses. Program level NTG ratios for the 2016 programs were estimated by ADM as part of the portfolio impact evaluation. Verified savings estimates at the meter were adjusted to account for line losses using a line loss adjustment factor of For gas savings estimates, a gas loss factor was included. To calculate the cost-effectiveness of each program, measure lives were assigned on a measure-by-measure basis. When available, measure life values came from the Oklahoma Deemed Savings Documents (OKDSD). When not available in the OKDSD, measure life values came from the Arkansas TRM. 69 Additionally, assumptions regarding incremental/full measure costs were necessary. Often, these costs were taken directly from the VisionDSM portfolio tracking database (PSO s program tracking database developed by Applied Energy Group, Inc.) or project specific invoices. When not available, ADM relied on PSO s 2009 Energy Efficiency Plan, 2013 Annual Report, and ADM estimates based on relevant industry sources. Avoided energy, capacity, transmission/distribution, and CO2 costs used to calculate cost-effectiveness were provided by PSO and are found in Section B.4 of this appendix. 68 California Standard Practice Manuel: Economic Analysis of Demand Side Management Programs, October Available at: CE56ADF8DADC/0/CPUC_STANDARD_PRACTICE_MANUAL.pdf 69 Appendix B: Portfolio Cost Effectiveness 365

369 Residential and commercial rates used to estimate certain cost-effectiveness tests were also provided by PSO. Table B-1 lists each program included in this analysis, along with the projected savings estimates and projected budget. Impacts show in Table B-1 are net-at-generator, reflecting the NTG projections and line losses. Table B-2 lists each program included in this analysis, along with the final verified savings estimates, total expenditures, Utility Cost Test (UCT) 70 results, and Total Resource Cost Test (TRC) results. Impacts shown in Table B-2 are net-at-generator, reflecting NTG assumptions and line losses as described above. Results from the UCT and TRC are focused on in this summary for the following reasons: The UCT results are a direct input to the shared savings component of the Demand Side Management Cost Recovery Rider (DSM Rider) as described in Oklahoma Corporate Commission PUD Oklahoma Administrative Code (OAC) 165: lists the goals of energy efficiency and demand response programs as (1) minimize the long-term cost of utility service, and (2) avoid or delay the need for new generation, transmission, and distribution investment. The TRC test best reflects these goals, as it looks at benefits and costs from the perspective of all utility customers in the utility s service territory (participants and non-participants). In addition to UCT and TRC results, results from the Ratepayer Impact Measure (RIM), Participant Cost Test (PCT) and Societal Cost Test (SCT) are included in the body of this appendix. Based on verified program impacts and spending during PY2016, PSO s overall portfolio is cost-effective based on both the UCT and TRC. 70 The UCT is also referred to as the Program Administrator Cost Test (PACT). 71 Cause No. PUD , Direct Testimony of Earlyne Reynolds. Appendix B: Portfolio Cost Effectiveness 366

370 Table B-1: Projections by Program, 2016 (Impacts are Net, at Generator) Projected Projected Program Peak Annual Annual Gas Total Program Demand Energy Savings Expenditures Reduction Savings (Therms) (kw) (kwh) High Performance Business 7,479 44,684, ,393 $10,711,689 Home Weatherization 1,280 4,041, ,752 $3,876,187 Energy Saving Products 2,670 23,065, ,100 $4,154,950 High Performance Homes 3,479 6,533, ,082 $8,746,440 Education 597 4,216,905-25,392 $2,127,000 Total EE Programs 15,504 82,541, ,949 $29,616,266 Business Demand Response 48, ,968 0 $3,116,200 Total DR Programs 48, ,968 0 $3,116,200 Total - Overall Portfolio 72 64, ,786, ,949 $32,894,466 Table B-2: Cost-Effectiveness by Program, 2016 (Impacts are Net, at Generator) Program Verified Verified Annual Peak Annual TRC UCT Gas Total Program Demand Energy (b/c (b/c Savings Expenditures Reduction Savings ratio) ratio) (Therms) (kw) (kwh) High Performance Business 11,111 54,125,859 0 $10,009, Home Weatherization 1,602 5,875, ,737 $3,805, Energy Saving Products 5,801 34,835,658 0 $3,894, High Performance Homes 2,814 7,120, ,345 $8,617, Education 1,466 7,954,224 0 $1,782, Total EE Programs 22, ,912, ,082 $28,110, Business Demand Response 46, ,373 0 $2,285, Total DR Programs 46, ,373 0 $2,285, Total - Overall Portfolio 73 69, ,817, ,082 $30,425, Includes Behavioral Modification budget of $162, Includes Behavioral Modification expenditure of $29,418. Appendix B: Portfolio Cost Effectiveness 367

371 5.2 Energy Efficiency Programs PSO s energy efficiency portfolio in 2016 consisted of five programs with a verified net peak demand reduction of 22,794 kw and verified net annual energy savings of 109,912,220 kwh (including line-loss estimates of 6.11%). Total spending in 2016 equaled $28,110,618. Table B-3 provides a summary of program participation and verified net impacts for each of the energy efficiency programs. Table B-4 provides a summary of program costs in Table B-3: Energy Efficiency Programs Verified Impacts (Net, at Generator) Program Number of Participants in 2016 Verified Peak Demand Reduction (kw) Verified Annual Energy Savings (kwh) Gas Savings (Therms) High Performance Business ,111 54,125,859 0 Home Weatherization 2,157 1,602 5,875, ,737 Energy Saving Products 74 1,638,440 5,801 34,835,658 0 High Performance Homes 75 5,542 2,814 7,120, ,345 Education 28,638 1,466 7,954,224 0 Total EE Programs 1,675,727 22, ,912, , The Energy Saving Products consists of the number of upstream CFL/LED bulbs and appliances discounted. For the downstream portion of the program, determining the number of participants is straight forward. For the upstream bulb discounts, the number of bulb packages sold is listed instead of number of participants. 75 The number of participants for High Performance Homes reflects to total number of customers and the total number of homes in the New Homes portion of the program. Appendix B: Portfolio Cost Effectiveness 368

372 Table B-4: Energy Efficiency Programs Reported Costs Program Annual Annual Annual Non- Annual Cash Non-EM&V EM&V Cash Marketing Inducement Admin Admin Inducement Costs ($) Costs ($) 77 Costs ($) 76 Costs ($) Costs ($) 78 High Performance Business $350,358 $418,877 $5,811,817 $2,921,282 $507,582 Home Weatherization $110,124 $126,741 $3,406,690 $52,907 $109,302 Energy Saving Products $114,788 $181,265 $2,608,766 $853,545 $136,599 High Performance Homes $185,254 $241,027 $5,460,582 $2,446,851 $284,045 Education $71,655 $26,425 $1,651,321 $26,027 $6,788 Total EE Programs $832,179 $994,335 $18,939,177 $6,300,612 $1,044,315 In the tables that follow, total costs and benefits, and cost-effectiveness test results are provided for each energy efficiency program in the PY2016 portfolio. Metric High Performance Business Program Table B-5:: High Performance Business Benefit/Cost Tests Utility Cost Test Total Resource Cost Test Ratepayer Impact Measure Societal Cost Test Participant Cost Test Benefit/Cost Ratio Net Benefits ($000s) 31, , (10,312.32) 29, , Total Benefits ($000s) 41, , , , , Total Costs ($000s) 10, , , , , Non-EM&V Admin Costs include PSO staff labor costs and overhead costs. 77 Cash inducement costs refer to customer rebate costs. 78 Non-cash inducement costs include third party implementation costs. Appendix B: Portfolio Cost Effectiveness 369

373 5.2.2 Home Weatherization Program Table B-6: Home Weatherization Benefit/Cost Tests Metric Total Ratepayer Utility Cost Societal Participant Resource Impact Test Cost test Cost Test Cost Test Measure Benefit/Cost Ratio Net Benefits ($000s) 2, , (2,929.76) 6, , Total Benefits ($000s) 6, , , , , Total Costs ($000s) 3, , , , Energy Saving Products and Services Program Table B-7: Energy Saving Products and Services Benefit/Cost Tests Metric Total Ratepayer Utility Cost Societal Participant Resource Impact Test Cost test Cost Test Cost Test Measure Benefit/Cost Ratio Net Benefits ($000s) 13, , (16,610.84) 19, , Total Benefits ($000s) 17, , , , , Total Costs ($000s) 3, , , , High Performance Homes Program Table B-8: High Performance Homes Benefit/Cost Test Metric Total Ratepayer Utility Cost Societal Participant Resource Impact Test Cost test Cost Test Cost Test Measure Benefit/Cost Ratio Net Benefits ($000s) 1, , (5,650.44) 7, , Total Benefits ($000s) 10, , , , , Total Costs ($000s) 8, , , , , Appendix B: Portfolio Cost Effectiveness 370

374 5.2.5 Education Program Table B-9: Education Benefit/Cost Test Metric Total Ratepayer Utility Cost Societal Participant Resource Impact Test Cost test Cost Test Cost Test Measure Benefit/Cost Ratio >1 Net Benefits ($000s) 4, , (2,025.61) 6, , Total Benefits ($000s) 6, , , , , Total Costs ($000s) 1, , , , Demand Response Programs PSO s demand response portfolio in 2016 consisted of one demand response program with a verified net peak demand reduction of MW and verified net energy savings of MWh. 79 Total spending in 2016 equaled $2,285,657. Table B-10 provides a summary of program participation and verified net impacts for the 2016 demand response portfolio. Table B-11 provides a summary of 2016 program costs. Table B-10: Demand Response Programs Verified Impacts (Net, at Generator) Number of Verified Peak Verified Annual Gas Program Participants in Demand Energy Savings 2016 Reduction (kw) Savings (kwh) (Therms) Business Demand Response , ,373 0 Total DR Programs , ,373 - Program Table B-11: Demand Response Programs Reported Costs Annual Non-EM&V Admin Costs ($) Annual EM&V Admin Costs ($) Annual Cash Inducement Costs ($) Annual Non-Cash Inducement Costs ($) Marketing Costs ($) Business Demand Response $87, $76, $1,771, $306, $44, Total DR Programs $87, $76, $1,771, $306, $44, In the table that follows, total costs and benefits, and full cost-effectiveness test results are provided for the Business Demand Response program. 79 The verified peak demand reduction shown here for the Business Demand Response program includes an adjustment for line-losses (6.11%). Appendix B: Portfolio Cost Effectiveness 371

375 5.3.1 Business Demand Response Program Table B-12: Business Demand Response Benefit/Cost Test Metric Total Ratepayer Utility Cost Societal Participant Resource Impact Test Cost test Cost Test Cost Test Measure Benefit/Cost Ratio Net Benefits ($000s) 4, , , , , Total Benefits ($000s) 7, , , , , Total Costs ($000s) 2, , Avoided Costs The avoided costs in the table below were developed for energy, capacity, T&D, and CO2 during the portfolio design process (PUD ). Appendix B: Portfolio Cost Effectiveness 372

376 Table B-13: Avoided Costs from PSO Portfolio Plan Year SPP - T&D SPP Capacity Energy Costs $/MWh $/MW-day $/kw-yr $/kw-yr CO2 ($/metric tonne) Natural Gas ($/Mcf) 2016 $43.66 $ $ $17.24 $0.00 $ $46.36 $ $ $17.53 $0.00 $ $47.64 $ $ $17.80 $0.00 $ $49.17 $ $ $18.08 $0.00 $ $51.25 $ $ $18.35 $0.00 $ $54.12 $ $ $18.63 $1.26 $ $62.48 $ $ $18.91 $15.10 $ $64.07 $ $ $19.19 $15.29 $ $66.39 $ $ $19.48 $15.49 $ $68.59 $ $ $19.77 $15.69 $ $69.91 $ $ $20.07 $15.90 $ $72.18 $ $ $20.37 $16.10 $ $74.00 $ $ $20.67 $16.31 $ $75.92 $ $ $20.98 $16.52 $ $78.07 $ $ $21.30 $16.74 $ $80.38 $ $ $21.53 $16.96 $ $83.77 $ $ $21.82 $17.18 $ $85.54 $ $ $22.11 $17.40 $ $81.01 $ $ $22.39 $17.62 $ $83.93 $ $ $22.68 $17.86 $ $85.22 $ $ $22.97 $18.09 $ $86.54 $ $ $23.26 $18.33 $ $87.89 $ $ $23.55 $18.58 $ $89.26 $ $ $23.83 $18.83 $ $90.66 $ $ $24.12 $19.08 $ $92.09 $ $ $24.41 $19.33 $ $93.56 $ $ $24.70 $19.59 $ $95.05 $ $ $24.99 $19.83 $8.62 Appendix B: Portfolio Cost Effectiveness 373

377 Appendix C. Identification of Program Implementers Table C-1 identifies program implementation contractors, their associated contact information, and the 2016 programs they were involved in. Table C-1: Program Implementer Identification Program(s) Implementat ion Contractor Contact Contact Title Contact Address Contact Phone Contact Business Demand Response PSO Jeff Brown Consumer Programs Manager 212 E. 6th St. Tulsa, OK Education Resource Action Programs Mike Gross Director of Program Management 976 United Circle, Sparks, NV com Energy Saving Products CLEAResult Karen Miller Program Manager 146 Chestnut Street, Springfield, MA om High Performance Business, High Performance Homes ICF International ICF International Michaela Martin Dale Hoenshell Vice President Energy Efficiency Principal 126 Center Park Lane, Oak Ridge, TN North Dallas Parkway, Plano, TX Home Weatherization Titan ES, LLC Rebuilding Tulsa Together Bradley Cockings Jennifer Barcus - Schafer President Chief Executive Officer 9700 S. Pole Road, Tulsa, OK P.O. Box 52201, Tulsa, OK bcockings@titanes.us jennifer.barcus_shafer@r ebuildingtogethertulsa Program Marketing Services VI Marketing and Branding Judi Startzman Vice President of Strategic Marketing 125 Park Avenue, Suite 200, Oklahoma City, OK jstartzman@thevibrand.c om Appendix C: Identification of Program Implementers 374

378 Appendix D. Training and Customer Outreach During 2016, PSO conducted several service provider recruitment and training events. Additionally, PSO sponsored various customer outreach events and stakeholder presentations. Table D-1 summarizes the in-store retail lighting promotional events; Table D-2 summarizes service provider recruitment and training events, customer outreach events, and other non-lighting promotion events throughout PY2016. Table D-1: Summary of In-Store Retail Lighting Promotional Events Date Event Name Location Sponsored By 01/09/2016 Lighting Event Lawton PSO/CLEAResult 01/23/2016 Lighting Event Hobart PSO/CLEAResult 01/24/2016 Lighting Event Tulsa PSO/CLEAResult 01/30/2016 Lighting Event Tulsa PSO/CLEAResult 02/13/2016 Lighting Event Lawton PSO/CLEAResult 02/20/2016 Lighting Event Tulsa PSO/CLEAResult 02/21/2016 Lighting Event Tulsa PSO/CLEAResult 02/27/2016 Lighting Event Duncan PSO/CLEAResult 02/27/2016 Lighting Event Owasso PSO/CLEAResult 03/11/2016 Lighting Event Lawton PSO/CLEAResult 03/12/2016 Lighting Event Elk City PSO/CLEAResult 03/12/2016 Lighting Event Tulsa PSO/CLEAResult 03/26/2016 Lighting Event Hobart PSO/CLEAResult 03/26/2016 Lighting Event Broken Arrow PSO/CLEAResult 04/04/2016 Lighting Event Chickasha PSO/CLEAResult 04/09/2016 Lighting Event Lawton PSO/CLEAResult 04/09/2016 Lighting Event Tulsa PSO/CLEAResult 04/09/2016 Lighting Event Tulsa PSO/CLEAResult 04/10/2016 Lighting Event Tulsa PSO/CLEAResult 04/20/2016 Lighting Event Tulsa PSO/CLEAResult 04/21/2016 Lighting Event Tulsa PSO/CLEAResult 04/23/2016 Lighting Event Owasso PSO/CLEAResult 04/29/2016 Lighting Event Lawton PSO/CLEAResult 04/30/2016 Lighting Event Hobart PSO/CLEAResult 05/14/2016 Lighting Event Lawton PSO/CLEAResult 05/21/2016 Lighting Event Lawton PSO/CLEAResult 05/28/2016 Lighting Event Tulsa PSO/CLEAResult 05/29/2016 Lighting Event Tulsa PSO/CLEAResult 06/04/2016 Lighting Event Elk City PSO/CLEAResult 06/17/2016 Lighting Event Lawton PSO/CLEAResult 06/18/2016 Lighting Event Lawton PSO/CLEAResult 06/18/2016 Lighting Event Tulsa PSO/CLEAResult 06/19/2016 Lighting Event Tulsa PSO/CLEAResult Appendix D: Training and Customer Outreach 375

379 PSO Training and Customer Outreach Date Event Name Location Sponsored By 06/24/2016 Lighting Event Lawton PSO/CLEAResult 07/01/2016 Lighting Event Hobart PSO/CLEAResult 07/09/2016 Lighting Event Tulsa PSO/CLEAResult 07/15/2016 Lighting Event Lawton PSO/CLEAResult 07/23/2016 Lighting Event Tulsa PSO/CLEAResult 07/23/2016 Lighting Event Hobart PSO/CLEAResult 07/24/2016 Lighting Event Tulsa PSO/CLEAResult 08/03/2016 Lighting Event Tulsa PSO/CLEAResult 08/13/2016 Lighting Event Lawton PSO/CLEAResult 08/20/2016 Lighting Event Lawton PSO/CLEAResult 08/20/2016 Lighting Event Tulsa PSO/CLEAResult 08/21/2016 Lighting Event Tulsa PSO/CLEAResult 08/26/2016 Lighting Event Hobart PSO/CLEAResult 09/17/2016 Lighting Event Tulsa PSO/CLEAResult 09/17/2016 Lighting Event Elk City PSO/CLEAResult 09/18/2016 Lighting Event Tulsa PSO/CLEAResult 09/24/2016 Lighting Event Lawton PSO/CLEAResult 10/05/2016 Lighting Event Lawton PSO/CLEAResult 10/08/2016 Lighting Event McAlester PSO/CLEAResult 10/09/2016 Lighting Event McAlester PSO/CLEAResult 10/11/2016 Lighting Event Tulsa PSO/CLEAResult 10/15/2016 Lighting Event Bartlesville PSO/CLEAResult 10/16/2016 Lighting Event Bartlesville PSO/CLEAResult 10/26/2016 Lighting Event Lawton PSO/CLEAResult 10/29/2016 Lighting Event Vinita PSO/CLEAResult 11/11/2016 Lighting Event Hobart PSO/CLEAResult 11/12/2016 Lighting Event Hobart PSO/CLEAResult 11/17/2016 Lighting Program Promotion Tulsa PSO/CLEAResult 11/19/2016 Lighting Event Elk City PSO/CLEAResult 11/19/2016 Lighting Event Tulsa PSO/CLEAResult 11/20/2016 Lighting Event Tulsa PSO/CLEAResult 11/21/2016 Lighting Event Tulsa PSO/CLEAResult 12/01/2016 Lighting Event Lawton PSO/CLEAResult 12/02/2016 Lighting Event Coweta PSO/CLEAResult 12/06/2016 Lighting Event Tulsa PSO/CLEAResult 12/07/2016 Lighting Event Tulsa PSO/CLEAResult 12/10/2016 Lighting Event Tulsa PSO/CLEAResult 12/10/2016 Lighting Event Lawton PSO/CLEAResult 12/11/2016 Lighting Event Tulsa PSO/CLEAResult 12/17/2016 Lighting Event Elk City PSO/CLEAResult 12/30/2016 Lighting Event Lawton PSO/CLEAResult Appendix D: Training and Customer Outreach 376

380 PSO Training and Customer Outreach Table D-2: Service Provider Recruitment & Training Events, Customer Outreach Events, and Other Non-Lighting Promotional Events Date Event Name Location Sponsored By # of Attendees Estimated 01/13/2016 HPH Program Updates Tulsa PSO/ICF /15/2016 HPH Service Provider Training Tulsa PSO/ICF /20/2016 PowerForward Overview McAlester PSO /21/2016 PowerForward Overview Tulsa PSO /25/2016 HPH Service Provider Training Tulsa ICF /26/2016 HPH Service Provider Training Tulsa ICF /26/2016 HPH Service Provider Training Tulsa ICF /04/2016 HPH Service Provider Training Tulsa ICF /09/2016 HPH Service Provider Training Tulsa ICF /09/2016 Trade Ally Training Tulsa ICF /10/2016 PowerForward Overview Delaware PSO /10/2016 HPH Service Provider Training Tulsa ICF /10/2016 Trade Ally Training McAlester ICF /11/2016 Trade Ally Training Oklahoma City ICF /18/2016 HPH Service Provider Training Sand Springs ICF /18/2016 HPH Service Provider Training Tulsa ICF /25/2016 HPH Service Provider Training Tulsa PSO/ICF /26/2016 HPH Service Provider Training Bixby ICF /26/2016 HPH Service Provider Training Broken Arrow ICF /29/2016 HPH Service Provider Training Cache ICF /02/2016 HPH Service Provider Training Tulsa ICF /02/2016 PowerForward Overview Jenks ICF /03/2016 PowerForward Overview Jay PSO /09/2016 PowerForward Overview Oologah PSO /10/2016 PowerForward Overview Grove PSO /10/2016 HPH Service Provider Training Tulsa ICF /11/2016 HPH Service Provider Training Tulsa ICF /22/2016 HPH Service Provider Training Tulsa ICF /22/2016 HPH Service Provider Training Tulsa ICF /23/2016 PowerForward Overview Tulsa PSO /23/2016 HPH Service Provider Training Lawton ICF /23/2016 PowerForward Overview Tulsa PSO /24/2016 PowerForward Overview Tulsa PSO /31/2016 HPH Service Provider Training Sperry ICF /05/2016 HPH Service Provider Training Tulsa ICF /05/2016 HPH Service Provider Training Broken Arrow ICF /05/2016 HPH Service Provider Training Jenks ICF /06/2016 PowerForward Overview Vinita PSO Appendix D: Training and Customer Outreach 377

381 PSO Training and Customer Outreach Date Event Name Location Sponsored By # of Attendees Estimated 04/06/2016 PowerForward Overview Vinita PSO /07/2016 PowerForward Overview Tulsa PSO /07/2016 PowerForward Overview Big Cabin PSO /08/2016 PowerForward Overview Chouteau PSO /11/2016 HPH Service Provider Training OKC ICF /12/2016 PowerForward Overview Lenapah PSO /14/2016 HPH Program Updates Tulsa ICF /16/2016 HPH Service Provider Training Clinton ICF /20/2016 HPB Service Provider Training Hugo ICF /20/2016 PowerForward Overview Tulsa PSO /25/2016 HPH Service Provider Training Chickasha ICF /05/2016 HPB Service Provider Training Bartlesville ICF /05/2016 PowerForward Overview Lawton ICF /05/2016 PowerForward Overview Lawton PSO /07/2016 PowerForward Overview Owasso PSO/ICF /12/2016 HPB Service Provider Training Oklahoma City ICF /18/2016 HPB Service Provider Training Tulsa PSO/ICF /02/2016 HPH Program Updates Broken Arrow PSO /28/2016 PowerForward Overview Tulsa PSO /12/2016 HPH Service Provider Training Tulsa ICF /15/2016 PowerForward Overview Jenks PSO /20/2016 PowerForward Overview McAlester PSO /21/2016 PowerForward Overview Tulsa PSO /26/2016 PowerForward Overview Tulsa PSO /27/2016 HPH Builder/Rater Training Tulsa PSO/ICF /27/2016 HPH Builder/Rater Training Tulsa ICF /02/2016 PowerForward Overview Inola PSO /04/2016 Efficiency Outreach Community Outreach Event Chouteau PSO /09/2016 PowerForward Overview McAlester PSO /09/2016 HPH Service Provider Training Sperry ICF /11/2016 PowerForward Overview Tulsa PSO /23/2016 HPB Service Provider Training Tulsa PSO/ICF /26/2016 PowerForward Overview Bartlesville PSO /29/2016 HPH Service Provider Training Henryetta ICF /29/2016 PowerForward Overview Vinita PSO /29/2016 HPH Service Provider Training Grove ICF /01/2016 PowerForward Overview Lawton PSO /08/2016 PowerForward Overview McAlester PSO /20/2016 HPH Service Provider Training Tulsa ICF /21/2016 PowerForward Overview McAlester PSO Appendix D: Training and Customer Outreach 378

382 PSO Training and Customer Outreach Date Event Name Location Sponsored By # of Attendees Estimated 09/22/2016 PowerForward Overview Henryetta PSO /23/2016 PowerForward Overview Grove PSO /27/2016 HPH Service Provider Training Tulsa ICF /03/2016 PowerForward Overview Tulsa PSO /04/2016 PowerForward Overview Afton PSO /10/2016 HPB SBES Training Tulsa PSO /11/2016 PowerForward Overview Barnsdall PSO /13/2016 PowerForward Overview Sand Springs PSO /13/2016 PowerForward Overview Tulsa PSO /18/2016 PowerForward Overview Tulsa PSO /21/2016 PowerForward Overview Tulsa PSO /22/2016 PowerForward Overview Tulsa PSO /25/2016 PowerForward Overview Ft. Towson PSO /01/2016 PowerForward Overview Tulsa PSO /02/2016 PowerForward Overview Chickasha PSO /08/2016 PowerForward Overview Krebs PSO /08/2016 PowerForward Overview Bartlesville PSO /10/2016 PowerForward Overview Lawton PSO /16/2016 PowerForward Overview Nowata PSO /01/2016 PowerForward Overview Muskogee PSO /06/2016 HPH Program Updates Tulsa PSO /08/2016 PowerForward Overview Bartlesville PSO /12/2016 PowerForward Overview Tulsa PSO /19/2016 PowerForward Overview Tulsa PSO Appendix D: Training and Customer Outreach 379

383 Appendix E. Marketing Synopsis The following pages of this appendix provide examples of marketing materials used to promote PSO s Demand Side Management portfolio in Appendix E: Marketing Synopsis 380

384 PSO Marketing Synopsis Appendix E: Marketing Synopsis 381

385 PSO Marketing Synopsis Appendix E: Marketing Synopsis 382

386 PSO Marketing Synopsis Appendix E: Marketing Synopsis 383

387 PSO Marketing Synopsis Appendix E: Marketing Synopsis 384

388 PSO Marketing Synopsis Appendix E: Marketing Synopsis 385

389 PSO Marketing Synopsis Appendix E: Marketing Synopsis 386

390 PSO Marketing Synopsis Appendix E: Marketing Synopsis 387

391 PSO Marketing Synopsis Appendix E: Marketing Synopsis 388

392 PSO Marketing Synopsis Appendix E: Marketing Synopsis 389

393 PSO Marketing Synopsis Appendix E: Marketing Synopsis 390

394 PSO Marketing Synopsis Appendix E: Marketing Synopsis 391

395 PSO Marketing Synopsis Appendix E: Marketing Synopsis 392

396 PSO Marketing Synopsis Appendix E: Marketing Synopsis 393

397 PSO Marketing Synopsis Appendix E: Marketing Synopsis 394

398 PSO Marketing Synopsis Appendix E: Marketing Synopsis 395

399 PSO Marketing Synopsis Appendix E: Marketing Synopsis 396

400 PSO Marketing Synopsis Appendix E: Marketing Synopsis 397

401 PSO Marketing Synopsis Appendix E: Marketing Synopsis 398

402 PSO Marketing Synopsis Appendix E: Marketing Synopsis 399

403 PSO Marketing Synopsis Appendix E: Marketing Synopsis 400

404 PSO Marketing Synopsis Appendix E: Marketing Synopsis 401

405 PSO Marketing Synopsis Appendix E: Marketing Synopsis 402

406 PSO Marketing Synopsis Appendix E: Marketing Synopsis 403

407 PSO Marketing Synopsis Appendix E: Marketing Synopsis 404

408 PSO Marketing Synopsis Appendix E: Marketing Synopsis 405

409 PSO Marketing Synopsis Appendix E: Marketing Synopsis 406

410 PSO Marketing Synopsis Appendix E: Marketing Synopsis 407

411 PSO Marketing Synopsis Appendix E: Marketing Synopsis 408

412 PSO Marketing Synopsis Appendix E: Marketing Synopsis 409

413 PSO Marketing Synopsis Appendix E: Marketing Synopsis 410

414 PSO Marketing Synopsis Appendix E: Marketing Synopsis 411

415 PSO Marketing Synopsis Appendix E: Marketing Synopsis 412

416 PSO Marketing Synopsis Appendix E: Marketing Synopsis 413

417 PSO Marketing Synopsis Appendix E: Marketing Synopsis 414

418 PSO Marketing Synopsis Appendix E: Marketing Synopsis 415

419 PSO Marketing Synopsis Appendix E: Marketing Synopsis 416

420 PSO Marketing Synopsis Appendix E: Marketing Synopsis 417

421 PSO Marketing Synopsis Appendix E: Marketing Synopsis 418

422 PSO Marketing Synopsis Appendix E: Marketing Synopsis 419

423 PSO Marketing Synopsis Appendix E: Marketing Synopsis 420

424 PSO Marketing Synopsis Appendix E: Marketing Synopsis 421

425 PSO Marketing Synopsis Appendix E: Marketing Synopsis 422

426 PSO Marketing Synopsis Appendix E: Marketing Synopsis 423

427 PSO Marketing Synopsis Appendix E: Marketing Synopsis 424