Volume I: Final PY2016 Evaluation of Consumer Programs

Transcription

1 Volume I: Final PY2016 Evaluation of Consumer Programs October 2 nd, 2017 Independent Electricity System Operator 120 Adelaide Street West Toronto, ON M5H 1T1

2 Prepared by: Cynthia Kan, PhD Laura James Jeremy Eckstein Andrew Rietz Anna Kelly Jane Colby The Cadmus Group, Inc. Audrey Yank Geneviève Lavigne Normand Michaud Econoler

3 Table of Contents Acronyms and Abbreviations... vi Executive Summary... viii 1. Introduction Coupons Program Program Description Impact Evaluation Methodology Findings Process Evaluation Methodology Findings Conclusions and Recommendations Heating and Cooling Program Program Description Impact Evaluation Methodology Impact Findings Direct Job Impacts Process Evaluation Methodology Findings Conclusions and Recommendations Residential New Construction Program Program Description Impact Evaluation Methodology Findings Direct Job Impacts Process Evaluation Methodology Findings i

4 4.4 Conclusions and Recommendations Cost-Effectiveness Introduction Calculation Methodology and Tests Total Resource Cost Test Program Administrator Cost Test Levelized Unit Electricity Costs Inputs and Assumptions Results Total Resource Cost Test Program Administrator Cost Test Levelized Unit Electricity Cost Conclusions List of Tables Table 1. Comparison of PY2016 to PY2015 Coupons Program Impact Evaluation Results... ix Table 2. Comparison of PY2016 to PY2015 Heating and Cooling Program Impact Evaluation Results... x Table 3. Comparison of PY2016 to PY2015 Residential New Construction Program Impact Evaluation Results... xi Table 4. Benefit/Cost Ratios for PY2016 Consumer Programs...xiii Table 5. Eligible Coupon Measures in PY Table 6. Comparison of PY2016 to PY2015 Coupons Program Performance... 3 Table 7. PY2016 Coupons Program Online Survey Distribution... 4 Table 8. Coupons Program Inputs Verified through Participant Surveys... 5 Table 9. General Scoring of Free Ridership Intention Component... 9 Table 10. General Scoring of Free Ridership Influence Component Table 11. Example Scoring Matrix for Free Ridership Influence Component (Hypothetical Respondent) 11 Table 12. Scoring Matrix for PY2016 Spillover Measures Table 13. Annual Coupons Initiative Participation in PY Table 14. Biannual Retailer Events Participation in PY Table 15. PY2016 Coupons Program Verified Gross Savings (MWh) Table 16. PY2016 Coupons Program Verified Gross Demand Reduction (MW) Table 17. Gross Reported and Verified Savings and Gross Realization Rates Table 18. PY2016 and PY2015 Measure Energy Savings Comparison Table 19. Survey-Based Algorithm Inputs for ENERGY STAR-Certified LEDs ii

5 Table 20. Coupons Program Free Ridership, Spillover and NTG Ratio by Region Table 21. Comparison of PY2016 and PY2015 NTG Components Table 22. Program-Level Free Ridership by Measure Table 23. PY2016 Spillover by Region* Table 24. Spillover by Measure Table 25. Rebound Usage Indicated in Survey* Table 26. NTG Components by Measure, at the Province Level Table 27. Coupons Program Process Evaluation Tasks Table 28. Coupons Program Documents Reviewed Table 29. Income Eligibility for Home Assistance Program Table 30. Participant Home Assistance Program Eligibility Based on Program Participation Table 31. Participant Home Assistance Program Eligibility Based on Income Table 32. Summary of PY2016 HVAC Program Results Table 33. HVAC Program Measure Counts and Incentive Levels Table 34. Load Profile: Alternative Peak Demand Factor Table 35. Comparison of Reported Savings by Savings Source Table 36. Gross Savings Inputs for Central Air Conditioners Table 37. Derating Factors for AHRI-Matched Central Air Conditioner and Furnace Pairs Table 38. Furnace with ECM Measure Permutations Table 39. HVAC Program Free Ridership Intention Component Scoring Table 40. HVAC Program Free Ridership Influence Component Scoring Table 41. HVAC Program Participation by Measure Table 42. Measure Counts, Gross Reported and Verified Savings and Gross Realization Rates Table 43. Comparison of PY2016 and PY2015 Energy Savings and Demand Reduction by Measure Table 44. HVAC Program Free Ridership, Spillover and NTG Ratio by Measure Table 45. HVAC Program Net-to-Gross Results by Region Table 46. HVAC Program Free Ridership Intention Score by Measure Category (n=284) Table 47. HVAC Program Free Ridership Influence Responses by Measure Category (n=284) Table 48. HVAC Program Influence Free-Ridership Score (n=284) Table 49. HVAC Program Intention and Influence Free-Ridership Scores by Measure Category Table 50. HVAC Program Spillover Estimates by Measure Category Table 51. HVAC Program Spillover Measures, Quantity and Savings Table 52. HVAC Program Job Impacts Table 53. HVAC Process Evaluation Tasks Table 54. HVAC Program Documents Reviewed Table 55. Survey Completes by Customer Category and LDC Reporting Region iii

6 Table 56. Customer Existing Air Conditioning Equipment Condition and Age* Table 57. Customer Existing Heating Equipment Condition and Age Table 58. Residential New Construction Program Measures and Incentives by Track Table 59. PY2016 Residential New Construction Program Results Table 60. PY2016 Prescriptive Measures Evaluation Approach Table 61. PY2016 Prescriptive Track Unit Savings Table 62. New Construction Effective Useful Life Review Table 63. PY2016 Performance Track Unit Savings Table 64. Residential New Construction Program Free Ridership Intention Component Scoring Table 65. Residential New Construction Program Free Ridership Influence Component Scoring Table 66: Residential New Construction Program Spillover Attribution Scoring Table 67. Number of Residential New Construction Program Measures per Track Table 68. Number of Houses per Performance Track Measure Table 69. PY2016 Residential New Construction Program Unit Savings and Effective Useful Life Table 70. Residential New Construction Program Gross Verified Savings per Program Track Table 71. Residential New Construction Program Net-to-Gross Results Table 72. Size of Participating Builder per Program Track Table 73. Residential New Construction Program Net-to-Gross Results per LDC Table 74. Residential New Construction Program Track Realization Rates Table 75. PY2016 Net Verified Savings for Prescriptive Track by Measure Table 76. PY2016 Net Verified Savings for Performance Track by Measure Table 77. PY2016 Net Verified Savings for Custom Track Table 78. Residential New Construction Program Net Verified Savings per Program Track Table 79. Residential New Construction Program Partner Interviews Table 80. Residential New Construction Program Builder Interviews per LDC Service Territory Table 81. Satisfaction with the Residential New Construction Program (Participating Builders) Table 82. Participating Builders by Size Table 83. Nonparticipating Builders by Size Table 84. Cost-Effectiveness Test Components Table 85. Total Resource Cost Test Ratio and Net Benefits for Each Consumer Program and Overall Table 86. Program Administrator Cost Test Ratio and Net Benefits Per Consumer Program and Overall128 Table 87. Levelized Unit Electricity Cost Results for Each Consumer Program and Overall List of Figures Figure 1. Customer Spillover Methodology... Error! Bookmark not defined. Figure 2. PY2013 through PY2016 General Purpose LED Baseline Lamp Types iv

7 Figure 3. PY2013 through PY2016 Specialty CFL Baseline Lamp Types Figure 4. Flow of Coupon Usage and Fulfillment Figure 5. Example of In-Store Signage Promoting Non-Energy Benefits Figure 6. Percentage of Respondents Participating in the Coupons Program Figure 7. Awareness of saveonenergy Figure 8. Sources of Awareness of saveonenergy by Marketing Channel Figure 9. Respondents Mentioning Coupon Event Marketing Channels Figure 10. Respondent Agreement with Impacts of saveonenergy Programs Figure 11. Respondent Agreement with Impacts of saveonenergy Programs by Region Figure 12. Participant Likelihood to Recommend the Coupons Program Figure 13. Percentage of Respondents Experiencing a Problem Redeeming Coupons Figure 14. Proportion of Measures Rebated, Energy Savings and Demand Reduction by Measure Figure 15. Efficiency Distribution for 15.0 SEER Central Air Conditioner Measure Figure 16. Capacity Distribution for 15.0 SEER Central Air Conditioner Measure Figure 17. Energy Savings and Demand Reduction Proportions by Furnace Fuel Type Permutation Figure 18. Average Per-Unit Energy Savings by Pre- and Post-Retrofit Furnace Fuel Type Figure 19. Increased Furnace Fuel Consumption by Fan Operation Figure 20. Reported Change in Winter Thermostat Setpoint Figure 21. Reported Change in Fan Operation When the HVAC System is Neither Heating Nor Cooling.. 70 Figure 22. Participant Reasons for Replacing Air Conditioning System Figure 23. Participant Reasons for Replacing Furnace Figure 24. How Customers Found Their Contractor Figure 25. Very Influential Factors in Customer Purchase Decisions Figure 26. Heating and Cooling System Replacement Need Figure 27. Customer Action in Absence of the Heating and Cooling Program Figure 28. Customer Satisfaction with Program Elements Figure 29. Customer Reasons for Dissatisfaction Figure 30. Customer Intention to Recommend the Heating and Cooling Program Figure 31. Categories of Customer Closing Comments Figure 32. Residential New Construction Prescriptive Measures Installed v

8 Acronyms and Abbreviations Acronym or Abbreviation Definition AHRI Air-Conditioning, Heating and Refrigeration Institute BOP Builder Option Package CAC central air conditioner CAI contractor adjustment influence CCHT Canadian Centre for Housing Technologies CEEA Canada Energy Efficiency Act CFF Conservation First Framework CSAT survey customer satisfaction survey DSM demand-side management ECM electronically commutated motor EER Energy Efficiency Ratio EM&V Protocols Evaluation, Measurement and Verification Protocols V2.0 EGNH EnerGuide New House ESNH Standard ENERGY STAR for New Homes Standard EUL effective useful life FTE full-time equivalent HAP Home Assistance Program HOU hours of use HRAI Heating, Refrigeration, and Air Conditioning Institute HVAC heating, ventilation and air conditioning HVAC Program Heating and Cooling Program IESO Independent Electricity System Operator LDC local distribution company LEAP Low-Income Energy Assistance Program LMMRS longitudinal mass market research survey LUEC Levelized Unit Electricity Costs NRCAN Natural Resources Canada NTG net-to-gross PAC Program Administrator Cost PIA prescriptive input assumption PY program year QA/QC quality assurance and quality control RNC Program Residential New Construction Program SEER Seasonal Energy Efficiency Ratio TRC Total Resource Cost TRM technical reference manual vi

9 UMP Acronym or Abbreviation VACS report Uniform Methods Project Value Added Consumer Services report Definition vii

10 Executive Summary The Independent Electricity System Operator (IESO) contracted with the Cadmus team (Cadmus, Econoler, and Nielsen Opinion Quest) to evaluate its province-wide saveonenergy branded consumer programs (Coupons, Heating and Cooling and Residential New Construction) for program year (PY) 2016 through PY2020 under the Conservation First Framework (CFF). This report describes the process, impact and cost-effectiveness evaluations of the PY2016 (January 1, 2016 through December 31, 2016) consumer programs. Program Descriptions and Methods Through the Coupons Program, the IESO and local distribution companies (LDCs) provide customers with coupons for use at participating retailers. Customers use these coupons to purchase various energy efficiency measures, such as LED bulbs, ENERGY STAR light fixtures, advanced power bars and electric baseboard programmable thermostats. The program is segmented into annual (year-round) and biannual (month-long events twice a year) offerings. To assess the Coupons Program, the Cadmus team conducted five evaluation activities: Online panel survey with 1,200 respondents (impact, process) Engineering analysis (impact) Interviews with IESO marketing and program staff (process) Review of program documents, including survey results from a longitudinal study managed by the IESO (process) and program redesign documents for PY2017 (process) Analysis of program tracking data (Value Added Central Services Report [VACS report]; impact) Through the Heating and Cooling (HVAC) Program, the IESO and LDCs offer customer rebates for the installation of high-efficiency furnaces with an electronically commutated motor (ECM) and highefficiency (SEER 14.5 or SEER 15 and greater) central air conditioners (CACs). Customers must work with a program contractor to initiate the rebate process. To assess the HVAC Program, the team conducted five evaluation activities: Telephone surveys with 214 participants, evenly split between customers who received an incentive for (1) a furnace with ECM measure, (2) a SEER 15+ CAC and (3) both the furnace and CAC measures combined (impact, process) Engineering analysis (impact) Interviews with IESO marketing and program staff and two implementer organizations (process) Review of program documents, including survey results from a longitudinal study managed by the IESO (process) and program redesign documents for PY2017 (process) Analysis of program tracking data (VACS report, HVAC Program raw data file; impact) Through the Residential New Construction (RNC) Program, the IESO and LDCs provide incentives to residential homebuilders who participate in one of three tracks: Prescriptive, Performance and Custom. viii

11 To assess the RNC Program, the team conducted six evaluation activities: Interviews with 13 participating and four nonparticipating builders (impact, process) Engineering analysis and desk reviews (impact) Review of HOT2000 models (impact) Interviews with IESO marketing and program staff, participating LDCs from PY2015 and PY2016 and their delivery agents, and Natural Resources Canada (NRCAN; process) Review of program documents (process) Analysis of program tracking data (monthly LDC project lists and project support documentation; impact) The Cadmus team used the IESO s CDM EE Cost Effectiveness Tool to calculate cost-effectiveness results for all programs, using evaluation results as inputs to the calculator. Key Findings and Conclusions Coupons Program Table 1 shows a comparison of the PY2016 to PY2015 Coupons Program evaluation results. The use of coupons, particularly during the Biannual Retailer Events, increased sharply in PY2016, resulting in an almost three-fold increase in net annual savings, despite a reduction in the net-to-gross (NTG). Table 1. Comparison of PY2016 to PY2015 Coupons Program Impact Evaluation Results Program Metric PY2016 Verified PY2015 Verified Program Reported Savings (GWh) Net-to-Gross 140% 164% Net Annual Energy Savings (GWh) Net Summer Peak Demand Reduction (MW) Increased LED coupon sales resulted in higher program savings in PY2016. In PY2016, retail prices for general purpose and specialty LEDs continued to decrease. In addition, manufacturers expanded the number of models available, which, coupled with dropping prices, made these products more of a mainstream option for consumers. Relative to PY2015, shoppers were less likely to report they intended to purchase a CFL in PY2016, which corresponds to a decrease in availability of CFLs across retailers. At the same time, consumers were more likely to report they intended to purchase an LED, indicating LEDs were filling some of the market vacuum as CFLs retreat. Finally, the new coupon fulfillment agent provided improved service to retailers through increased retailer communication and engagement, which may have led retailers to place more coupons near eligible products, including LEDs. Changes in the lighting market led to substantial changes in per-unit savings. Because more people claimed an incandescent or halogen baseline than a CFL baseline, the baseline wattage for general purpose LEDs increased by about 9 watts from the PY2015 value. This resulted in an increase for the perunit kilowatt-hour savings by 74%. Similar baseline shifts led to decreased per-unit savings for specialty CFLs and lighting fixtures. ix

12 While there are a variety of program measures, nearly all savings 96% came from general purpose and specialty LEDs. Even specialty CFLs, which have previously driven large volumes of savings, made a negligible contribution in PY2016. Nevertheless, there are separate coupons for 11 additional measures, in different forms (single and multi-packs) that require disproportionate marketing, processing and evaluation efforts compared to the savings they generate. Changes in the NTG methodology and survey responses led to a decrease in the NTG for most regions. PY2016 regional NTG estimates dropped by 13% on average from the PY2015 regional values, with decreases in all but two regions. Two factors accounted for most of the reduction. First, PY2016 survey results indicated less spillover than in previous years. Second, the Cadmus team added analysis of rebound usage to the NTG ratio formula. Rebound usage reduces the NTG ratio, but had not previously been evaluated. Heating and Cooling Program Table 2 shows a comparison of the PY2016 to PY2015 HVAC Program evaluation results. The higher PY2016 verified energy savings are due to increased participation (projects and measure quantities) and a higher NTG. However, demand reduction decreased due to the higher baseline for the SEER 15 CAC measure and adjustments to the demand calculation for the furnace with ECM measure. Table 2. Comparison of PY2016 to PY2015 Heating and Cooling Program Impact Evaluation Results Program Metric PY2016 Verified PY2015 Verified Program Reported Savings (GWh) 96.6 Not Available Net-to-Gross Ratio 70% 51% Net Annual Energy Savings (GWh) Net Summer Peak Demand Reduction (MW) Inconsistencies in the VACS report resulted in discrepancies in the total reported program savings. The Cadmus team identified inconsistency issues within the VACS report, specifically regarding the number of significant figures recorded for demand reduction values. As a result, the final reported demand reduction in the final impact reporting template from PY2016 does not exactly match the demand reduction in the VACS report, by a difference of approximately 44 kw. The program tracking data does not include all the primary inputs necessary for program evaluation. Currently, the information collected for the SEER 15 CAC measure is adequate to check whether the equipment qualifies for a program rebate. However, one of the key inputs for the savings evaluation cooling capacity is not tracked in the program data. The Cadmus team identified cooling capacities for most equipment rebated through the program based on the recorded Air-Conditioning, Heating and Refrigeration Institute reference number, but this process is not always successful. Opportunities exist to expand customer awareness of the program. Only 27% of the surveyed customers knew about the HVAC Program before contacting a contractor and 19% of customer comments suggested that the program expand its marketing effort. While some customers replaced equipment that was in good working order (34% for CACs and 39% for furnaces), most sought to replace x

13 either completely broken or malfunctioning equipment. Making customers aware of program offerings and the economic and environmental benefits of efficient heating and cooling equipment could increase the proportion who replace older and less efficient equipment before it experiences problems. Contractors are a key driver of the program. Survey data indicate that most customers hear about the program from a contractor and that most customers replaced equipment with existing problems. Customers seek out contractors that are either known to them or known to an acquaintance and that a contractor recommendation is very influential in a customer s decision to upgrade equipment. Residential New Construction Program Table 3 shows a comparison of the PY2016 to PY2015 RNC Program evaluation results. Savings decreased in PY2016 despite a higher overall NTG. The savings difference from PY2015 is due to lower participation (in LDCs, Performance projects, and Prescriptive measure quantitiesand to changes in the verified unit savings. Table 3. Comparison of PY2016 to PY2015 Residential New Construction Program Impact Evaluation Results Program Metric PY2016 Verified PY2015 Verified Program Reported Savings (GWh) Net-to-Gross Ratio Net Annual Energy Savings (GWh) Net Summer Peak Demand Reduction (MW) Track builders exceeded building code requirement due to program influence. The track builders who constructed the most ENERGY STAR homes and implemented the most Prescriptive measures under the RNC Program reported that they would have built houses only to code in PY2016 had the program not existed. As such, the free-ridership level was lowest in the Prescriptive track (13%) and for ENERGY STAR homes (19%). Conversely, smaller custom builders said they would have likely built homes to the same efficiency level in absence of the program. The Performance track performed best, generating 41% of overall program net verified savings. Within the Performance track, most homes (96%) were ENERGY STAR-certified. Reduced networking efforts between LDCs and builders may be negatively impacting participation. During the interviews, some LDCs shared that they have little contact with builders due to limited staff. In addition, these LDCs reported that they assumed most builders already knew about the program and therefore spent less time reaching out to builders than in the past. However, in PY2016, the number of submitted projects decreased from PY2015. In addition, implemented projects were distributed over fewer LDCs. Follow up with builders is insufficient. Several participating builders expressed concern regarding sparse communications with LDCs, especially related to preliminary and final applications. Once they submitted the required documentation, builders said they did not always receive confirmation that their xi

14 application was received. Moreover, when receiving payment, builders noted that there was no documentation that specified the associated measure(s) to explain the incentive amount. While builders struggle on their own with aspects of the application process, delivery agent involvement can ensure a smooth process. Builders identified the paperwork required to complete the application process, especially for the Prescriptive track, as a participation barrier. Specifically, builders said it was complex and cumbersome. In contrast, builders generally reported that the Performance track application process was more convenient and straightforward. For all tracks, builders reported they appreciated when delivery agents helped them complete the application process. Participating LDCs find the RNC Program reporting process labour intensive. During interviews, LDCs said they faced challenges with RNC Program reporting. Small LDCs in particular expressed concerns with the time required to complete reporting, and larger LDCs expressed displeasure with the complexity associated with reporting a large number of projects. Specifically, these larger LDCs said the manual data entry made the current reporting process tedious. The Cadmus team noted inconsistencies in the project list, likely due to the manual, burdensome reporting process. A measure review and adjustment could reenergize participation and increase savings. According to interviewed LDCs, the RNC Program lags behind the market in terms of the energy efficiency measures offered. In addition, the team found that the current Prescriptive track products do not generate significant per-unit savings, especially given the current market baseline identified as part of the PY2016 per-unit savings review. Therefore, significant energy savings can only be achieved if builders install a high number of products. Moreover, some Prescriptive track products overlap with Builder Option Package (BOP) measures implemented by builders to meet ENERGY STAR certification, further reducing savings opportunities through the Prescriptive track. Because the IESO does not currently track non-electricity energy savings, the program benefits may be understated. As highlighted in the Spillover section, only 14% of Ontario houses are primarily heated with electricity, meaning most houses are heated with natural gas. In addition, all 35 reviewed HOT2000 simulation files represented houses with natural gas as primary heating systems. Despite this, the IESO does not currently track natural gas savings. The reduced program scale and savings potential negatively influenced cost-effectiveness. Participation dropped significantly in PY2016 with the smaller budgets allocated for the RNC Program by LDCs compared to the previous framework. Furthermore, the current program offerings generated lower per-unit savings especially given the current market baseline and share of electric savings identified as part of the PY2016 per-unit savings review resulting in low overall program costeffectiveness. Cost-Effectiveness The Cadmus team used the impact results and program costs to assess the cost-effectiveness of province-wide consumer programs. The team assessed performance from the perspective of numerous cost-effectiveness tests, namely the Total Resource Cost (TRC), the Program Administrator Cost (PAC) xii

15 and the Levelized Unit Electricity Costs (LUEC) tests. These tests include benefits and costs that persist over the effective useful life (EUL) of the measures implemented through the Coupons, RNC and HVAC programs. For a program to be considered cost-effective, the benefits must exceed the costs, with a program benefit to cost ratio greater than 1.0. The overall portfolio has a TRC test result of 4.94 and a PAC test result of Table 4. Benefit/Cost Ratios for PY2016 Consumer Programs Program TRC PAC LUEC ($/kwh) Coupons Program Heating and Cooling Program Residential New Construction Program Consumer Programs xiii

16 1. Introduction The IESO contracted with the Cadmus team to evaluate its province-wide saveonenergy branded consumer programs (Coupons, Heating and Cooling and Residential New Construction) for PY2016 through PY2020 under the CFF. The CFF began in PY2015 and runs through PY2020, with a goal of reducing the electricity consumption in Ontario by 7 TWh by December Among other changes, the CFF gives LDCs a more prominent role in achieving the conservation target, as well as the flexibility to tailor program offerings to the LDC s unique mix of customers. Because PY2015 was a transition year for the 70 or more IESO-connected LDCs, PY2016, which is the same as the 2016 calendar year, was the first full year of operation under the CFF. This report covers the evaluation of the PY2016 province-wide consumer programs; however, as these programs have been offered and evaluated over multiple years the results highlight differences between PY2015 and PY2016. The Cadmus team performed the activities below: Impact Evaluation Determine with 90/10 confidence and precision the annual verified gross energy savings and demand reduction at the project track or measure category level. Measure NTG ratios at the province, regional or individual LDC level, based on ability to achieve statistical significance Update the IESO s Prescriptive Measures and Assumptions List with new prescriptive inputs and assumptions (PIA) Determine direct job impacts due to the programs Process Evaluation Develop recommendations for program improvement Examine program and marketing effectiveness Identify reasons for variations in program performance among LDCs Assess participant satisfaction Cost-Effectiveness Calculate program-level TRC, PAC and LUEC test results This report is structured with a chapter for each of the three programs, followed by a chapter of costeffectiveness results. Each program chapter includes a program description (including incentives and eligibility requirements), methodology and results of the impact evaluation, methodology and results of the process evaluation and conclusions and recommendations. 1 This value includes LDC-connected business and residential customers. 1

17 2.1 Program Description 2. Coupons Program The Coupons Program consists of the Biannual Retailer Events and the year round (Annual) Coupons Initiative. During the Biannual Retailer Events, which occur in the spring and fall of each year, participating retailers offer point-of-sale coupons for eligible products. The IESO, the LDCs and participating retailers promote the event through direct mail, online advertising and in-store signage. Retailers also train their staff to promote the Biannual Retailer Events. Annual Coupons are available year-round on the saveonenergy website. Some LDCs participate in the Annual Coupons Initiative by mailing coupons booklets (that have a special barcode identifying the LDC) to their customers. The coupons offer discounts on high-efficiency items including CFL and LED lighting, light fixtures, lighting controls, water heater blankets, weatherstripping, outdoor timers, power bars and other items. Table 5 shows the coupons available in PY2016. Table 5. Eligible Coupon Measures in PY2016 Measure ENERGY STAR-certified general purpose LED light bulbs (single or double pack) ENERGY STAR-certified LED light bulbs (multipack of three or more bulbs, general or specialty) Coupon Discount ($) Annual Coupons Initiative Biannual Retailer Events 3 8 ENERGY STAR-certified specialty LED light bulbs (single or double pack) 5 ENERGY STAR-certified specialty CFL light bulbs (single or double pack) 3 ENERGY STAR-certified specialty CFL light bulbs (multipack of three or more) ENERGY STAR-certified light fixtures (indoor hard wired) with one or two sockets ENERGY STAR-certified light fixtures (indoor hard wired) with three or more sockets ENERGY STAR-certified ceiling fans 10 Indoor motion sensors/dimmer switches/timers (hard wired, single pack) 3 Indoor motion sensors/dimmer switches/timers (hard wired, multipack of two or more) 6 Weatherstripping (door frame kits) 3 Weatherstripping (foam or V-strip package) 2 Insulation blanket for electric water heaters 4 Heavy-duty outdoor timers 4 Power bars with integrated timer or auto-shutoff 4 2

18 Measure Coupon Discount ($) Annual Coupons Initiative Outdoor clothesline umbrella stands or clothesline kits 10 Programmable thermostats for electric baseboard heaters (single or multipack) Programmable thermostats for electric baseboard heaters (multipack of three or more) Biannual Retailer Events In PY2016, consumers redeemed 9.7 million coupons, compared to the 4.1 million coupons redeemed in PY2015. Due to an increase in the gross per-unit savings for general purpose LEDs, and despite an overall reduction in the NTG ratio, the net annual energy savings for PY2016 was 428 GWh, higher than the previous year. Detailed program savings, realization rates and NTG ratios in PY2016 and PY2015 are shown in Table 6. Table 6. Comparison of PY2016 to PY2015 Coupons Program Performance Program and Track Level Metric PY2016 Verified PY2015 Verified Program Reported Savings (GWh) Realization Rate (%) 167% 100% Gross Verified Demand Reduction (MW) Gross Verified Annual Energy Savings (GWh) Gross Verified Lifetime Energy Savings (GWh) 3, ,564.7 Net-to-Gross Ratio 140% 164% Net Peak Demand Reduction (MW) Net Peak Demand Reduction Persisting at PY2020 (MW) Net Annual Energy Savings (GWh) Net Energy Savings Persisting at PY2020 (GWh) Net Lifetime Energy Savings (GWh) 5, , Impact Evaluation The Cadmus team determined the gross and net energy savings and demand reduction for each measure available through the PY2016 Coupons Program. We assessed savings from the Biannual Retailer Events and the Annual Coupons Initiative consistent with the CFF Evaluation Measurement and Verification Protocols V (EM&V Protocols). 2 Ontario Power Authority. Conservation First Evaluation Measurement and Verification (EM&V) Protocols and Requirements, v2.0. Available online: 3

19 2.2.1 Methodology The Cadmus team conducted a general population survey to collect primary data to inform the impact evaluation for PY2016. For key measures, we calculated the per-unit savings values using updated inputs to the engineering algorithms applied in previous evaluations, which conform to CFF EM&V Protocols. We determined net savings using a similar approach to the PY2015 evaluation, but with the added component of a rebound analysis. Data Collection: General Population Survey The Cadmus team conducted a survey of the general population to identify participants and nonparticipants, to collect data to update inputs to the gross and net savings calculations and to inform the process evaluation. We issued the survey to a sample designed to achieve 90% confidence at ±10% precision at the program level and for key program measures (light bulbs and lighting controls). Table 7 shows the target number of completes and the final sample for both participants and nonparticipants. Table 7. PY2016 Coupons Program Online Survey Distribution Respondent Type Target Completes Participants Nonparticipants No target 622 Total ,239 The Cadmus team designed and administered the general population survey online to two proprietary panels of Ontario residents. We distributed the survey in late April 2017, and questions specifically addressed the PY2016 year. Gross Savings Methodology For key coupon measures, the Cadmus team used engineering analyses to determine gross energy savings. For four measures (ENERGY STAR-certified ceiling fans, heavy-duty outdoor timers, water heater insulation blankets and programmable thermostats for baseboard heaters) that together contributed only 0.3% of program savings in PY2016, the team applied the existing PIA values. We followed the same methodology for key measures as used in the PY2015 Coupons Program evaluation, using the updated measure inputs shown in Table 8. For other inputs not shown in the table, the Cadmus team relied on deemed or previously assessed values. 4

20 Table 8. Coupons Program Inputs Verified through Participant Surveys Measure Input Source LED General Purpose Lamps Baseline wattage Participant responses, lumen equivalency standards CFL and LED Specialty Lamps Baseline wattage Participant responses, lumen equivalency standards Distribution of measure specialty lamp type Participant responses ENERGY STAR Fixtures Sockets per fixture Participant responses Fixture lamp types Participant responses Lighting Controls Control type Participant responses Days of use per year Participant responses Clothesline Kits and Stands Fuel type of supplemental clothes dryer Participant responses Power Bars and Outdoor Timers Items connected to timers Participant responses All Measures In-service rate Verified participation Participant responses Participant responses Lighting Measures: LEDs and CFLs To determine per-unit gross savings for the general-purpose LEDs, specialty LEDs and specialty CFLs discounted through the Coupons Program, the Cadmus team applied the energy savings algorithm used in the PY2015 evaluation and consistent with the CFF EM&V Protocols, and industry standard guidelines for lumen equivalency. For each program measure, the team summed savings from the following equation to calculate measure savings at the program level: Where: kwh RES = (Watt Base Watt EE ) Stepdown ISR Hours RES 1,000 W/kW Watts Base = Average wattage of displaced bulbs Watts EE = Average wattage of program bulb Stepdown = ISR = In-service rate A factor that accounts for the impact of the Canada Energy Efficiency Act (CEEA) lighting efficiency standards on the maximum baseline wattage that can be assumed in PY2016 Hours RES = Average daily hours of use (HOU) for residential applications The Cadmus team calculated per-unit demand reduction estimates by multiplying the estimated per-unit energy savings by the measure-specific summer peak demand factor. The following sections detail the assumptions and equation inputs we developed to calculate lighting savings. 5

21 Watts EE and Watts Base The Cadmus team used survey responses to determine the appropriate Watts Base and Watts EE values for each measure. We calculated a Watts Base value based on the distribution of displaced bulb technologies, as reported by survey respondents. Surveyed participants answered what bulb technology they would have installed had the coupon-incented bulb not been available, selecting an incandescent, halogen, CFL, LED or other type of bulb. We used guidance from NRCAN on lumen equivalence across technologies to determine the average wattage of each displaced bulb type, and weighted the wattages by the response frequency to calculate the average baseline wattage. For the specialty LED and CFL bulbs, we also determined average wattages across four types of bulb function: reflectors, globes, candles and 3-way bulbs. To determine Watts EE, the Cadmus team used the average wattage for all products of the same technology and function type available in the Canadian market (per the ENERGY STAR Qualified Products list). 3 For specialty LEDs and CFLs, we also weighted the average wattages for different specialty functions by the frequency of purchase reported through the surveys, to calculate a single composite Watts EE value. Stepdown To account for regulatory changes enacted in 2012 by the CEEA, the Cadmus team calculated an adjustment factor to represent the difference in per-unit savings before and after the regulation took effect. The regulation, which established minimum lamp efficacy standards above what is achieved by standard incandescent bulbs, resulted in the virtual elimination of incandescents from the market by Therefore, a percentage of respondents who claimed that they would have installed an incandescent if the program bulb was not available would not actually have had that opportunity. The team identified these no opportunity respondents as those indicating an incandescent baseline who also indicated they did not know about the CEEA standards, and who did not indicate that their baseline bulb was one they already had in storage. We calculated a modified per-unit savings, assigning a halogen baseline to these respondents, then divided that per-unit value by the unmodified per-unit savings to determine the stepdown factor. The Cadmus team used the stepdown approach consistently with the PY2015 evaluation. In past years, this approach has provided the flexibility to assume one per-unit savings value for the first year, then a second value for future years as incandescents became less available. Incandescents phased out of retail stores over the course of the PY2015 year. To reflect this progressive decrease, the final per-unit savings value was a weighted average of the per-unit savings value when assuming incandescent availability in 3 ENERGY STAR Qualified Products List Accessed April 28, 2017: Specifications/ENERGY-STAR-Certified-Light-Bulbs-Version-2-0/ebgj-qsf7/data 6

22 the first half of the year, and the value when assuming no incandescent availability in the second half of the year. For PY2016, we assumed that no incandescents were available in stores at any point. In-Service Rate The Cadmus team used the participant survey responses to determine the number of lighting products purchased with coupons that were subsequently installed or placed in storage with the intent to install at some point in the future, versus being lost, disposed of or otherwise not available for use. Residential Hours of Use For LEDs and CFLs, the Cadmus team used the residential HOU determined through the PY2013 and PY2014 Coupons Program participant surveys, consistent with the HOU used in the PY2015 evaluation. Verified Participation The team used the participant survey results to estimate the average number of eligible products purchased with each type of coupon. We calculated the total number of measures incented through the program as the total number of coupons, multiplied by the average number of units purchased per coupon. We applied the per-unit savings value calculated using the engineering analysis, as appropriate, to the total estimated number of items purchased for each measure, to calculate program gross savings. For lighting measures, the per-unit savings algorithm accounts for the in-service rate, so no further adjustments were necessary to determine participation. For all other measures, we reviewed PIA algorithms to determine whether to account for the in-service rate. Lighting Measures: Controls and Fixtures For lighting controls and ENERGY STAR lighting fixtures, the Cadmus team applied the energy savings algorithms used in the PY2015 evaluation and consistent with the CFF EM&V Protocols and industry standard guidelines for lumen equivalency to determine per-unit gross savings for each program measure. For lighting controls measures, the Cadmus team used participant survey responses to determine the distributions of types of hard-wired controls (motion sensors, indoor timers or dimmer switches) purchased by participants with a coupon. We also used survey responses to determine the installation rate and the types of lamps controlled. We used the PY2015 HOUs for baseline (no controls) and measure (with controls) scenarios for each type of control. For ENERGY STAR fixtures, the team used participant survey responses to determine the types of lamps participants would have installed if they had not purchased fixtures with a coupon, the number of sockets per fixture and the installation rate. Non-Lighting Measures The Cadmus team used the IESO PIAs to evaluate program savings for programmable thermostats, water heater blankets, ENERGY STAR-certified ceiling fans and heavy-duty outdoor timers. For the remaining non-lighting measures, the team followed the PY2015 evaluation methodology to evaluate savings, 7

23 using key inputs determined from PY2016 participant survey responses. For weatherstripping measures, we used participant surveys to determine the type of weatherstripping installed (door frame kits or V- strip packages), the baseline (whether used as new, replacement, or additional weatherstripping) and whether participants had central cooling at home. For power bars, we used participant surveys to determine the types of power bars purchased and the type of equipment plugged into the power bars. Lastly, for clotheslines, we used participant surveys to estimate clothesline usage patterns and types of clothes dryers used in place of clotheslines. For weatherstripping, power bars and clotheslines, the Cadmus team used participant surveys to determine the average installation rates and the number of measure units purchased with each coupon. Net Savings Methodology The Cadmus team used self-report data from the general population survey to assess free ridership and spillover for the PY2016 evaluation, and to evaluate NTG for each region. We derived net savings the savings directly attributable to the program by applying the appropriate regional NTG ratio to each LDC s gross savings, and summing impacts across LDCs. We used the following equation to calculate the PY2016 Coupons Program NTG ratio: NTG Ratio = 1 - Free Ridership + Spillover - Rebound Free ridership refers to program participation by those who would have acquired the energy efficiency measure in the absence of the program s influence. The effect of free riders reduces the net savings attributable to the program. Spillover represents the additional energy savings that occur when customers as a result of the program s influence install other energy efficiency measures without rebates or change their energy use behaviour. For example, participating customers may experience the energy-saving benefits of a rebated product, which encourages them to adopt energy-saving products or behaviours elsewhere in their home or business. A nonparticipating customer may respond to general marketing or advertising and purchase energy-savings products without an incentive. Rebound is the decrease in energy savings resulting from program participants using their new equipment more than they used the historic equipment because the increased efficiency lowers the cost of increased use. While the previous evaluator did not measure rebound, the Cadmus team added this factor in PY2016 to improve accuracy. 8

24 Free Ridership The Cadmus team used data from two questions in the residential participant surveys to calculate free ridership. One question asked about the participant s intention at the time they made the purchase, and the other asked about factors that might have influenced the participant. Intention is what the participant would most likely have done if not receiving program assistance. A free rider would have made the same purchase at the same time, without the coupon. Influence is the effect of factors that may have influenced a participant s decision to take energy-efficient actions. If the participant noted any influence from any of the program marketing or another program activity, they are not a free rider. Only non-lighting participants answer the intention question, since the lighting baseline calculation incorporates the participant s intention to purchase an eligible measure. All participants answer the influence question. For the PY2016 analysis, the team used PY2016 survey data to assess free ridership for all non-lighting measures. For lighting measures, we used the PY2015 rate, weighted to reflect PY2016 lighting participation. The Cadmus team computed the free-ridership score for each participant, ranging from 0% (no free ridership) to 100% (total free ridership), by summing the intention and influence scores, each worth 50% of the total. The following sections provide more detail on how we scored each response. Intention Scoring Table 9 shows the intention question from the survey and each response option, along with the intention score corresponding to each response. Question Which of the following would have been most likely if there had been no Coupons Program? Table 9. General Scoring of Free Ridership Intention Component Response Intention Score (Max 50.0%) Purchased the exact same product in the same quantity without the coupon 50.0% Purchased the exact same product in the same quantity without the coupon, but delayed the purchase by a few months 37.5% Done something else [Specify: ] 25.0% Don t know 25.0% Purchased a cheaper product 0.0% Purchase whatever was on sale 0.0% Purchased a smaller quantity of the exact same product 0.0% Delayed the purchase by one year or more 0.0% Reporting an alternative that likely would have produced no energy savings resulted in an intention score of 0%; reporting something that likely would have produced some energy savings, but lower savings than the incented purchase, resulted in an intention score of 25% or 37.5% (depending on the action); and reporting the same outcome as the incented purchase resulted in an intention score of 50% 9

25 (the maximum). The Cadmus team scored don t know responses as 25% to allow for the possibility that the respondent would have made the same purchase. We gave an intention score of 0% for respondents who added detail to the response done something else that indicated they would not have purchased as efficient a measure without the coupon. Influence Scoring To estimate program influence, the Cadmus team asked respondents how various program elements influenced their decision about the energy efficiency measure they purchased. The influence of any one of several aspects of the program implementation such as coupon discounts, retailer advertising or program marketing on a buyer s decision to purchase a more efficient product over a less efficient product indicates that the savings from that product are at least partially attributable to the program. The program influence score was equal to the minimum rating of any single program element, rather than an average, because the Cadmus team assumed that if any given element had a significant influence on the respondent s decision, then the program itself was successful in influencing the respondent. For the influence question, we asked participants about the importance of the program, rather than using the word influence, to avoid any bias the respondent might have against being perceived as not fully in charge of their own decision making. High program influence and free ridership have an inverse relationship: the greater the program influence, the lower the free ridership (see Table 10). In other words, if a respondent cites a program element as very important to their decision, the Cadmus team assumed that they took an energyefficient action they would not have taken without the market intervention; therefore, we assigned these participants a low or zero free-ridership score for that program element, and we attributed the savings from that participant s purchase to the program. Table 10. General Scoring of Free Ridership Influence Component Maximum Program Influence Rating Influence Score (non-lighting) (Max 50.0%) 1 Not at all important 50.0% 2 Not very important 37.5% 3 Somewhat important 12.5% 4 Very important 0.0% Don t know 25.0% Table 11 shows the scoring matrix, with hypothetical data, that we used to determine program influence. The specific program elements addressed in the survey are shown on the left. 10

26 Table 11. Example Scoring Matrix for Free Ridership Influence Component (Hypothetical Respondent) Program Element Coupon discount In-store information from a sales associate, in-store event or in-store display Other marketing and/or advertising about the program (mailing, website, online ad) LDC or IESO information about energy efficiency in general Not At All Important (50.0%) Not Too Important (37.5%) X Somewhat Important (12.5%) X Very Important (0.0%) X Don t Know (25.0%) Minimum Score 0.0% Aggregated Score As stated earlier, the Cadmus team determined each respondent s total free ridership by summing the intention and influence components, resulting in a score between 0% and 100% for each of the selected measures purchased with a coupon. To determine the regional free ridership, we averaged the freeridership scores for all respondents in that region, weighting each response by the free-rider savings contributed by that respondent for that measure. The Cadmus team applied the region-level freeridership rates to calculate net energy savings and demand reduction. Spillover Spillover refers to savings from energy-efficient purchases by participants or nonparticipants who have been influenced in some way by program activities, such as through enhanced awareness of energyefficient options due to program marketing, even though they do not use a coupon for their purchase. Like spillover is the additional savings generated by the purchase of a coupon-eligible measure, without using a coupon and for which the respondent indicates that the program had some influence. Non-like spillover is the purchase of any energy-efficient item not eligible for a coupon, but still motivated by the program activity. The Cadmus team evaluated like and non-like spillover for the Coupons Program using responses to the participant and nonparticipant surveys. Error! Reference source not found. shows the Cadmus team s approach to identifying spillover measures using survey responses. X 11

27 Figure 1. Customer Spillover Methodology Since participating in the program, have you installed any additional energy-efficient equipment or made any other changes to improve the energy efficiency of your home? (Change for which you did NOT receive a rebate or discount.) What equipment or improvements? How many/how much? How important was the program in your decision? The survey asked respondents who purchased energy-efficient measures, but did not use a coupon, to use a four-point scale to rate the degree of program influence on their purchase: (1) very important, (2) somewhat important, (3) not very important or (4) not at all important. Where the respondent indicated that the program was very important, we assigned 100% of the measure savings to the program. We adjusted the attributable savings percentage downward as program importance diminished, as shown in Table 12. This methodology is a revision of the methodology used in previous evaluations, in which we applied a 5-point scale and diminished program attribution by 25% at each level (100%, 75%, 50%, etc.). While there is no empirical test evidence of which method is more accurate, the Cadmus team adopted the revised scoring as a more conservative approach, with a reduced risk of over-reporting spillover savings. Table 12. Scoring Matrix for PY2016 Spillover Measures Response Percentage Savings Attributed to Program Very important 100% Somewhat important 50% Not too important 0% Not at all important 0% Don t know 0% The Cadmus team screened the survey data to remove any reported spillover measures for which participants reported receiving a rebate through another IESO or LDC program, and those for which they reported using a coupon from the IESO. To determine like spillover savings in each region, the Cadmus team applied the appropriate attribution percentage to the per unit savings for each program-eligible measure purchased without a coupon reported by survey respondents from the particular region. The Cadmus team calculated the region spillover rate as the total attributable savings from all program-eligible measures purchased without a coupon reported by the survey sample divided by the total savings from all coupon-discounted 12

28 measures reported by the survey sample. The team calculated participant and nonparticipant like spillover separately for each region. The Cadmus team calculated the rate of non-like spillover at the program level, and applied the program rate to each region. To determine the program rate, the team identified all survey respondents that had completed an energy efficiency action or purchased an energy efficient item not eligible for a coupon, and attributed the appropriate portion of the savings to the program based on how the respondent rated the programs importance on their decision (see Table 12). The sum of attributable savings from these nonlike actions and measures reported by the respondents, divided by the total savings from all coupon-discounted measures reported by respondents, equals the rate of non-like spillover. The team determined savings from like spillover measures using the evaluated per-unit savings. For nonlike spillover measures, we used the PY2015 evaluation assumptions, based on various sources, using updated values where available. Rebound The final component in the NTG ratio is the percentage of rebound generated by participants. Rebound is when a consumer increases their usage of more efficient products since they are using less energy per hour of use than previously. For example, someone who purchases an LED light bulb may not be as careful to turn the lights off, because they know the bulb is wasting less energy when left on. The Cadmus team evaluated rebound using participant survey responses to rebound questions about whether and how purchased ENERGY STAR LEDs, ENERGY STAR specialty CFLs and ENERGY STAR light fixtures changed their energy usage behaviour; specifically, whether they were less likely to turn off lights as a result of their purchase. The team also asked weatherstripping participants whether they were more or less likely to turn their thermostat down in the summer or up in the winter (using more energy to heat or cool their home). However, these questions did not receive a response sample sufficient to determine a percentage of rebound different from 0%. As weatherstripping generated a very small percentage of overall program savings, this type of rebound is likely minimal, if it exists at all. The team determined total rebound usage by asking respondents how many hours per day they left the program-discounted lighting turned on, relative to the prior product that was installed. Similar to the approach we used to determine free ridership and spillover, we divided the total rebound usage by survey respondents and by the survey sample s total gross savings to determine the percentage rebound for the program Findings Using the methodology described above, the Cadmus team determined the gross and net savings achieved with the PY2016 Coupons Program. Gross Savings Impact Total participation increased to over 9.6 million coupons in PY2016 from just over 4.1 million in PY2015, across both the Annual Coupons Initiative and Biannual Retailer Events. Sales of LEDs during the 13

29 biannual events accounted for most of the increase, though sales of fixtures (which often contain LED bulbs) also increased substantially. Coupon purchases of ENERGY STAR-certified specialty CFLs, formerly a major driver of program savings, decreased in both the Annual Coupons Initiative and Biannual Retailer Events, as shown in Table 13 and Table 14, respectively. Table 13. Annual Coupons Initiative Participation in PY2016 Measure Number of PY2016 Coupons Redeemed Percentage Change from PY2015 ENERGY STAR-certified LEDs (general or specialty) 1,838,437 18% ENERGY STAR-certified specialty CFLs 63,654-11% ENERGY STAR-certified light fixtures (indoor hard wired) 69, % ENERGY STAR-certified ceiling fans 722-9% Indoor motion sensors/dimmer switches/timers (hard wired) 30,135 39% Weatherstripping 3,924 16% Insulation blanket for electric water heaters % Heavy-duty outdoor timers % Power bars with integrated timer and auto-shutoff % Outdoor clothesline umbrella stands or clothesline kits 1,708 61% Programmable thermostats for electric baseboard heaters (single or double packs) 1,128-44% Total* 2,011,363 19% * The percentage change in total coupons shows a comparison of the total number of PY2016 coupons to the total number of PY2015 coupons, including measures not continued in PY

30 Table 14. Biannual Retailer Events Participation in PY2016 Measure PY2016 Coupons Redeemed Percentage Change from PY2015 ENERGY STAR-certified LEDs (general or specialty) 7,208, % ENERGY STAR-certified specialty CFLs 53,046-65% ENERGY STAR-certified light fixtures (indoor hard wired) 283, % ENERGY STAR-certified ceiling fans % Indoor motion sensors/dimmer switches/timers (hard wired) 90,349 79% Weatherstripping 29,419 12% Insulation blanket for electric water heaters % Heavy-duty outdoor timers 5, % Power bars with integrated timer and auto-shutoff 3,227 68% Outdoor clothesline umbrella stands or clothesline kits 3,278-22% Programmable thermostats for electric baseboard heaters (single or double packs) 1,332-23% Total* 7,679, % * The percentage change in total coupons shows a comparison of the total number of PY2016 coupons to the total number of PY2015 coupons, including measures not continued in PY2016. Table 15 shows the program-level gross savings by measure for both the Annual Coupons Initiative and Biannual Retailer Events. As in PY2015, the biannual event accounted for most PY2016 energy savings, at 79%. The lighting measures (ENERGY STAR-certified LEDs, specialty CFLs and fixtures) comprised 98.2% of total savings for PY2016, with the LED lighting measures alone accounting for nearly 96% of the program total energy savings. The non-lighting measures comprised the remaining 2% of Coupons Program savings. 15

31 Table 15. PY2016 Coupons Program Verified Gross Savings (MWh) Measure Annual Coupons Initiative Biannual Retailer Events PY2016 Total Percentage of Total ENERGY STAR-certified LEDs (general purpose) 42, , , % ENERGY STAR-certified LEDs (specialty) 18,014 46,913 64, % ENERGY STAR-certified specialty CFLs , % ENERGY STAR-certified light fixtures (indoor hard wired) 1,221 4,511 5, % ENERGY STAR-certified ceiling fans % Indoor motion sensors/dimmer switches/timers (hard wired) 405 1,327 1, % Weatherstripping 176 1,359 1, % Insulation blanket for electric water heaters % Heavy-duty outdoor timers % Power bars with integrated timer and autoshutoff Outdoor clothesline umbrella stands or clothesline kits Programmable thermostats for electric baseboard heaters (single or double packs) % , % % Coupons Program Total 63, , , % Table 16 shows the PY2016 demand reduction by measure. Similar to the energy savings, LEDs led to about 94% of total demand reduction. 16

32 Table 16. PY2016 Coupons Program Verified Gross Demand Reduction (MW) Measure Annual Coupons Initiative Biannual Retailer Events Total Energy Savings Percentage of Total ENERGY STAR-certified LEDs (general purpose) % ENERGY STAR-certified LEDs (specialty) % ENERGY STAR-certified specialty CFLs % ENERGY STAR-certified light fixtures (indoor hard wired) % ENERGY STAR-certified ceiling fans % Indoor motion sensors/dimmer switches/timers (hard wired) % Weatherstripping % Insulation blanket for electric water heaters % Heavy-duty outdoor timers % Power bars with integrated timer and autoshutoff Outdoor clothesline umbrella stands or clothesline kits Programmable thermostats for electric baseboard heaters (single or double packs) % % % Coupons Program Total % Per-Unit Savings and Realization Rates Table 17 shows the per-unit reported and verified energy savings and demand reduction for each measure. Realization rates varied widely, due primarily to changes in the baseline, installation rate and other inputs to the energy savings algorithms. Among the lighting measures, ENERGY STAR-certified general purpose LEDs had the highest realization rate, at 144%, due to the sharp increase in per-unit saving. This was also the only realization rate among the lighting measures to exceed 100%. The realization rates for specialty LEDs, specialty CFLs and light fixtures were below 100%. In some cases, realization rates other than 100% are the result of data anomalies in the VAS report data. For example, the verified per-unit savings for ceiling fans, programmable thermostats and baseboard heaters did not change from the PY2015 evaluation to the PY2016 evaluation, but the VAS report data applied a different value. 17

33 Table 17. Gross Reported and Verified Savings and Gross Realization Rates Measure ENERGY STAR-certified LEDs (general purpose) Per-Unit Reported Annual Gross Savings Per-Unit Verified Annual Gross Savings Annual Gross Realization Rate kwh kw kwh kw kwh kw % 144% ENERGY STAR-certified LEDs (specialty) % 78% ENERGY STAR-certified specialty CFLs % 70% ENERGY STAR-certified light fixtures (indoor hard wired) % 81% ENERGY STAR-certified ceiling fans % 102% Indoor motion sensors/dimmer switches/timers (hard wired) % 71% Weatherstripping (door frame kits) % 91% Weatherstripping (foam or V-strip packages) Insulation blanket for electric water heaters % 82% % 100% Heavy-duty outdoor timers % 100% Power bars with integrated timer and auto-shutoff Outdoor clothesline umbrella stands or clothesline kits Programmable thermostats for electric baseboard heaters % 99% % 75% % 200% The Cadmus team followed the same methodology used in PY2015 to calculate verified measure savings for PY2016. We updated several measure calculation inputs using results of the PY2016 participant surveys or more recent market data (in the case of LEDs). Table 18 summarizes the changes in measure savings from PY2016 to PY2015. The per-unit savings decreased in PY2016 for most measures; however, the savings increased considerably for general purpose LEDs, which was the largest savings measure for the Coupons Program in PY

34 Table 18. PY2016 and PY2015 Measure Energy Savings Comparison Measure ENERGY STAR-certified LEDs (general purpose) PY2016 Gross Verified Values PY2015 Gross Verified Values Percentage Change kwh kw kwh kw kwh kw % 74% ENERGY STAR-certified LEDs (specialty) % -8% ENERGY STAR-certified specialty CFLs % -26% ENERGY STAR-certified light fixtures (indoor hard wired) % -12% ENERGY STAR-certified ceiling fans % 0% Indoor motion sensors/dimmer switches/timers (hard wired) % -29% Weatherstripping (door frame kits) % -10% Weatherstripping (foam or V-strip packages) Insulation blanket for electric water heaters % -19% % 0% Heavy-duty outdoor timers % 0% Power bars with integrated timer and autoshutoff Outdoor clothesline umbrella stands or clothesline kits Programmable thermostats for electric baseboard heaters % -7% % -25% % 0% The Cadmus team investigated the driving factors resulting in measure savings changes in PY2016 for the highest impact measures, which were ENERGY STAR-certified general purpose and specialty LEDs, ENERGY STAR-certified specialty CFLs and ENERGY STAR-certified fixtures. ENERGY STAR-Certified General Purpose LEDs As described above, general purpose LED energy savings per measure were 74% higher in PY2016 than in PY2015. This increase was largely the result of a change in baseline wattage between the two years. As expected given the decrease of CFL bulbs in the marketplace, the percentage of survey respondents indicating that they would have used a CFL to substitute for the coupon LED dropped substantially, from 54% in PY2015 to 29% in PY2016. Although the percentage of respondents indicating that they would substitute a non-coupon LED increased by 4%, the majority of the decrease in CFL substitutes was made up for by respondents who selected incandescents and halogens as the likely baseline. Figure 2 shows that 40% of respondents indicated that they would have substituted one of the two higher-wattage bulbs, double the percentage from PY2015. This resulted in an increase in the estimated baseline wattage from 22 watts in PY2015 to 31 watts in PY

35 Figure 2. PY2013 through PY2016 General Purpose LED Baseline Lamp Types Source: Participant Surveys, PY2013 PY2016. ENERGY STAR-Certified Specialty LEDs Unlike general purpose LEDs, the per-unit savings for specialty LEDs was 8% lower than the PY2015 value. This modest change was the net result of several minor shifts from PY2015 to PY2016. In PY2016, due to several new models being listed in the ENERGY STAR Qualified Products List, the Cadmus team added 3-way LEDs to the specialty LED category. (In previous evaluation years, 3-way bulbs were considered a CFL specialty bulb type, but were not readily available as an LED.) Because few survey respondents indicated an incandescent or halogen as the baseline product for the 3-way LED, the baseline wattage was heavily weighted to CFL and LED substitutes. As a result, the 3-way LED provided a much smaller differential between the baseline and efficient wattage than other bulb types, and reduced the per-unit savings for the specialty LED category. Since only 4% of respondents used a program coupon to purchase a 3-way LED, the overall impact on the per-unit savings was minimal. Table 19 shows key inputs to the savings algorithm for specialty LEDs that are based on the annual participant and nonparticipant survey results. (General purpose bulbs are also included in the table for comparison; however, general purpose values are not included in the Specialty LED Average shown in the table.) In addition to the reduced baseline wattage, PY2016 survey respondents also indicated a slightly lower likelihood to have already installed their product, with 81% of all coupon-purchased LEDs installed, compared to 86% the prior year. 20

36 PY2016 Survey Results Table 19. Survey-Based Algorithm Inputs for ENERGY STAR-Certified LEDs Average Baseline Watts Average Measure Watts Delta Watts Percentage Installed Measure Count Measure Distribution General purpose % 3,711 79% Flood/Reflector % 427 9% Globe/Vanity % 55 1% Candle/Chandelier % 338 7% 3-way % 165 4% Specialty LED Average % PY2015 Survey Results General purpose % % Flood/Reflector % % Globe/Vanity % 76 7% Candle/Chandelier % 104 9% 3-way N/A N/A N/A N/A N/A N/A Specialty LED Average % ENERGY STAR-Certified CFLs In PY2016, the per-unit energy savings for specialty CFLs decreased 26% from the previous year, largely due to two significant shifts in the coupon sales of specialty CFLs. First, as shown in Figure 3, there was a shift in the distribution of baseline bulb types from being a majority CFLs (56%) in PY2015 to a greater number of LEDs, and small increase in halogen baseline bulbs, in PY2016. In general, LEDs have slightly lower wattages than CFLs, so this shift resulted in a drop in the average baseline from 29.2 watts in PY2015 to 22.3 watts in PY2016. The second change was in the types of specialty CFL bulbs purchased. The share of dimmable CFLs dropped from 36% to 21% of all specialty CFLs purchased with a coupon, and the share of 3-way CFLs dropped from 17% to 12%. Instead, participants purchased more candle CFLs (18% in PY2016 compared to 8% the prior year) and globe CFLs (17% in PY2016 compared to 11% the prior year). Dimmable and 3- way CFLs had the highest delta watts (difference between the baseline watts and the efficient measure watts) of all bulb types in both PY2015 and PY2016, while candles and globes had the lowest delta watts. As a result, the overall delta watts for specialty CFLs dropped from 10.7 watts in PY2015 to 9.0 watts in PY

37 Figure 3. PY2013 through PY2016 Specialty CFL Baseline Lamp Types Source: Participant Surveys, PY2013 PY2016 ENERGY STAR-Certified Light Fixtures The per-unit savings for ENERGY STAR-certified light fixtures in PY2016 dropped by 12% compared to PY2015, primarily due to a reduction in the baseline wattage. In PY2016, 43% of survey respondents indicated that they would have installed an LED as the baseline measure for the fixture, compared to 25% in PY2015. The percentage of respondents indicating an incandescent or halogen baseline decreased correspondingly. As a result, the baseline decreased from 30.0 watts in PY2015 to 27.9 watts in PY2016. Net-to-Gross The Cadmus team used data from the participant and nonparticipant surveys to determine the NTG ratio for the program and for each region in the territory for PY2016. Table 20 shows each component of the NTG ratio for each region, as well as at the program level. 22

38 Table 20. Coupons Program Free Ridership, Spillover and NTG Ratio by Region Region Free Ridership Spillover Rebound Total NTG Ratio Percentage Gross Energy Savings (Weight) Central Ontario 15% 46% 13% 118% 8% Eastern Ontario 17% 103% 13% 173% 4% Enersource Hydro Mississauga Inc. 13% 55% 13% 130% 5% Horizon Utilities Corporation 16% 76% 13% 147% 4% Hydro One Networks Inc. 13% 66% 13% 140% 26% Hydro Ottawa Limited 14% 61% 13% 134% 6% Northern Ontario 16% 96% 13% 167% 3% PowerStream Inc. 14% 72% 13% 146% 7% Southwest Ontario 17% 79% 13% 150% 16% Toronto Hydro-Electric System Limited 13% 60% 13% 135% 18% Veridian Connections Inc. 13% 45% 13% 120% 2% Program Net-to-Gross (Weighted) 140% For the program overall and for most regions, the NTG ratio in PY2016 was lower than the NTG ratio in PY2015, as a result of market factors and changes to the NTG methodology. The average, regional NTG ratio dropped by 13%; however, the actual change by region ranged from a 32% reduction in Central Ontario to a 2% increase in Northern Ontario. Only two regions, Northern Ontario and Eastern Ontario, had an increase in the NTG ratio from PY2015 to PY2016. In all other regions, the NTG ratio decreased. As shown in Table 21, the changes were caused by shifts in both free ridership and spillover, as well as the addition of rebound. Table 21. Comparison of PY2016 and PY2015 NTG Components Component PY2016 PY2015 Free Ridership 14% 21% Spillover 67% 85% Rebound 13% N/A Total NTG Ratio 140% 164% Free Ridership The Cadmus team assessed free ridership at the region level as described in the Methodology section, and applied these values to the final coupon allocation for each LDC. Using the participant survey results, we scored free ridership based on intent and influence for each measure, by region. For LEDs and specialty CFLs, the Cadmus team used the average PY2015 measure-level free-ridership values and calculated a savings-weighted average using PY2016 estimates for per-unit savings. The team applied the resulting average free-ridership value of 15.7% to all LED and CFL savings for each region. Because these measures resulted in the majority of savings in each region, and were therefore weighted 23

39 more heavily than the estimates for other measures, free-ridership rates in each region converged to this value (see Table 20). Table 22 shows the weighted average of measure-level free ridership across regions, ranging from 2.1% for clothesline kits to 38.2% for power bars. Table 22. Program-Level Free Ridership by Measure Measure Respondent Free- Rider Savings Free Ridership ENERGY STAR-certified general purpose LEDs (n=454) 10, % ENERGY STAR-certified specialty LEDs (n=192) 3, % ENERGY STAR-certified specialty CFLs (n=165) % ENERGY STAR-certified light fixtures (indoor hard wired; n=138) 2, % Indoor motion sensors/dimmer switches/timers (hard wired; n=76) % Heavy-duty outdoor timers (n=56) % ENERGY STAR-certified ceiling fans (n=40) % Power bars (n=79) % Clothesline kits (n=32) % Programmable thermostats (n=46) % Weatherstripping (door frames) (n=51) % Weatherstripping (V-strip packages) (n=33) % Insulation for hot water tanks (n=16) % Average Free Ridership 14.2% Spillover Spillover refers to savings from program-eligibile measures purchased without a coupon (like spillover) or from purchases of non-program measures that provide energy savings, or efficient actions (non-like spillover). Savings from a measure or action are only considered spillover if the respondent was influenced by program marketing or a program activity. The Cadmus team evaluated both like and nonlike spillover from participants, and like spillover from nonparticipants. Table 23 presents the regional values for like spillover from participants and nonparticipants, the program-level value for non-like spillover from participants and the total spillover by region. While spillover remained high relative to gross savings, averaging 66% across regions, it was lower than in PY2015, when it averaged 85% across regions. Like spillover is calculated at the region level, meaning that each region s like spillover rate reflects the actual spillover purchases by survey respondents from that region. Non-like spillover, due to a lower incidence in the survey population, is calculated at the province level, and the province rate is applied uniformly to each region. More detail on these calculations is provided in the Methodology section. 24

40 Region Table 23. PY2016 Spillover by Region* Like Spillover: Participants (n=123) Like Spillover: Nonparticipants (n=681) Non-Like Spillover: Participants (n=312) Total Spillover Central Ontario 0.8% 16.5% 28.7% 46.0% Eastern Ontario 8.0% 66.0% 28.7% 102.6% Enersource Hydro Mississauga Inc. 3.9% 22.9% 28.7% 55.5% Horizon Utilities Corporation 3.4% 43.6% 28.7% 75.7% Hydro One Networks Inc. 4.3% 33.1% 28.7% 66.1% Hydro Ottawa Limited 4.0% 28.3% 28.7% 61.0% Northern Ontario 0.0% 67.3% 28.7% 96.0% PowerStream Inc. 4.1% 39.4% 28.7% 72.1% Southwest Ontario 12.0% 38.3% 28.7% 78.9% Toronto Hydro-Electric System Limited 5.0% 26.8% 28.7% 60.4% Veridian Connections Inc. 5.1% 11.7% 28.7% 45.5% Coupons Program Overall** 5.2% 31.7% 28.7% 65.5% * The n values represent the number of respondents who purchased an eligible measure without a coupon and who rated the program somewhat important or very important to their purchase decision. ** Overall values are weighted by region gross savings. Overall values are presented for information purposes only, and are not used to assess net savings. The Cadmus team assessed participant and nonparticipant status by measure. Participant spillover only resulted from individuals who purchased units of a particular measure with a coupon and also purchased units of that measure without a coupon. Across all measures, 123 respondents had purchased measures both with and without a coupon, and reported that the program was either somewhat important or very important in their decision to purchase measures without a coupon. The eligible units purchased without a coupon by these respondents resulted in savings equal to 5% of the total savings from all measures purchased with a coupon by survey respondents. Nonparticipant like spillover only resulted from participants who had purchased units of a particular measure without a coupon (and purchased none of that measure with a coupon). A total of 681 respondents indicated that they purchased a coupon-eligible measure without a coupon, purchased none of that measure with a coupon and attributed credit to the program for their purchase. The total nonparticipant spillover was larger than the participant spillover, equal to 32% of the total savings from all measures purchased with a coupon by survey respondents. Participant and nonparticipant like spillover varied significantly by region. Participant like spillover ranged from 0.0% in Northern Ontario to 12.0% in Southwest Ontario, and nonparticipant spillover ranged from 11.7% in the Veridian Connections territory to 67.3% in Northern Ontario. Among participants, the majority of spillover savings came from general purpose and specialty LEDs for every region except Southwest Ontario, which achieved similar spillover from general purpose LEDs (at 1.6% of region gross savings compared to 1.9% average), and also achieved high spillover from weatherstripping 25

41 (4%, compared to the 0.7% average) and programmable thermostats (1.8%, compared to the 0.3% average). In nine of 11 regions, general purpose LEDs had either the highest (six regions) or second highest (three regions) nonparticipant spillover savings across all products, measured as a percentage of total gross savings for the region. In seven regions, clotheslines had either the highest (three regions) or second highest (four regions) nonparticipant spillover savings. Northern Ontario had the highest total nonparticipant spillover, due to higher than average savings from heavy duty timers, representing 25.6% of region gross savings, compared to an average of 5.7% for this measure across all regions. High nonparticipant spillover in Eastern Ontario was the result of higher than average savings from specialty CFLs (10.2% compared to 2.7% average) and light fixtures (10.4% compared to 2.5% average). Table 24 shows overall like spillover from participants and nonparticipants by measure. Measure Table 24. Spillover by Measure Participant Like Spillover Nonparticipant Like Spillover Total Like Spillover ENERGY STAR-certified LEDs (general purpose) 3.9% 16.8% 20.7% ENERGY STAR-certified LEDs (specialty) 3.7% 14.2% 17.9% ENERGY STAR-certified specialty CFLs 7.1% 28.4% 35.5% ENERGY STAR-certified light fixtures (indoor hard wired) 5.7% 26.3% 32.0% Indoor motion sensors/dimmer switches/timers (hard wired) 10.7% 53.0% 63.8% Heavy-duty outdoor timers 3.5% 55.6% 59.0% ENERGY STAR-certified ceiling fans 4.4% 85.1% 89.5% Power bars with an integrated timer or auto shut-off 18.1% 90.7% 108.8% Outdoor clothesline umbrella stands or clothesline kits 4.2% 88.9% 93.1% Programmable thermostats for electric baseboard heaters 9.2% 66.2% 75.4% Weatherstripping (door frame kits) 19.0% 89.9% 108.9% Weatherstripping (V-strip packages) 16.7% 103.3% 120.0% Insulation blanket for electric water heaters 9.1% 52.3% 61.4% Program Total (weighted) 5.2% 31.7% 36.9% Rebound The Cadmus team assessed rebound (increased usage of an efficient measure, relative to usage of the less efficient measure replaced) for lighting measures and light fixtures. Forty-seven respondents who had purchased an LED, specialty CFL or light fixture through the program indicated that they used their lighting more frequently following the purchase and installation of their efficient measure. 26

42 Free Ridership Participant Like Spillover Nonparticipant Like Spillover Participant Non- Like Spillover Rebound Measurespecific NTG Factor Measure Table 25. Rebound Usage Indicated in Survey* Lighting Rebound (kwh) a ENERGY STAR-certified light fixtures (indoor hard wired) 2,521 Gross Savings from Respondent Measures Rebound ENERGY STAR-certified LEDs (general purpose) 8,386 13% 135,218 ENERGY STAR-certified LEDs (specialty) 4,677 20% ENERGY STAR-certified specialty CFLs 1,501 28% Total Participants (Survey Sample) 17, ,218 13% * Results are based on a sample of 47 respondents who indicated rebound usage. Table 26 presents all NTG ratio components by measure, which the Cadmus team calculated and applied by region. This summary table provides province-level context. Table 26. NTG Components by Measure, at the Province Level* 28% Measure ENERGY STAR-certified LEDs (general purpose) 16% 4% 17% N/A 13% 92% ENERGY STAR-certified LEDs (specialty) 16% 4% 14% N/A 20% 82% ENERGY STAR-certified specialty CFLs 16% 7% 28% N/A 28% 92% ENERGY STAR-qualified light fixtures (indoor hard wired) 24% 6% 26% N/A 28% 80% Indoor motion sensors/dimmer switches/timers (hard wired) 30% 11% 53% N/A N/A 133% Heavy-duty outdoor timers 4% 3% 56% N/A N/A 155% ENERGY STAR-qualified ceiling fans 9% 4% 85% N/A N/A 180% Power bars with an integrated timer or auto shut-off 38% 18% 91% N/A N/A 171% Outdoor clothesline umbrella stands or clothesline kits 2% 4% 89% N/A N/A 191% Programmable thermostats for electric baseboard heaters 8% 9% 66% N/A N/A 167% Weatherstripping (door frame kits) 8% 19% 90% N/A N/A 201% Weatherstripping (V-strip packages) 11% 17% 103% N/A N/A 209% Insulation blanket for electric water heaters 3% 9% 52% N/A N/A 158% All Coupon Program Measures 14% 5% 32% N/A 13% 110% *These measure-specific province-level NTG values are provided for information only and are not used to calculate program net savings. Non-like spillover is derived from non-eligible measures, and therefore is not applicable. 2.3 Process Evaluation For the process evaluation, the Cadmus team assessed the effectiveness of the program design, implementation and marketing in terms of driving customer participation. 27

43 2.3.1 Methodology The Cadmus team conducted a comprehensive review of program materials, marketing materials, reporting templates and program data to help us understand the program intent, as well as how the program activities may have impacted program delivery and customer participation. We interviewed program staff and surveyed participants to understand program operations and to gain insight into customer awareness and perceptions of the program, as well as their overall satisfaction. Table 27 shows the data collection activities we completed for our process evaluation of the Coupons Program. Table 27. Coupons Program Process Evaluation Tasks Task Audience Completes Document Review N/A N/A Stakeholder Interviews Surveys Program Manager 1 Marketing staff (cross-program) 1 Participants 617 Nonparticipants* 622 Ipsos customer satisfaction survey (CSAT survey; analysis only) 16,877 Ipsos longitudinal mass market research survey (LMMRS; analysis only) 2,881 * Nonparticipants in this table represent the number of respondents who made no purchases at all with a saveonenergy coupon. However, some respondents who were participants for one measure also answered questions as a nonparticipant for other measures. See Appendix C in Volume II for the data collection instruments. Document Review The Cadmus team reviewed the program documents listed in Table 28, which were provided by the IESO on behalf of the LDCs, to inform our development of the data collection instruments. Document Type Table 28. Coupons Program Documents Reviewed Document Name PDF Conservation First Framework LDC Toolkit, V3.0 (December 18, 2015) Workbooks Photos Website 2017 SKU list template Data reported from participating retailers Eight examples of advertising and coupon displays at participating retailers Stakeholder In-Depth Interviews The Cadmus team conducted two telephone interview with key staff from the IESO. The first interview, with the program manager, provided insight into how the program was administered in PY2016, including the role of delivery partners and the objectives driving changes to the program design and administration in PY2016 and PY2017. The second interview, with a program marketing staff, provided information on the target audience and the marketing and promotional activities over the year. 28

44 Participant and Nonparticipant Surveys The team defined a participant as a customer of a participating LDC who purchased an eligible measure using a coupon in PY2016. A nonparticipant for any particular measure was an eligible customer that did not purchase an eligible product using a coupon. A nonparticipant may have purchased an eligible product without using a coupon, or may not have purchased an eligible product at all. A participant for one measure could also be a nonparticipant for another measure. The Cadmus team completed 1,239 online surveys with the general population in Ontario, using a proprietary contact list. Of these respondents, 617 were participants and the remaining 622 were nonparticipants. With the online survey, we collected information to inform both the impact and process evaluations. For the process evaluation, we collected information about program awareness and attitudes toward energy efficiency. Longitudinal General Population Surveys The IESO contracted with Ipsos to conduct two separate online surveys on a monthly basis during PY2016. With the CSAT survey, Ipsos collected information on IESO customer satisfaction, particularly with the consumer programs. Ipsos conducted the CSAT survey the first week of each month, from January through December 2016, targeting approximately 1,600 responses in each month. With the LMMRS, Ipsos asked questions related to awareness of and attitudes toward saveonenergy and the consumer programs. Ipsos issued the LMMRS monthly from August 2016 through December 2016, for a sample of approximately 580 respondents each month. Ipsos issued both surveys to a proprietary sample, and identified Coupons Program participants through questions in the survey. Results were weighted by age and gender to be representative of the Ontario population. The Ipsos CSAT survey and LMMRS questions are available in Appendix C of Volume II Findings Findings from the process evaluation are presented below. Program Design and Delivery The PY2016 Coupons Program consisted of two separate initiatives, the Annual Coupons Initiative and the Biannual Retailer Events. The Annual Coupons Initiative allowed LDCs to deliver coupons (coded for the LDC) direct to their customers, and allowed customers to print coupons off the IESO website that can be redeemed all year long. During the Biannual Retailer Events in April and October, retailers placed limited-duration coupons (provided by Summerhill) directly in their stores and the IESO initiated a strong marketing push. These major components of the program design were carried over from the previous five-year cycle. Changes to the Program in PY2016 In PY2016, changes included eliminating some measures and reducing some coupon incentives. Specifically, the IESO eliminated pipe wrap and general purpose CFLs; reduced the coupon discount for 29

45 single or double packs of general purpose LEDs and specialty CFLs to $3, from $5 in PY2015; and added a coupon for packages of three or more general purpose LEDs or specialty LEDs. Program Responsibilities In PY2016, the IESO designed, marketed and managed the Coupons Program and executed reimbursement checks to retailers for redeemed coupons. For retailer specialist services and coupon fulfillment, which together comprise the majority of program administrative requirements, the IESO contracts with Summerhill. The IESO program manager oversees the Summerhill contract, monitors results and engages stakeholders such as the LDCs. As retailer specialist, Summerhill recruits retail partners, including obtaining their signature on the participation agreement that details the program requirements. Summerhill also provides retailers with physical packets of coupons for placement near eligible products. Summerhill staff periodically visit participating retailers to check that the retailer is complying with program rules. As the fulfillment agent, Summerhill receives batches of redeemed coupons from retailers, counts and scans them and vets them against retailer sales data. Summerhill then submits approved invoices to the IESO for payment. Summerhill submits tracking data to the IESO, which houses all program data, and manages reporting to other stakeholders. Figure 4 shows the flow of coupon fulfillment in the program. Figure 4. Flow of Coupon Usage and Fulfillment Role of Summerhill PY2016 was the first year the IESO contracted with Summerhill for the fulfillment role. According to the program manager, the IESO decided to consolidate the retailer specialist and coupon fulfillment roles under one company, in order to provide a more streamlined, positive experience for retailers. In 30

46 particular, the IESO wanted to avoid payment delays to retailers. The program manager reported that Summerhill performed well overall and successfully reduced payment delays. Retailers appeared to be more satisfied with the program, more engaged with marketing and more likely to submit timely, complete invoices and coupons for processing. However, the program manager said Summerhill did have difficulty accounting for returned items. Although the participation agreement specifies that retailers must reimburse customers only the amount the customer paid after the coupon discount, some retailers were not able to manage these transactions. These retailers then invoiced the IESO for the coupon amount for returned items, because the retailer would otherwise experience a net loss. In addition, some retailers submitted coupons for returned items, but did not claim those coupons in their invoice. Throughout the year, retailers invoiced for both more than and less than the amount of coupons provided, due to issues related to returns. In these cases, the IESO paid the retailers based on sales data. Although transitioning to a different fulfillment agent probably led to improvements in program operations, it also resulted in some fragmentation of the annual tracking data. Summerhill was onboarded in March 2016, so the previous fulfillment agent managed coupon payments under their own data management system for the first two months of PY2016. Summerhill required a month to fully launch their coupon fulfillment management systems, include coupon tracking. For that month, they hand-keyed and tracked data in spreadsheets. This shift from one system to another resulted in small discrepancies in the calculation of reported savings, and in the allocation of coupons across LDCs in the IESO s quarterly reports of the final coupons from the PY2015 year and the initial months of PY2016. The overall number of processed coupons was not affected. In order to have a consistent dataset, the Cadmus team relied on the IESO s VAS report for PY2016, published in April Program Marketing The program is marketed through several channels. Some LDCs contribute by developing their own coded versions of the coupons and delivering booklets of these coupons to their customers through the mail. Some retailers advertise the coupons and eligible products using co-brandable materials and instore signage developed by the IESO. The IESO is responsible for most program marketing, as well as for managing the IESO and saveonenergy brand. The IESO contracts advertising to a marketing firm, Rain 43. The IESO maintained a similar marketing strategy in PY2015 and PY2016, but increased overall marketing activity in PY2016 to generate more awareness of the IESO and its programs. In early PY2016, the IESO relaunched the saveonenergy brand and built on the shift to more digital ads in PY2015 by creating more digital banner ads, native site ads and videos, and distributing these through social media and YouTube. The IESO also had more radio and TV advertising in PY2016. Rain 43 tested responses to various TV and video ads, and selected those with the best response for full distribution. The IESO also helped reduce barriers to entry by adding a store locator to the website, so that customers could easily identify participating retailers. Most ads in PY2016 were issued to promote the Biannual Retailer Events. 31

47 For the Annual Coupons Initiative, advertising was limited to occasional social media posts referring customers to the IESO website to download coupons. Beginning in PY2015 and continuing in PY2016, the IESO marketing staff shifted the messaging in ads from promoting only the discount to promoting the non-energy benefits of the eligible products, especially comfort. Figure 5 shows an example of non-energy benefit messaging on in-store signage. Figure 5. Example of In-Store Signage Promoting Non-Energy Benefits Participation and Awareness Although the advertising strategy has not changed much since PY2015, IESO marketing staff noted the significant increase in usage as evidence of marketing success. Use of coupons, based on the CSAT surveys, ranged from 17% to 34% of the general population over the course of the year. Figure 6 shows the percentage of survey respondents each month who reported participating in the Coupons Program. The increase in usage is not unexpected, given that autumn is the peak time for lighting sales, generally, and coincides with the fall Biannual Retailer Event. 32

48 Figure 6. Percentage of Respondents Participating in the Coupons Program Source: Ipsos CSAT surveys, PY2016. The LMMRS showed that between 58% and 74% of customers, by region, and 67% across the territory, had brand awareness of saveonenergy. Awareness of the Coupons Program was similar, with 61% of Ontario respondents indicating they were aware of the program. However, region by region, awareness of the saveonenergy brand and the Coupons Program varied widely. For example, while awareness of saveonenergy was roughly equivalent to the Ontario average among Toronto Hydro customers, awareness of the Coupons Program was lower than average. Figure 7 shows awareness of the saveonenergy brand and the Coupons Program, by region. 33

49 Figure 7. Awareness of saveonenergy Source: PY2016 Ipsos LMMRS Question A3. saveonenergy [...] provides Ontarians with a wide variety of incentive programs and tips to help manage electricity use in the home, save money and help the environment. Having read this description... and QB1. Have you heard of [the Coupons Program] program by either name or description? As shown in Figure 8, most customers learned about the saveonenergy brand through the saveonenergy coupon or a bill insert from their electric provider, both of which were mentioned by 28% of respondents. The next most common source, direct mail, was also provided by the LDCs. However, 10 additional channels were mentioned by between 5% and 20% of respondents, showing that while not all the marketing campaigns are having a deep impact, customers are learning about the brand in numerous ways. 34

50 Figure 8. Sources of Awareness of saveonenergy by Marketing Channel Source: PY2016 Ipsos LMMRS Question A4. Where did you learn about saveonenergy? (n=2,881; multiple response) Customers learned about the Coupons Program through similar channels, with the notable difference that many more customers learned about the Coupons Program at a retail store (Figure 9). This difference demonstrates that the Coupons Program design, which intends to promote the coupons through direct placement in stores, is effective. 35

51 Figure 9. Respondents Mentioning Coupon Event Marketing Channels Source: PY2016 Ipsos LMMRS Question C1. Where did you learn about the saveonenergy Coupon Event initiative? (n=1,761; multiple response) Attitudes To test customer attitudes toward the saveonenergy brand, Ipsos included a question in the LMMRS about whether respondents agreed with a series of statements about saveonenergy. The statements ranged from very focused on energy efficiency, to more general benefits to one s self or the community. Respondents were given a 5-point scale: strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree or strongly disagree. Figure 10 shows the ratings from respondents for each statement. In general, respondents were more likely to agree with statements focused on energy efficiency, and less likely to agree with statements about general well-being. For example, 81% of respondents agreed with the statement, Following advice from saveonenergy and participating in saveonenergy programs encourages people to make energy efficient choices for their homes, while only 53% agreed with the statement, Following advice from saveonenergy and participating in saveonenergy programs helps people and businesses in Ontario be more successful. Although a larger percentage of respondents failed to agree with statements that were less focused on energy efficiency, they did not disagree, tending instead to rate themselves as neutral (neither agree nor disagree). 36

52 Figure 10. Respondent Agreement with Impacts of saveonenergy Programs Source: PY2016 Ipsos LMMRS Question A6. Based on your understanding of saveonenergy, please rate how strongly you agree or disagree with the following statements: Following advice from saveonenergy and participating in saveonenergy programs... (n=1,925) Respondent attitudes varied somewhat by region. Although respondents in most regions were generally about equally likely to agree with the statement that saveonenergy helps people make efficient choices, agreement with the statement that saveonenergy helps people and businesses be successful varied more widely, from 43% among Hydro One customers to 64% among Enersource customers. Figure 11 shows respondent agreement with these two statements at the province level. 37

53 Figure 11. Respondent Agreement with Impacts of saveonenergy Programs by Region Central (n=107) Eastern (n=102) Enersource (n=172) Horizon (n=156) Hydro One (n=340) Hydro Ottawa (n=150) Mid and Southwestern (n=105) Northern (n=106) Powerstream (n=193) Toronto Hydro (n=329) Veridian (n=166) Ontario (n=1,925) Source: PY2016 Ipsos LMMRS Question A6. Based on your understanding of saveonenergy, please rate how strongly you agree or disagree with the following statements: Following advice from saveonenergy and participating in saveonenergy programs... Customer and Market Actor Experience Ipsos conducted monthly surveys throughout PY2016 to collect information on customer response to program marketing and participation in the program. According to Ipsos data, attitudes toward the program remained fairly stable throughout the year. Most participants said they were likely to recommend the Coupons Program to a friend. Figure 12 shows that 70% to 75% of participants ranked their likelihood to recommend the program as a 8 or higher on a 10-point scale, where 10 represented being extremely likely. 43% 47% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 50% 51% 50% 50% 53% 57% 59% 56% 57% 64% saveonenergy helps people and businesses in Ontario be more successful saveonenergy encourages people to make energy efficient choices for their homes 78% 78% 80% 80% 78% 82% 83% 82% 81% 84% 88% 86% 38

54 Figure 12. Participant Likelihood to Recommend the Coupons Program Source: PY2016 Ipsos CSAT surveys. Considering your experience, how likely would you be to recommend the saveonenergy Coupons Program to a friend or colleague using a scale from 1 to 10, where 1 is not at all likely and 10 is extremely likely? (Totals may not sum to 100% due to rounding.) Some participants experienced problems redeeming their coupons during PY2016. As shown in Figure 13, between 13% and 16% of respondents reported experiencing some problem, and between 5% and 6% of respondents said their problem was not resolved. Surveys did not capture details of the problems that customers experienced. Figure 13. Percentage of Respondents Experiencing a Problem Redeeming Coupons Source: PY2016 Ipsos CSAT surveys. Did you experience any problems while participating in the saveonenergy Coupons Program and, if so, was this problem fully resolved? 39

55 Success, Challenges and Future Planning Successes PY2016 was a highly successful year in terms of participation, with the number of coupons redeemed doubling from PY2015. Even more noteworthy, the savings for the program nearly tripled, even as the NTG ratio decreased. The savings increase is due in large part to a surge in the redemption of coupons for general purpose and specialty LEDs. Savings from these measures increased by 18% in the Annual Coupons Initiative and by 245% in the Biannual Retailer Events. The program manager reported that it is not clear what is driving the increase in participation. One possibility is that as LED prices fall, the coupon is a larger percentage of the retail price, making the offer a better deal despite the coupon value also dropping in PY2016. Challenges The program manager noted that measures other than general purpose and specialty LEDs did not perform as well. Four measures had an overall decrease in total savings in PY2016: specialty CFLs, ENERGY STAR-certified ceiling fans, clotheslines and programmable thermostats for baseboard heaters. Other measures, such as heavy duty timers, had a significant increase in uptake. However, even with a 172% increase in savings contributed to the program total, heavy duty outdoor timers made up just 0.2% of program savings. The program manager reported that the more than 500% increase in savings from light fixtures may be due in large part to the single-socket can light kits, which contain an LED bulb. Although the bulb does come with an apparatus for fixing it in the ceiling, this product is essentially more about the lightbulb than a lighting fixture. However, the program manager was not sure how to distinguish fixture kits from other fixtures in a consistent and meaningful way, or whether that was necessary. Future Planning The program manager reported that although the coupon fulfillment agent was able to integrate fairly seamlessly with program operations, the act of processing millions of physical coupons continued to be difficult and expensive over the year. For this reason, among others, the program manager reported that the IESO will change the program in mid In the first half of PY2017, the Coupons Program will follow the same model of having both the Annual Coupons Initiative and Biannual Retailer Event. In the second half of the year, the IESO will replace the Coupons Program with a biannual instant discount program. Rather than rely on consumers to bring a coupon to the store or pick up a coupon in the aisle, the instant discount program will rely on retailers to automatically discount eligible products. As with the Coupons Program, retailers will submit an invoice to the IESO for the discounts. 40

56 According to the IESO program manager and the business case for the instant discount program, there are several factors driving the change to the instant discount program: An instant discount design is more cost-effective due to eliminating the costs associated with issuing, tracking and processing physical coupons. The instant discount removes barriers for customers, who will no longer need to remember their coupons. Because the discount is issued through the retailers existing point-of-purchase system, retailers will be able to provide much more accurate program data in terms of the number and type of products discounted. Instant discounts provided through online sales will expand the reach of the program to rural areas, and will make coverage more consistent across LDCs. The SKU-based instant discount can more easily be customized to the specific product, to optimize the savings achieved for each program dollar spent. The instant discount design requires retailers to identify SKUs, which Summerhill and the IESO will approve before the program launches. This process will help eliminate confusion with overlap among products like LEDs and fixtures, which had different discounts in the Coupons Program. The instant discount program will not have an annual initiative. The program manager reported that the Annual Coupons Initiative allowed for more participation, but confused both customers and retailers regarding the expiration date of coupons. In addition, according to the program manager, retailers reported that they liked the sense of urgency and importance created by the biannual limited time event. However, by making the same discount available all year, the annual event mitigated the effectiveness of the limited-time design. The instant discount program is designed to capitalize on the limited-time aspect of the Biannual Retailer Events. For PY2017, the IESO does not plan any major changes to its marketing strategy, assuming that the major drivers of customer behaviour will be similar between the Coupons Program and the instant discount program. However, IESO marketing staff reported that the general population in the province is concerned about the high price of electricity, and consumers perceive TV ads as expensive. To avoid the perception that the IESO is wasting money on advertising, they decided to discontinue TV ads for PY

57 Cross-participation with the Home Assistance Program The Cadmus team used the results of the participant and nonparticipant surveys to determine the percentage of participants in the Coupons Program who could qualify for the Home Assistance Program (HAP). HAP requires that participants meet one of the following criteria to be eligible: Have received a Low-Income Energy Assistance Program (LEAP) grant in the past 12 months Have received assistance from a specified Ontario assistance program Have a household income relative to number of people in the home is at or below the level noted in Table 29. Table 29. Income Eligibility for Home Assistance Program Number of People in Home After-Tax Household Income 1 $21,773 2 $30,792 3 $37,712 4 $43,546 5 $48,686 6 $53,333 7 $57,606 8 $61,583 9 $65, $68,852 The Cadmus team also considered respondents that had participated in a natural gas program for lowincome households as likely eligible for HAP. As shown in Table 30, 6% of surveyed participants were eligible for either LEAP or had used a low-income natural gas program. Table 30. Participant Home Assistance Program Eligibility Based on Program Participation Program Participants (n=617) LEAP 3% Natural Gas Demand-Side Management Program 1% Both 2% Total 6% Based on the survey respondents answers to questions about income and occupancy, 8.9% (lower bound) were eligible for HAP based on income, as shown in Table 31. Respondents for whom we could not definitively determine eligibility are shown in the table as Unknown. 42

58 Table 31. Participant Home Assistance Program Eligibility Based on Income Income Status Participants (n=528) Nonparticipants (n=518) Low Income 8.9% 13.5% Not Low Income 84.1% 77.6% Unknown 7.0% 8.9% Using both criteria (income and participation in a natural gas program or LEAP), 13% of participants and 16% of nonparticipants are HAP eligible. 2.4 Conclusions and Recommendations Increased LED coupon sales resulted in higher program savings in PY2016. In PY2016, retail prices for general purpose and specialty LEDs continued to decrease. In addition, manufacturers expanded the number of models available, which, coupled with dropping prices, made these products more of a mainstream option for consumers. Relative to PY2015, shoppers were less likely to report they intended to purchase a CFL in PY2016, which corresponds to a decrease in availability of CFLs across retailers. At the same time, consumers were more likely to report they intended to purchase an LED, indicating LEDs were filling some of the market vacuum as CFLs retreat. Finally, the new coupon fulfillment agent provided improved service to retailers such increased retailer communication and engagement, which may have led retailers to place more coupons near eligible products, including LEDs. Recommendation: Monitor product retail prices to ensure that the coupon discount does not drop below an acceptable minimum floor. The incentive should strike a balance between being too high, which may result in stockpiling by shoppers and depress the in-service rate (ISR), and too low, which may result in increased free ridership. The current $3 per single pack discount has contributed to high sales volumes without resulting in high free ridership. Consider the current incentive level (measured as a percentage of the retail price, rather than an absolute dollar value) as the target, and reduce the incentive (whether as an instant discount or a coupon) as needed to maintain the current balance. Changes in the lighting market led to substantial changes in per-unit savings. Because more people claimed an incandescent or halogen baseline than a CFL baseline, the baseline wattage for general purpose LEDs increased by about 9 watts from the PY2015 value. This resulted in an increase for the perunit kilowatt-hour savings by 74%. Similar baseline shifts led to decreased per-unit savings for specialty CFLs and lighting fixtures. Recommendation: Goal setting for future program years should account for gradual reductions in per-unit savings from LEDs. The IESO should anticipate that the baseline wattage for general purpose LEDs will decline over time, as LEDs become more affordable and familiar to consumers. In addition, changes in the regulated standard for general service lamps in the United States may spill over into Ontario once they take effect in The new United States regulations may cause large, international manufacturers to phase out some incandescent, halogen or CFL products in favor of more LED options for all North America, even if Canada does not adopt a similar standard. 43

59 While there are a variety of program measures, nearly all savings 96% came from general purpose and specialty LEDs. Even specialty CFLs, which have previously driven large volumes of savings, made a negligible contribution in PY2016. Nevertheless, there are separate coupons for 11 additional measures, in different forms (single and multi-packs) that require disproportionate marketing, processing and evaluation efforts compared to the savings they generate. Recommendation: As the IESO transitions from the Coupons Program to the Instant Discount Program, consider limiting the number of available products to those that generate the most savings; in particular, LEDs. Changes in the NTG methodology and survey responses led to a decrease in the NTG for most regions. PY2016 regional NTG estimates dropped by 13% on average from the PY2015 regional values, with decreases in all but two regions. Two factors accounted for most of the reduction. First, we estimated less spillover than in previous years. Second, the Cadmus team added analysis of rebound usage to the NTG ratio formula. Rebound usage reduces the NTG ratio, but had not previously been evaluated. 44

60 3.1 Program Description 3. Heating and Cooling Program Through the HVAC Program (marketed as saveonenergy s Heating & Cooling Incentive), the IESO has provided incentives for energy-efficient HVAC equipment since PY2006. Residents (and businesses with residential-type systems) who purchase qualifying new or replacement equipment are eligible for a program incentive if the equipment is installed by a participating contractor. The program operates province-wide and is managed by the Heating, Refrigeration, and Air Conditioning Institute (HRAI) of Canada. HRAI operates in a dual capacity as the program implementation contractor and as a national association representing HVAC and refrigeration equipment manufacturers, distributors and contractors. HRAI has been supported by Summerhill, an energy efficiency engagement contractor, since early PY2016. Summerhill manages the program data processing systems and manages incentive payments. The IESO offered the following incentives through the HVAC Program in PY2016: Heating: $250 incentive for installing a qualifying electric, natural gas, propane or oil-fueled furnace equipped with an ECM Cooling: $250 incentive for installing a CAC with a rating of at least 14.5 SEER and 12.0 EER $400 incentive for installing a CAC that meets or exceeds 15.0 SEER and 12.5 EER As shown in Table 32, the program achieved a PY2016 net annual energy savings of 76.2 GWh and a net summer peak demand reduction of 22.4 MW. Compared to PY2015, this is an overall increase in net energy savings and decrease in net demand reduction. Table 32. Summary of PY2016 HVAC Program Results Program and Track Level Metric PY2016 Verified PY2015 Verified Program Reported Savings (GWh) 96.6 Not Available Realization Rate (%) 112% Not Available Gross Verified Demand Reduction (MW) Gross Verified Annual Energy Savings (GWh) Gross Verified Lifetime Energy Savings (GWh) 2, ,068.8 Net-to-Gross Ratio 70% 51% Net Peak Demand Reduction (MW) Net Peak Demand Reduction Persisting at 2020 (MW) Net Annual Energy Savings (GWh) Net Energy Savings Persisting at 2020 (GWh) Net Lifetime Energy Savings (GWh) 1, ,

61 The IESO offered three HVAC Program measures in PY2016, though only two of those measures were eligible for savings. The 14.5 SEER CAC measure was not eligible for savings in PY2016 due to a change in the baseline assumption for CAC measures: Starting January 1, 2016, the baseline increased from 13.0 SEER to 14.5 SEER. The measures offered in PY2016 are shown in Table 33. The IESO chose to continue with the 14.5 SEER CACs until July 2017 for several reasons: it did not want to disengage with customers who were interested in this measure, removing measures from the program is a complex change management process and the IESO thought that there were relatively few of these measures in the program. Table 33. HVAC Program Measure Counts and Incentive Levels Measure Measure Count Incentive Amount CAC SEER ,538 $250 CAC SEER ,119 $400 Furnace with ECM 77,960 $250 Total 136, Impact Evaluation For the impact evaluation, the team determined verified net energy savings and demand reduction and compared these to the reported (program tracked) impacts Methodology The Cadmus team used engineering analysis to determine gross verified impacts for the HVAC Program using the EM&V Protocols as an overarching framework for the evaluation. The team used algorithms from the Uniform Methods Project (UMP) 4 to evaluate energy savings for the CAC measure. We evaluated furnace with ECM measures by adapting the deemed savings approach of the previous evaluator. After determining the gross verified impacts, we applied the NTG ratio, estimated through participant survey responses, to compute net verified savings. The team derived summer peak demand 4 U.S. Department of Energy, National Renewable Energy Laboratory. Uniform Methods Project: Determining Energy Efficiency Savings for Specific Measures. Available online: Chapters 4 and 5 of the UMP are directly relevant to the HVAC Program evaluation and provide the recommended approach to evaluating savings. 46

62 reduction using the alternative 5 summer peak demand factors provided by the IESO, 6 as shown in Table 34, and applying it to the energy savings. Table 34. Load Profile: Alternative Peak Demand Factor Load Profile Name Alternative Summer Peak Demand Factor AC_Central 0.112% Furnace_ECM_ % Furnace_ECM_ % Furnace_ECM_ % Ventilation_And_Circulation 0.011% Reported Savings The Cadmus team obtained reported savings by summarizing information from the IESO s VAS report and PIA list. As the report shows information at the project level, it was necessary to use the PIA list to disaggregate savings to the measure level to develop measure-level realization rates. This approach was successful for program energy savings, which matched at both the project and measure levels. However, due to inconsistencies in measure-level demand reduction within the VAS report, the team s disaggregated demand reduction does not match the VAS reported demand reduction. As shown in Table 35, the discrepancy in reported savings between the program tracking data and the Cadmus teams measure-level disaggregation was approximately 43.7 kw. Table 35. Comparison of Reported Savings by Savings Source Source of Savings Reported Savings kwh kw VAS Report 96,613,606 29,032.2 Prescriptive Input Assumptions 96,613,606 28,988.5 Difference Gross Verified Savings The Cadmus team used program tracking data, algorithms from the UMP and adapted the deemed savings methodology used in previous evaluations to determine gross verified savings for the HVAC 5 6 The EM&V Protocols outline an alternate approach for estimating summer peak demand reduction of weather-sensitive measures. The load impact characteristics of these measures improve coincidence with the system peak, because the system peak is also weather sensitive, so the alternate approach provides a more accurate estimation of peak demand reduction. The alternate approach uses a weighted average of the maximum monthly demand reduction in each of the three summer months instead of the average of the entire block of hours. File name: CE Tool Load Profiles with 8760; Load Profiles tab. 47

63 Program. Equipment specifications in the program database included unit type, model number, efficiency level and measure installation locations. One of the key inputs required for evaluating the CAC measure cooling capacity was not available in the program database. As such, we used the AHRI reference number to look up the capacity in the AHRI database. The team identified the cooling capacities for approximately 90% of the CAC measures rebated in PY2016. We used a program average capacity for those projects where we could not determine the cooling capacity. Approximately 500 CAC projects included neither a SEER value nor a cooling capacity, and the team assigned them each the average savings for the measure. Similarly, approximately 900 of the furnace with ECM projects did not include one of the key inputs for evaluation heating fuel type so the team assigned these projects the average savings for the measure. The Cadmus team developed verified savings for each unit of equipment rebated through the program, accounting for the overall size and efficiency, as well as replaced and new equipment type and fuel. The team used key information from the program tracking data such as CAC SEER, furnace fuel types and the type of equipment installed to determine equipment-level savings. Wherever possible, we also verified the inputs recorded in the program tracking data against the AHRI database. Central Air Conditioners The Cadmus team evaluated gross verified savings for the CAC measures using the approach outlined in Chapter 4 of the UMP, 7 in conjunction with the SEER 14.5 baseline the IESO adopted in PY2016. Beginning on January 1, 2016, the IESO updated the baseline assumption for CACs to 14.5 SEER because of the change in ENERGY STAR requirements. Prior to January 1, 2016, the IESO had used the provincial efficiency standard of 13.0 SEER. This shift reduced the energy savings and demand reduction achieved from installing units that meet or exceed 15.0 SEER and eliminated savings from SEER 14.5 units. Key savings inputs such as equipment efficiency and cooling capacity were provided in the program tracking data or identified as part of the evaluation. As listed in Table 36, the Cadmus team also leveraged savings inputs from previous IESO program evaluation reports and field studies as appropriate. We used the following algorithm to evaluate savings: 1 1 kwh Saved = (Cooling Capacity) ( ) (EFLH) SEER baseline SEER installed The specific inputs used in the PY2016 evaluation are provided in Table U.S. Department of Energy Chapter 4: Small Commercial and Residential Unitary and Split System HVAC Cooling Equipment-Efficiency Upgrade Evaluation Protocol. 48

64 Table 36. Gross Savings Inputs for Central Air Conditioners Variable Description Value Source kwh Saved Energy savings Program average is 31.6 kwh Calculated Cooling Capacity Rated cooling capacity (Btuh) Varies by project. Program average is 25,628 Btuh Cadmus team evaluation SEER baseline SEER of baseline unit 14.5 SEER PY2015 program evaluation SEER installed SEER of installed unit Varies by project. Program average is 15.5 SEER Tracking data EFLH Equivalent full-load cooling hours EFLH PY2015 program evaluation The team evaluated demand reduction in two steps. First, we checked for consistency between the values the IESO provided for alternative peak demand factors in Table 34 and the value we derived using the 8,760 hourly load shapes and alternate definition of peak 8 from the EM&V Protocols for weathersensitive measures. Second, we applied the verified alternate summer peak demand factor to the energy savings to derive summer peak demand reduction. We found the values in Table 34 to be consistent with our derivation of the alternate summer peak factors and used those values in our analysis of demand reduction. High-Efficiency Furnace with Electronically Commutated Motor The Cadmus team s approach to evaluating savings for the furnace with ECM measure was consistent with the methodology used in the PY2015 evaluation. We relied on a report published by the Canadian Centre for Housing Technologies (CCHT) 9 that used energy metering and energy simulation to estimate the savings of ECM furnace fan motors in the Canadian climate. The study estimated the electric savings resulting from replacing a non-ecm motor with an ECM fan motor, and accounts for additional natural gas consumption resulting from the more efficient motor operation. For situations where the newly installed furnace represented a change in primary heating fuels from electricity to fossil fuel, we derived additional electric savings and natural gas penalties due to fuel switching based on the natural gas consumption of a high-efficiency furnace. The Cadmus team supplemented the CCHT study with research available in previous IESO reports, including an update in the PY2015 evaluation report that reduced the estimated energy savings associated with ECM motors. This update is based on the IESO metering field study conducted in PY2013 and PY2014, which revealed that ECM fan motors save less energy than originally proposed in the CCHT study. 8 9 The alternate definition of summer peak uses a weighted average of the monthly maximum load reduction for June through August (from 1:00 p.m. to 7:00 p.m.). Natural Resources Canada. Final Report on the Effects of ECM Furnace Motors on Electricity and Gas Use: Results from the CCHT Research Facility and Projections. Last modified August 3, Available online: 49

65 The Cadmus team calculated savings for the furnace with ECM measure in two parts, using the algorithm below. kwh Saved = (ECM Savings) (1 Adjustment Factor) + (Furnace Fuel Switching Savings) (10.345) Where: kwh Saved = Kilowatt-hours saved ECM Savings = Deemed ECM savings based on participant characteristics as modelled in the CCHT study and updated to reflect the findings of the PY2013 and PY2014 IESO field study Adjustment Factors = Factor to adjust for ECM savings that are already accounted for in AHRI matched-pair furnace and air conditioning systems Furnace Fuel Switching Savings = For participants who switch from an electric to natural gas heating systems, deemed savings are based on furnace natural gas consumption as modelled in the CCHT study = Conversion factor to adjust CCHT study furnace savings from cubic metres of natural gas to kilowatt hours Following the equipment segmentation scheme used in the previous evaluation, the Cadmus team recreated 54 savings permutations based on the output of the CCHT study models, participant characteristics derived from participant surveys and adjustment factors that account for AHRI matched pair systems (more details are provided in Table 37). The key participant characteristics we used to develop the separate permutations are outlined below. Fuel Type. The team used three categories of fuel type: participants who continue to use a natural gas furnace, participants who continue to use an electric furnace and participants who switch from an electric furnace to a natural gas furnace. All three scenarios are eligible for ECM savings; however, furnace savings due to fuel switching are only applicable to those customers replacing an electric furnace with a fossil fuel furnace. Similar to previous evaluations, the Cadmus team also evaluated a heating penalty for customers who installed a fossil fuel furnace to account for increased fossil fuel consumption (as a result of a more efficient fan that produces less waste heat). Construction Vintage. The team used two different housing categories: homes built after 1980 (newer homes) and those built before 1980 (older homes). End Use. The end-use characteristic indicates whether the participant received a rebate for a CAC system in addition to the furnace with ECM measure rebate; this information is available in the VACS report. The team developed savings permutations for scenarios where the customer HVAC system includes only heating components and for scenarios where the customer system includes both heating and cooling components. The savings for these permutations differ 50

66 because homes with CAC systems have higher furnace fan usage than homes with only a furnace. Fan Setting. The factor with the largest impact on ECM savings, fan usage, is not recorded in the program tracking data. The three fan settings are: (1) continuous operation both pre- and postretrofit, (2) non-continuous or automatic operation both pre- and post-retrofit and (3) continuous operation pre-retrofit and automatic operation post-retrofit. The ECM savings permutations also include adjustment factors to account for furnace and CAC matched pairs. These matched pairs are air handler and condenser units that are tested together during the AHRI certification process. The testing process accounts for the energy consumption of the furnace fan motor and interactive effects between more efficient fan operation and increased heating or decreased cooling load. Thus, the savings associated with these equipment are reduced slightly to avoid double counting of savings. The adjustment factors used are shown in Table 37. Table 37. Derating Factors for AHRI-Matched Central Air Conditioner and Furnace Pairs Construction Type Fan Usage AHRI-Matched Derating Factor Existing New Continuous 1.7% Non-Continuous 13.1% Continuous to Non-Continuous 1.6% Continuous 2.1% Non-Continuous 22.5% Continuous to Non-Continuous 1.7% Since information on participant fan setting and home vintage is not currently collected, the Cadmus team was unable to assign equipment savings directly from the 54 savings permutations. Instead, we combined the 54 permutations using participant survey results to create nine savings permutations, which we then applied to the furnace with ECM projects (shown in Table 38): The percentage of newer homes (54%) versus older homes (46%) The percentage of customers who did not receive a CAC rebate in PY2016, but already have a CAC installed at their home (79%) The percentage of customers who operate their furnace fan continuously both pre- and postretrofit (23%), the percentage who operate their fan non-continuously both pre- and postretrofit (75%) and the percentage who change their fan operation from continuous to noncontinuous (2%) 51

67 Fuel Type Non-Electric to Non-Electric Electric to Non-Electric Electric to Electric HVAC System End Use Table 38. Furnace with ECM Measure Permutations AHRI- Matched Set Quantity Rebated* Weighted Unit Savings (kwh) Weighted Unit Peak Savings (kw) Increased Natural Gas Consumption (m 3 ) Heat and cool Matched 32, Heat and cool Unmatched 1, Unknown Unmatched 41, Heat and cool Matched , , Heat and cool Unmatched 45 25, , Unknown Unmatched 1,465 24, , Heat and cool Matched Heat and cool Unmatched Unknown Unmatched * This table only includes projects with recorded furnace characteristics. For approximately 900 furnace projects, the Cadmus team assigned a measure average savings and demand reduction of 1,366.8 kwh and kw, respectively, because detailed measure information was not available in the program tracking database. The team evaluated demand reduction in two steps. First, we checked for consistency between the values the IESO provided for alternative peak demand in Table 34 and the value we derived using the 8,760 hourly load shapes and alternate definition of peak from the EM&V Protocols for weathersensitive measures. Second, we applied the verified alternate summer peak demand factor to the energy savings to derive summer peak demand reduction. We found the values in Table 34 to be consistent with our derivation of the alternate summer peak factors and used those values in our analysis of demand reduction. The team made one deviation from the PY2015 methodology, specifically for the scenario where a nonelectric furnace replaced an electric furnace. Previously, the summer peak demand reduction was calculated based on the entirety of the unit s savings, which included components for savings resulting from the ECM as well as savings due to fuel switching. However, the furnace should not be providing heat during peak hours of the summer, so the Cadmus team calculated PY2016 demand reduction for these permutations as the amount resulting from the ECM installation only. Net Savings The Cadmus team derived net savings those savings directly attributable to the program by adjusting the gross savings using the NTG ratio, as noted in the following equation: Net Savings = NTG Ratio * Gross Savings NTG ratio estimates serve a critical role in impact evaluations, allowing program administrators to determine the portion of gross energy savings influenced by and attributable to their programs. The Cadmus team used telephone surveys to collect data from participants to determine free ridership, 52

68 spillover and rebound, and to assess program delivery and participant satisfaction. The Process Evaluation section of this chapter includes a more detailed description of the survey methodology. The team calculated the NTG ratio based on free ridership, spillover and rebound as shown in the following equation: NTG Ratio = 1 Free Ridership + Spillover Rebound Free Ridership Free ridership refers to the likelihood that a participant would have installed energy-efficient equipment without program assistance. Self-report surveys are commonly used to estimate free ridership in downstream rebate programs by assessing participants perception of the program s influence on their decisions, and their intentions to install energy-efficient equipment or practice energy-efficient behaviours had the program not existed. Each component (intention and influence) is scored from 0% to 50%; the sum of the two components is the total free-ridership score, which ranges from 0% to 100%. This approach is consistent with the PY2015 HVAC Program evaluation, although we worded some of the questions and response options differently. Intention. The Cadmus team assessed HVAC Program participants intentions to determine the likelihood they would have purchased and installed the energy-efficient equipment without program support. To do this, we developed free-ridership questions and possible response combinations that resulted in an intention score specific to the HVAC Program. Table 39 presents the intention freeridership questions, possible responses and intention scores for the HVAC Program. Question Q1. Did your existing equipment require immediate replacement or could you have repaired your equipment or postponed the purchase for a year or more? Q2. [If immediate] Which of the following would you most likely have done if there were no Heating and Cooling Program? Table 39. HVAC Program Free Ridership Intention Component Scoring Immediate replacement Response Could have repaired or postponed the purchase Intention Score (maximum of 50%) Go to Q2 Go to Q3 Purchased less expensive or less efficient equipment 0% Purchased whatever equipment was recommended by contractor Purchased the exact same equipment and paid the full cost without the rebate 50% (100%-CAI%)* 50% Don t know 25% 53

69 Question Q3. [If postpone] Which of the following would have you have most likely done if there were no Heating and Cooling Program? Response Cancelled or postponed the purchase at least one year Intention Score (maximum of 50%) Kept or repaired the old equipment 0% Purchased less expensive or less efficient equipment within one year Purchased whatever equipment was recommended by my contractor within one year Purchased the same equipment and paid the full cost without the rebate within one year 0% 0% 50% (100%-CAI%)* 50% Don t know 25% * The Cadmus team adjusted this value by a contractor adjustment influence (CAI) score of 76% for furnace measures and 78% for CAC measures, derived from the PY2015 contractor interviews regarding the level of program influence on contractor recommendations. Reporting an alternative that likely would have produced no energy savings results in an intention score of 0%; reporting something that likely would have produced some, but lower, energy savings than the incented purchase results in an intention score of 25% and reporting the same outcome as the incented purchase results in an intention score of 50%. Influence. The Cadmus team assessed HVAC Program influence on participants decisions to purchase and install the energy-efficient equipment. To estimate program influence, we asked respondents to identify how the following program elements influenced their decisions about the energy efficiency measure they implemented: Program incentives Contractor recommendation 10 Marketing and/or advertising about the program LDC or IESO information about energy efficiency in general Previous experience with another energy efficiency program from the LDC or IESO High program influence and free ridership have an inverse relationship: the greater the program influence, the lower the free ridership (see Table 40). In other words, if respondents cited at least one element of the program as very important, we assumed they took an energy-efficient action they would not have taken without the market intervention; therefore, the program participant received a low or 10 If the contractor recommendation influence was the highest scoring program element, we adjusted the score by the value of CAI from PY2015 multiplied by 50% to account for the program s influence on contractors. The team applied a CAI of 76% for furnace measures and 78% for CAC measures. 54

70 zero influence free-ridership score. Table 40 shows how the Cadmus team rated and scored HVAC Program influence. Table 40. HVAC Program Free Ridership Influence Component Scoring Maximum Program Influence Rating Influence Score 1 Not at all important 50.0% 2 Not very important 37.5% 3 Somewhat important 12.5% 4 Very important 0.0% Don t know 25.0% Final Free Ridership. After quantifying the intention and influence scores, the Cadmus team summed them to compute the overall free-ridership score for each participant, with each score representing 50% of the total free-ridership score. The final free ridership for each participant ranges from 0% (no free ridership) to 100% (total free ridership). Spillover We used the following equation to calculate the percentage of spillover, at the measure level, in which the spillover savings reported by respondents is divided by the total verified gross savings achieved by all survey respondents in the measure category: Spillover % = Spillover Measure Energy Savings for All Survey Respondents Program Measure Energy Savings for All Survey Respondents We calculated spillover savings for survey respondents who indicated that they installed additional energy savings measures after participating in the HVAC Program. The Cadmus team determined spillover savings by multiplying the measure savings by a percentage that reflects the program s influence on the respondent s decision to install additional measures. The team asked participants who purchased energy-efficient measures, but did not receive a rebate, to use a four-point scale to rate the degree of program influence on that purchase: (4) very important, (3) somewhat important, (2) not very important or (1) not at all important. We assigned 100% of the measure savings to those measures that received the very important rating, 50% of the savings to those with a somewhat important rating and 0% savings to those with a not very important or not at all important rating. We calculated measure savings values for each individual spillover measure using the best information available from the PY2015 Consumer Programs Evaluation, the PIA list or ENERGY STAR calculators. Rebound The final component in the NTG ratio is the percentage of rebound generated by participants. Rebound occurs when a consumer increases their usage of more efficient products because of their awareness that the new product is more efficient. For example, someone who purchases a high efficiency CAC may set their thermostat to a lower temperature in the summer because they know the new CAC is more efficient. 55

71 The Cadmus team evaluated rebound using participant survey responses to questions about whether and how a customer s energy usage behaviour may have changed after they purchased the new unit. We specifically asked whether the customer had changed thermostat setpoints or increased the furnace fan runtime. Direct Job Impacts The Cadmus team determined direct job impacts by asking interviewed IESO staff and program implementers about the number of full-time employees they each have as a result of the HVAC Program Impact Findings The next sections detail the verified per-unit and program-level gross impact findings. Gross Impact Findings Overall program participation increased in PY2016 across each of the three measure categories, as well as in the overall number of projects submitted to the program. The largest increase in participation, both in absolute and relative terms, was realized by the CAC SEER 15 measure. Approximately 9,445 more customers installed a CAC SEER 15.0 in PY2016 than in PY2015, yielding a 22.1% relative increase. The furnace with ECM measure also saw moderate growth, with an additional 4,661 customers installing a new furnace in PY2016 (6.4% increase over PY2015). Despite not being eligible for savings under the new baseline assumptions, the CAC SEER 14.5 measure also realized a 1.9% increase in participation in PY2016 (see Table 41). Measure Table 41. HVAC Program Participation by Measure Measure Counts Total Projects* PY2016 PY2015 Relative Change PY2016 PY2015 Relative Change CAC SEER ,538 6, % CAC SEER ,119 42, % Furnace with ECM 77,960 73, % 100,975 93, % Total 136, , % 100,975 93, % * A project may include one or more measures. Table 42 shows annual reported and verified per-unit energy savings, demand reduction and realization rates for the CAC and ECM measures. Measure-level energy realization rates varied from 111% to 189%, resulting in a program gross realization rate of 112%. Measure-level demand realization rates varied from 103% to 360%, resulting in a program gross realization rate of 108%. 56

72 Table 42. Measure Counts, Gross Reported and Verified Savings and Gross Realization Rates Measure Measure Counts Per-Unit Reported Annual Gross Savings Per-Unit Verified Annual Gross Savings Annual Gross Realization Rate kwh kw kwh kw kwh kw CAC SEER , CAC SEER , % 360% Furnace with ECM 77,960 1, , % 103% Overall 136, % 108% Furnaces with an ECM was the largest measure category in PY2016, accounting for 99% of the programreported savings and approximately 57% of equipment rebated through the program. These proportions are consistent with the previous evaluation and are similar for the verified savings. Figure 14 shows a comparison of the PY2016 proportions of measure counts, energy savings and demand reduction results for each measure. Figure 14. Proportion of Measures Rebated, Energy Savings and Demand Reduction by Measure The Cadmus team estimated savings for every reported piece of equipment through a combination of engineering algorithms and application of deemed savings. Energy and demand realization rates realization rates varied from 100% for the following reasons: Verified savings for the furnace with ECM measure were slightly higher than reported due to the specific measure mix installed in PY2016. Approximately 2.7% of customers switched from an electric furnace to a fossil fuel furnace; while this represents a small proportion, the installations 57

73 include electricity savings from fuel switching, which is a significant contribution to the program energy savings. Verified savings for the CAC SEER 15.0 measure were higher than reported. This is a result of higher than actual savings assumptions for the baseline in the program planning documents, leading to lower than actual savings. The reported savings used approximately an 82% reduction in energy savings as a result of the shift from a baseline of 13.0 SEER to 14.5 SEER. Based on the size and efficiency of equipment installed in PY2016, the actual reduction is closer to 66%. The realization rate for the CAC SEER 15.0 measure is 360% because of the program planning assumptions. When compared to the PY2015 evaluation, the reduction in verified demand for PY2016 is consistent with the reduction in measure energy savings. Table 43 shows a comparison of the year-overyear savings by measure. Table 43. Comparison of PY2016 and PY2015 Energy Savings and Demand Reduction by Measure Measure Per-Unit Savings PY2016 Per-Unit Savings PY2015 Percentage Change kwh kw kwh kw kwh kw CAC SEER % -100% CAC SEER % -66% Furnace with ECM 1, , % -46% Overall % -50% Overall, the average per-unit energy savings decreased by approximately 11% and the average per-unit demand reduction decreased by approximately 50% between PY2015 and PY2016. The measure-level savings vary for the following reasons: CAC SEER 14.5 measures did not receive any savings in PY2016 due to a change in the program baseline assumptions. Starting January 1, 2016, the IESO increased the baseline assumption for CACs from 13.0 SEER to 14.5 SEER. As a result, the CAC SEER 14.5 measure was not eligible for energy savings or demand reduction in PY2016. Energy savings and demand reduction for the CAC SEER 15.0 measure decreased by approximately 66% between PY2015 and PY2016, primarily due to the change in baseline assumptions, but also to the installed equipment capacity. The 2015 evaluation evaluated savings using an installed unit size of 26,000 Btuh; in PY2016, we calculated the average capacity from the AHRI database model number lookup and determine unit size, as 25,628 Btuh. The 58

74 baseline shift accounted for an approximate 66% decrease in savings while the change in capacity accounted for roughly 1%. 11 Energy savings for the furnace with ECM measures decreased by approximately 4% from PY2015 levels, primarily due to a change in the measure mix. The proportion of fuel-switching customers was slightly lower in PY2016 than in PY2015, at 2.7% versus 2.9%, respectively. This is a seemingly small change, but the impact on overall per-unit savings is significant due to the larger savings associated with fuel switching. Demand reduction for the furnace with ECM measures decreased by approximately 46% between PY2015 and PY2016, almost entirely due to a change to the savings calculation for fuelswitching installations. Previously, the summer demand reduction for fuel switching applications was calculated off the entire possible load of the system (connected load of the furnace and the furnace fan). For the PY2016 evaluation, we updated the estimated summer demand reduction to reflect only furnace fan savings during the summer. Central Air Conditioners The IESO offered two CAC measures through the HVAC Program in PY2016, though one measure was not eligible for savings under the updated program baseline assumptions. Starting on January 1, 2016, the IESO updated the baseline assumption for air conditioning efficiency from 13.0 SEER to 14.5 SEER. This adjustment was prompted by an update to the ENERGY STAR standard, which requires that CACs meet 15.0 SEER for inclusion in the program. As a result of the baseline adjustment, the SEER 14.5 CAC measure was not eligible for savings and the savings associated with the SEER 15.0 measure decreased. To be eligible for a rebate, CACs submitted to the program must meet or exceed 14.5 SEER. To be eligible for savings in PY2016, the equipment must meet or exceed 15.0 SEER. Most CAC measures rebated through the program in PY2016 met or exceeded the requirement of 15.0 SEER, though a few projects did not qualify for savings. Overall, the team identified 11 projects submitted as 15.0 SEER CACs that did not meet the program efficiency requirements. We did not determine verified savings for these projects. Of the equipment that did qualify for savings, most rebated measures (96.9%) were between 15.0 and 17.0 SEER. Approximately 2.7% of equipment was between 17.0 and 19.0 SEER and 0.4% of projects rebated exceeded 19.0 SEER. As the 14.5 SEER CAC measure was not eligible for savings, the Cadmus team did not include them in the efficiency distribution. Overall, we determined the average CAC 11 Note that these factors are interrelated, and it is challenging to disaggregate the precise contribution due to each of these factors individually. However, when the factors are applied to the PY2015 savings, the expected per-unit savings for PY2015 are approximately the same as the verified value of 31.6 kwh. The team used the same EFLH value in PY2016 as we used in PY2015, so the EFLH Factor is 0: Expected Savings = (PY2015 Verified Savings) * (Capacity Factor) * (Efficiency Factor) * (EFLH Factor) Expected Savings = (94.0 kwh) * ( ) * ( ) * ( ) = 31.6 kwh 59

75 efficiency level as 15.5 SEER, similar to the 15.6 value determined in PY2015. Figure 15 shows the distribution of efficiency levels within the 15.0 SEER CAC measure category. Figure 15. Efficiency Distribution for 15.0 SEER Central Air Conditioner Measure The Cadmus team performed a similar review of the equipment size (capacity), with the distribution shown in Figure 16. Nearly half (46.4%) of the equipment rebated through the program was between 24,000 and 30,000 Btuh (2.0 and 2.5 cooling tonnes). Nearly 95% of the equipment was smaller than 36,000 Btuh (3.0 cooling tonnes) and the average equipment capacity was approximately 25,600 Btuh (2.14 cooling tonnes). Figure 16. Capacity Distribution for 15.0 SEER Central Air Conditioner Measure 60

76 High-Efficiency Furnace with Electronically Commutated Motor The ECM measure had the largest per-unit measure savings and the largest impact on the overall program energy realization rate. Savings for this measure are driven by a small group of fuel-switching customers that replaced their electric furnaces with a fossil fuel furnace. In these installations, the claimed fuel switching savings range from approximately 70 to 450 times larger than the savings associated with the ECM. These installations represent approximately 2.7% of furnace with ECM installations, but account for approximately 49% of the energy savings associated with the measure. Figure 17 shows the proportion of program equipment, energy savings and demand reduction attributable to each of the three furnace with ECM pre- and post-retrofit fuel combinations. Figure 18 shows the average per-unit savings by furnace fuel permutation. Figure 17. Energy Savings and Demand Reduction Proportions by Furnace Fuel Type Permutation Note: The Electric to Electric permutation is not visible because it represents a very small percentage of total measures (0.15%), energy savings (0.01%), and demand reduction (0.03%). 61

77 Figure 18. Average Per-Unit Energy Savings by Pre- and Post-Retrofit Furnace Fuel Type The electric to non-electric permutations have the largest per-unit electric energy savings; however, the majority of these savings are not the result of an ECM alone. The savings for these measures is so high because a working electric furnace is removed from service and replaced with a fossil fuel furnace. These measures represent a fuel switching scenario in which the decrease in electric consumption is offset by an equivalent increase in natural gas consumption. The energy savings resulting from installing an ECM are approximately the same for the electric to non-electric measure permutations as for the non-electric to non-electric measure permutations. The balance of savings for the electric to non-electric measure permutations are a result of fuel switching. For this reason, the electric to non-electric measures have the largest per-unit electricity energy savings, and also represent the largest increase in per-unit natural gas consumption. Though the electric to non-electric measures comprise a significant portion of furnace with ECM energy savings, the verified demand reduction is similar to the other fuel type permutations and comprises a small proportion of total measure demand reduction. This is different from the previous evaluation, in which demand reduction associated with this fuel type permutation were large relative to the other permutations, and is due to a change in the peak demand reduction calculation. Previously, the summer peak demand reduction for fuel switching applications was calculated using the entire possible load of the furnace system (connected load of the furnace and the furnace fan). For the PY2016 evaluation, the Cadmus team calculated the summer demand reduction to reflect only furnace fan operation during the summer. As discussed previously, the overall impact of this change was a 46% decrease in demand reduction for the furnace with ECM measure between PY2015 and PY

78 Six of the nine furnace permutations we used include natural gas penalties resulting from the installation of a new furnace with ECM. This factor accounts for additional heating fuel consumption from the more efficient furnace fan motor. In a typical furnace assembly, the fan motor is embedded in the furnace cabinet, and as air is moves through the system, it absorbs heat given off by the fan motor. ECM fans are more efficient than traditional motors, so more of the input energy is converted into useful work spinning the fan and moving air through the home and less waste heat is created. The overall heating load of the home does not change as the result of an ECM installation. Rather, the source of the heat generation shifts. ECM fans generate less waste heat than non-ecm fans, and the difference in waste heat generation is compensated for by the furnace itself. The reduction in waste heat is compensated for through an increase in furnace output the burner fires longer than it would have without an ECM resulting in increased heating fuel consumption by the furnace. The natural gas penalty resulting from installing an ECM is the same for all six of the affected fossil fuel permutations (-59.3 cubic metres of natural gas), but these values represent a weighted average of three different savings values shown in Figure 19. The increase in natural gas consumption is driven by the occupant s operation of the furnace fan in the pre- and post-retrofit scenarios, as well as the operating characteristics of those fans. Figure 19. Increased Furnace Fuel Consumption by Fan Operation 63

79 The three fan operation scenarios the team considered are outlined below. Fan operation information is not available at the project-level, so the Cadmus team used survey data to estimate the percentage of program participants in each category. Continuous to Non-Continuous. Following the installation of the new furnace, the customer changes how they operate their furnace fan, shifting from continuous operation to noncontinuous operation. This represents an overall decrease in fan run time and the ECM fan can operate at lower speeds, both of which lead to less waste heat generation from fan operation. This scenario represents the largest increase in natural gas consumption, at 209 cubic metres. Continuous to Continuous. The occupant runs their furnace fan continuously in both the preand post-retrofit scenarios. The fan runs for the same amount of time as it did previously, but at lower speeds. When running at lower speeds, the ECM fan produces less waste heat than a non- ECM fan. This scenario represents the second largest increase in natural gas consumption, at 180 cubic metres. Non-Continuous to Non-Continuous. The occupant runs their fan non-continuously in both the pre- and post-retrofit scenarios. This scenario is similar to the continuous to continuous scenario, with the primary difference being the amount of time the fan is operating. In this noncontinuous to non-continuous scenario, the fan runs intermittently rather than continuously. The ECM fan runs at lower speeds than the non-ecm fan, resulting in less waste heat generation. This scenario represents the smallest incremental increase in natural gas consumption, at 25 cubic metres. Net-to-Gross Findings For the PY2016 evaluation of the HVAC Program, the Cadmus team used responses from surveys with PY2016 participating consumers and information from interviews with contractors in PY2015 to derive NTG ratio estimates for the program. We used survey questions to assess consumers purchase intentions in the absence of an IESO incentive and to measure program influence. We used the interviews conducted in PY2015 with program contractors to assess the extent to which the program influenced their practice of recommending qualifying equipment. We used established algorithms, previously used in PY2015 and described in the Methodology section, to determine free ridership and spillover for furnace and CAC program measures. The Cadmus team weighted the measure-level NTG ratio estimates by the verified population energy savings to arrive at an overall program-level NTG estimate of 70%, as shown in Table 44. Table 44. HVAC Program Free Ridership, Spillover and NTG Ratio by Measure Measure Category n Free Ridership Spillover NTG Ratio Total Program Verified MMBtu Savings Central Air Conditioner % 33% 100% 1,658,437 Furnace % 1% 70% 106,555,962 Overall % 1% 70% 108,214,399 64

80 The NTG for each region, in aggregate, is shown in Table 45. Table 45. HVAC Program Net-to-Gross Results by Region Region NTG Number of Projects Enersource Hydro Mississauga Inc. 71% 5,747 Horizon Utilities Corporation 71% 4,872 Hydro One Networks Inc. 70% 19,597 Hydro Ottawa Limited 70% 6,620 PowerStream Inc. 71% 10,312 Toronto Hydro-Electric System Limited 70% 12,465 Veridian Connections Inc. 70% 2,770 Central Ontario 71% 11,518 Eastern Ontario 70% 6,696 Northern Ontario 70% 2,335 Southwestern Ontario 70% 19,133 Overall 70% 102,065 Free-Ridership Findings As described in the Methodology section, the Cadmus team asked how participants purchasing decisions would have differed in absence of the program. Table 46 shows the distribution of responses to the question: Which of the following would you have most likely done if there were no Heating and Cooling Program rebate? We used the responses to determine each participant s final intention score, then we averaged these individual scores to arrive at overall free-ridership estimates for the furnace and CAC measures. 65

81 Table 46. HVAC Program Free Ridership Intention Score by Measure Category (n=284) Intention Question Response Options Intention Score CAC Furnace Cancelled or postponed the purchase at least one year 0% 0 0 Purchased less expensive or less efficient equipment within one year Purchased whatever equipment was recommended by contractor within one year Would have purchased the exact same equipment and paid the full cost without the rebate within one year 50% * (CAI%) = 11% for CAC, 12% for furnace* 0% % Would have stuck with or repaired the old equipment 0% Don t know 25% Refused 25% 2 3 Total Average Intention Score 28% 26% * The team adjusted this value by a CAI score of 78% for CAC measures and 76% for furnace measures, derived from the PY2015 program contractor interviews, and which reflects the level of program influence on contractor recommendations. Table 47 shows the distribution of responses to the question: I m going to read a list of factors that you may have considered when buying the energy-efficient [MEASURE] that you had installed. For each one, please indicate how important it was in your purchase decision using a number from 1 to 4, where 1 means not at all important, 2 means not too important, 3 means somewhat important and 4 means very important. From responses to this question, we obtained data about the importance of equipment rebates, recommendations from the contractor, information about energy efficiency from a utility or the IESO and marketing or advertising of the program on the customers purchasing decisions. Response Options 1 - Not at all important 2 - Not very important 3 - Somewhat important 4 - Very important Table 47. HVAC Program Free Ridership Influence Responses by Measure Category (n=284) Influence Score Heating and Cooling Program Rebate Contractor Recommendation Program Marketing and/or Advertising Information from Utility or the IESO about Energy Efficiency in General CAC Furnace CAC Furnace CAC Furnace CAC Furnace 50.0% % % % Don t know 25% Average Rating

82 The Cadmus team used the maximum rating given by each participant for any factor in Table 47 to determine their individual influence score, presented in Table 48. We weighted individual influence scores by the measure respective total survey sample verified gross savings to arrive at savings-weighted average influence scores by measure category. Table 48. HVAC Program Influence Free-Ridership Score (n=284) Influence Rating Influence Score Maximum Influence Rating CAC Furnace 1 - Not at all important 50.0% Not very important 37.5% Somewhat important 12.5% Very important 0.0% Average Maximum Influence Rating Average Influence Score 5% 5% The team then summed the intention and influence components to estimate the total average freeridership by measure. The higher the free-ridership score, the more savings are deducted from the gross savings estimates. Table 49 summarizes the intention, influence and free-ridership scores for each measure category. Table 49. HVAC Program Intention and Influence Free-Ridership Scores by Measure Category Measure Category n Intention Score Influence Score Free-Ridership Score CAC % 5% 33% Furnace % 5% 31% Spillover Findings Twenty-one participants reported installing a total of 51 high-efficiency measures after participating in the program; these respondents did not receive an incentive and said participation in the program was important in their decision to install additional measures. The measures types to which we attributed spillover savings included high-efficiency clothes washers, dishwashers, refrigerators, water heaters, insulation, furnaces, thermostats, CACs and high-efficiency windows. We calculated savings for spillover measures based using the best information from either the PY2015 Consumers Program Evaluation, the PAI list or ENERGY STAR calculators. We divided the total survey sample spillover savings for each measure category by the gross program savings for measure category survey sample to obtain the measure category spillover estimates, shown in Table

83 Measure Category Table 50. HVAC Program Spillover Estimates by Measure Category Survey Sample Spillover kwh Savings Survey Sample Program kwh Savings Percentage Spillover Estimate CAC 1, , % Furnace 1, , % Overall, the program was very important or somewhat important in 21 participants decisions to install additional high-efficiency equipment for which they did not receive a rebate. The Cadmus team assigned 100% of the measure savings to those measures that received the very important rating, 50% of the savings to those with a somewhat important rating and 0% savings to those with a not very important or not at all important rating. Table 51 shows these additional spillover measures and the total resulting energy savings by measure category. Measure Category CACs Furnace Table 51. HVAC Program Spillover Measures, Quantity and Savings Spillover Measures Quantity Total kwh Savings CFLs Furnace PIA list kwh Source PY2015 Consumer Programs Evaluation report Humidifier Added to Furnace ENERGY STAR* LEDs Consumer Program Weatherstripping Evaluation Report Windows CMUA POU TRM** Overall 8 1,516.4 N/A Central Air Conditioner PIA list CFLs Dishwasher Furnace PY2015 Consumer Programs Evaluation report PIA list Humidifier Added to Furnace ENERGY STAR* Water Heater Pipe Wrap Water Heater Tank Blanket Weatherstripping Windows Overall 15 1,300.1 N/A PY2015 Consumer Programs Evaluation report * U.S. Environmental Protection Agency (EPA). ENERGY STAR Market & Industry Scoping Report Residential Humidifiers. Available online: ENERGY_STAR_Scoping_Report_Residential_Humidifiers.pdf ** California Municipal Utilities Association. POU Technical Resource Manual. Available online: 68

84 Rebound Findings The Cadmus team did not find evidence of rebound based on the results of the participant survey. The team asked respondents to quantify whether their winter thermostat setpoints had changed since installing their new HVAC equipment (Figure 20). The team found that of the survey respondents who increased their heating setpoint in the winter, the CAC respondents actually increased their heating setpoints more than furnace with ECM respondents; thus, we cannot attribute the increase in setpoint to the installation of the furnace. Figure 20. Reported Change in Winter Thermostat Setpoint Source: Participant Survey Questions 13e and 13f. How did you change your [thermostat] settings? (n=46) and What were your thermostat settings [before/after] the change in the winter? (n=15) Similarly, the team found that none of the new CAC respondents lowered their cooling setpoints in the summer. Based on these findings, the team found no evidence of rebound due to increased heating or cooling operation. The Cadmus team also looked at whether participants increased their furnace fan usage following the installation of a new furnace with ECM. As shown in Figure 21, 42% of respondents said that their furnace fans operated the same as they had previously and 41% of respondents indicated that their furnace fans operated less than they had previously. While the remaining 18% did increase usage, on average this would be offset by the 41% who are operating less. Based on these responses, the team did not calculate any rebound for either the CAC or furnace with ECM measures. 69

85 Figure 21. Reported Change in Fan Operation When the HVAC System is Neither Heating Nor Cooling Source: Participant Survey Question 13l. Since installing the system would you say the amount of time your fan runs when not heating or cooling has...? (n=159) Direct Job Impacts According to feedback from our interviews with the IESO, HRAI and Summerhill, the program added 10.2 direct full time jobs to support the HVAC Program, as illustrated in Table 52. Organization Table 52. HVAC Program Job Impacts Number of Jobs Percentage of Time Spent on HVAC Program Full Time Equivalent IESO 3 40% 1.2 HRAI 3 100% 3 Summerhill 6 100% 6 Total 10.2 Because the Cadmus team did not interview retailers or LDCs for this evaluation, we were unable to estimate the number of jobs created in those organizations as a result of the HVAC Program. 3.3 Process Evaluation Methodology The Cadmus team designed the process evaluation to answer the following questions: How effective was program marketing at generating participant awareness and engagement? What motivated customers to participate in the HVAC Program? How effective was the administration and delivery of the HVAC Program? To what extent did the program adhere to its design processes? 70

86 How effective was the overall HVAC Program design? How can future programming for the residential sector be improved? The Cadmus team used data from three main sources for the process evaluation: a review of program materials, in-depth interviews with key stakeholders and participant surveys. We designed the in-depth interviews with the IESO, HRAI and Summerhill staff to provide us with a comprehensive understanding of the program. The participant survey measured the influence of the program on purchase decisions and behaviour, and collected data on equipment and home characteristics, as well as customer awareness and satisfaction. The materials review provided details on the program and its eligibility requirements. Table 53 shows the number of interviews and surveys the team completed for the process evaluation. Table 53. HVAC Process Evaluation Tasks Task Role Completed interviews/surveys Stakeholder Interviews IESO Program Lead 1 Program implementers (HRAI and Summerhill) 2 Participant Surveys Program participants 214 See Volume II of this report for the data collection instruments. Document Review The Cadmus team reviewed the program documents listed in Table 54 to inform the development of the data collection instruments and provide context on program changes in PY2016 and in the future. Document Type IESO program rules Program redesign documents PY2015 evaluation report Program website Table 54. HVAC Program Documents Reviewed Document Name or Link saveonenergy.ca/consumer/programs/hvac-rebates/getting-started.aspx saveonenergy.ca/consumer/programs/hvac-rebates/terms---conditions.aspx saveonenergy. Conservation First Framework LDC Toolkit Draft version 2. March 16, Research Into Action. Final Report, 2015 Consumers Program Evaluation Volume 1. September 1, saveonenergy.ca/consumer/programs/hvac-rebates.aspx Data files VACS report, 2015 project true-up file and PIA list updated in March 2017 Quarterly Tracking Report (Q2) Ipsos. Residential Customer CSTAT Survey, Heating and Cooling Incentives Q Quarterly Tracking Report. July Stakeholder In-Depth Interviews The Cadmus team conducted three telephone interviews with key staff from the IESO, HRAI and Summerhill. These interviews provided insight into how the HVAC Program was delivered, changes to 71

87 the program, the relationship between the various parties responsible for managing and implementing the program and challenges and successes. Participant Surveys In April 2016, the Cadmus team surveyed 214 customers who received incentives from the program in PY2016. We used the survey results to assess program awareness, purchasing considerations, program satisfaction, delivery and marketing, NTG and customer demographics. We randomly sampled customers in regions with 200 or more participants, and used a census for regions with fewer than 200 participants. We called participants until we reached approximately 70 completes for each measure category (Tier 2 CAC, furnace with ECM and furnace with ECM and Tier 2 CAC), as shown in Table 55. We did not survey customers who received an incentive for SEER 14.5 CAC units due to low participation and the fact that these units do not receive savings. Table 55. Survey Completes by Customer Category and LDC Reporting Region Region Customers in Sample Tier 2 CAC (SEER 15) Furnace Equipped with ECM + Tier2 CAC Furnace Equipped with ECM Central Ontario Eastern Ontario Enersource Hydro Mississauga Inc. Total Horizon Utilities Corporation Hydro One Networks Inc Hydro Ottawa Limited Northern Ontario PowerStream Inc Southwest Ontario Toronto Hydro-Electric System Limited Veridian Connections Inc Total 3,

88 3.3.2 Findings After analyzing the results of our interviews with program managers and stakeholders, as well as participant surveys, we came to several conclusions: Onboarding of Summerhill and introducing a new data processing system in PY2016 did not appear to cause significant disruption to program operations Contractor marketing efforts are the primary way customers learn about the program, and contractors are a critical element to influencing customers to adopt energy-efficient heating and cooling equipment Customer satisfaction with the program is high, although when customers expressed dissatisfaction it was primarily in regards to rebate processing issues Program Design and Delivery The program is managed by the IESO, who contracts with HRAI and Summerhill to implement the program. According to the IESO, the implementation team s division of labour works well: The IESO sets program strategy, conducts marketing, manages implementers and resolves escalated compliance issues with contractors or customer HRAI is responsible for recruiting and retaining equipment contractors, for assisting them with claims and incentive processing and for determining equipment eligibility Summerhill manages the data processing system, prepares reports on the disposition and number of customer applications processed, prepares status reports for the IESO and HRAI and processes customer rebate payments 73

89 In PY2016, the IESO contracted with Summerhill to manage the HVAC Program application and rebate processing data. As part of this change, Summerhill introduced a new Internet-based application processing system, which was launched in March The data management and rebate processing system includes the following steps and quality control checks, and it typically takes between six and eight weeks for an application to be processed (from the time an application is initiated by a contractor to the time a rebate check is issued): 1. The contractor submits an application online, providing customer and equipment information Upon receipt of the application, Summerhill flags it as pending in the database, and a. Cross-references submitted equipment model numbers against a list of program-qualified equipment and flags applications with duplicate equipment serial numbers or customer addresses as declined b. If Summerhill flags applications as declined, it identifies errors and notifies HRAI, who then follows up with contractors directly to resolve issues 3. Upon resolution of declined flags or passing quality control checks, Summerhill flags applications as submitted and sends an or mail notification to customer asking them to upload or mail their equipment proof of purchase in order to complete the application 13 (errors at this step can again trigger declined flags that require resolution with a customer or contractor) 4. After the application is complete, Summerhill flags it as IESO payment/approval and sends a weekly report with these applications to the IESO for approval 5. After the IESO approves the applications, Summerhill mails rebate checks to customers on behalf of the IESO Summerhill generates a series of reports throughout the data management process described above. In addition to the weekly report of declined applications provided to HRAI, Summerhill provides the IESO with several reports: Weekly updates on the program, including applications processed, rebates processed, applications that are pending and declined and submitted flags Reasons for declined applications 14 Applications awaiting approval Contractors must submit the entire application at one time; they cannot save and return to an application before submitting it. Customers can mail or upload their application materials online. This report helps the IESO understand if errors are administrative or egregious (such as if a contractor installs equipment that is not qualified in the program). 74

90 In addition, the IESO, HRAI and Summerhill communicate regularly by telephone to discuss program management issues. HRAI also regularly updates the IESO on the volume of communications it has with contractors. Contractors HRAI is responsible for managing, recruiting and retaining the large network of contractors who are, according to the IESO, key influencers and a central design element of the program. According to HRAI, new contractor recruitment occurs via customer or contractor inquiries, rather than through active outreach. Contractors provide advice to customer, initiate the online application process (including inputting pre- and-post-installation equipment details) and provide the necessary proof of purchase that customers need to receive their rebates. According to HRAI, almost 80% of licenced contractors are registered to participate in the program. According to program tracking data (VAS report), 1,374 contractors completed 102,065 projects in PY2016. The number of projects completed by each contractor varied greatly, from only one project to one contractor who completed 4,305 projects. Although 52% of program contractors completed 20 or fewer projects, these projects accounted for only 6% of the total volume completed in PY2016. Five percent of contractors completed over 300 projects each in PY2016: these contractors completed over half the total volume of projects in PY2016. These numbers illustrate that although a large number of individual contractors are involved with the program, the program is primarily implemented by a relatively small group of contractors. HRAI registers (or re-registers) contractors every time the IESO develops a new program contractor participation agreement. The program released the latest participation agreement in March 2016, and plans to release the next iteration of the agreement with the program s re-design in July While HRAI maintains communication with contractors via a range of channels, including quarterly newsletters, posts on an online message board and training sessions, HRAI staff also exchange s directly with contractors on an individual basis. According to HRAI, the volume of weekly s never falls below 60, can rise to about 300 when a new agreement is introduced and reached approximately 1,000 when Summerhill introduced the new online application system. The HRAI provides an orientation training for newly registered contractors 15 as well as required training on CAC installations. The orientation training is focused on program logistics and outlines how the program operates. While the orientation training is mandatory for first time program contractors (and for all contractors when significant changes to the program occur), it is also available at any time to any contractor who is registered with the program. The CAC installation training is also required and is 15 The orientation training is required the first time that contractors register for the program, but not necessarily every time that that a new participation agreement is released, unless there are significant changes to the program design. When the HVAC Program launches a new program element in July 2017, HRAI will re-train all participating contractors. 75

91 intended to ensure quality control when contractors install equipment rebated through the HVAC Program. HRAI is redesigning the CAC installation training based on planned program changes and to include information on how to engage with customers regarding energy efficiency practices. In addition to ongoing communications between HRAI and contractors, HRAI collects feedback from HVAC Program contractors and customers via surveys. According to the IESO, challenges related to working with contractors include ensuring that they read and understand the HVAC Program materials, use collateral when selling equipment and that they do not use aggressive sales tactics that can erode customer confidence in the program. HRAI can recommend that a contractor be audited if they are identified as regularly not installing qualifying equipment. The IESO can terminate contractors participation agreement, and has done so in the past. In addition to contractors, HRAI engages with equipment distributors and retailers to update them on program-qualifying equipment to ensure that contractors have access to this equipment. While the HVAC Program implementers do not formally recruit them into the program, HRAI maintains an distribution list of approximately 30 distributors and retailers. Customers Existing Equipment, Program Awareness and Motivation Most customers (92%) who installed a new CAC already had cooling equipment (5% of whom had room air conditioning units). As in previous evaluations, the Cadmus team asked respondents about the age and disposition of their existing air conditioning equipment. As illustrated in Table 56, 87% of customers replaced equipment that was older than 10 years and 66% of customers replaced equipment that was either totally broken or did not work properly. Table 56. Customer Existing Air Conditioning Equipment Condition and Age* It was totally broken and no longer provided cool air It did not work properly or as good as it used to, but still provided cool air It was in good working condition Percentage One to five years old 1% 0% 2% 3% Six to 10 years old 3% 3% 6% 11% Eleven to 15 years old 8% 11% 8% 27% Sixteen years or older 15% 25% 19% 60% Total percentage 27% 39% 34% 100% Source: Participant Survey Questions 5 and 6. Which of the following best describes the age of the air conditioner you were using before you installed your new air conditioner? (n=123) and Which of the following best describes the condition of the previous air conditioning system that you replaced? (n=124) * Totals in the table may not sum due to rounding. We also asked survey respondents why they chose to replace their air conditioning system. As illustrated in Figure 22, the most frequently stated reasons for replacing an air conditioner were to save money (21%), to take preventative action before a unit failed (18%) and to replace broken equipment (14%). 76

92 Figure 22. Participant Reasons for Replacing Air Conditioning System Source: Participant Survey Question 7. Why did you decide to replace your air conditioning system? (n=119; multiple response) The Cadmus team also asked respondents about the age and condition of the furnace they replaced. As illustrated in Table 57, 90% of the heating equipment that customers replaced was over 10 years old and 61% of the equipment replaced was either broken or not working properly. Table 57. Customer Existing Heating Equipment Condition and Age It was totally broken and no longer provided heating It did not work properly or as good as it used to, but still provided heat It was in good working condition Percentage One to five years old 0% 0% 0% 0% Six to 10 years old 2% 3% 5% 10% Eleven to 15 years old 4% 10% 7% 21% Sixteen years or older 11% 31% 27% 69% Total percentage 17% 44% 39% 100% Source: Participant Survey Questions 5 and 6. Which of the following best describes the age of the furnace you were using before you installed your new furnace? (n=123) and Which of the following best describes the condition of the previous furnace that you replaced? (n=124) 77

93 The Cadmus team also asked survey respondents why they chose to replace their furnace. As illustrated in Figure 23, the most frequently stated reasons for replacing a furnace were to replace an old unit before it broke (28%), to save money (21%) and to replace broken equipment (11%). Figure 23. Participant Reasons for Replacing Furnace Source: Participant Survey Question 4. Why did you decide to replace your heating system? (n=124; multiple response) In PY2016, the IESO only conducted online marketing for the HVAC Program. Nonetheless, 27% of respondents knew about equipment rebates before contacting their contractor and the remaining customers learned about equipment rebates from their contractor. The Cadmus team asked customers how they found their contractor. As illustrated in Figure 24, most customers found their contactor via a personal connection, either through a family member or friend (24%) or because they had a prior relationship with that contractor (18%). Other ways customers most frequently found their contractor was through Internet research (14%), the phone book (9%) or a local advertisement (9%). 78

94 Figure 24. How Customers Found Their Contractor Source: Participant Survey Question 11a. How did you find your contractor? (n=205; multiple response) We asked customers about the influence of their contractor s recommendation, the rebate, information about energy efficiency from their utility or the IESO and program marketing on their decision to purchase their new heating and cooling equipment. Figure 25 illustrates the percentage of customers who rated each of these factors as very influential on their purchase decision. The contractor s recommendation was the most influential factor for both CAC and furnace purchases and the rebates were the second most influential factor. Program marketing or advertising had relatively little influence. 79

95 Figure 25. Very Influential Factors in Customer Purchase Decisions Contractor recommendation (n=140, 141) 51% 48% Rebate (n=141, 143) 40% 41% Information from my utility or the IESO about energy efficiency (n=138, 142) 21% 32% Marketing or advertising about the program (n=134, 140) 12% 21% 0% 10% 20% 30% 40% 50% 60% Central Air Conditioner Furnace Source: Participant Survey Questions 17a-18d: Indicate how influential the following was in your purchase decision:...? Customers also provided feedback about what they would have done without the HVAC Program. Because customers have different options, depending on whether they immediately needed to buy a new system, the Cadmus team first asked if their existing system required immediate replacement or if they could have repaired the equipment or postponed the purchase of a new system for one year or more. As illustrated in Figure 26, most program participants who installed a furnace or CAC could have replaced their system later or could have repaired their existing system. 80

96 Figure 26. Heating and Cooling System Replacement Need Source: Participant Survey Questions 14a and 14b. Did your existing furnace require immediate replacement or could you have repaired your equipment or postponed the purchase of a new system for a year or more? and Did your existing central air conditioner require immediate replacement or could you have repaired your equipment or postponed the purchase of a new system for a year or more? Most customers stated that without the HVAC Program they would have either purchased the same system or would have purchased whatever system their contractor recommended. Sixteen percent of the customers who did not have to immediately make a purchase said that they would have continued to use their equipment or would have repaired that equipment (Figure 27). 81

97 Figure 27. Customer Action in Absence of the Heating and Cooling Program Source: Participant Survey Questions 15 and 16. Which of the following would you have most likely done if there were no Heating and Cooling Program? (each question asked of participants whose system was not in need of immediate replacement and was in need of immediate replacement, respectively) Customer Experience Customers expressed high levels of satisfaction with the HVAC Program. As shown in Figure 28, at least 70% of respondents rated themselves as very satisfied with the equipment, the HVAC Program overall and the contractor s installation work. Customer were slightly less satisfied with the contractors sales process, although 66% still rated themselves as very satisfied. Customers expressed significantly less satisfaction with the time it took for them to receive their rebate and with the electricity savings after installing their new equipment. 82

98 Figure 28. Customer Satisfaction with Program Elements Source: Participant Survey Question 20. How satisfied are you with the...? The Cadmus team asked customers who rated themselves as a little satisfied or not at all satisfied with any category to explain why they expressed dissatisfaction. The most frequently cited driver of dissatisfaction was rebate processing delays, mentioned by 43% of respondents (Figure 29). Other frequently noted drivers of dissatisfaction were contractor performance (21%) and concerns about the efficiency of the equipment (10%). 83

99 Figure 29. Customer Reasons for Dissatisfaction Source: Participant Survey Question 21. What in particular were you dissatisfied with, and why? (n=82) The following are some customer quotes about their dissatisfaction with rebate processing: It took a long time to get the money from the rebate. I only found out about the rebate through the contractor and he took care of all the paperwork. Therefore, their marketing isn t that good. [The dissatisfaction came from] the length of time it took for the rebate to arrive. But also not being able to get an answer from the company that it [the rebate] was coming from due to the person being overwhelmed. Why couldn t it have been deposited electronically? It took a long time to get it [the rebate]. I had to phone the contractor and ask if they had applied for it. And they had, it was just slow in coming. In some cases, respondents stated concerns about rebate processing and contractor performance at the same time. For example, one customer said, We had to go back in a few times for putting in information on the application. I felt the contractor put that on us. The following is an example from one customer about concerns with the efficiency of their new system: I thought I would see more of an improvement. It s difficult to compare the rates of electric because the prices have increased year over year. While the comments above describe some of the reasons for customer dissatisfaction, customers were generally very satisfied with the program, and 96% said they would either recommend the HVAC Program to a family member or friend or had already done so (Figure 30). 84

100 Figure 30. Customer Intention to Recommend the Heating and Cooling Program Source: Participant Survey Question 33: Have you, or would you, recommend the saveonenergy Program to a friend or colleague? (n=208) At the end of the survey, we asked program participants to offer closing comments about their experience with the HVAC Program. As illustrated in Figure 31, 33% of customer comments (the largest majority of responses) indicated that they were satisfied with the HVAC Program. Additionally, 19% of comments were suggestions that the IESO expand program marketing and 11% were expressing satisfaction with the rebates. Figure 31. Categories of Customer Closing Comments Source: Participant Survey Question 34. Do you have any other comments for us? (n=82) 85

101 The following are examples of customer comments expressing satisfaction with the program: Keep the program going. It s a valuable program to those where money does make a difference, and if we can help make people make energy-efficient decisions and be environment-cautious, it s always a good thing. It was a very easy process everything was very clear and it came relatively quickly so I was very pleased with it overall. The following are examples of customer comments suggesting an expansion of marketing efforts: You need to get your name out there; most people don t know about [the program]. I pay a lot of attention to the news pertaining to HVAC, but it isn t out there. If people knew they could save, people might proceed right away [with] reducing their carbon footprint. I wish [the program was] more advertised. If the information was more available to every household, it would be really great. Success, Challenges and Future Planning While the HVAC Program has no formal participation or energy-savings goals, the IESO program manager and the program implementers said the program performed well in PY2016, despite the challenge of introducing a new data management platform. The stakeholders shared the following observations about the PY2016 HVAC Program: Processed over 100,000 applications, greatly exceeding expectation of 60,000 applications Met goals of processing rebates within four to eight weeks of receiving the application (four weeks if no issues are identified in the application and eight weeks if errors were resolved) Maintained high customer satisfaction (per Ipsos customer survey completed in PY2016) Introduced the new online data processing system without disrupting program processes Stakeholders also outlined several challenges, including that the 14.5 SEER CACs do not generate savings and therefore cause cost-effectiveness issues for the program. This measure will be phased out of the program in PY2017. Additionally, the IESO plans to remove furnaces from the program at the end of PY2017, due to increases in the measure s baseline performance. Additionally, while two stakeholders noted that the introduction of the new data processing system went smoothly, one said it caused a significant hiccup and presented contractors with a steep learning curve. Other challenges noted were the large volume of paperwork that Summerhill must process, since many materials received from the customer for the application are in paper form 16 and applications sometimes contain errors that must be resolved. 16 Both contractors and customers have the option of submitting documents by , fax or mail. Customers often submit documents by mail or fax, requiring Summerhill to scan and upload documents. 86

102 While the program staff made no significant changes to measure offerings in PY2016, the IESO plans to introduce a suite of new measures to the HVAC Program in July 2017: Regular and cold climate air-source heat pumps Regular and cold climate ductless heat pumps Retrofit variable speed fan motors for existing furnaces Circulator pumps with variable speed fan motors Adaptive thermostats 18.0 SEER and 19.0 SEER CACs HRAI noted that it hopes to increase contractor participation in the program. 3.4 Conclusions and Recommendations Inconsistencies in the VACS report resulted in discrepancies in the total reported program savings. The Cadmus team identified inconsistency issues within the VACS report, specifically regarding the number of significant figures recorded for demand reduction values. As a result, the final reported demand reduction from PY2016 does not match the demand reduction in the VACS report, by a difference of approximately 44 kw. Recommendation: Update the program tracking database so that each piece of equipment is entered as an individual line item, rather than having each line correspond to a project. The IESO requires that evaluators report claimed and verified savings at the measure level, but the program data is reported at the project level. Attempting to disaggregate the reported projectlevel information into measure-level results is prone to error and, as the program begins to offer more measures, this issue will compound. Recommendation: Check the program tracking database for consistency with respect to savings assignments and ensure that values match the PIA list. The program tracking data does not include all the primary inputs necessary for program evaluation. Currently, the information collected for the SEER 15 CAC measure are adequate to check whether the equipment qualifies for a program rebate. However, one of the key inputs for the savings evaluation cooling capacity is not tracked in the program data. The Cadmus team identified cooling capacities for most equipment rebated through the program based on the recorded AHRI reference number, but this process is not always successful. Recommendation: To accurately report and verify measure savings, include the key specifications for each measure in the program tracking data. As new measures are added to the program, confer with the Cadmus team to identify key specifications to track. Opportunities exist to expand customer awareness of the program. Only 27% of the customers knew about the HVAC Program before contacting a contractor and 19% of customer comments suggested that the program expand its marketing effort. While some customers replaced equipment that was in good 87

103 working order (34% for CACs and 39% for furnaces), most sought to replace either completely broken or malfunctioning equipment. Making customers aware of program offerings and the economic and environmental benefits of efficient heating and cooling equipment could increase the proportion who replace older and less efficient equipment before it experiences problems. Recommendation: Expand the online marketing of program offerings, including new measured offered in PY2017, to include bill inserts and other effective marketing techniques, such as direct mailings. Ensure that contractors have collateral and training to explain the benefits of new measure offerings in PY2017. Contractors are a key driver of the program. Survey data indicate that most customers hear about the program from a contractor, that most customers replaced equipment with existing problems, that customers seek out contractors that are either known to them or a known to an acquaintance of theirs and that a contractor recommendation is very influential in a customer s decision to upgrade equipment. Customers are likely to continue to engage heating and cooling contractors for specific, existing reasons, and that engagement presents the primary opportunity for a customer to learn about the program and the benefits of energy-efficient equipment. HRAI stated that approximately 80% of eligible contractors are registered with the program and that it does not actively recruit contractors to the program. Recommendation: Prioritize the recruitment of unregistered contractors into the program by identifying unregistered contractors, identifying reasons they have not registered and developing a strategy to bring them into the program. Recommendation: Conduct research to understand if contractors face barriers in marketing the program and identify ways in which the IESO can further assist them to promote program measures and rebates. 88

104 4.1 Program Description 4. Residential New Construction Program Launched in PY2011, the RNC Program encourages residential homebuilders to include energy-efficient design and technologies in new and substantially renovated homes. Eligible buildings had to meet several qualifications: Be located in Ontario Be three or fewer stores in height Have a building area not exceeding 600 square meters Be used for residential occupancy Be connected to or behind the metre of another electricity consumer connected to the LDC distribution system There are no RNC Program eligibility requirements for the type of primary heating systems: houses heated with electric and fossil fuel systems are eligible. Residential homebuilders can receive incentives to participate in three RNC Program tracks: The Prescriptive track, which offers pre-established incentive amounts for preapproved products installed. The Performance track, which provides three pre-established incentives for homes receiving either an ENERGY STAR certification, an EnerGuide rating of 83 or 84 or an EnerGuide rating of 85 or more from a NRCAN-certified energy auditor. o o The NRCAN ENERGY STAR for New Homes Standard 17 (ESNH Standard) establishes the ENERGY STAR certification minimum requirements, granting an RNC incentive if a house meets the ESNH Standard Core BOP for Ontario. For EnerGuide houses, efficiency ratings are determined using the NRCAN HOT2000 energy modelling software. The Custom track, which uses pre-established calculation worksheets, basing incentive amounts on the level of energy performance improvement over the Ontario Building Code. Table 58 presents measures eligible for incentives under each RNC Program track. 17 Natural Resources Canada. ENERGY STAR for New Homes Standard. Version April Available online: 89

105 Table 58. Residential New Construction Program Measures and Incentives by Track Track Measure Incentive Prescriptive Performance Custom All-off switch $50.00 Lighting control products (timers, motion sensors, dimmers) $3.00 per Product ENERGY STAR-certified CACs $30.00 ENERGY STAR-certified niche lighting: Recessed lighting Under-the-counter lighting LED lighting ENERGY STAR-certified indoor light fixtures: 1 or two sockets 3 or more sockets ENERGY STAR-certified home, which consists of:* Required building envelop and domestic hot water measures (i.e., the Core BOP for Ontario) Selected building envelope and domestic hot water measures among options, up to the minimum number of points required per climatic zone (BOP Options for Ontario) A minimum of 400 kwh per year of electrical savings from the following specified options: ENERGY STAR-certified CACs ENERGY STAR-certified appliances ENERGY STAR-certified lighting (CFL or LED) Efficient ventilation system (ENERGY STAR-certified fan and hood, or a heat recovery ventilation with at least 75% efficiency) EnerGuide 83 or 84 home EnerGuide 85 or more home Space and water heating, building envelope and/or space cooling** $15.00 per Product $3.00 $15.00 $500 per Home $500 per Home $1,000 per Home $0.10 per kwh or $800 per kw * From Version 12.6 ENERGY STAR for New Homes Standard requirements enforced during PY2016. ** For Custom track requirements, see: IESO. Conservation First Framework LDC Tool Kit. October Available online: Table 59 shows a comparison of PY2016 to PY2015 RNC Program results. Overall, PY2016 savings decreased due to lower participation and lower verified per-unit savings for most measures. 90

106 Table 59. PY2016 Residential New Construction Program Results Program Metrics PY2016 Results PY2015 Results Program Reported Energy Savings (GWh) Program Reported Peak Demand Reduction (MW) Energy Savings Realization Rate 20% 76% Peak Demand Reduction Realization Rate 89% 187% Gross Verified Annual Energy Savings (GWh) Gross Verified Demand Reduction (MW) Gross Verified Lifetime Energy Savings (GWh) Net-to-Gross Ratio 82% 49% Net Annual Energy Savings (GWh) Net Peak Demand Reduction (MW) Net Energy Savings Persisting at 2020 (GWh) Net Peak Demand Reduction Persisting at 2020 (MW) Net Lifetime Energy Savings (GWh) Impact Evaluation Methodology This subsection presents the Cadmus team s approach for the PY2016 RNC Program impact evaluation, with our methods and activities based on the CFF EM&V Protocols. The Cadmus team used engineering analysis, secondary research and desk reviews of project files to review unit savings values and estimate gross verified savings. The team used different approaches for each program track, which are outlined below. Prescriptive Track The Cadmus team reviewed the reported unit savings values (for energy savings and peak demand reduction), as well as the EUL, for each prescriptive measure. When necessary, we updated unit savings and EUL values according to the most recent literature. For measures already outlined in the most recent version (March 2017) of the IESO Prescriptive Measures and Assumptions List, the team reviewed and updated assumptions based on engineering algorithms and secondary research of HOU, baseline and unit energy consumption values. For measures that are not present in the list, we calculated unit savings values from secondary research, including a review of technical reference manuals (TRMs) from other jurisdictions. The team also conducted a unit savings review of the prescriptive measures to provide greater consistency among the other consumer programs offering the same measures (the Coupons and HVAC programs). Table 60 presents the evaluation approach we used for each prescriptive measure. 91

107 Dimmer switch ENERGY STAR-certified LED Table 60. PY2016 Prescriptive Measures Evaluation Approach Prescriptive Measure Unit Savings and EUL Evaluation Approach ENERGY STAR-certified light fixture (one or two sockets) ENERGY STAR-certified light fixture (three or more sockets) Indoor motion sensor ENERGY STAR-certified recessed LED lighting ENERGY STAR-certified under-the-counter lighting SEER 15 central air conditioner Updated to be consistent with the Coupons Program (see section 2.2.1)* Secondary research Updated to be consistent with the HVAC Program (see section 3.2) * Unit savings values borrowed from the Coupons or HVAC programs do not include the Coupon or HVAC specific ISRs. The RNC evaluation included updating unit savings and EUL values for ENERGY STAR-certified, recessed lighting fixtures and ENERGY STAR-certified, under-the-counter lighting fixtures, using the equations outlined in the next subsections. ENERGY STAR-Certified Recessed Lighting Fixture The Cadmus team established the unit savings value of ENERGY STAR-certified, recessed lighting fixtures using the average savings achieved by replacing common halogen recessed lamps with LED recessed lamps, per the following equation: Annual Unit Savings [ kwh h Displaced Wattage [W] HOU [ ] 365 [day yr ] = day yr ] 1,000 [ W kw ] The reported residential recessed lighting fixture baseline uses halogen lamps. 18 According to the team s best knowledge, the most common halogen lamps are 40 watts and 60 watts. The literature did not confirm which baseline halogen fixtures are most common in Ontario. Therefore, the team assumed that 40-watt and 60-watt lamps would be installed at equal rates and used a baseline of 50 watts. For replacement LED fixtures, the team used an average value of 9.5 watts, in accordance with a database of LED fixtures, distributed under a Nova Scotia demand-side management (DSM) program that offered direct rebates for such products in PY2016 (Efficiency Nova Scotia 2017). Hence, 40.5 watts 18 Efficiency Nova Scotia. Residential Efficient Product Rebates Program 2016 DSM Evaluation. Page

108 were displaced. The team assumed 2.9 HOU per day, based on a 2014 study. 19 As a result, the energy savings value increased from 32.0 kwh to 42.9 kwh per year. The team calculated unit peak demand reduction using the following equation: Displaced Wattage [W] Summer Peak Coincidence Peak Demand Reduction [kw] = 1,000 [ W kw ] Based on the residential lighting load profile in the IESO s cost-effectiveness tool, the team used a summer peak coincidence factor of 2.66%. We updated the peak demand reduction value from kw to kw per year. We left the EUL value at 20 years, which is used in other jurisdictions. 20 ENERGY STAR-Certified Under-the-Counter Lighting Fixture No study has been conducted on under-the-counter lighting fixtures. As these lamps have purposed and HOU similar to recessed lighting lamps, we used the same unit energy savings and peak demand reduction for both fixture types. Prescriptive Track Unit Savings Table 61 presents reported and updated annual unit savings and EUL values for prescriptive measures NMR Group Inc. and DNV GL. Northeast Residential Lighting Hours-of-Use Study. Page Available online: Study-Final-Report1.pdf NMR and DNV GL Efficiency Nova Scotia Efficiency Partnerships. Mid-Atlantic Technical Reference Manual. Version 6.0. Page 56. May Efficiency Vermont. Technical Reference User Manual Measure Savings Algorithms and Cost Assumptions. Page 344. March

109 Prescriptive Measures Table 61. PY2016 Prescriptive Track Unit Savings Unit Energy Savings (kwh) Reported Values Unit Peak Demand Reduction (kw) EUL (yr) Unit Energy Savings (kwh) Verified Values Unit Peak Demand Reduction (kw) Dimmer switch ENERGY STAR-certified LED ENERGY STAR-certified light fixture (one or two sockets) ENERGY STAR-certified light fixture (three or more sockets) EUL (yr) Indoor motion sensor ENERGY STAR-certified recessed LED lighting ENERGY STAR-certified underthe-counter lighting SEER 15 central air conditioner Most verified unit savings were significantly lower than reported values. The team conducted a baseline review for all measures and adjusted the new baseline values. These adjustments significantly lowered the per-unit savings. During this baseline review, we also sought to ensure use of the same baseline values used for the same measures offered across all consumer programs in PY2016. Performance Track The team verified Performance track gross savings using two evaluation activities: (1) a baseline review to validate use of an EnerGuide rating of 80 for the current market; and (2) a unit savings review of Performance track measures namely ENERGY STAR-certified houses, EnerGuide 83 or 84 houses and EnerGuide 85 and more houses. The team determined unit savings using the average electrical savings achieved by each of the three Performance track measures. The team also updated EUL values for ENERGY STAR-certified houses and EnerGuide houses based on values found in TRMs from other regions. The team conducted the baseline review using the NRCAN HOT2000 software, 21 selecting a sample of 35 HOT2000 simulation files to achieve 90% confidence at ±10% precision. The team modified these 35 efficient house simulation files, which originally achieved an EnerGuide rating of at least 83, to just meet Ontario Building Code minimum requirements enforced during PY The average EnerGuide rating Natural Resources Canada HOT2000 Software, Version 9.34 or Version Ministry of Municipal Affairs, Building and Development Branch. MMA Supplementary Standard SB-12: Energy Efficiency for Housing. July Available online: 94

110 of up-to-code energy simulations confirmed a baseline value of EnerGuide 80 for the current residential market, further confirmed during the interviews with builders. Only five of the 13 interviewed builders reported that, outside of the program, their current building practices exceeded the minimum code requirements. We interviewed builders from a mix of small- and large-sized firms (seetable 82). Of five builders whose practices exceed code requirements, three were from a small firm (below 50 employees) and two were from a large firm (above 250 employees). EnerGuide 83 or 84 Houses and EnerGuide 85 or More Houses For the two EnerGuide measures, the Cadmus team conducted a savings review using the same sample of 35 HOT2000 files used to confirm the above-mentioned baseline value of EnerGuide 80. First, we calculated annual electrical savings for each house by subtracting the total as-built electrical consumption from the up-to-code electrical consumption (with only electrical energy consumption considered). Then, the team divided the annual electrical savings for each house by the number of EnerGuide New House (EGNH) points between the up-to-code simulation rating and the as-built simulation rating. We used the following formula to determine the savings value per EGNH point: kwh kwh yr Home Electrical Savings [ Energy Savings per EGNH Point [ Point ] = yr ] (As Built To Code EGNH Rating) The team determined that the average savings value per EGNH point is 230 kwh for all 35 homes. Next, using the following formula, we calculated unit savings by multiplying the number of EGNH points achieved above the baseline by each Performance track measure: Energy Unit Savings [ kwh yr ] kwh yr = Average Savings per EGNH Point [ ] (As Built EGNH Rating 80) Point = 230 kwh yr Point (83 80) = 691 kwh yr for EnerGuide 83 The average unit savings value achieved by EnerGuide 83 houses and EnerGuide 84 houses is 806 kwh per year (this is an average of 691 kwh per year for EnerGuide 83 houses and 921 kwh for EnerGuide 84 houses). Similarly, an EnerGuide 85 or more house saved 1,267 kwh per year, which we calculated based on the average savings for EnerGuide 85 and 86 houses, representing the majority of projects reviewed in this category. The Cadmus team estimated unit peak demand reduction using the IESO peak demand reduction definition and the latest load profiles. According to the IESO EM&V Protocols, peak demand occurred from 1:00 p.m. to 7:00 p.m. for June, July and August, totaling 396 hours. Based on the IESO load 95

111 profiles, a duct-sealed residential building heated with natural gas had a summer peak demand factor of The following formula provided unit peak demand reduction: Unit Peak Demand Reduction [ kw ] = Unit Savings [kwh yr yr ] The team established that the unit peak demand reduction value is kw per year for houses with an EnerGuide rating of 83 or 84 and is kw per year for houses with an EnerGuide rating of 85 or more. To review the EUL, the Cadmus team conducted a literature review and compared EUL values used by the IESO to those applied by other jurisdictions in North America. As shown in Table 62, two different EUL values emerged. Table 62. New Construction Effective Useful Life Review Jurisdiction EUL Value (years) IESO 23 Efficiency Vermont* 25 Efficiency Maine** 20 * Efficiency Vermont. Technical Reference User Manual No ** Efficiency Maine. Retail/Residential Technical Reference Manual, Version Custom Path, page Based on these values, the team considered the EUL of 23 years (used by the IESO) as reasonable, and did not change the EUL value of EnerGuide-rated houses. ENERGY STAR-Certified Houses The Cadmus team reviewed project files covering 1,191 ENERGY STAR houses to achieve 90% confidence at ±10% precision. These files detailed the options builders selected to comply with the Core Building Option Package (BOP) for Ontario. The BOP required savings of at least 400 kwh per year, consisting of measures in the ESNH Standard required to achieve ENERGY STAR certification. For the majority of projects we reviewed, the builders installed two measures from the eligible electrical savings options efficient lighting (which required 75% of installed light bulbs per house to be CFLs or LEDs) and heat recovery ventilation systems with a sensible heat-recovery efficiency of 75% or less. According to the ESNH Standard, these two measures produced annual savings of 405 kwh, so the team updated average unit savings for ENERGY STAR houses to 405 kwh per year. The team calculated unit peak demand reduction using the same methodology as we used to determine reduction from EnerGuide-rated houses, establishing this as 0.13 kw per year. 96

112 For the EUL, the Cadmus team conducted a literature review of recent TRMs. The Pennsylvania Public Utility Commission 23 used an EUL of 15 years for ENERGY STAR homes. As the efficient electrical measures installed in ENERGY STAR-certified houses mostly consisted of lighting and ventilation equipment, an EUL of 15 years is realistic, as is an EUL of 23 years for building envelope measures, so the team updated the EUL accordingly. Performance Track Unit Savings Results Table 63 presents the reported and verified unit savings values and EUL for each performance measure. Performance Measures Table 63. PY2016 Performance Track Unit Savings Unit Energy Savings (kwh) Reported Values Unit Peak Demand Reduction (kw) EUL (year) Unit Energy Savings (kwh) Verified Values Unit Peak Demand Reduction (kw) EUL (year) ENERGY STAR-certified house 1, EnerGuide 83 or 84 house 1, EnerGuide 85 or more house 3, , The verified unit energy savings for Performance track measures are all lower than the reported values. The team determined verified unit energy savings using only electrical savings as opposed to all energy savings, which included natural gas savings. If all energy sources were considered, the verified savings would be nearly similar to the reported savings, indicating that the revised savings may not have been based on electrical savings only. For all the simulation files we reviewed, the houses use natural gas as the primary heating system. In Ontario, the overall percentage of houses using electricity as their main heating systems is only 14%. 24 Based on the simulation files, the primary electrical savings resulted from installing more efficient lighting, ventilation system motors, heat recovery ventilation systems, natural gas furnace motors and central air-conditioning systems. Despite lower verified unit energy savings, the verified unit peak demand reduction was higher than the reported value. Custom Track In PY2016, only one custom RNC project was completed, for an application form initially submitted in PY2013. The Cadmus team conducted a desk review of the calculation worksheets, hourly peak load, Pennsylvania Public Utility Commission. Technical Reference Manual, State of Pennsylvania, Act 129 Energy Efficiency and Conservation Program & Act 213 Alternative Energy Portfolio Standards. June Available online: Statistics Canada. Type of Main Heating Fuel Used, by Province, Accessed April 28,

113 and site visit photos for that project to understand how the gross savings were reported. The project involved a high velocity, centrally zoned combination system with electronically commutated fan motor (ECM). The system included one thermostat per zone (two in total), a damper control on air flow in ducts, a natural gas-fired hot water boiler and an electric air-conditioning unit. The reported savings for the system were based on research conducted by Canmet on test houses with controlled settings. For example, if zone 1 was being cooled to a certain temperature but zone 2 was only cooled to a temperature above zone 1, then the savings was the difference in temperature between the two zones for that volume of the home. The PY2013 application form used a reported baseline that did not have zoning and included a permanent split capacitor fan motor. Although no longer appropriate for projects designed in PY2016, this may have been an appropriate baseline at the time the application was submitted. Upon analyzing documentation, it was not possible for the Cadmus team to definitively verify all calculation assumptions and determine how they refer to the project characteristics (occupancy level, regional weather conditions, insulation level, etc.). However, because of the involvement of Canmet, a reputable research institution, in developing the savings calculation worksheets, the Cadmus team elected to accept the savings submitted for this project in the PY2016 evaluation results. Net-to-Gross NTG estimates are a calculation of the portion of gross savings influenced by and attributable to DSM programs. NTG consists of two main components free ridership and spillover calculated according to the following equation: NTG = 1 Free Ridership + Spillover Free ridership refers to the likelihood that a participant would have installed energy-efficient products without program assistance. Spillover equals the additional savings obtained by participants investing in non-program energy efficient measures as a result of their program participation. The Cadmus team derived net savings the savings directly attributable to the program by adjusting gross energy savings using the NTG, as shown in the following equation: Net Savings = NTG * Gross Savings Due to the small number of PY2016 RNC participants, the team established NTG values only at the program and track levels, then within the Performance track, we determined a separate NTG for ENERGY STAR homes and EnerGuide homes (the participation process differed for these measures and was expected to affect the NTG). The team then weighted program track-level NTG estimates by the program track s verified energy savings to calculate the overall program-level NTG. In this year s impact evaluation, the Cadmus team based RNC free ridership and spillover assessments on 17 interviews (13 with participating builders and four with nonparticipating builders). Among the 13 participating builders interviewed, four were large builders, three were mid-sized and six were small (see Table 82). The four nonparticipating builders interviewed were small builders. The team modified 98

114 the PY2016 free ridership and spillover questionnaires and algorithms from those used in the previous RNC evaluation to better capture the program s influence. Appendix E provides the participating and nonparticipating builder interview guides. The following subsection details the free ridership and spillover algorithms used for the PY2016 impact evaluation. Free Ridership Free ridership consists of two components: program intention and influence. With each component scored from 0% to 50%, the sum of the components equals the total free ridership score, ranging from 0% to 100%, as follows: Free Ridership = Intention Score + Influence Score The team used a self-report approach during participant interviews to estimate their perceptions of the program s influence on their participation decisions and on their intentions to practice energy-efficient behaviours. Intention. The Cadmus team assessed participant builders intentions to determine the likelihood that they would have implemented energy-efficient products and built energy-efficient homes without program support. The team also developed free ridership questions and possible response combinations that would result in an intention score. Table 64 presents the intention free ridership questions, possible responses and intention scores used for the RNC Program. We calculated a total intention score by multiplying the scores of the two questions, and dividing the result by two (for a maximum potential result of 50%). Table 64. Residential New Construction Program Free Ridership Intention Component Scoring Question Response Intention Score Q1. Thinking about the houses you built in Ontario in 2016 that include incented measures, if you had not received incentives and assistance from the Residential New Construction program: A. Would you have implemented these measures 1. In the same number of homes 100% 2. In most homes 75% 3. In close to half homes 50% 4. In a few homes only 25% 5. In no homes at all 0% B. Without the RNC Program, would you have implemented 1. All of the measures 100% 2. Most of the measures 75% 3. Some of the measures 50% 4. A few measures only 25% 5. None of these measures 0% Influence. The Cadmus team assessed the program s influence on participants decisions to build houses with energy-efficient measures. To estimate this influence, we asked respondents to identify how program elements influenced their decisions regarding the energy-efficient measures they implemented. 99

115 The program influence level and free ridership have an inverse relationship: the greater the program influence, the lower the free ridership (as shown in Table 65). In other words, if respondents cited at least one program element as very important, the team assumed they performed an energy-efficient action that they would not have taken without the program s intervention; therefore, the program participant received a low or zero influence score. Table 65 presents RNC elements and influence scores, with a maximum potential total influence score of 50%. Table 65. Residential New Construction Program Free Ridership Influence Component Scoring Question How important is each of the following factors in your decision to build and implement these measures? A. Program incentive Response B. Program marketing and advertising about energy efficiency in new homes C. Education assistance offered by the program D. Decision to obtain your NRCAN certification? Influence Score 1. Very important 0% 2. Somewhat important 12.5% 3. A little important 37.5% 4. Not at all important 50% Final Free Ridership After quantifying intention and influence, the Cadmus team computed the total free-ridership score for each participant by summing the intention and influence scores, each representing 50% of the total free-ridership score. Total free ridership ranged from 0% (no free ridership) to 100% (total free ridership). The team analysed free-ridership levels on a per-track basis, and for ENERGY STAR and EnerGuide homes within the Performance track to better assess variation in program attribution within this track. We weighted the number of homes by each builder (shown in the RNC project list) to determine NTG for the Performance track. For the Prescriptive track, we determined NTG by weighting each builder s verified savings. Spillover For spillover, the team asked participating and nonparticipating builders about the following topics: The number and description of energy-efficient measures they incorporated into nonparticipating houses. The number of houses they built outside the program that were certified as ENERGY STAR or have an EnerGuide rating above 83. The program s influence on their decision to construct efficient houses or implement the identified energy-efficient measures. (We counted those houses as spillover where the builder indicated that the program was important to their decisions to build above code.) 100

116 The team used the following formula to obtain the spillover level, dividing the proportion of nonparticipating, electrically heated homes meeting RNC requirements due to the program s influence by the total number of participating houses: Spillover = Where: #NP Houses % Eligible NP Houses % Attribution % Electric # Participating Houses #NP Houses = Number of nonparticipating new houses built in PY2016 % Eligible NP House = Percentage of eligible nonparticipating houses % Attribution = Level of RNC Program influence in decision to build nonparticipating houses to a program-eligible level % Electric = Percentage of houses electrically heated in Ontario #Participating Houses = Number of PY2016 participating houses The team used Canada Mortgage and Housing Corporation data, 25 which listed that a total of 38,093 new houses were built in Ontario during To determine the number of nonparticipating eligible houses (#NP Houses = 36,048), the team subtracted the number of participating houses (#Participating Houses = 2,045) from the total number of new houses built in Ontario. Then, the team determined the percentage of nonparticipating houses eligible for the program (% Eligible NP Houses), based on the number of nonparticipating eligible houses meeting program requirements (those with eligible RNC prescriptive measures installed and an ENERGY STAR-certified rating or an EnerGuide rating above 83), which we derived from participating and nonparticipating builder interview results. Also using builder interview results, the team established an overall attribution level (% Attribution), which is the RNC Program s influence level in decisions to build these nonparticipating houses to a program eligible level of energy efficiency. We estimated the overall RNC Program attribution level was estimated by weighting the builders individual attribution score by their number of efficient houses built outside the program. Table 66 presents the attribution question, possible responses and scores used. Table 66 presents the attribution question, possible responses and scores used. Table 66 presents the attribution question, possible responses and scores used. Table 66 presents the attribution question, possible responses and scores used. 25 Canada Mortgage and Housing Corporation. Housing Market Information Portal. Accessed April 28, www03.cmhc-schl.gc.ca/hmiportal/en/#tablemapchart/35/2/ontario 101

117 Table 66: Residential New Construction Program Spillover Attribution Scoring Question Response Attribution Score How important was your experience with the Residential New Construction program in your decision to implement those energy efficiency measures or build houses to that level of energy efficiency? 1. Very important 100% 2. Somewhat important 66.6% 3. A little important 33.3% 4. Not at all important 0% Finally, we established the percentage of houses in Ontario using electricity as their main heating systems (% Electric) according to Statistics Canada. Then we used this number, as outlined in the formula above, to consider only the electricity spillover savings. See the Spillover Findings section for more details. Builder Size Table 72. Size of Participating Builder per Program Track Prescriptive Performance ENERGY STAR Performance EnerGuide Custom Large (250 employees or more) Mid-sized ( employees) Small (1 49 employees) RNC Program* * Six interviewed builders participated in more than one program track Findings Program Participation In PY2016, the RNC Program led to installations of 45,098 prescriptive products and the construction of 2,045 efficient houses. As shown in Table 67, these results are below those from PY2015. Table 67. Number of Residential New Construction Program Measures per Track Program Track PY2016 PY2015 Percentage Difference Custom Track 1 Prescriptive Track (products) 45,098 91,934-51% Performance Track (houses) 2,045 2,276-10% Participation decreased for both the Prescriptive and Performance tracks. Prior to CFF, the Legacy Framework had dedicated funding for the RNC Program. PY2016 represented the first full year under the CFF, with LDCs granted greater flexibility in their program offerings. The LDCs allocated a smaller portion of their DSM budgets to the RNC Program, and as a result, projects implemented in PY2016 were distributed over 10 LDCs, compared to 25 LDCs in PY2015. The participation decrease proved more significant for the Prescriptive track. Interviewed builders reported that this program track required onerous paperwork. In PY2016, ENERGY STAR-certified light 102

118 fixtures (one or two sockets and three sockets or more) made up the most popular measures, as shown in Figure 32. Combined, they accounted for 81% of installed prescriptive products and 73% of net verified savings generated under the Prescriptive track. Figure 32. Residential New Construction Prescriptive Measures Installed The vast majority of builders who participated in the PY2016 Prescriptive track also received an incentive for ENERGY STAR-qualified houses. We theorize that builders installed efficient light bulbs through the Performance track to achieve ENERGY STAR certification, while fixtures were funneled through the Prescriptive track. For the Performance track, the number of efficient houses built under the RNC Program decreased by 10% from PY2015 to PY2016. The number of ENERGY STAR-certified houses rose by 11% in PY2016, while the number of houses receiving an EnerGuide rating decreased significantly, as shown in Table 68. Performance Track Measure Table 68. Number of Houses per Performance Track Measure Number of Houses PY2015 Percentage of Houses Number of Houses PY2016 Percentage of Houses Percentage Difference ENERGY STAR-certified homes 1,778 78% 1,966 96% 11% EnerGuide 83 or 84 homes % 59 3% -86% EnerGuide 85 or more homes 86 4% 20 1% -77% Total 2, % 2, % -10% All 35 of the EnerGuide projects reviewed were for houses with a natural gas primary heating system. ENERGY STAR-certified homes remained by far the most popular Performance track measure in PY2016, accounting for 96% of homes built and 41% of the net verified savings for the overall RNC Program in PY

119 Unit Savings Review Table 69 shows a comparison of the reported and verified values for Prescriptive and Performance track measures. To determine the verified per-unit savings and EUL values, the Cadmus team used up-to-date baseline values. The details of each per-unit savings and EUL values are presented in the Methodology section. For most measures, the unit energy savings and demand reduction decreased. Table 69. PY2016 Residential New Construction Program Unit Savings and Effective Useful Life Program Measures Prescriptive Track Unit Energy Savings (kwh) Reported Values Unit Peak Demand Reduction (kw) EUL (yr) Unit Energy Savings (kwh) % Diff. (kwh) Verified Values Unit Peak Demand Reduction (kw) % Diff. (kw) Dimmer switch % % 10 ENERGY STAR-certified LED % % 17 ENERGY STAR-certified light fixture (one or two sockets) ENERGY STAR-certified light fixture (three or more sockets) EUL (yr) % % % % 16 Indoor motion sensor % % 20 ENERGY STAR-certified recessed LED lighting ENERGY STAR-certified under-the-counter lighting SEER 15 central air conditioner Performance Track ENERGY STAR-certified house % % % % % % 18 1, % % 15 EnerGuide 83 or 84 house 1, % % 23 EnerGuide 85 or more house 3, , % % 23 Verified Gross Savings The RNC Program achieved total gross savings of GWh and 0.51 MW, as shown in Table 70. The majority of energy savings was achieved through the Prescriptive track, while the Performance track had slightly higher demand reduction due to higher per-unit peak demand reduction. The peak demand reduction for the one Custom track project represent a greater proportion of the total gross RNC Program peak demand reduction, largely because this project included air conditioning and efficient ventilation motors. Also, the savings achieved by the Custom project are mostly during the IESO 104

120 peak demand period, more than compared to the other program tracks (which had much low peak demand ratio). Table 70. Residential New Construction Program Gross Verified Savings per Program Track Program Track Gross Energy Savings (kwh) PY2016 Verified Gross Peak Demand Reduction (kw) Prescriptive 5,770, Performance 3,879, Custom 404, Total 10,054, Net-to-Gross Findings Table 71 shows the free ridership, spillover and NTG results at the program and track levels. The team established the overall weighted average program NTG estimate as 81% for PY2016. Program Track Table 71. Residential New Construction Program Net-to-Gross Results Number of builders Number of Projects Free Ridership Spillover NTG PY2016 NTG PY2015 Prescriptive % 0% 87% 50% Performance ENERGY STAR % 0% 81% Performance EnerGuide % 0% 40% Custom % 0% 78% RNC Program 13* 82** 19% 0% 81% 49% * Six interviewed builders participated in more than one program track. ** Numerous projects included measures from more than one program track. In PY2016, the program realized higher NTG values than in PY2015. The Cadmus team modified the survey questionnaires and NTG algorithms in PY2016 to better captured program influence. We also calculated separate NTG values for the two main types of Performance track houses, which also increased the precision of track-level NTG results by capturing the variation in NTG among home rating types. The following subsections present free ridership and spillover findings in greater detail. Free Ridership Findings Among all program tracks, the Prescriptive track had the lowest free ridership. In PY2016, most builders who implemented prescriptive measures under the RNC were production or track builders, driven to install the measures mostly by program incentives. Prescriptive track builders construct large number of homes that would normally only meet minimum building code requirements. However, the RNC Program provided an incentive for these builders to include additional energy-efficient measures in their new residential construction that otherwise would not have occurred. Similarly, Performance track ENERGY STAR homes had relatively low free ridership. The track builders who constructed the most ENERGY STAR homes and implemented the most prescriptive measures 49% 105

121 under the RNC Program (Table) reported that they would have built houses only to code in PY2016 had the program not existed. Conversely, Performance track EnerGuide homes had higher free ridership compared to ENERGY STAR homes and Prescriptive track measures. Most builders who constructed EnerGuide homes said they likely would have built homes to the same efficiency level in the absence of the RNC Program. These smaller custom builders recognized for their efficient building practices and design built fewer homes, and built homes for clients who requested high-efficiency houses (hence, they were less influenced by program incentives). Builder Size Table 72. Size of Participating Builder per Program Track Prescriptive Performance ENERGY STAR Performance EnerGuide Custom Large (250 employees or more) Mid-sized ( employees) Small (1 49 employees) RNC Program* * Six interviewed builders participated in more than one program track. Spillover Findings The interviewed participating and nonparticipating builders revealed that they built approximately 800 houses in PY2016 without receiving an RNC incentive, that were eligible for an incentive. Still, this represents a small proportion of the provincial residential market. The team determined that 2% of nonparticipating houses were eligible for the program (% Eligible NP Houses), based on the number of nonparticipating eligible houses meeting program requirements. Then, also using builder interview results, the team established a 4% overall attribution level (% Attribution). Nonparticipating builders said they were not influenced by the RNC Program in their decision to build program-eligible energy-efficient houses. For participating builders, who completed 14% of new residential constructions in Ontario outside the program in PY2016, the team established the average attribution level at 25%. We calculated the overall attribution level using the following formula: % Attribution = (% NP Houses Buit by P Builders % Average Attribution P Builders ) + (% NP Houses Buit by NP Builders % Average Attribution NP Builders ) = (14% 25%) + (86% + 0%) = 4% The percentage of houses in Ontario using electricity as their main heating systems (% Electric) was established to 14% according to Statistics Canada. Overall, the team found negligible final spillover, established at zero for all program tracks based on the following formula: 106

122 Spillover = #NP Houses % Eligible NP Houses % Attribution % Electric # Participating Houses = 36,048 Nonparticipating Houses 2% 4% 14% 2,045 Participating Houses = 0% LDC-Level Net-To-Gross Findings The Cadmus team calculated NTG estimates at the LDC level and compared results to PY2015 estimates, as shown in Table 73. Changes in an LDC s NTG from one year to the next is due to variations in the implemented measure types. The LDCs who primarily submitted Prescriptive track or Performance track ENERGY STAR measures had higher NTGs than those who submitted more Performance track EnerGuide measures. Table 73. Residential New Construction Program Net-to-Gross Results per LDC LDCs PY2016 NTG PY2015 NTG Essex Powerlines Corporation 40% 49% Horizon Utilities Corporation 83% 49% Hydro One Brampton Networks Inc. 85% 50% Hydro One Networks Inc. 74% 50% Lakefront Utilities Inc. 40% 49% Milton Hydro Distribution Inc. 83% 49% Niagara Peninsula Energy Inc. 84% 49% PowerStream Inc. 78% 49% Oakville Hydro Electricity Distribution Inc. 84% Waterloo North Hydro Inc. 85% 49% RNC Program 81% 49% Realization Rates The team calculated realization rates by dividing verified gross savings by reported gross savings at the program and track levels. Table 74 shows energy savings and peak demand reduction program realization rates. Program Tracks Table 74. Residential New Construction Program Track Realization Rates Energy Savings PY2016 Peak Demand Reduction Energy Savings PY2015 Peak Demand Reduction Prescriptive 13% 24% 55% 38% Performance ENERGY STAR 21% 146% Performance EnerGuide 40% 114% 109% 921% Custom 100% 100% Total 20% 89% 76% 187% 107

123 In PY2016, the RNC Program achieved a lower energy savings realization rate than in PY2015, primarily due to updates to the verified per unit energy savings for the Prescriptive and Performance tracks, which were found to be lower overall compared to PY2015 per unit savings values. Specifically, we updated the Performance track unit savings to only include electricity savings; this resulted in lower perunit savings, because most homes in Ontario are heat using natural gas (as stated above, according to Statistics Canada, only 14% of houses use electricity as their main heating fuel). The Prescriptive track, also had updates to per unit savings, which reflect baseline shifts in the market. In PY2016, the RNC Program achieved a lower peak demand reduction realization rate than in PY2015. The lower peak demand savings seen in PY2016 compared to PY2015 for Performance track is due to the Cadmus team updating the verified unit demand reduction value. Also, during the baseline evaluation activity, the team identified five simulation files that only achieved an EnerGuide rating of 82 or less and therefore did not comply with minimum EnerGuide rating eligibility requirements for the program. As such, these projects received zero verified gross savings. 26 Verified Net Savings Table 75 presents the net verified savings results for the Prescriptive track measures, while Table 76 outlines the net verified results for Performance track measures and Table 77 outlines the net verified results for the Custom track. Prescriptive Measures Table 75. PY2016 Net Verified Savings for Prescriptive Track by Measure Number of Units Verified Net Savings (kwh) Percentage of Energy Savings Verified Net Peak Demand Reduction (KW) Percentage of Peak Demand Reduction Dimmer switch 1,515 13, % % ENERGY STAR-certified LED 5, , % % ENERGY STAR-certified light fixture (one or two sockets) ENERGY STAR-certified light fixture (three or more sockets) 28, , % % 7,729 99, % % Indoor motion sensor 1,109 41, % % ENERGY STAR-certified recessed LED lighting ENERGY STAR-certified underthe-counter lighting 87 3, % % 538 4, % % SEER 15 central air conditioner % % Total 45, , % % 26 The team included these five houses in the measure count for PY

124 Performance Measures Table 76. PY2016 Net Verified Savings for Performance Track by Measure Number of Houses Verified Net Savings (kwh) Percentage of Energy Savings Verified Net Peak Demand Reduction (KW) Percentage of Peak Demand Reduction EnerGuide 83 or 84 house 59 17, % % EnerGuide 85 or more house 20 10, % % ENERGY STAR-certified house 1, , % % Total 2, , % % Table 77. PY2016 Net Verified Savings for Custom Track Number of Projects Verified Net Savings (kwh) Verified Net Peak Demand Reduction (kw) Custom track 1 315, Total 1 315, In total, the PY2016 RNC Program achieved 1.62 GWh in energy savings and 0.36 MW in peak demand reduction. Table 79 shows a comparison of PY2016 and PY2015 net verified savings per program track, and overall. Program Track Table 78. Residential New Construction Program Net Verified Savings per Program Track Net Energy Savings (kwh) PY2016 Verified Net Peak Demand Reduction (kw) Net Energy Savings (kwh) PY2015 Verified Net Peak Demand Reduction (kw) Prescriptive 636, ,838, Performance 672, ,467, Custom 315, Total 1,624, ,305, , In PY2016, the Performance track achieved 41% of the RNC Program net verified savings. Despite lower participation in PY2016, the savings difference from PY2015 is mostly due to changes in the verified unit savings. More specifically, the Cadmus team updated unit savings for Prescriptive track measures using new baseline values, and we updated unit savings for Performance track measures to reflect only electricity savings Direct Job Impacts The Cadmus team estimated the number of full-time equivalent (FTE) jobs attributable to the RNC Program based on information gathered from interviewed builders and LDC staff. Participating builders attributed 74 FTE jobs to the RNC Program, with builders reporting an average of 2.4 FTE staff being involved with RNC. To estimate the total number of FTE jobs created or maintained at participating builders, the team multiplied the average number of FTE jobs by the number of participating builders in PY

125 At the LDC level, there were 11 FTE jobs due to the RNC Program, with LDCs reporting an average of 1.1 FTE staff being involved with RNC. The team assumed that nonparticipating LDCs did not have an increase in jobs attributable to RNC. Similar to the approach we used for builders, the team multiplied the average number of FTE jobs by the number of participating LDCs in PY2016 to estimate the total number of FTE jobs created or maintained at participating LDCs. The total direct job impacts from builders and LDCs is Process Evaluation Methodology For the PY2016 RNC process evaluation, the Cadmus team conducted a number of activities. The following subsections present the details of each process evaluation activity. Appendix E provides the detailed interview guides used in the RNC process evaluation. Program Documentation Review The Cadmus team completed a comprehensive review of RNC Program documentation, which mainly consisted of the program rules document, the project application form and a project list. The team assessed the project list to formulate recommendations for rendering it more comprehensive and easier to use for evaluation purposes and for LDC staff members involved in program delivery. When interviewing program partners and participating builders, the team assessed their views regarding documentation and obtained feedback on their overall experiences with the program application form and reporting template, as well as their improvement suggestions. Interviews with Program Partners The team conducted in-depth phone interviews with five LDC staff and two IESO program team members one marketing staff member and the RNC Program lead. We used the interviews to gather insights into the effectiveness of the RNC Program s design, operations and performance. Overall, these interviewees provided us with a better understanding of program goals, design and delivery, actual implementation, successes, challenges, satisfaction, awareness and communication processes. Certain LDCs hired third-party delivery agents to take charge of program delivery. When applicable, our interviews with these LDCs included their delivery agent staff. One of the five interviewed LDCs did not offer the RNC Program in PY2016, although they had in previous years. This interview provided further insight on the lower PY2016 participation levels and the challenges associated with program delivery. The Cadmus team also spoke with ENERGY STAR program officers at NRCAN to better understand the ESNH Standard used for ENERGY STAR-certified homes. Table 79 summarizes the list of interviews with the various program partners. 110

126 Table 79. Residential New Construction Program Partner Interviews Program Partners Number of Interviews IESO Program Staff 1 IESO Marketing Staff 1 PY2016 LDCs 4 PY2015 LDC 1 NRCAN ENERGY STAR Program Staff 1 Total 8 Interviews with Builders As part of the PY2016 evaluation, the Cadmus team conducted 13 in-depth phone interviews with builder staff in charge of managing company participation in the RNC Program. Among the 13 participating builders we interviewed, four were from large firms, three were from mid-sized firms and six were from small firms (see Table 82). As shown in Table 80, some builders had more than one office participate in the RNC Program, in multiple LDC service territories. The team also interviewed four nonparticipating builders who did not participate in the PY2016 RNC Program. Table 80. Residential New Construction Program Builder Interviews per LDC Service Territory LDCs Participating Builders (in Final Project List) Participating Builders (at Time of Scheduling Interviews) Participating Builders Interviewed Essex Powerlines Corporation Horizon Utilities Corporation Hydro One Brampton Networks Inc Hydro One Networks Inc Lakefront Utilities Inc Milton Hydro Distribution Inc Niagara Peninsula Energy Inc PowerStream Inc Oakville Hydro Electricity Distribution Inc Waterloo North Hydro Inc Total Total (unique builders)* * The total number of unique builders combines various participating offices for a given builder. Five of these builders had more than one participating office in PY2016. Due to the small number of participating builders in PY2016, the team adopted a census approach to reach as many builders as possible. Interviews with participating builders lasted from 20 minutes to one hour, with an average duration of 30 minutes. The IESO provided contact information for participating builders, while the team worked with LDC representatives to obtain nonparticipating builders contact 111

127 information. The team also attempted to contact nonparticipating builders using the Ontario Home Builders Association member directory. 27 The participating builder interviews helped the Cadmus team assess general views on the program and gather feedback on builders experiences, participation barriers and program satisfaction, as well as free ridership and spillover. The nonparticipating builder interviews helped the team gathered further insights on respondents experiences, participation barriers, program awareness and spillover. (As stated above, nonparticipating builders were those who did not receive an incentive or submit a project under the RNC Program in PY2016.) We collected some firmographic information from participating and nonparticipating builders Findings For the PY2016 RNC Program, the team focused our process evaluation activities on determining program satisfaction and effectiveness, specifically related to program documentation, awareness, delivery, participation barriers and the application process. For the evaluation, we also determined RNC successes and challenges, along with opportunities for future planning. Program Design and Delivery The IESO targeted the RNC Program to homebuilders, with a goal to influence the upstream residential building market and to help increase the overall energy efficiency of new houses in Ontario. The IESO does not target RNC to homeowners, which is under the purview of other consumer portfolio programs. By directing RNC Program incentives to builders, the IESO seeks to achieve the greatest impact per dollar invested in the program. In fact, the current incentive amounts prove more relevant to builders especially larger track builders than to homeowners. Despite targeting homebuilders, the IESO offers RNC under its consumer programs portfolio. Through RNC, incentives are available for builders who install energy-efficient products and/or meet energy performance criteria in new or substantially renovated houses (see Table 58). Eligible houses may be heated using any primary heating system (e.g., electricity, natural gas). The program design is dependent on well-established energy-efficient brands (ENERGY STAR) and rating schemes (EnerGuide) in the residential market. The ENERGY STAR and EnerGuide labels have their own promotional materials, certification processes, requirements and training or educational resources for builders. Through financial support, the RNC encourages builders to adopt energy-efficient practices promoted by these labels. The IESO and LDCs have different responsibilities within the RNC Program. The IESO provides program rules and fulfills a facilitator and support role at the provincial level for LDCs. LDCs take charge of program delivery, including marketing, application intake, follow-up and reporting. To deliver the 27 Ontario Home Builders Association. Member Directory. Accessed March 31,

128 program, some LDCs contract with a delivery agent, while others do it themselves. Most LDCs transitioned to the CFF at the beginning of PY2016, taking responsibility for establishing their specific RNC savings target and budget; in the previous framework, the IESO allocated the budget. Currently, the program has no provincial-level savings goal. As part of program delivery, LDCs and delivery agents conduct program marketing, though few such marketing activities were conducted in PY2016 to promote the RNC Program. The Program Awareness, Motivation and Understanding section outlines these activities in greater detail. Builders usually enquire about the RNC through their LDC, which in turn provides information on the participation processes. Builders must submit preliminary application forms that include the expected program tracks and measures. These program tracks vary according to regional contexts or company types. For example, track or production builders tend to construct ENERGY STAR houses under the Energy Star Prescriptive track, while custom builders mostly construct EnerGuide rated house under the Performance track. After accepting the preliminary application, the LDC reach out to the participating builder and keeps the application form on file until construction is done and the final application is submitted. The construction cycle lasts from a few months to two or three years. Along with the application, the process requires a number of documents (such as building permits, receipts, EnerGuide reports and ENERGY STAR certificates). The Builders Experience section discusses challenges regarding the application process. The RNC Program also includes quality assurance and control (QA/QC) processes for 20% of RNC homes, during which IESO visits a certain percentage of houses to validate compliance with program requirements. For some LDCs, QA/QC occurs in collaboration with NRCAN-certified auditors, who already visit the houses for certification purposes. According to some interviewed LDCs, having NRCANcertified auditors conduct the program QA/QC facilitates the process, as the time between the end of construction and occupancy is often very narrow and limits the capacity of LDC and delivery agent staff to conduct the QA/QC. Some LDCs consider the QA/QC step burdensome, especially for the Prescriptive track. The IESO knows of these challenges and plans to reduce the percentage of homes that are required to take part in the QA/QC process. Finally, LDCs conduct monthly program reporting using a template provided by the IESO. With implementation of the CFF in PY2016, the reporting process changed. Some LDCs expressed concerns about RNC reporting requirements, as discussed in the Program Documentation section. Communication between LDCs and the IESO mostly occurs for program reporting, but the IESO also remains in contact with LDCs to answer questions about the program or to communicate program changes via s or webinars., The LDCs regard the level of communication as effective and do not consider additional communication required, especially among smaller LDCs seeking simple communications that avoid overburdening their teams. 113

129 Program Documentation The RNC Program documentation reviewed mostly consists of the program rules document, application form and program tracking database. The Builders Experience section discusses the application form. The program rules document is simple and informative; as the only document provided to LDCs who offer RNC, these rules describe key program elements (program measures and incentives, eligibility criteria and program requirements for QA/QC and reporting). The program tracking database lists projects completed during the program year, and it includes data on other provincial programs (e.g., Home Assistance Program) in addition to the RNC. The database is presented as an Excel spreadsheet containing two tabs (Projects and Measures), with both tabs including general information on each measure and project (e.g., the LDC, framework, funding mechanism, implementation year, identification number). The Projects tab details applicant contact information as well as project addresses, important dates and total incentive amounts. The Measures tab outlines program tracks, measure names, number of units, completion dates and gross reported savings. The database includes the primary information required for evaluation activities, it does not currently contain the corresponding HOT2000 simulation file numbers for EnerGuide measures. This additional information would facilitate transparency with follow-up and evaluation activities. The database also reports applicant and company names in the same field, which required additional manipulations as part of evaluation activities. Usually, the team found that measures could be associated with projects using the identification numbers listed under both tabs. However, the database contained these inconsistencies: Approximately 5% of project identification numbers appeared under the Projects tab more than once, making it impossible to use those numbers to identify which measures were completed under which projects (without assuming that the various project entries listed under the same identification numbers actually constituted a single project). One measure under the Measures tab was not associated with a project under the Projects tab. Similarly, two projects did not have any associated measures. Though all reported dates had the same format, the team identified the following issues: A few measure completion dates were not listed in the same implementation year as the associated project completion dates. This led to confusion when grouping savings by framework and program year. The listed frameworks were inconsistent with project completion dates. All projects completed under the previous framework were incorrectly identified as having been completed under the CFF. One project had a completion date entry of One Performance track measure was not identified with the correct measure name, based on the provided documentation. 114

130 More than one applicant name, address and company name were reported with slight variations or typos, which caused confusion about the actual number of applicants. Two different unit peak demand reduction values were used for EnerGuide 83 versus EnerGuide 84 homes. LDCs also mentioned that they currently populate the reporting template manually, a process they consider very labour intensive and easily leading to reporting mistakes, particularly if requiring a high number of entries. According to LDCs, automating the reporting process would reduce input errors and, therefore, the number of inconsistencies in the program tracking database. Due to recent changes made to the application form, the current program tracking database does not allow for defining the total number of houses involved in the program. The changes were made to simplify the form and ease application process burdens for builders. This produced an unintended result, making it no longer possible to determine the number of houses in which a builder installed prescriptive measures, though it is still possible to determine the number of houses built under the Performance track using the number of units reported. Knowing the total number of houses in which Prescriptive measures were installed under the RNC Program for a given year may not be necessary as long as the number of Performance track houses can be identified and the IESO establishes other program metrics (such as number of prescriptive measures installed, number of builders, number of project, etc.) to monitor participation over time. Program Awareness, Motivation and Understanding As part of the interviews with participating and nonparticipating Ontario builders, the team asked questions about their awareness of the RNC Program. One of the 13 participating builders said that all large builders are aware of the program, two said 50% to 75% of all builders are aware and six builders said fewer than 50% of all builders are aware (with responses ranging from very few builders to less than 50%). Finally, four of the 13 respondents had no idea about the level of program awareness among other Ontario builders. Of the four nonparticipating builders interviewed, one had previously participated in RNC, two had never heard of the program and one remembered hearing about it: Oh yeah, I had someone coming in about a month ago from Hydro Ottawa to talk about that program, I haven t looked at it yet. The team noted some confusion among certain builders regarding which organization the IESO or NRCAN actually offers the program. Large builders learned of the RNC Program from their LDC or delivery agent staff. Participating builders said the incentives was their main reason for program participation: It makes it more affordable for us to deliver energy-efficient homes. The cost of building ENERGY STAR is fairly [high] and the RNC Program helps. 115

131 The proportion of builders who are NRCAN certified to build ENERGY STAR or EnerGuide houses provided another indicator of program awareness and of motivations for participating in the RNC. While the Ontario Home Builders Association website reports 1,000 members, only 150 are NRCAN certified. Six of the 13 participating builders said the RNC Program somewhat influenced or highly influenced their decision to obtain NRCAN certification. Some builders participated in the RNC Program due to the ENERGY STAR brand. Label awareness carried value and represented an opportunity for builders to differentiate themselves from other builders by offering more sustainable, higher-quality houses. Most interviewed LDCs considered ENERGY STAR as less of a selling point as much as it helps builders stand out: The GTA [Greater Toronto Area] is an extremely competitive market with extremely savvy and knowledgeable consumers. So we find that building ENERGY STAR elevates us among those builders that do not. We believe in the program, we ve started from the ground up, our trade and our site people are educated and up to date because we started what we need to achieve and we use it as a marketing tool. We find that a lot of people recognize it and want to differentiate themselves in homeownership. In terms of marketing, there were very few promotional activities or advertising for the RNC Program. Interviewed LDCs mostly relied on word of mouth, though some LDCs worked with NRCAN energy auditors to promote the program and to increase awareness. Other LDCs remained in contact with builders in their area regarding program changes, especially LDCs with less staff turnover and a greater ability to maintain relationships with builders over time. Other LDCs also cited reaching out to builders through networks or builder events. Overall, the Cadmus team found that promoting the program and increasing RNC awareness among builders is a low priority for LDCs, mostly due to perceptions that large builders who are most likely to participate in the RNC Program already know about it. Considering the program nature, marketing activities might not increase program awareness and participation, although reaching out to individual builders or through their networks appears to be of value. LDCs reported having a good understanding of the program, except for the reporting process (which they considered onerous). One LDC said training on the application form and reporting template would be appreciated. Another mentioned that LDCs generally have very low understanding of ENERGY STAR and EnerGuide concepts, despite familiarity with the RNC Program delivery processes. Builders Experience The team asked participating builders about their satisfaction with the program (with the results shown in Table 81). Most builders expressed overall satisfaction, especially with the incentive amounts. Half of the builders, however, expressed dissatisfaction with the application process, and unsatisfied participating builders were among the largest contributors to program participation and savings. 116

132 Table 81. Satisfaction with the Residential New Construction Program (Participating Builders) Satisfaction Application Process Incentive Amount Overall Satisfaction Very satisfied Somewhat satisfied A little satisfied 3 Not at all satisfied Total 12* * One builder did not answer because the delivery agents in the area fill out the paperwork for applicants. Participating builders expressing dissatisfaction with the application process primarily noted concerns with the following elements: Onerous paperwork or numerous documents required Communication with LDCs High number of program representatives with whom to deal One builder expressed this last concern in the following statement: [When participating in the program] you sort of kick-off into a long administrative process where we have different people to interface with. My company has a number of different LDCs that we have to deal with. And each of those LDCs may or may not choose to use a third-party. So we might have to deal with one of any number of the third-party and we might have to deal with one of any number of the LDCs, each of them having a different team and a different process. A fewer would be simpler. Each of these concerns in outlined in further detail in the following subsections. Application Process and Paperwork According to builders, the application process and required paperwork are the main program barriers: many builders found the application process onerous and difficult to manage. Builders also found other program paperwork too cumbersome, although they recognized the importance of proving that work had been accomplished. The documentation requested did not always match builders reality: I m sorry, I need to express this to you now, but part of it [paperwork and amount of hoops you have to jump through] seems to have been written by someone who has no idea how builders build homes, quite frankly. One builder explained issues that arise from having to provide the building permit along with the preliminary application: Well, lots of time[s], we might build a site with 50 houses in a given year, for example, but yet we don t have the building permit in January. So we write down 50 houses and do all the figuring out of all the different model types of houses that we build in the application form. But lots of times, the house construction might not start before June, so you don t have the building permit in January, when you fill in the preliminary application. 117

133 Another builder explained the difficulty encountered when providing the occupancy permit: To get the money [incentive], you have to get an occupancy permit, and this can be difficult in some municipalities. So sometimes we build a whole bunch of streets, and homeowners might be in houses 1, 2, 3 of any street, and then the municipality might say you know what, there are too many numbers on that corner; so we want them to be on 4, 5, 6 of the other corner of the other street. Since the occupancy permits have an address associated with them, the existing occupancy permits that you might have become invalid. This happens fairly frequently. So you have to kind of chase that occupancy permit. Moreover, large- and mid-sized builders found that providing invoices or receipts to LDCs for the Prescriptive track was less than simple. Builders did not necessarily have these documents on hand for the specific number of measures installed under the RNC, as they could have been purchased as part of a larger equipment order or as part of an agreement with large contractors: As a production builder, we tend to work out trades in large contracts. We don t receive individual receipts for LED lights or all-off switches. Instead, these items would be listed in the scope of work issued to the trades. Builders also submit similar documents to NRCAN to obtain ENERGY STAR certifications and EnerGuide ratings, thus increasing the overall amount of paperwork required. As for the QA/QC process, the IESO and LDCs could envision collaborating with NRCAN to reduce the program documentation burdens. RNC documentation requirements could also be adapted to better reflect the reality of builders, while still providing proof of installation to the IESO. Communications A few builders indicated that there was scant communication with LDCs after the preliminary application submission. Once the required documentation had been submitted or the energy advisor had completed a visit for QA/QC purposes, builders did not always receive confirmation of their application submissions or feedback on eligible measures. Moreover, builders reported that no statement accompanied payment to specify measures covered by the payment. Current communication created frustration and confusion among builders, with some feeling a lack of transparency: So once we have finished off, you know, giving all the documentation, it becomes very opaque, and so it goes into some kind of a black box, and there is no communication that comes out of that black box. And so you might be submitting multiple invoices on multiple different properties, and you don t hear anything for a period of time. So let s say, for example, we are submitting an invoice for a thousand dollars, and then another invoice for a thousand dollars, and then a third one and at some point somebody will give you a payment of $ You have no way of knowing what that is for. Are we going to get more? Were some of them rejected? What happened? So there is a total lack of transparency about what s happening. 118

134 Program Stakeholders According to interviewed participating builders, LDCs did not deliver the program in a consistent manner, and builders experiences varied from one LDC to another. Generally, experience improved when a delivery agent was involved. Importantly, due to staff turnover, LDCs and delivery agents may hire new staff and contract different delivery agents over time: this led some builders to say they had bad experiences because of the lack of continuity among LDC and delivery agent staff: My company deals with different LDCs. And each of those LDCs may or may not choose to use a third party, and so we might have to deal with one of any number of the third party and LDCs, each of them having a different team and a different process. In 2016, in some third parties, a couple of key people left and they lost the documentation. So, you know, we did send the documents but they lost it! While a higher number of program stakeholders can add greater complexity to any process, having more people involved in the process allows for greater support. For some builders, the support received by LDCs or delivery agent staff eased the participation burdens: But you know, I have a couple of people that work with us. They provide so much support, you know, like I m not even sure we probably would not be doing this if it wasn t for the support that we get from them. Builder Suggestions for Improvement When asked about suggestions to improve the program, five builders recommended simplifying the paperwork and streamlining the submission process, while four builders recommended improving communications (given in the following example statements): If they could streamline the submission process, I think that would be most helpful. [... ] It s just that there are expectations from my bosses when they see that I log in so much time figuring things out. And then we wait for months. I probably just want to hear back from the utilities, knowing that they received my things because I don t hear back from them all the time. Success, Challenges and Future Planning During LDC interviews, respondents expressed varied perspectives regarding what works or does not work under the RNC Program. The Cadmus team also asked for improvement recommendations as well as potential opportunities to increase participation. Success Interviewed participating LDCs generally agreed that, though low, incentives proved meaningful for builders. In fact, 12 of 13 interviewed builders were very satisfied or somewhat satisfied with incentive amounts, as was shown in Table 81. According to one of the four interviewed LDCs, directing funding to builders represented a key element of the RNC Program design. Another said the RNC Program can influence high-level decisions made by builders and create collaboration among various building 119

135 industry stakeholders to discuss energy efficiency when planning efficient houses, which could change how market actors do business. One LDC said there is less misalignment now that LDCs have taken charge of program delivery. However, according to this person, the previous CFF delivery structure allowed for a better understanding of intent and made it easier to work closely with builders. Respondents noted that involving delivery agents was successful. With support from delivery agents, LDCs could better handle paperwork and provide support to builders, especially when several projects were involved. LDCs also noted that, overall, the RNC was a consistent and easy-to-run program. One respondent highlighted the benefits of the new and simplified application form. As for program offers, all interviewed LDCs expressed strong, positive feelings about the Performance track and cited the success of and their satisfaction with the process and measures offered. They considered it much easier to process applications for the Performance track than for the Prescriptive track, as the process did not require as much paperwork and provide similar level of incentives. LDCs also said that builders found it easier to apply for the Performance track, since the application process is less onerous and builders can be more certain of receiving incentive payments. Challenges Overall, LDCs do not consider RNC as very cost-effective (see the Cost-Effectiveness section for more details). At the time of the evaluation, the IESO knew of differences between certain lighting measure incentives and their market values, and planned to address this issue by reducing the incentive values. Some LDCs reported, however, that low cost-effectiveness is a misconception, as RNC delivery costs are low (with low marketing cost, few staff required, very few program documents and little monitoring required). Nonetheless, some LDCs worried that the program will be discontinued, as it has not reach its greater potential. The Future Planning section outlines program design opportunities. Furthermore, four of the five LDCs cited challenges associated with the Prescriptive track in terms of the required paperwork, builders difficulty in complying, the application complexity and the QA/QC processes. In addition, they said the measures currently offered lag behind the market and do not really push the market to adopt the most efficient options. All LDCs agreed that the reporting sheets, which currently require manual population by LDCs, is a process they consider labour intensive, extremely onerous, and sufficiently time consuming to potentially prevent small LDCs from delivering the program. As for the QA/QC, LDCs said the narrow window between the end of construction and the new house occupancy complicates the process. Two participating LDCs expressed concerns that the RNC Program had been designed without involvement from the industry. As a result, the program requirements and process did not correlate with the builders actual experiences. Another LDC cited challenge regarded the RNC Program s lack of 120

136 being adapted to how builders make decisions in terms of extended long-term business cycles an important issue to understand since the RNC seeks to influence these builders decisions: The RNC is more of a business-to-business program, but it runs like a consumer program. It is hard to change the way the builders make their decisions, so the RNC should not be designed to run as a regular consumer program. Three LDC raised issues regarding how the market will react to upcoming building code changes. According to LDCs, the energy-efficient construction cycle reaches a peak just before a code upgrade, when builders are familiar with the previous code changes and are building efficient houses for less cost (with the required materials being available). It will be more difficult to build ENERGY STAR and EnerGuide houses at least 20% more efficient than the upcoming building code, presenting limited opportunities. LDCs know this may affect Performance track participation in PY2017. Finally, according to some builders, a few challenges resulted from regional factors. Outside of the Greater Toronto Area, the availability and cost of materials could be an issue. A few builders reported higher costs to build ENERGY STAR houses in Ottawa and rural areas. Some builders also said the selected materials (e.g., windows) required to meet ENERGY STAR and EnerGuide standards are not always available from manufacturers or suppliers in their areas. (Notably, during PY2016, about 60% of new houses in Ontario were built in the Greater Toronto Area.) Builders also reported that in the Greater Toronto Area, the consumers are particularly knowledgeable. Future Planning LDCs offered various suggestions to improve the RNC Program and future planning. They suggested including a greater variety of measures in the Prescriptive track, outside the current focus on lighting. They also suggested that the IESO expand the RNC Program to include other types of efficient equipment not covered by the upcoming building code update, such as PV solar, peak savers (e.g., controls, information flow), plug loads, heat pumps, heat recovery ventilation systems and electric vehicle plugs and net-zero ready. LDCs suggested including Performance track measures to further improve building envelope efficiency and to ensure that the RNC Program remains relevant to the new building code while continuing to fostering market development, with the intention that the program increase its upstream influence. According to LDCs, greater savings can be generated when building a new house rather than by retrofitting it later, and new building measures could be included (e.g., net-zero houses, R-2000, solar passive design, photovoltaic): The RNC needs to be more strategic to be effective and set targets further out. We are at a maturity stage, and we should be ready to move ahead, above the low-hanging fruits. And we should do so in a holistic approach. Firmographics Among the 13 participating builders we interviewed, four are large (250 employees or more), three are mid-sized (100 to 249 employees) and six are small (fewer than 50 employees; Table 82). Large- and 121

137 mid-sized builders considered themselves production or track builders. Some small builders construct custom and track houses, while others only build custom houses. Table 82. Participating Builders by Size Size n Number of Employees Average Number of Houses Built in PY2016 Large or more 1,250 Mid-sized to Small 6 fewer than The four nonparticipating builders we interviewed are small builders, with one building only building custom houses, another building only tract houses and the remaining two building both types of houses. One nonparticipating builder participated in the RNC Program from PY2013 to PY2015, but chose not to participate in PY2016, as the houses they built would not have met RNC requirements. Table 83. Nonparticipating Builders by Size Size n Number of Employees Average Number of Houses Built in PY2016 Small 4 Fewer than Conclusions and Recommendations Changes to the verified unit savings resulted in lower program savings. Despite lower participation in PY2016, the savings difference from PY2015 is mostly due to changes in the verified unit savings. More specifically, the Cadmus team updated unit savings for Prescriptive track measures using new baseline values, and updated unit savings for Performance track measures to reflect only electricity savings. We confirmed that EnerGuide homes should use a baseline rating of 80. A measure review and adjustment could reenergize participation and increase savings. According to interviewed LDCs, the RNC Program lags behind the market in terms of the energy efficiency measures offered. In addition, the team found that the current Prescriptive track products do not generate significant per-unit savings, especially given the current market baseline identified as part of the PY2016 per-unit savings review. Major energy savings, therefore, can only be achieved if builders install a high number of products. Moreover, some Prescriptive track products (CACs and LED lighting) overlap with BOP measures implemented by builders to meet ENERGY STAR certification, further reducing savings opportunities through the Prescriptive track. Recommendation: Revise and expand the list of eligible products to drive the market beyond current practices and achieve greater per-unit and total savings. For example, the program could offer new Performance track measures, such as those with a higher EnerGuide rating, or could create a new whole-building Performance approach that fosters the development of more energy-efficient building practices. Such an incentive offering would recognize builders who construct houses with higher EnerGuide ratings and encourage other builders to achieve comparable efficiency levels. 122

138 Recommendation: Include a greater variety of Prescriptive track measures (beyond lighting products and ENERGY STAR BOP measures) to generate electricity savings (e.g., heat pumps). Recommendation: Continue updating the efficiency requirements of Prescriptive track products and remove products from eligibility that have an efficiency level similar to the current market. Recommendation: Eliminate measure overlap between prescriptive and performance tracks such as ENERGY STAR CACs and lighting products. In PY2016, most ENERGY STAR homes were built by track builders and all EnerGuide homes were built by smaller custom home builders. Among the eight participating builders interviewed who constructed ENERGY STAR homes under the RNC Program, four were from a large firm (with 250 employees or more) and three were from a mid-sized firm (with 50 to 250 employees). The ENERGY STAR incentive level proved more relevant to track builders who construct a larger number of homes. Conversely, all five builders who constructed EnerGuide homes under the RNC Program were from a small firm (with less than 50 employees). These small custom builders tend to build fewer but more efficient homes. Participating track builders exceeded minimum building code requirement due to program influence. The track builders who constructed the most ENERGY STAR homes and implemented the most prescriptive measures under the RNC Program reported that they would have built houses only to code in PY2016 had the program not existed. As such, the free-ridership level was lowest with the Prescriptive track (13%) and Performance track ENERGY STAR homes (19%). Conversely, Performance track EnerGuide homes had higher free ridership (60%), as custom builders said they likely would have built homes to the same efficiency level in absence of the program. Reduced networking efforts between LDCs and builders may be negatively impacting participation. During the interviews, some LDCs shared that they have little contact with builders due to limited staff. In addition, these LDCs reported that they assumed most builders already knew about the program and therefore spent less time reaching out to builders than in the past. However, in PY2016, the number of submitted projects decreased from PY2015. In addition, implemented projects were distributed over fewer LDCs. Recommendation: Increase networking with builders to establish or maintain relationships. Leverage builder networks to reach builders who might not know of the program. Recommendation: Encourage track builders to increase the share of ENERGY STAR homes to 100 percent. In addition, explore opportunities with track home builders to foster the construction of EnerGuide rated homes through various mechanisms such as capacity building and technical support, revising the incentives, etc. Recommendation: Work directly with small builders to encourage the construction of higher EnerGuide efficiency homes (EnerGuide 85+) and possibly net zero ready homes. 123

139 Follow up with builders is insufficient. Several participating builders expressed concern regarding sparse communications with LDCs, especially related to preliminary and final applications. Once they submit the required documentation, builders said they did not always receive confirmation that their application was received. Moreover, when receiving payment, builders noted that there was no statement that specified the associated measure(s) to explain the incentive amount. Recommendation: Confirm application receipt with builders and provide status updates regarding the application process. Recommendation: Consider including an explanatory note with incentive payments to document how the payment amount was determined. While builders struggle on their own with aspects of the application, delivery agent involvement can ensure a smooth process. Builders identified the paperwork required to complete the application process, especially for the Prescriptive track, as a participation barrier. Specifically, builders said it was complex and cumbersome, especially for track builders who install Prescriptive track products in a high number of homes. Four of the five participating builders we interviewed who installed Prescriptive measures were large track builders. In contrast, builders generally reported that the Performance track application process was more convenient and straightforward. For all tracks, builders reported they appreciated when delivery agents helped them complete the application process. Recommendation: Adapt the Prescriptive track documentation requirements to reflect the business operation process of track builders. Organize a focus group with track builders and gather their input to improve the application process. Recommendation: Consider working more with delivery agents to help LDCs deliver the RNC Program, either at the LDC or regional level. Specifically to support builders complete the application process. Participating LDCs find the RNC Program reporting process labour intensive. During interviews, LDCs said they faced challenges with RNC Program reporting. Small LDCs in particular expressed concerns with the time required to complete reporting, and larger LDCs expressed displeasure with the complexity associated with reporting a large number of projects. Specifically, these larger LDCs said the manual data entry made the current reporting process tedious. The Cadmus team noted inconsistencies in the project list, likely due to the manual, burdensome reporting process. Recommendation: Simplify the process of completing the project list for LDCs by adding more input masks and linking project and measure lists, where applicable. Further automate the reporting process by linking final application forms with the project list. Increased automation would not only facilitate the reporting process, but would reduce inconsistencies in the reported information. Recommendation: Establish an online system to enable LDCs to centralize project documents (e.g., HOT2000 files, receipts, application forms, building permits, certifications). Such a centralized system would facilitate follow up between LDCs and the IESO, while increasing 124

140 consistency in the information reported and documentation provided. Further, this would ensure that documentation remains readily available and accessible for evaluation purposes. Participating LDCs do not necessarily have a good understanding of ENERGY STAR and EnerGuide concepts. During the interviews, one LDC said that LDCs generally have a low understanding of ENERGY STAR and EnerGuide concepts, despite familiarity with the RNC Program delivery processes. These two certifications, managed by NRCAN, have specific sets of requirements that are soon expected to evolve following upcoming changes to the Ontario Building Code. As part of the evaluation, the Cadmus team spoke with ENERGY STAR program officers at NRCAN to better understand the ESNH Standard used for ENERGY STAR-certified homes. Recommendation: Provide a webinar or other training for LDCs so they can better understand the ENERGY STAR and EnerGuide concepts. Because the IESO does not currently track non-electricity energy savings, the program benefits may be understated. As highlighted in the Spillover section, only 14% of Ontario houses are primarily heated with electricity meaning, most houses are heated with natural gas. In addition, all 35 reviewed HOT2000 simulation files represented houses with natural gas as primary heating systems. Despite this, the IESO does not currently track natural gas savings. Recommendation: Consider options to account for all savings, including non-electrical savings (e.g., natural gas). The reduced program scale and savings potential negatively influenced cost-effectiveness. Participation dropped significantly in PY2016 with the smaller budgets allocated for the RNC Program by LDCs compared to the previous framework. Furthermore, the current program offerings generated lower per-unit savings especially given the current market baseline and share of electric savings identified as part of the PY2016 per-unit savings review resulting in low overall program costeffectiveness. 125

141 5.1 Introduction 5. Cost-Effectiveness This chapter presents cost-effectiveness results for the PY2016 province-wide consumer programs. The Cadmus team used the IESO s internal CDM Energy Efficiency Cost Effectiveness Tool, inputting energy savings and peak demand reduction, measure EULs, NTG ratios and incremental costs for each project. The IESO tool calculates the net present value of benefits and costs based on the inflation and discount rates provided in the tool (of 2% and 4%, respectively). The team also selected an industry- and projectspecific load profile for each project implemented, taking various formulas from the Conservation & Demand Management Energy Efficiency Cost Effectiveness Guide. 5.2 Calculation Methodology and Tests The Cadmus team calculated the TRC, PAC and LUEC. For the TRC test, we estimated program cost and benefits from a total societal perspective, while for the PAC test we used the program administrator s perspective of cost and benefits (Table 84). Table 84. Cost-Effectiveness Test Components Component TRC PAC LUEC Avoided Energy Costs Benefit Benefit - Non-Energy Benefits Benefit - - Secondary Fuel Savings (Gas) Benefit - Incremental Participant Costs Cost - - Program Administration Costs Cost Cost Cost Incentive Payments - Cost Cost Participant Bill Savings Discounted Lifetime Energy Savings - - Benefit Total Resource Cost Test The TRC test measures the overall impacts of program benefits and costs. The test compares all the program benefits and costs to the province to determine if the benefits received by the populace outweigh the total costs incurred by the residents, the LDCs and the IESO. The TRC uses the following benefit/cost ratio equation: TRC B C = PV [(Value of Gross Saved Energy + Value of Gross Non Energy Benefits) NTG] PV [Program Adminstrative Costs + (Incremental Participant Cost NTG)] Where: B = Benefits C = Costs PV = Present value NTG = Net to gross 126

142 The value of gross saved energy is the gross savings multiplied by utility-avoided energy and capacity costs. Incremental participant costs are the additional costs incurred by participants to install the energy-efficient technology over the baseline or standard technology (i.e., equipment typically installed in the absence of the efficient technology) Program Administrator Cost Test The PAC test examines program benefits and costs solely from the program administrators (i.e., the IESO and LDCs ) perspective. The PAC test uses the following benefit/cost ratio equation: PAC B C = PV [Value of Gross Saved Energy NTG] PV [Adminstrative Costs + Incentive Payments] Levelized Unit Electricity Costs The LUEC measures the overall competitiveness of different electricity sources, which allows for the comparison of DSM programs, programs over different timeframes or different supply-side options. It represents the annualized costs (discounted costs and lifetime savings) per lifetime kwh from the PAC test perspective (program administrative, delivery and incentive costs). The LUEC is determined using the following equation (costs divided by discounted lifetime kwh savings): LUEC = PV [Adminstrative Costs + Incentive Payments] PV [Gross Lifetime kwh NTG] Inputs and Assumptions For the costs-effectiveness analysis, the Cadmus team relied on several PY2016 evaluation impact results: Net energy savings Peak demand reduction Measure EUL Measure incremental lifecycle costs We combined the evaluation data with the following program financial data provided by the IESO to calculate cost-effectiveness: IESO administrative costs Program incentive payments LDC administrative costs 5.3 Results Total Resource Cost Test Table 85 shows the TRC test results for individual programs and for the consumer portfolio, with a benefit/cost ratio of 4.94 resulting from benefits of $544,008,565 and costs of $110,225,562. The 127

143 PY2015 TRC test ratio for the province-wide residential sector programs combined was The higher cost-effectiveness in PY2016 may be attributed to lower program costs per kwh. Table 85. Total Resource Cost Test Ratio and Net Benefits for Each Consumer Program and Overall Program Ratio Benefits Costs Net Benefits Coupons Program $432,708,345 $23,315,754 $409,392,591 Heating and Cooling Program 1.36 $109,568,097 $80,442,668 $29,125,429 New Home Construction Program 0.27 $1,732,122 $6,467,140 $-4,735,017 Consumer Programs 4.94 $544,008,565 $110,225,562 $433,783, Program Administrator Cost Test Table 86 shows the PAC test results for individual programs and for the consumer portfolio, with a benefit/cost ratio of 3.40 resulting from benefits of $355,595,713 and costs of $104,508,822. The PY2015 PAC test for the province-wide residential sector programs combined was A contributing factor to the increase in the PAC test ratio in PY2016 is the decrease in the program costs per kwh. Table 86. Program Administrator Cost Test Ratio and Net Benefits Per Consumer Program and Overall Program Ratio Benefits Costs Net Benefits Coupons Program 4.67 $258,812,913 $55,468,791 $203,344,122 Heating and Cooling Program 2.05 $95,276,606 $46,558,748 $48,717,858 New Home Construction Program 0.61 $1,506,193 $2,481,282 $-975,089 Consumer Programs 3.40 $355,595,713 $104,508,822 $251,086, Levelized Unit Electricity Cost Table 87 shows the LUEC for the consumer programs with a dollar per kwh ratio of $0.019/kWh resulting from costs of $104,508,822 and benefits (present value) of 5,441,946,465 kwh. The PY2015 LUEC dollars per kilowatt-hour for the province-wide residential sector programs combined was Table 87. Levelized Unit Electricity Cost Results for Each Consumer Program and Overall Program Ratio ($/kwh) Costs Benefits (PV kwh) Coupons Program $0.012 $55,468,791 $4,501,616,914 Heating and Cooling Program $0.050 $46,558,748 $922,702,610 New Home Construction Program $0.141 $2,481,282 $17,626,941 Consumer Programs $0.019 $104,508,822 $5,441,946, Independent Electricity System Operator Conservation Results Report. Available online: 128

144 5.4 Conclusions From the perspective of the IESO, LDCs, and participating customers, a TRC ratio greater than 1.0 indicates that the program was cost-effective. The consumer portfolio was cost-effective in PY2016 (with a TRC test ratio of 4.94 and PAC test ratio of 3.40). The LUEC dollars per kilowatt-hour is $0.019 for energy savings. The Coupons Program continued to be cost-effective, with a TRC test ratio of and a PAC test ratio of 4.67 in PY2016. The TRC test ratio for this program was largely driven by negative incremental costs, which were treated as benefits in the TRC test calculations. In PY2015, the TRC test ratio was and the PAC test ratio was 2.3. The LUEC dollars per kilowatt-hour for the Coupons Program decreased from $0.02 in PY2015 to $0.01 in PY2016. The Heating & Cooling program continued to be cost-effective with a TRC ratio of 1.36 in PY2016 and a PAC ration of Per unit ex post energy savings decreased in PY2016 for the HVAC program decreasing benefits resulting in the TRC ratio declining from 1.8 in PY2015 to 1.36 in PY2016, while the PAC ratio fell from 2.2 to Decreases to ex post savings at the measure level were: CAC SEER 14.5 decreased 100% and did not claim savings in PY2016; CAC SEER 15 savings decreased 66%; and ECM savings decreased 4% from the previous program year. The LUEC dollars/kwh ratio decreased from 0.06 $/kwh in PY2015 to 0.05 $/Kwh in PY2016 resulting from both a decline in fixed program costs (-43%) and an increase in total program energy savings. 29 The New Construction program was not cost-effective in PY2016 with a TRC ratio of 0.27 and a PAC ratio of 0.62, while in PY2015, the program had a TRC ratio of 1.3 and PAC ratio of 1.9. The primary drivers of the decrease in cost-effectiveness were a decrease in program participation, reported savings decreased 40% compared to PY2015, combined with a reduction in evaluated savings that resulted in an 84% decrease in gross verified savings from PY2015. The LUEC dollars/kwh ratio increased from 0.04 $/kwh in PY2015 to 0.14 $/kwh in PY2016; as program costs only decreased 13% from PY2105 while net verified savings decreased 74%. The overall consumer portfolio was cost-effective in PY2016, and the net benefit results have increased compared to PY2015 for both the TRC and PAC tests. Program costs per kwh decreased for the consumer programs in PY2016, driving higher cost-effectiveness. 29 Levelized energy benefits increased in PY2016 to 922,702 MWh from 679,836 MWh in PY

145 Volume II: Appendices October 2, 2017 Independent Electricity System Operator 120 Adelaide Street West Toronto, ON M5H 1T1

146 Table of Contents True Up Tables... A 1 Province Wide Marketing Interview Guide... B 3 Coupons Program Supplemental Appendices... C 6 PY2016 Coupons Program Demographics... C 6 PY2016 Coupons Program Stakeholder Interview Guide... C 14 PY2016 Coupons Program Participant and Nonparticipant Survey... C 17 Ipsos Customer Satisfaction (CSAT) Survey... C 50 Ipsos Longitudinal Mass Market Research Survey (LMMRS)... C 58 Heating and Cooling Program Supplemental Appendices... D 65 PY2016 Heating and Cooling Program Demographics... D 65 PY2016 Heating and Cooling Program Participant Survey... D 69 PY2016 Heating and Cooling Program Savings Permutations for High Efficiency Furnace with Electronically Commutated Motor... D 88 Residential New Construction Program Supplemental Appendices... E 91 PY2016 Residential New Construction Program Stakeholder Interview Guide... E 91 PY2016 Residential New Construction Program Nonparticipating LDC Interview Guide... E 97 PY2016 Residential New Construction Program Builder Interview Guide... E 100 PY2016 Residential New Construction Program Nonparticipating Builder Interview Guide... E 112 Example EnerGuide 80 Model... E 122 i

147 True Up Tables The following tables provide PY2015 true up values reported during the PY2016 evaluation, along with PY2016 and PY2015 results. The Cadmus team applied the evaluation results from the PY2015 evaluation to the true up projects/measures to obtain these results. Program Year PY2016 PY2015 PY2015 True Up Table 1. Province Wide Detailed Impact Findings Summary Net Energy Net Summer Peak Persistence of 2016 Program Savings (GWh) Demand Reduction (MW) Savings in 2020 (GWh) Coupons HVAC New Construction HAP Total Coupons HVAC New Construction HAP Total Legacy CFF Total True up Program Year PY2016 PY2015 PY2015 True Up Program Table 2. Coupons Program True Up Net Energy Net Summer Peak Participation Savings Demand Reduction (GWh) (MW) Persistence of 2016 Savings in 2020 (GWh) Annual 2,011, Bi Annual 7,679, Coupon CFF 9,691, Annual 1,687, Bi Annual 2,475, Coupon CFF + Legacy 4,163, Legacy Annual 272, Legacy Bi Annual 19, CFF Annual 121, CFF Bi Annual 9, Total True up 422, Appendix A. True Up Tables Page A 1

148 Program Year PY2016 PY2015 PY2015 True Up Program Table 3. HVAC Program True Up Participation Net Energy Savings (GWh) Net Summer Peak Demand Reduction (MW) Persistence of 2016 Savings in 2020 HVAC CFF + Legacy 136, HVAC CFF + Legacy 122, Legacy 2, CFF 2, Total True Up 4, Program Year Program Table 4. Residential New Construction Program True Up Net Energy Savings (GWh) Net Summer Peak Demand Reduction (MW) Persistence of 2016 Savings in 2020 (GWh) PY2016 RNC CFF + Legacy PY2015 RNC CFF + Legacy PY2015 True Up Legacy CFF Total True Up Appendix A. True Up Tables Page A 2

149 Province Wide Marketing Interview Guide Research Topic Introduction and Background Province Wide Marketing Strategy Coupons Program HVAC Residential New Construction HAP Question Map Section A Section B Section C Section D Section E Section F A. Introduction and Background Thank you for making the time to speak with me about the 2016 marketing plan and strategy. Cadmus is gathering feedback from key program staff and other stakeholders to make sure we have a thorough understanding of the program operations. We ll also get your perspective on things that are working well or any areas where you have experienced challenges so far. We will use the information you provide to inform our understanding of the program, so that we can provide well rounded and balanced observations and recommendations. The interview will take about 45 minutes to an hour. A1. To start, please tell me about your role and main responsibilities with IESO. B. Province Wide Marketing Strategy I d like to talk with you about marketing activities for the IESO portfolio. [PROBE FOR: HAP, COUPONS, HVAC, RES NC, PILOTS] B1. What are the marketing responsibilities of the LDCs versus the IESO? [PROBE FOR: COUPONS, HVAC, RES NC, PILOTS] 1. How familiar are customers with IESO compared to their LDCs? B2. Have there been any changes to the marketing mix from 2015? B3. Who is responsible for creating marketing materials? How are the materials optimized/tested? B4. What is the role of manufacturers and retailers related to program marketing? Do they have any existing materials that you leverage? B5. What are the biggest challenges to marketing the programs? Is there anything you would change about how the programs are marketed that you think would increase participation? B6. What are your plans for the marketing strategy in 2017 and beyond? Province Wide Marketing Interview Guide Page B 3

150 C. Coupons Program C1. We understand that the Coupons Program was marketed using radio ads and flyers. Is there anything else we are missing? C2. How does this compare to previous program years? C3. What are the Coupons Program marketing successes? What are the Coupons Program marketing challenges? C4. How does the program coordinate with participating retailers to ensure that in store materials are in place? C5. My understanding is that the LDCs handle most marketing for the program, but, in the sales data, we see that LDC specific coupons make up only a small fraction of all coupons. Do you have any insight as to why this is the case? C6. How does marketing differ between the annual and biannual events? D. HVAC D1. What kind of marketing took place in 2016? [PROBE: DISTRIBUTOR, CONTRACTOR, CUSTOMER] D2. How does this level and type of marketing compare to previous program years? D3. What are the goals of your marketing strategies in 2017? How effective have the marketing strategies and activities been in 2017 related to meeting goals? D4. Do you have any plans to change marketing practises in the future as a response to the redesigned program? E. Residential New Construction E1. What are the main marketing activities for the Residential New Construction? E2. Are there any LDCs that lead the marketing for the Residential New Construction? E3. The incentives are actually for the builders. Besides promoting the program to the builders, do you promote it to the buyers too? E4. What are the Residential New Construction program s marketing successes? 1. What are Residential New Construction marketing challenges? 2. Where do you see the most opportunity for Residential New Construction marketing? Province Wide Marketing Interview Guide Page B 4

151 F. HAP F1. What are the main marketing activities for the HAP? F2. Are there LDCs that lead marketing for the HAP? F3. Are participating delivery agents involved with marketing the HAP? F4. How are income eligible participants targeted in marketing? F5. What are HAP marketing successes? What are HAP marketing challenges? 1. Where do you see the most opportunity for HAP marketing? Thank you for your time! Province Wide Marketing Interview Guide Page B 5

152 PY2016 Coupons Program Demographics Coupons Program Supplemental Appendices The Cadmus team used the participant and nonparticipant surveys to assess the demographic profile of Coupon participants. The survey asked questions related to economic status, housing, and other factors. The sections below also compare the results from the participant and nonparticipant surveys to demographic data on Ontario as a whole, where data was available. Economic Factors Figure 1 shows household income ranges among participants and nonparticipants compared to the Ontario population. Figure 1. Household Income Source: Coupons Program Participant Survey Question M14. Please tell me which of the following categories applies to your total household income for the year Source for population of Ontario: Statistics Canada Census of Population. Population 15 years and over by highest degree, certificate or diploma, by province and territory (Quebec, Ontario, Manitoba, Saskatchewan). Last modified July 29, The income distribution skewed more heavily toward wealthier households among program participants and nonparticipants than for the Ontario population. The largest percentage of respondents earned $120,000 or more annually for both participant (23%) and nonparticipant (19%) households, whereas only 6% of Ontario residents earn that much. Meanwhile, only 4% of participant and 9% of nonparticipant households earned less than $20,000 annually compared to 33% of Ontario residents. Figure 2 breaks down home ownership among participants and nonparticipants compared to the Ontario population. Appendix C. Coupons Program Supplemental Appendices Page C 6

153 Figure 2. Home Ownership Source: Coupons Program Participant Survey Question M7. Do you own or rent your current place of residence? Source for population of Ontario: Statistics Canada Census of Population. Population 15 years and over by highest degree, certificate or diploma, by province and territory (Quebec, Ontario, Manitoba, Saskatchewan). Last modified July 29, Participants were more likely than non participants to own their own home, with 87% of participants indicating they were homeowners compared to 77% of nonparticipants. Both groups were more likely to own their homes compared to the general population, since across Ontario 68% of households own their home. Participants and nonparticipants rented homes slightly more than half as often as typical Ontario residents (17% to 32%, respectively). Figure 3 shows how frequently participants and nonparticipants received government benefits. Appendix C. Coupons Program Supplemental Appendices Page C 7

154 Figure 3. Receipt of Government Benefits Source: Coupons Program Participant Survey Questions M1. Do you or does anyone in your household receive any of the following types of benefits from any of the following government programs? (multiple response; responses sum to more than 100%); M2. Have you or anyone in your household received Utility Low Income Energy Assistance Program (LEAP) grant within the past 12 months? ; M3. Have you or anyone in your household qualified to participate in a natural gas low income Demand Site Management program during the past 12 months? Participants and nonparticipants received benefits or assistance most frequently from Allowance for Seniors (32% and 30%, respectively) and the Ontario Electricity Support Plan (24% and 27%, respectively). A small percentage of households had residents who qualified for the Low Income Energy Assistance Program (3% of participants, 5% of nonparticipants) or Demand Side Management (1% of participants, 3% of nonparticipants). Housing Figure 4 shows participant and nonparticipant home types. Appendix C. Coupons Program Supplemental Appendices Page C 8

155 Figure 4. Home Type Source: Coupons Program Participant Survey Question M8. What type of home do you live in? Most respondents live in single family detached houses (69% of participants, 61% of nonparticipants). Figure 5 shows the typical age of participants and nonparticipants homes. Figure 5. Home Age Source: Coupons Program Participant Survey Question M12. How old is your home (estimates are fine)? Appendix C. Coupons Program Supplemental Appendices Page C 9

156 The largest percentage of participants and nonparticipants owned homes build from 1975 to 1984 (15% and 16%, respectively). Nonparticipants homes tended to be older, with 58% of homes at least 32 years old compared to 51% of participants. Figure 6 shows the number of bedrooms among participants and nonparticipants compared to the Ontario population. Figure 6. Number of Bedrooms Source: Coupons Program Participant Survey Question M9. How many bedrooms are in your home? Source for population of Ontario: Statistics Canada Census of Population. Population 15 years and over by highest degree, certificate or diploma, by province and territory (Quebec, Ontario, Manitoba, Saskatchewan). Last modified July 29, The largest percentage of participants and nonparticipants owned homes with three bedrooms (45% and 43%, respectively), which aligns with the Ontario population (36%). Participants generally had larger homes than nonparticipants. The number of bedrooms for nonparticipants closely resembles that for the Ontario population. Figure 7 shows the number of bathrooms among participants and nonparticipants compared to the Ontario population. Appendix C. Coupons Program Supplemental Appendices Page C 10

157 Figure 7. Number of Bathrooms Source: Coupons Program Participant Survey Question M10. How many bathrooms are in your home? Source for population of Ontario: Statistics Canada Census of Population. Population 15 years and over by highest degree, certificate or diploma, by province and territory (Quebec, Ontario, Manitoba, Saskatchewan). Last modified July 29, Participants and nonparticipants were most likely to have homes with at least two bathrooms (64% and 58%, respectively), which aligns with the Ontario population (52%), with participants most likely. Figure 8 shows the range of home sizes by square footage among participants and nonparticipants. Figure 8. Home Size Source: Coupons Program Participant Survey Question M13. How many square feet is your home? More than half of participants (52%) and nonparticipants (53%) live in homes ranging from 1,000 to 1,999 square feet in size. Participants generally reported larger homes than nonparticipants. Appendix C. Coupons Program Supplemental Appendices Page C 11

158 Other Demographics Figure 9 shows the primary language spoke in participants and nonparticipants homes. Figure 9. Primary Language Spoken in Home Source: Coupons Program Participant Survey Question M4. What is the primary language used in your household? Source for population of Ontario: Statistics Canada Census of Population. Population 15 years and over by highest degree, certificate or diploma, by province and territory (Quebec, Ontario, Manitoba, Saskatchewan). Last modified July 29, Most participants and nonparticipants spoke English as their primary language in their homes (93% and 96%, respectively) compared to just 79% of the Ontario population, which frequently spoke a language other than English or French (19%). Figure 10 shows the highest level of education completed by participants and nonparticipants compared to the Ontario population. Appendix C. Coupons Program Supplemental Appendices Page C 12

159 Figure 10. Highest Level of Education Completed Source: Coupons Program Participant Survey Question M5. What is the last level of education that you have completed? Source for population of Ontario: Statistics Canada Census of Population. Population 15 years and over by highest degree, certificate or diploma, by province and territory (Quebec, Ontario, Manitoba, Saskatchewan). Last modified July 29, Eighty three percent of participants and nonparticipants achieved additional education beyond a high school diploma compared to 51% of the Ontario population. Figure 11 shows the percentage of homes with wireless Internet among participants and nonparticipants compared to the Ontario population. Appendix C. Coupons Program Supplemental Appendices Page C 13

160 Figure 11. Homes with Wireless Internet Source: Coupons Program Participant Survey Question M13a. Do you have wireless internet in your home? Most participants and nonparticipants have wireless Internet in their homes (92% and 90%, respectively) compared to just 19% of the Ontario population. PY2016 Coupons Program Stakeholder Interview Guide Introduction and background Researchable Questions Gather insight into the drivers of program design Identify whether the program is operating according to design and where obstacles may exist with regard to program operation Assess partner satisfaction with program and obstacles to partner participation Identify areas working well Identify areas that could be improved Identify and assess program goals Question Section A Section B, C Sections B and C Section D Section B, C and D Section B, C and D Section B Audience: IESO program lead and Summerhill Purpose: Review 2016 objectives, goals and program processes, assess what is working well and assess areas where challenges exist. Cadmus staff will conduct these in depth interviews. We will schedule interviews in advance of the call via e mail. Interviews will take between 30 and 45 minutes (we will include this detail in the e mail invitation). A. Introduction Thank you for making the time to speak with me about the 2016 Coupons Program. Cadmus is gathering feedback from key program staff and other stakeholders for our evaluation. We will use the information Appendix C. Coupons Program Supplemental Appendices Page C 14

161 you provide to inform our understanding of the program, so that we can provide well-rounded and balanced observations and recommendations. The interview will take about 45 minutes to an hour. A1. To start, please tell me about your role and main responsibilities working on the Coupons Program in A2. And please describe how you work with Summerhill on coupons. B. Program Design B1. What are the objectives of the Coupons Program? [PROBE: ENERGY SAVINGS TARGETS, EQUITY GOALS, CROSS PROMOTION GOALS, ETC.] B2. [IF NOT MENTIONED ABOVE] Does the program have energy saving targets for 2016? If so, how do these compare to 2015? How did the program perform against its targets in 2015? B3. Like many residential lighting program administrators, IESO dropped standard CFLs as an eligible measure for How did you expect this to impact program savings and how did you plan to compensate for the reduction in savings? B4. For several years, the program has operated with both a biannual and annual initiatives. Can you describe why the program has both a biannual event and annual initiatives? B5. In 2015 and 2016, some LDCs offered their own coded coupons. Can you describe how you coordinated with the LDCs to offer coded coupons? [PROBE: DID IESO HAVE ADVANCED NOTICE OF WHICH LDCS WOULD OFFER COUPONS AND WHAT COUPONS THEY WOULD OFFER? WAS THERE ANY FORECASTING REGARDING UPTAKE OF LDC VERSUS IESO COUPONS? DID IESO COORDINATE MARKETING WITH THE LDCS?] B6. Why did some LDCs participate and not others? Do some LDCs face barriers to offering their own coupons? B7. One suggestion in the 2015 evaluation was to use coupons to serve the low income population, in addition to or instead of HAP. What are your thoughts in regard to this recommendation? C. Program Performance and Operations C1. Based on our early analysis, coupon redemptions increased in 2016 for the second year in a row. Does this match your understanding? Do you have any thoughts as to what is driving the increased coupon redemption? C2. In 2015, IESO commissioned an evaluation of compliance with the Coupons Program rules by participating retailers. Has IESO taken any action as a result of this report? Why or why not? C3. Did the program implementer, Summerhill, make any changes to the way they processed coupons in 2016? What were their challenges or difficulties with regard to processing coupons in 2016, related to returns or confusion with manufacturer coupons? C4. Did processing LDC coded coupons result in any challenges? Appendix C. Coupons Program Supplemental Appendices Page C 15

162 D. Retailer Partnerships D1. Did the number of enrolled retailers change from 2015 to 2016? What about the number of active retail locations? By active, I mean locations that actually submitted coupons relative to the number of locations registered to participate. D2. In 2015, the program had 28 retailers participating in the biannual event and 18 in the annual event. How did the number of participating retailers in 2016 compare? [IF CHANGED] Why do you think the participation level changed? [PROBE: CHANGES TO RETAILER REQUIREMENTS; CHANGES TO INCENTIVE LEVELS OR OVERALL BUDGET; CHANGES IN LIGHTING OR OTHER MARKETS] D3. Do you feel the program had sufficient retailers to achieve the desired level of savings and to reach all parts of the IESO territory? If not, why do you say that? D4. What are the basic requirements for retailers to participate in the Coupons Program? Have retailer requirements changed at all since 2015? D5. How are retailer staff trained? Have there been any changes to retailer training in 2016? E. Closing E1. Is there anything else you would like to cover that we did not discuss? Thank you for your input. We appreciate your time. Have a nice day. Appendix C. Coupons Program Supplemental Appendices Page C 16

163 PY2016 Coupons Program Participant and Nonparticipant Survey Research Objectives Question Survey Screening Section A Like Spillover Section C Purchase/Participation Motivations and Barriers Section B Marketing and Segmentation Section E Free Ridership Section D Energy Efficient Actions and Non Like Spillover Section E Input Assumptions for CFLs and LEDs Section H Input Assumptions for Fixtures and Controls Section I Power Bars, Clotheslines, Weatherstripping Sections J, K, L Participant Characterization Section P A. Screener A1. In which province/territory do you reside? 1. Newfoundland and Labrador [THANK AND TERMINATE] 2. Nova Scotia [THANK AND TERMINATE] 3. Prince Edward Island [THANK AND TERMINATE] 4. New Brunswick [THANK AND TERMINATE] 5. Quebec [THANK AND TERMINATE] 6. Ontario 7. Manitoba [THANK AND TERMINATE] 8. Saskatchewan [THANK AND TERMINATE] 9. Alberta [THANK AND TERMINATE] 10. British Columbia [THANK AND TERMINATE] 11. Northwest Territories [THANK AND TERMINATE] 12. Nunavut [THANK AND TERMINATE] 13. Yukon Territory [THANK AND TERMINATE] 14. Outside Canada [THANK AND TERMINATE] 99. (Refused) [THANK AND TERMINATE] A2. What is your postal code? A3. Which of the following electricity distribution companies is your electricity provider? 1. [INSERT ALL LDC NAMES LISTED IN POSTAL CODE MAPPING (AS OF JAN 2014) LDC LIST WORKSHEET] 2. Other [THANK AND TERMINATE] 98. (Don t know) [THANK AND TERMINATE] Appendix C. Coupons Program Supplemental Appendices Page C 17

164 B. Awareness and Participation [FORCE RESPONSE] B1. Have you ever heard of the Save on Energy Coupons Program? The Save on Energy Coupons Program provides downloadable coupons, coupon booklets mailed to Ontario residences and coupons at participating retailer events for energy efficient products such as ENERGY STARqualified CFL and LED light bulbs, dimmer switches, outdoor clotheslines and power bars with integrated timers or auto shutoff. 1. Yes 2. No [THANK AND TERMINATE] B2. Have you participated in the Save on Energy Heating and Cooling Incentive Program? Through this program, homeowners can receive up to $650 in incentives on eligible replacement central heating systems and ENERGY STAR qualified central cooling systems purchased from and installed by a participating contractor. 1. Yes, have participated in this program 2. No, have not participated in this program B3. For each of the following products, please indicate how many you purchased in 2016, whether you used a coupon or not. If none, please enter 0. Please be sure to indicate how many individual units of the product you bought and not the number of packages. Your best estimates are fine. For example, if you purchased three two packs of CFLs, please input six. B C D ENERGY STAR qualified light fixtures Lighting control products (including indoor motion sensors, indoor lighting timers, indoor dimmer switches) Heavy duty plug in outdoor timers E F G ENERGY STAR qualified ceiling fans Power bars with an integrated timer or auto shut off Outdoor clothesline kit H Programmable thermostats for electric baseboard heater I Weatherstripping (door frame kits) J Weatherstripping (foam or V strip packs) Appendix C. Coupons Program Supplemental Appendices Page C 18

165 L N P Insulation blankets for electric water heaters ENERGY STAR qualified specialty CFLs (3 way bulbs or reflectors, floods, globes, candle shape) ENERGY STAR qualified general purpose LED light bulb(s) (standard incandescent shape) Q ENERGY STAR qualified specialty LED light bulb(s) (3 way bulbs or reflectors, floods, globes, candle shape) R General purpose or specialty LED light bulb(s) that are NOT ENERGY STARqualified [IF B3B TO B3R = 0, SKIP TO E1] B4. [ASK IF B1=1] You indicated that you had purchased the following product(s) in Please tell us how many you purchased using a Save on Energy coupon, and how many you purchased without using a coupon. Again, please be sure to indicate how many individual units of the product you bought using a coupon and not the number of packages. Your best estimates are fine. For example, if you used a Save on Energy coupon to purchase three two packs of CFLs, please input six. [FORCE RESPONSE. SHOW ONLY THOSE PRODUCTS WITH 1 RESPONSE AT B3 Appendix C. Coupons Program Supplemental Appendices Page C 19

166 CODE Measure Qty With Coupon (i) Qty Without Coupon (ii) Total (iii) B ENERGY STAR qualified light fixtures C Lighting control products (including: indoor motion sensors, indoor timers, indoor dimmer switches) D Heavy duty plug in outdoor timers E ENERGY STAR qualified ceiling fans F Power bars with an integrated timer or auto shut off G Outdoor umbrella stands or clothesline kit H Programmable thermostats for electric baseboard heater I Weatherstripping (door frame kits) J Weatherstripping (foam or V strip packs) L Insulation blankets for electric water heaters N ENERGY STAR qualified specialty CFL light bulbs (downlights, reflectors, floods, globe, candle shapes) P ENERGY STAR qualified general purpose LED light bulb(s) (standard incandescent shape) Q ENERGY STAR qualified specialty LED light bulb(s) C. Like Spillover C1. [ASK FOR MEASURES IN B4 WHERE COLUMN II>0 AND B1=1] You indicated that you purchased the following product(s) in 2016 without a Save on Energy coupon. How important were the benefits promoted by Save on Energy in your decision to purchase this/these product(s)? 1. Save on Energy was not at all important in your decision 2. Save on Energy was a little important in your decision 3. Save on Energy was somewhat important in your decision 4. Save on Energy was very important in your decision 98. (Don t know) B C D E F G H I J L N P Q ENERGY STAR qualified light fixtures Lighting control products (including: indoor motion sensors, indoor timers, indoor dimmer switches) Heavy duty plug in outdoor timers ENERGY STAR qualified ceiling fans Power bars with an integrated timer or auto shut off Outdoor umbrella stands or clothesline kit Programmable thermostats for electric baseboard heater Weatherstripping (door frame kits) Weatherstripping (foam or V strip packs) Insulation blankets for electric water heaters ENERGY STAR qualified specialty CFL light bulbs (downlights, reflectors, floods, globe, candle shapes) ENERGY STAR qualified general purpose LED light bulb(s) (standard incandescent shape) ENERGY STAR qualified specialty LED light bulb(s) Appendix C. Coupons Program Supplemental Appendices Page C 20

167 D. Save on Energy Free Ridership D1. [ASK FOR MEASURES IN B4 WHERE COLUMN I>0 AND B1=1] [ASK D1 D3 AS A BLOCK; REPEAT FOR EACH PRODUCT] Think back to when you made your purchase using the Save on Energy coupon. At what point in your shopping trip did you receive or pick up the coupon that you used? 1. Downloaded before arriving at the store 2. Cut from booklet before arriving at store 3. Received in the store prior to selecting the item purchased 4. Received at check out counter or after selecting the item I intended to purchase 98. (Don t know) 99. (Refused) D2. [ASK IF D1=4] Were you aware of the Save on Energy coupons before you selected the item you purchased? 1. Yes 2. No [SKIP TO E1] D3. [ASK IF D2=1] When did you become aware of the Save on Energy coupons? 1. Before entering the store 2. After entering the store 98. (Don t know) D4. [ASK PARTICIPANTS WHO PURCHASED ONE OR MORE PRODUCT(S) IN B4 B L COLUMN I>0, AND EITHER: D1=1,2,3, 98, 99 OR D2=1] [KEEP D4 AND D5 AS ONE BLOCK AND ASK FOR ALL PRODUCTS EXCEPT N P Q; ASK BLOCK FOR EACH PRODUCT TYPE, AND REPEAT QUESTION UNTIL ALL PRODUCTS HAVE BEEN SHOWN.] We know you have many choices in the products you purchase. If you had not had a Save on Energy coupon when you purchased your [SEE INSERT BELOW], which of the following actions do you think you would have taken? Please select the one option that is the most likely. Would you have? 1. Purchased the exact same product in the same quantity without the coupon 2. Purchased a cheaper product 3. Purchase whatever was on sale 4. Purchased a smaller quantity of the exact same product 5. Purchased the exact same product in the same quantity without the coupon, but delayed the purchase by a few months 6. Delayed the purchase by a year or indefinitely 7. Done something else [SPECIFY: ] 98. (Don t know) Appendix C. Coupons Program Supplemental Appendices Page C 21

168 IF INSERT B4Bi>0 ENERGY STAR qualified light fixture(s) B B4Ci>0 Lighting control product(s) (including indoor motion sensors, indoor timers, indoor dimmer switches) B4Di>0 Heavy duty outdoor timer(s) D B4Ei>0 ENERGY STAR qualified ceiling fans E B4Fi>0 Power bars with an integrated timer or auto shut off F B4Gi>0 Outdoor clothesline kit G B4Hi>0 Programmable thermostats for electric baseboard heater H B4Ii>0 Weatherstripping (door frame kits) I B4Ji>0 Weatherstripping (foam or V strip packs) J B4Li>0 Insulation blankets for electric water heaters L C D5. [ASK OF PARTICIPANTS WHO PURCHASED ONE OR MORE PRODUCT(S) IN B4 COLUMN I>0, INCLUDING PRODUCTS N P Q. REPEAT QUESTION UNTIL ALL PRODUCTS HAVE BEEN SHOWN.] For each of the following, please indicate how important it was in your decision to purchase your [SEE INSERT BELOW] using a Save on Energy coupon. 1. Not at all important 2. A little important 3. Somewhat important 4. Very important 99. Not applicable IF INSERT B4Bi>0 ENERGY STAR qualified light fixture(s) B B4Ci>0 Lighting control product(s) (including indoor motion sensors, indoor timers, indoor dimmer switches) B4Di>0 Heavy duty outdoor timer(s) D B4Ei>0 ENERGY STAR qualified ceiling fans E B4Fi>0 Power bars with an integrated timer or auto shut off F B4Gi>0 Outdoor clothesline kit G B4Hi>0 Programmable thermostats for electric baseboard heater H B4Ii>0 Weatherstripping (door frame kits) I B4Ji>0 Weatherstripping (foam or V strip packs) J B4Li>0 Insulation blankets for electric water heaters L B4Ni>0 B4Pi>0 ENERGY STAR qualified specialty CFL light bulbs (downlights, reflectors, floods, globe, candle shapes) ENERGY STAR qualified general purpose LED light bulb(s) (standard incandescent shape) B4Qi>0 ENERGY STAR qualified specialty LED light bulb(s) Q C N P Appendix C. Coupons Program Supplemental Appendices Page C 22

169 A B E F G H IF The Save on Energy coupon INSERT Information from a store display promoting the benefits of buying efficient products Promotional materials from Save on Energy on the benefits of energy efficient products (not received in store) Information from the website Save on Energy.ca on the benefits of efficient products Radio messaging about the benefits of energy efficiency and availability of Save on Energy coupons Information from a local utility promoting the benefits of coupon products D6. [ASK OF PARTICIPANTS WHO PURCHASED ONE OR MORE PRODUCT(S) IN B4 COLUMN I>0] Were there any other aspects of the Save on Energy coupons program that influenced your decision to purchase a rebated product? Please explain. 1. Open ended response 2. No other elements influenced my decision 98. (Don t know) E. Energy Efficient Actions and Non Like Spillover E1. [ASK IF B1=1] Which of the following actions did you take in 2016 to save energy in your home? And for each one, please indicate if you took this action before you first learned about Save on Energy coupons, after first learning about Save on Energy coupons or both before and after learning about Save on Energy coupons. [RANDOMIZE] 1. Before 2. After 3. Both before and after 4. Have not performed this action in my home E1.1. Switched to washing laundry with cold water instead of hot water E1.2. Turn off / reduced use of lights E1.3. Turned down the thermostat setting on your heating system E1.4. Unplugged devices usually plugged into outlets E1.5. Installed seasonal LED lights (e.g., C7 LED Lights, Mini LED lights) rather than a C7 incandescent light string or mini incandescent light string E1.6. Bought ENERGY STAR qualified computers (desktop) rather than a non ENERGY STAR computer E1.7. Bought an energy efficient television (CEE Tier 4) rather than a standard television or ENERGY STAR qualified set top box rather than a standard set top box E1.8. Bought ENERGY STAR qualified notebooks, monitors/displays or A/V equipment (e.g., audio amplifiers, optical disc players) rather than non ENERGY STAR equipment E1.9. Installed an ENERGY STAR clothes washer rather than a standard clothes washer E1.10. Installed an ENERGY STAR dishwasher rather than a standard dishwasher Appendix C. Coupons Program Supplemental Appendices Page C 23

170 E1.11. Installed solar landscape lights rather than conventional electric powered lights E2. [ASK E2.1 E2.11 FOR EACH E1.1 TO E1.11=2.] How important would you say your experience with the Save on Energy program was in each behaviour you indicated? Please give your response of a scale of 1 to 4, where 1 is not important at all, 2 is a little important, 3 is somewhat important and 4 is very important. 1. Not important at all 2. A little important 3. Somewhat important 4. Very important E2a. E2b. E2c. E2d. E2e. E2f. E2g. E2h. E2i. E2j. E2k. E2l. [ASK IF E1.2=2 AND E2.2>2] How many lights do you turn off manually per day that would otherwise stayed on? [ASK IF E1.2=2 AND E2.2>2] On average, how many hours per day are the lights off rather than staying on? [ASK IF E1.3=2 AND E2.3>2] What is your primary type of heating? _ Electric furnace _ Electric baseboard _ Gas furnace _ Other [ASK IF E1.3=2 AND E2.3>2] How much did you set back the thermostat? degrees F or degrees C? [ASK IF E1.4=2 AND E2.4>2] How many devices? [ASK IF E1.4=2 AND E2.4>2] What types of devices? [ASK IF E1.4=2 AND E2.4>2] How many hours per day unplugged? ) [ASK IF E1.5=2 AND E2.5>2] How many strings of lights? [ASK IF E1.6=2 AND E2.6>2] How many? [ASK IF E1.7=2 AND E2.7>2] How many? [ASK IF E1.8=2 AND E2.8>2] How many? [ASK IF E1.11=2 AND E2.11>2] How many? Appendix C. Coupons Program Supplemental Appendices Page C 24

171 E3. [ASK IF E1.1=2 AND E2.1>2 OR IF E1.9=2 AND E2.9>2 OR IF E1.10=2 AND E2.10>2)] Do you have electric or gas water heating? 1. Electric 2. Gas 98. (Don t know) 99. (Refused) E4. Did your participation in the Save on Energy Coupons Program influence you to take any other energy efficient actions or make any other energy efficiency purchases for which you did not use a coupon? This should not include any HVAC upgrades for which you received a Save on Energy Heating and Cooling rebate. 1. Yes 2. No E5. [ASK IF E4=1] What other energy efficient actions did you take in 2016? 1. After only 2. Both before and after 1 Open ended response 2 Open ended response 3 Open ended response 4 Open ended response 5 Open ended response 6 Open ended response 7 Open ended response 8 Open ended response 9 Open ended response 10 Open ended response E6. [ASK FOR EACH E5=1 RESPONSE] How important would you say your experience with the Save on Energy program was in your decisions or behaviour? Please give your response of a scale of 1 to 4, where 1 is not important at all, 2 is a little important, 3 is somewhat important and 4 is very important. 1. Not important at all 2. A little important 3. Somewhat important 4. Very important [FOR SECTIONS F K, TOTAL VALUES SHOULD BE CAPPED AS INDICATED. PARTICIPANTS AND NONPARTICIPANTS ARE DEFINED AS FOLLOWS: PARTICIPANTS: WHERE B4, COL I>0, IGNORING COL II NONPARTICIPANTS: WHERE B4, COL I=0 AND WHERE B4, COL II>0] Appendix C. Coupons Program Supplemental Appendices Page C 25

172 F. Input Assumptions Specialty CFLs [ASK IF (PARTICIPANTS: B4N, COL I 1/NONPARTICIPANTS: B4N, COL II 1), OTHERWISE SKIP TO G1] F1. Earlier you indicated that you purchased [INSERT # RESPONSE FROM (PARTICIPANTS: B4N, COL I/NONPARTICIPANTS: B4N, COL II)] specialty CFLs in 2016 [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON]. Thinking of these specialty CFLs [IF PARTICIPANT, INCLUDE: THAT YOU PURCHASED USING A SAVE ON ENERGY COUPON], how many and what types of bulbs did you purchase? Please enter a number in each box, including 0 if you did not purchase any of that type of bulb. [INSERT PICTURE NEXT TO RESPONSE BOX. SUMMED RESPONSES SHOULD EQUAL (PARTICIPANTS: B4N, COL I/NONPARTICIPANTS: B4N, COL II)] B Flood / reflector C Globe/Vanity D Candle/Chandelier E Covered F Dimmable CFLs (Used with a dimmer switch to adjust brightness) G 3 way CFLs (Has three brightness levels in a 3 way lamp) F2. [ASK IF F1G>0] Of the [INSERT # RESPONSE FROM F1G] 3 way CFLs that you purchased in 2016, how many were [ENSURE SUM OF NUMERIC RESPONSES EQUAL RESPONSE AT F1G] 1. Low wattage (approximately equivalent to 60/75/100 watt 3 way incandescents) 2. High wattage (approximately equivalent to 60/100/150 watt 3 way incandescents) Appendix C. Coupons Program Supplemental Appendices Page C 26

173 F3. [ASK IF F1X>1] [REPEAT QUESTION UNTIL ALL VALID PRODUCTS (F1X > 1) HAVE BEEN SHOWN.] Thinking about the [SEE INSERT 1 BELOW] [SEE INSERT 2 BELOW] CFL bulbs [IF PARTICIPANT, INCLUDE: THAT YOU PURCHASED USING A SAVE ON ENERGY COUPON], how many of each of the following package sizes did you purchase? IF F1B>1 F1C>1 F1D>1 F1E>1 F1F>1 F1G>1 IF F1B>1 F1C>1 F1D>1 F1E>1 F1F>1 F1G>1 INSERT 1 INSERT RECALL NUMERIC RESPONSE AT F1B RECALL NUMERIC RESPONSE AT F1C RECALL NUMERIC RESPONSE AT F1D RECALL NUMERIC RESPONSE AT F1E RECALL NUMERIC RESPONSE AT F1F RECALL NUMERIC RESPONSE AT F1G Flood/Reflector Globe/Vanity Candle/Chandelier A Shaped/Covered Dimmable CFLs 3 way CFLs INSERT 2 INSERT [FOR EACH PRODUCT, ENSURE TOTAL BULBS INDICATED (F3 RESPONSE PACKAGE SIZE) DOES NOT EXCEED RESPONSE AT F1. FOR MORE THAN 6 IN PACK OPTION, ASSUME PACK SIZE IS 7 BULBS.] Appendix C. Coupons Program Supplemental Appendices Page C 27

174 SPECIALTY CFL TYPE 1 Pack 2 Pack 3 Pack 4 Pack 5 Pack 6 Pack More than 6 in pack B Flood/Reflector C Globe/Vanity D Candle/Chandelier E A Shaped/Covered F G Dimmable CFLs Used with a dimmer switch to adjust brightness 3 way CFLs Has three brightness levels in a 3 way lamp F4. [ASK IF F1B TO F1G>1] How many of the [INSERT F1B TO F1G] specialty CFLs that you purchased in 2016 [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON] are currently installed in lamps or fixtures at your home? [ENSURE NUMERIC RESPONSE DOES NOT EXCEED F1B TO F1G] 1. Open ended numeric question [SPECIFY: ] Appendix C. Coupons Program Supplemental Appendices Page C 28

175 F5. [ASK IF F4>0] If you had not installed [INSERT # RESPONSE FROM B4N] specialty CFL bulbs [IF PARTICIPANT, INCLUDE: THAT YOU PURCHASED USING A SAVE ON ENERGY COUPON] in 2016, how many of the following bulbs types would you have installed instead? [RANDOMIZE; ENSURE SUM OF NUMERIC RESPONSE EQUALS F4] 1. LED bulbs 2. Incandescent bulbs (an incandescent bulb is the most familiar type of bulb used in homes but is no longer commonly available in stores due to national restrictions on maximum bulb wattage) 3. Halogen light bulbs (a halogen bulb is a newer type of incandescent lamp with a longer life and brighter light) 4. CFL bulbs 5. No bulbs at all (left socket empty) 6. Other [SPECIFY: ] 98. (Don t know) F6. [ASK IF F4>0] Of the [INSERT # RESPONSE FROM F4] specialty CFL bulbs [IF PARTICIPANT, INCLUDE: YOU PURCHASED USING A SAVE ON ENERGY COUPON] in 2016 that you have installed, how many For a definition, please roll your mouse over the blue, italicized words. [ENSURE SUM OF NUMERIC RESPONSE DOES NOT EXCEED F4] 1. Replaced an incandescent bulb? (AN INCANDESCENT BULB IS THE TYPE OF BULB THAT MOST HOMES CURRENTLY USE A REGULAR LIGHT BULB) 2. Replaced a halogen bulb? (A HALOGEN BULB IS A NEWER TYPE OF INCANDESCENT LAMP WITH A LONGER LIFE AND BRIGHTER LIGHT) 3. Replaced a CFL bulb? 4. Replaced an LED bulb? 5. Were installed in new fixtures or fixtures that did not have bulbs before? 6. Replaced some other type of bulb? F7. [ASK IF F6.6>0] You ve indicated that [INSERT NUMERIC RESPONSE FROM F6.6] specialty compact fluorescent bulbs replaced some other type of bulb. What type of bulb was this? 1. Open ended [SPECIFY: ] F8. [ASK IF ( F1B TO F1G) F4>1] Of the [INSERT ( F1B TO F1G) F4] specialty CFLs that you purchased [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON] but do not currently have installed in your home, how many did you install but later remove? [ENSURE NUMERIC RESPONSE DOES NOT EXCEED [( F1B TO F1G) F4] 1. Open ended [SPECIFY: ] Appendix C. Coupons Program Supplemental Appendices Page C 29

176 F9. [ASK IF ( F1B TO F1G) F4 F8>1] Based on the information you have provided, there are a total of [INSERT ( F1B TO F1G) F4 F8] specialty CFLs that you have purchased [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON] but have never installed. What is the main reason the remaining bulb(s) have not been installed? [RANDOMIZE] 1. Threw them away 2. Storing for later use 3. Returned them 4. Gave them away 5. Something else [SPECIFY: ] F10. [ASK IF F9=2] How many of the [INSERT ( F1B TO F1G) F4 F8] specialty CFLs you have yet to install do you plan on installing as described below? [ENSURE NUMERIC RESPONSE DOES NOT EXCEED [( F1B TO F1G) F4 F8] 1. Will be installed in the next two months 2. Will be installed in the next three to six months 3. Will be installed in the next seven to 12 months 4. Will be installed more than one year from now 5. Will be installed as current incandescent bulbs burn out 6. Will be installed as current CFL bulbs burn out 7. Not planning on installing the specialty CFLs 98. (Don t know) G. General Purpose and Specialty LEDs [ASK IF (PARTICIPANTS: B4P, COL I 1 OR B4Q, COL I 1/NONPARTICIPANTS: B4P, COL II 1 OR B4Q, COL II 1), OTHERWISE SKIP TO H1] Appendix C. Coupons Program Supplemental Appendices Page C 30

177 G1. [ASK IF (PARTICIPANTS: B4P 1 SET G1A=B4P, COL I/NONPARTICIPANTS: B4P 1 SET G1A=B4P, COL II)] Earlier you indicated that you purchased [INSERT # RESPONSE FROM B4P (COL I FOR PARTICIPANTS AND COL II FOR NONPARTICIPANTS) + B4Q (COL I FOR PARTICIPANTS AND COL II FOR NONPARTICIPANTS)] ENERGY STAR qualified LED bulbs in 2016 [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON]. Thinking of these LED bulbs that you purchased [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON], how many and what types of bulbs did you purchase? [INSERT PICTURE NEXT TO RESPONSE BOX. SUMMED RESPONSES A F SHOULD = B4P + B4Q] A General Purpose/Standard Incandescent Shape B Flood/Reflector C Globe/Vanity D Candle/Chandelier E 3 Way LED F Other [SPECIFY ] G2. [ASK IF G1E>0] Of the [INSERT # RESPONSE FROM G1E] 3 way LEDs that you purchased in 2016, how many were [ENSURE SUM OF NUMERIC RESPONSES EQUAL RESPONSE AT G1E] 1. Low wattage (approximately equivalent to 60/75/100 watt 3 way incandescents) 2. High wattage (approximately equivalent to 60/100/150 watt 3 way incandescents) [SET G3X=1 PACK IF G1X=1] [KEEP G3 TO G8 AS ONE BLOCK. ASK BLOCK OF QUESTIONS FOR EACH LED TYPE AND REPEAT QUESTIONS UNTIL ALL VALID PRODUCTS (G1X>1) HAVE BEEN SHOWN] Appendix C. Coupons Program Supplemental Appendices Page C 31

178 G3. [ASK IF G1>1] Thinking about the [SEE INSERT 1 BELOW] [SEE INSERT 2 BELOW] ENERGY STARqualified LED bulbs that you purchased, how many of each of the following package sizes did you purchase [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON]? IF G1A>1 G1B>1 G1C>1 G1D>1 G1E>1 IF G1A>1 G1B>1 G1C>1 G1D>1 G1E>1 INSERT 1 INSERT RECALL NUMERIC RESPONSE AT G1A RECALL NUMERIC RESPONSE AT G1B RECALL NUMERIC RESPONSE AT G1C RECALL NUMERIC RESPONSE AT G1D RECALL NUMERIC RESPONSE AT G1E INSERT 2 INSERT General purpose/standard incandescent shape Flood/reflector Globe/vanity Candle/chandelier 3 way [FOR EACH PRODUCT, ENSURE PRODUCT OF NUMERIC RESPONSE AND PACK SIZES DOES NOT SUM TO EXCEED RESPONSE AT G1X. FOR MORE THAN 4 IN PACK OPTION, ASSUME PACK SIZE IS 5.] LED TYPE 1 Pack 2 Pack 3 Pack 4 Pack More than 4 in pack A General purpose/standard incandescent shape B Flood/reflector C Globe/vanity D Candle/chandelier E 3 way Appendix C. Coupons Program Supplemental Appendices Page C 32

179 G4. [ASK IF G1X>1] How many of the [INSERT # RESPONSE FROM G1X] [INSERT LED BULB TYPE] ENERGY STAR qualified LED bulbs that you purchased in 2016 [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON] are currently installed in lamps or fixtures at your home? [PROGRAMMER NOTE: ENSURE NUMERIC RESPONSE DOES NOT EXCEED G1X] 1. Open ended numeric [SPECIFY: ] G5. [ASK IF PARTICIPANT AND G4>0] If you had not purchased [INSERT # RESPONSE FROM G4] [INSERT LED BULB TYPE] ENERGY STAR qualified LED bulbs using Save on Energy coupons in 2016, how many of the following bulbs types would you have installed instead? [RANDOMIZE; ENSURE SUM OF NUMERIC RESPONSE EQUALS G4.] 1. Incandescent bulbs (an incandescent bulb is the most familiar type of bulb used in homes a regular light bulb) 2. Halogen light bulbs (a halogen bulb is a newer type of incandescent lamp with a longer life and brighter light) 3. LED bulbs 4. CFL bulbs 5. Something else [SPECIFY: ] 98. (Don t know) G6. [ASK IF PARTICIPANT AND G4>0] Of the [INSERT # RESPONSE FROM G4] [INSERT LED BULB TYPE] ENERGY STAR qualified LED bulbs you purchased using Save on Energy coupons in 2016, how many For a definition of incandescent bulb, please roll your mouse over the blue, italicized words. [ENSURE SUM OF NUMERIC RESPONSE DOES NOT EXCEED G4] 1. Replaced an incandescent bulb? (AN INCANDESCENT BULB IS THE TYPE OF BULB THAT MOST HOMES CURRENTLY USE A REGULAR LIGHT BULB) 2. Replaced a halogen bulb? (A HALOGEN BULB IS A NEWER TYPE OF INCANDESCENT LAMP WITH A LONGER LIFE AND BRIGHTER LIGHT) 3. Replaced a CFL bulb? 4. Replaced an LED bulb? 5. Were installed in new fixtures or fixtures that did not have bulbs before? 6. Replaced some other type of bulb? G7. [IF G6.6>0] You ve indicated that [INSERT NUMERIC RESPONSE FROM G6.6] LED bulbs replaced some other type of bulb. What type of bulb was this? 1. Open ended [SPECIFY: ] Appendix C. Coupons Program Supplemental Appendices Page C 33

180 G8. [ASK IF G1X>0] Thinking just of the [INSERT # RESPONSE FROM G1X] [INSERT LED BULB TYPE] ENERGY STAR qualified LED bulbs that you purchased in 2016 [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON], how many bulbs did you purchase at each of the following watts? [ENSURE SUM OF NUMERIC RESPONSES =G1X] 1. Less than 5 watts [DO NOT INCLUDE FOR G1A] 2. 5 to less than 10 watts to less than 15 watts [DO NOT INCLUDE FOR G1C OR G1D] to less than 20 watts [DO NOT INCLUDE FOR G1C OR G1D] watts or more[do NOT INCLUDE FOR G1C OR G1D] 98. (Don t know) G9. [ASK IF ( G1A TO G1E G4) 1] Of these [INSERT ( G1A TO G1E G4)] ENERGY STAR qualified LED bulbs that you purchased but do not currently have installed in your home, how many did you install but later remove? [ENSURE NUMERIC RESPONSE DOES NOT EXCEED G1A TO G1E G4] 1. Open ended [SPECIFY: ] G10. [ASK IF G9>0 AND IF ( G1A TO G1E G4 G9) 1)] Based on the information you have provided, there are a total of [INSERT RESPONSE FROM ( G1A TO G1E G4 G9)] ENERGY STAR qualified LED bulbs that you purchased [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON] but have never installed. What is the main reason why the remaining bulb(s) have never been installed? [RANDOMIZE] 1. Threw them away 2. Storing for later use 3. Returned them to store 4. Gave them away 5. Other [SPECIFY: ] G11. [ASK IF G10=2] Thinking of the ENERGY STAR qualified LED bulbs that you have yet to install, how many do you plan on installing as described below? [ENSURE SUM OF NUMERIC RESPONSES DOES NOT EXCEED ( G1A TO G1E G4 G9)] 1. Will be installed in the next two months 2. Will be installed in the next three to six months 3. Will be installed in the next seven to 12 months 4. Will be installed more than one year from now 5. Will be installed as current incandescent bulbs burn out 6. Will be installed as current CFL bulbs burn out 7. Not planning on installing the LEDs 98. (Don t know) Appendix C. Coupons Program Supplemental Appendices Page C 34

181 G12. [ASK IF B4N P Q>0] Canada s Energy Efficiency Regulations restricted the sale of standard 100 and 75 watt incandescent light bulbs manufactured after January 1, 2014, and standard 60 and 40 watt incandescent light bulbs manufactured after December 31, Had you heard about this restriction before today? 1. Yes 2. No 98. (Don t know) H. Input Assumptions Fixtures [ASK IF (PARTICIPANTS: B4B, COL I 1/NONPARTICIPANTS: B4B, COL II 1), OTHERWISE SKIP TO I1) H1. Earlier you indicated that you purchased [INSERT RESPONSE FROM (PARTICIPANTS: B4B, COL I/NONPARTICIPANTS: B4B, COL II)] ENERGY STAR qualified light fixtures in 2016 [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON]. Thinking of these fixtures, how many light bulb sockets does each of them have? [NUMBER OF FIXTURES FOR WHICH RESPONSE IS ASKED SHOULD EQUAL (PARTICIPANTS: B4B, COL I/NONPARTICIPANTS: B4B, COL II)] 1. Fixture 1: 2. Fixture 2: 3. Fixture 3: 4. Fixture 4: 5. Fixture 5: 6. Fixture 6: 7. Fixture 7: 8. Fixture 8: 9. Fixture 9: 10. Fixture 10: H2. Earlier you indicated that you purchased [INSERT RESPONSE FROM (PARTICIPANTS: B4B, COL I/NONPARTICIPANTS: B4B, COL II)] ENERGY STAR qualified light fixtures in 2016 [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON]. Thinking of these fixtures, how many are currently installed in your home? [ENSURE NUMERIC RESPONSE DOES NOT EXCEED (PARTICIPANTS: B4B, COL I/NONPARTICIPANTS: B4B, COL II] 1. Open ended [SPECIFY: ] Appendix C. Coupons Program Supplemental Appendices Page C 35

182 H3. [ASK IF H2 < (PARTICIPANTS: B4B, COL I/NONPARTICIPANTS: B4B, COL II)] How many of the [INSERT (PARTICIPANTS: B4B, COL I/NONPARTICIPANTS: B4B, COL II)) H2] ENERGY STARqualified light fixtures that you have yet to install do you plan on installing as described below? [ENSURE NUMERIC RESPONSE DOES NOT EXCEED (PARTICIPANTS: B4B, COL I/NONPARTICIPANTS: B4B, COL II) H2] 1. Will be installed in the next two months 2. Will be installed in the next three to six months 3. Will be installed in the next seven to 12 months 4. Will be installed more than one year from now 5. Not planning on installing the light fixtures 98. (Don t know) H4. Lighting Fixture Baseline Question [ASK IF PARTICIPANTS: B4B, COL I>0 AND H2>0] If you had not purchased and installed [INSERT # RESPONSE FROM H2] ENERGY STAR qualified light fixtures using Save on Energy coupons in 2016, what type of bulbs would you have used in the fixtures that you would have installed instead? [NOT MULTIPLE RESPONSE; RANDOMIZE] 1. CFLs (compact fluorescent light bulbs) 2. Incandescent bulbs (AN INCANDESCENT BULB IS THE MOST FAMILIAR TYPE OF BULB USED IN HOMES A REGULAR LIGHT BULB) 3. Halogen light bulbs (A HALOGEN BULB IS A NEWER TYPE OF INCANDESCENT LAMP WITH A LONGER LIFE AND BRIGHTER LIGHT) 4. LEDs 5. Linear fluorescent tubes 6. Something else [SPECIFY: ] 98. (Don t know) I. Input Assumptions Controls [ASK IF (PARTICIPANTS: B4C, COL I 1/NONPARTICIPANTS: B4C, COL II 1), OTHERWISE SKIP TO J1] Appendix C. Coupons Program Supplemental Appendices Page C 36

183 I1. [ASK IF (PARTICIPANTS: B4C, COL I 1/NONPARTICIPANTS: B4C, COL II 1) Earlier you indicated that you purchased [INSERT RESPONSE FROM (PARTICIPANTS: B4C, COL I 1/NONPARTICIPANTS: B4C, COL II 1)] lighting control products (indoor motion sensors, indoor lighting timers and/or indoor dimmer switches) in 2016 [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON]. How many of each type of control product did you purchase in each of the following package sizes? [ENSURE PRODUCT OF NUMERIC RESPONSE AND PACK SIZES DOES NOT SUM TO EXCEED RESPONSE AT (PARTICIPANTS: B4C, COL I/NONPARTICIPANTS: B4C, COL II). FOR MORE THAN 6 IN PACK OPTION, ASSUME PACK SIZE IS 7 FOR TESTING PURPOSES. TOTAL PRODUCTS ROW SHOULD AUTOCALCULATE BASED ON COLUMN ENTRIES] 1 pack (one control device per package) 2 pack (two control devices per package) 3 pack (three control devices per package) 4 pack (four control devices per package) 5 pack (five control devices per package) 6 pack (six control devices per package) More than 6 in pack Indoor Motion Sensors Indoor Lighting Timers Indoor Dimmer Switches Total Products At least At least At least I2. Thinking of each of these lighting control products, how many are currently installed in your home? [SHOW ONLY PRODUCTS WITH RESPONSES OF AT LEAST 1 IN I1. ENSURE NUMERIC RESPONSE DOES NOT EXCEED CORRESPONDING PRODUCT RESPONSES IN I1] 1. Indoor motion sensors 2. Indoor lighting timers 3. Indoor dimmer switches I3. Lighting Controls Baseline Question [ASK IF PARTICIPANT AND I1>0] How many of each type of lighting is controlled by the [INSERT # RESPONSE FROM I1=INDOOR MOTION SENSORS] indoor motion sensor(s), [INSERT # RESPONSE FROM I1=INDOOR LIGHTING TIMERS] indoor lighting timer(s) and [INSERT # RESPONSE FROM I1=INDOOR DIMMER SWITCHES] indoor dimmer switch(es) currently installed in your home? [RANDOMIZE; ENSURE SUM OF NUMERIC RESPONSES FOR EACH CONTROL TYPE IS EQUAL TO THE # RESPONSE TO H5 FOR THAT CONTROL TYPE] Appendix C. Coupons Program Supplemental Appendices Page C 37

184 Lighting Type Controlled by Device Indoor Motion Sensor Indoor Lighting Timer Indoor Dimmer Switch 1 CFLs (compact fluorescent light bulbs) 2 Incandescent bulbs (an incandescent bulb is the most familiar type of bulb used in homes a regular light bulb) 3 Halogen light bulbs (a halogen bulb is a newer type of incandescent lamp with a longer life and brighter light) 4 LEDs 5 Linear fluorescent tubes 8 Something else. If so, what? [SPECIFY: ] 98 (Don t know) I4. [ASK IF I1<I3] How many of the [INSERT I1 I3] lighting control products that you have yet to install do you plan on installing as described below? [SHOW ONLY PRODUCTS WITH RESPONSES OF AT LEAST 1 IN I1. ENSURE NUMERIC RESPONSE DOES NOT EXCEED I1 I3 FOR EACH PRODUCT] 1. Will be installed in the next two months 2. Will be installed in the next three to six months 3. Will be installed in the next seven to 12 months 4. Will be installed more than one year from now 5. Not planning on installing the lighting control products 98. (Don t know) J. Power Bars, Clotheslines and Weatherstripping [ASK IF (PARTICIPANTS: B4F, B4G, B4I OR B4J, COL I 1), OTHERWISE SKIP TO K1] Power Bars J1. [ASK IF B4F, COL I 1] Earlier you indicated that you purchased [INSERT RESPONSE FROM B4F] power bars with an integrated timer or auto shut off in Thinking of these power bars, how many of each type did you purchase with a Save on Energy coupon? [INSERT PICTURE NEXT TO RESPONSE BOX. SUMMED RESPONSES SHOULD EQUAL B4F, COL I] Appendix C. Coupons Program Supplemental Appendices Page C 38

185 A Power bar with an integrated timer (turns off power on a pre set schedule to reduce standby loses) B Power bar with auto shut off/smart power bar (One outlet is for controlling equipment, and several outlets are for dependent equipment. When controlling equipment is turned off, associated dependent equipment that would normally stay on standby are also turned off. There are usually some always on outlets in addition to the control and dependent outlets) J2. [ASK IF J1B>0] How many of the [INSERT RESPONSE FROM J1B] power bars with auto shut off are you currently using? [ENSURE NUMERIC RESPONSE DOES NOT EXCEED J1B] 1. Open ended numeric [SPECIFY: ] J3. [ASK IF J2<J1B] How many of the [J1B J2] power bars with auto shut off that you are not currently using do you plan on starting to use as described below? [ENSURE NUMERIC RESPONSE DOES NOT EXCEED J1B J2] 1. Will start using in the next two months 2. Will start using in the next three to six months 3. Will start using in the next seven to 12 months 4. Will start using more than one year from now 5. Not planning on using 98. (Don t know) Appendix C. Coupons Program Supplemental Appendices Page C 39

186 J4. [ASK IF J2>0] Thinking of your power bar with auto shut off, please indicate how many of each of the following types of equipment are plugged into the control outlet, and how many are plugged into the dependent outlets. Do not include equipment that is plugged into any of the outlets that are always on. [ASK SEPARATELY FOR EACH UNIT IN J2. ENSURE NUMERIC RESPONSE FOR CONTROL OUTLET DOES NOT EXCEED 1] Control Outlet (When the item plugged into the control outlet is turned off, equipment plugged into dependent outlets that would normally stay on standby are also turned off. The control outlet is often color coded to differentiate it from the dependent outlets and outlets that are always on.) Televisions Televisions DVD players DVD players Blu ray disc players Blu ray disc players VCRs VCRs Dependent Outlets (Equipment plugged into dependent outlets will switch off when the item plugged into the control outlet is switched off. Dependent outlets are often color coded to differentiate them from the control outlet and outlets that are always on.) DVR or set top boxes (cable/satellite boxes) DVR or set top boxes (cable/satellite boxes) Streaming set top boxes Streaming set top boxes Streaming sticks Streaming sticks Amplifier/receivers Amplifier/receivers Game consoles Game consoles Stereos Stereos Speakers (for stereo) Speakers (for stereo) Lamps Lamps Room air conditioners Room air conditioners Desktop computers Desktop computers Monitors Monitors Laptops Laptops Printers Printers Fax machines Fax machines Scanners Scanners Computer speakers Computer speakers Wireless routers Wireless routers Cable modems Cable modems Other [SPECIFY: ] Other [SPECIFY: ] Appendix C. Coupons Program Supplemental Appendices Page C 40

187 J5. On average, for a typical weekday, how many hours per day is the equipment plugged into the control outlet switched on? How many hours per day is the equipment plugged into the control outlet switched on during a typical weekend day? [ASK SEPARATELY FOR EACH UNIT IN J2. NUMERIC RESPONSE RANGE FOR EACH DAY IS 0 TO 24] 1. Hours per typical weekday 2. Hours per typical weekend day Clothesline Kits J6. [ASK IF B4G, COL I 1] Earlier you indicated that you purchased [INSERT RESPONSE FROM B4G, COL I] outdoor clothesline umbrella stands or outdoor clothesline kits in Do you use these outdoor clotheslines to dry laundry at your home? 1. Yes 2. No [SKIP TO J13] J7. On average, for a typical week, how many days during the week and how many days on the weekend do you use your outdoor clotheslines/umbrella stands in the each of the following months? Enter 0 for none. [MAX NUMBER OF WEEKDAYS FOR RESPONSE IS 5 AND MAX NUMBER OF WEEKEND DAYS IS 2] Month Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Number of WEEK days each week use clotheslines/ umbrella stands (0, 1, 2, 3, 4 or 5) [IF J7 WEEKDAY=0 AND J7 WEEKEND=0 FOR ALL MONTHS THEN SKIP J8 AND J9] Number of WEEKEND days each week use clotheslines/umbrella stands (0, 1 or 2) J8. On average, how many loads of laundry do you dry on your outdoor clotheslines/umbrella stands in a day? 1. Open ended numeric [SPECIFY: ] Appendix C. Coupons Program Supplemental Appendices Page C 41

188 J9. On average, which hours of the day do you use your outdoor clotheslines/umbrella stands in the following months? AM J10. If you do not use your outdoor clotheslines/umbrella stands, what type of dryer do you use to dry your laundry? 1. Electric dryer 2. Gas dryer 3. I always dry my clothes on a clothesline 4. Other [SPECIFY: ] 98. (Don t know) 12 1 Weatherstripping Door Frame Kits J11. [ASK IF (PARTICIPANTS: B4I, COL I 1/NONPARTICIPANTS: B4I, COL II 1), OTHERWISE SKIP TO J14] Earlier you indicated that you purchased [INSERT RESPONSE FROM (PARTICIPANTS: B4I, COL I/NONPARTICIPANTS: B4I, COL II)] door frame weatherstripping kits in 2016 [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON]. Thinking of these kits, how many are currently installed in your home? [ENSURE NUMERIC RESPONSE DOES NOT EXCEED (PARTICIPANTS B4I, COL I/NONPARTICIPANTS: B4I, COL II] 1. Open ended numeric [SPECIFY: ] J12. [ASK IF J11>0] How many of the [INSERT J11] installed door frame weatherstripping kits were new for the home, replaced existing door frame weatherstripping or were added to existing door frame weatherstripping? [ENSURE SUM OF NUMERIC RESPONSES MATCHES J11] 1. # new door frame weatherstripping 2. # replaced existing door frame weatherstripping 3. # added to existing door frame weatherstripping PM Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Appendix C. Coupons Program Supplemental Appendices Page C 42

189 J13. [ASK IF J11< (PARTICIPANTS B4I, COL I/NONPARTICIPANTS: B4I, COL II)] How many of the [INSERT (PARTICIPANTS B4I, COL I/NONPARTICIPANTS: B4I, COL II) J11] door frame weatherstripping kits that you have yet to install do you plan on installing as described below? [ENSURE NUMERIC RESPONSE DOES NOT EXCEED (PARTICIPANTS B4I, COL I/NONPARTICIPANTS: B4I, COL II) J11] 1. Will be installed in the next two months 2. Will be installed in the next three to six months 3. Will be installed in the next seven to 12 months 4. Will be installed more than one year from now 5. Not planning on installing the door frame weatherstripping 98. (Don t know) Weatherstripping Foam/V Strip J14. [ASK IF (PARTICIPANTS: B4J, COL I 1/NONPARTICIPANTS: B4J, COL II 1), OTHERWISE SKIP TO K1] Earlier you indicated that you purchased [INSERT RESPONSE FROM (PARTICIPANTS: B4J, COL I/NONPARTICIPANTS: B4J, COL II)] foam or V strip weatherstripping packs in 2016 [IF PARTICIPANT, INCLUDE: USING A SAVE ON ENERGY COUPON]. Thinking of these foam or V strip packs, how many are currently installed in your home? [ENSURE NUMERIC RESPONSE DOES NOT EXCEED (PARTICIPANTS: B4J, COL I/NONPARTICIPANTS: B4J, COL II] 1. Open ended numeric [SPECIFY: ] J15. [ASK IF J14>0] How many of the [INSERT J14] installed foam or V strip weatherstripping packs were new for the home, replaced existing weatherstripping or were added to existing weatherstripping? [ENSURE SUM OF NUMERIC RESPONSES MATCHES J14] 1. # new foam or V strip weatherstripping 2. # replaced existing weatherstripping 3. # added to existing weatherstripping J16. [ASK IF J14< (PARTICIPANTS: B4J, COL I/NONPARTICIPANTS: B4J, COL II)] How many of the [INSERT (PARTICIPANTS: B4J, COL I/NONPARTICIPANTS: B4J, COL II) J14] foam or V strip weatherstripping packs that you have yet to install do you plan on installing as described below? [ENSURE NUMERIC RESPONSE DOES NOT EXCEED (PARTICIPANTS: B4J, COL I/NONPARTICIPANTS: B4J, COL II) J14] 1. Will be installed in the next two months 2. Will be installed in the next three to six months 3. Will be installed in the next seven to 12 months 4. Will be installed more than one year from now 5. Not planning on installing the foam or V strip weatherstripping 98. (Don t know) Appendix C. Coupons Program Supplemental Appendices Page C 43

190 J17. What is the primary source of heating for your house? [DO NOT READ] 1. Gas Furnace 2. Electric Baseboard 3. Electric Furnace 4. Gas Hydronic (forced hot water) 5. Oil Hydronic (forced hot water) 6. Other [SPECIFY: ] J18. Do you have central air conditioning? [DO NOT READ] 1. Yes 2. No 98. (Don t know) 99. (Refused) K. Timers K1. [ASK IF B4D 1), OTHERWISE SKIP TO L1] Earlier you indicated that you purchased [INSERT RESPONSE FROM B4D] heavy duty outdoor timer(s) in Thinking of these timers, how many of them are you currently using or have you used at any point in 2016? 1. Open ended numeric [SPECIFY: ] K2. [ASK IF K1<B4D] How many of the [B4D K1] timers that you have not yet used do you plan on starting to use as described below? [ENSURE NUMERIC RESPONSE DOES NOT EXCEED B4D K1] 1. Will start using in the next two months 2. Will start using in the next three to six months 3. Will start using in the next seven to 12 months 4. Will start using more than one year from now 5. Not planning on using 98. (Don t know) K3. [ASK IF K1>0] Thinking of your heavy duty outdoor timers, please indicate how many of each of the following types of equipment are currently plugged into a timer. (Or, for holiday or seasonal lighting, were plugged into a timer during their period of use in 2016.) Do not include equipment that is plugged into any outlet that is always on. [ENSURE NUMERIC RESPONSE DOES NOT EXCEED K1] 1. Pool Pumps 2. Spa Pumps/Heaters 3. Irrigation or Landscaping Pumps 4. Outdoor Lighting 5. Car Block Heaters 6. Holiday/Seasonal Lighting 7. Other [SPECIFY: ] Appendix C. Coupons Program Supplemental Appendices Page C 44

191 K4. On average, for a typical weekday, how many hours per day is the equipment plugged into the timer switched on? How many hours per day is the equipment plugged into the timer switched on during a typical weekend day? [ASK SEPARATELY FOR EACH UNIT IN K3 WITH RESPONSE OF AT LEAST 1. NUMERIC RESPONSE RANGE FOR EACH DAY IS 0 TO 24] 1. Hours per typical weekday 2. Hours per typical weekend day L. Rebound L1. [ASK IF PARTICIPANT AND (F4, G4 OR H2>0); REPEAT QUESTION FOR EACH CODE UNTIL ALL PRODUCTS HAVE BEEN SHOWN] You indicated you installed [INSERT 1] [INSERT 2] you purchased with a Save on Energy coupon. Since you installed your item(s), do you tend to have your new lights turned on more often, the same amount of time or less often than the lights you had installed previously? 1. Turned on more often 2. No change to hours of use 3. Turned on less often 4. I did not have a product installed previously 98. (Don t know) 99. (Refused) CODE IF INSERT 1 B H2>0 [RESPONSE FROM H2] N F4>0 [RESPONSE FROM F4] P/Q G4>0 [RESPONSE FROM G4] CODE INSERT 2 B N P/Q ENERGY STAR qualified light fixtures ENERGY STAR qualified specialty CFL light bulbs (downlights, reflectors, floods, globe, candle shapes) ENERGY STAR qualified LED light bulb(s) (standard or specialty) L2. [ASK IF L1=1; REPEAT FOR EACH PRODUCT] On an average day, how long are the lights on rather than staying off? 1. Hours Min Appendix C. Coupons Program Supplemental Appendices Page C 45

192 L3. [ASK IF J11 OR J14>0] You indicated you installed weatherstripping that you purchased with a Save on Energy coupon. Since you installed your weatherstripping, do you tend to set your thermostat to the same setting or a different setting than before you installed it? 1. I tend to set it to a higher temperature in the summer or lower temperature in the winter 2. I tend to set it to the same temperature as before 3. I tend to set it to a lower temperature in the summer or higher in the winter 98. (Don t know) 99. (Refused) L4. [ASK IF L3=1 3] What is your primary type of heating? 1. Electric furnace 2. Electric baseboard 3. Gas furnace 4. Other L5. By how many degrees have you changed your temperature setting? 1. [ ] degrees [DROPDOWN (FORCE RESPONSE): CELCIUS, FAHRENHEIT] M. Demographics M1. Do you or does anyone in your household receive any of the following types of benefits from any of the following government programs? [RANDOMIZE] 1. National Child Benefit Supplement (NCBS) 2. Allowance for the Survivor 3. Guaranteed Income Supplement 4. Allowance for Seniors 5. Ontario Works 6. Ontario Disability Support Program (ODSP) 7. Healthy Smiles Ontario Child Dental Program 8. Ontario Electricity Support Plan 9. None of the above M2. Have you or anyone in your household received Utility Low Income Energy Assistance Program (LEAP) grant within the past 12 months? 1. Yes 2. No M3. Have you or anyone in your household qualified to participate in a natural gas, low income, demand side management program during the past 12 months? 1. Yes 2. No Appendix C. Coupons Program Supplemental Appendices Page C 46

193 M4. What is the primary language used in your household? 1. English 2. French 3. Chinese 4. Spanish 5. German 6. Italian 7. Arabic 8. Other 99. (Refused) M5. What is the last level of education that you have completed? (Select one) 1. Grade school or less 2. Some high school 3. High school grad 4. Vocational/technical school 5. College 6. Some university 7. University grad 8. Postgraduate degree 99. (Refused) M6. How many people, including yourself, live in the household full time? (Select one) 1. One 2. Two 3. Three 4. Four 5. Five 6. Six 7. Seven or more 99. (Refused) M7. Do you own or rent your current place of residence? 1. Own 2. Rent 3. Occupy rent free 99. (Refused) Appendix C. Coupons Program Supplemental Appendices Page C 47

194 M8. What type of home do you live in? 1. Single family detached house 2. Single family semi detached house 3. Townhouse or rowhouse 4. Duplex, triplex or fourplex 5. Condominium 6. Apartment 7. Mobile home 8. Other [SPECIFY: ] 99. (Refused) M9. How many bedrooms are in your home? 1. 1 bedroom 2. 2 bedrooms 3. 3 bedrooms 4. 4 bedrooms or more 99. (Refused) M10. How many bathrooms are in your home? 1. 1 bathroom bathrooms 3. 2 bathrooms or more 99. (Refused) M11. How many stories is your home? 1. One 2. Two 3. Three or more 99. (Refused) M12. How old is your home? (estimates are fine) 1. Less than two years old (built in 2015 or after) 2. Two to less than seven years old ( ) 3. Seven to less than 12 ( ) to less than 17 ( ) to less than 22 ( ) to less than 27 ( ) to less than 32 ( ) to less than 42 ( ) to less than 52 ( ) Appendix C. Coupons Program Supplemental Appendices Page C 48

195 to less than 67 ( ) to less than 92 ( ) years or more (Built in 1924 or earlier) 99. (Refused) M13. How many square feet is your home? 1. <1,000 square feet 2. 1,000 1,999 square feet 3. 2,000 2,999 square feet 4. 3,000 4,999 square feet 5. >5,000 square feet 99. (Refused) M14. Do you have wireless internet in your home? 1. Yes 2. No 98. (Don t know) 99. (Refused) M15. Please tell me which of the following categories applies to your total household income for the year 2016? 1. Under $20, $20,000 to under $30, $30,000 to under $40, $40,000 to under $50, $50,000 to under $60, $60,000 to under $80, $80,000 to under $100, $100,000 to under $120, $120,000 or more 98. (Don t know) 99. (Refused) Thank you for your time. Appendix C. Coupons Program Supplemental Appendices Page C 49

196 Ipsos Customer Satisfaction (CSAT) Survey 1 S1_1. [Began participating in the saveonenergy peaksaver PLUS Program] Which, if any, of the following have you done since...? 2 S1_3. [Participated in the saveonenergy Heating and Cooling Incentive Program] Which, if any, of the following have you done since...? 3 S1_4. [Used a saveonenergy Coupon] Which, if any, of the following have you done since...? 4 1_1. [Saveonenergy Peaksaver PLUS Program] Considering your experience, how likely would you be to recommend the... to a friend or colleague using a scale from 1 to 10 where 1 is not at all likely and 10 is extremely likely? 5 1_3. [Saveonenergy Heating and Cooling Incentive Program] Considering your experience, how likely would you be to recommend the... to a friend or colleague using a scale from 1 to 10 where 1 is not at all likely and 10 is extremely likely? 6 1_4. [Saveonenergy Coupons Program] Considering your experience, how likely would you be to recommend the... to a friend or colleague using a scale from 1 to 10 where 1 is not at all likely and 10 is extremely likely? 7 1. [SUMMARY TOP4BOX (7,8,9,10)] Considering your experience, how likely would you be to recommend the... to a friend or colleague using a scale from 1 to 10 where 1 is not at all likely and 10 is extremely likely? 8 1. [SUMMARY TOP3BOX (8,9,10)] Considering your experience, how likely would you be to recommend the... to a friend or colleague using a scale from 1 to 10 where 1 is not at all likely and 10 is extremely likely? 9 1. [SUMMARY MID3BOX (5,6,7)] Considering your experience, how likely would you be to recommend the... to a friend or colleague using a scale from 1 to 10 where 1 is not at all likely and 10 is extremely likely? [SUMMARY LOW3BOX (1,2,3)] Considering your experience, how likely would you be to recommend the... to a friend or colleague using a scale from 1 to 10 where 1 is not at all likely and 10 is extremely likely? [SUMMARY LOW4BOX (1,2,3,4)] Considering your experience, how likely would you be to recommend the... to a friend or colleague using a scale from 1 to 10 where 1 is not at all likely and 10 is extremely likely? [SUMMARY MEAN] Considering your experience, how likely would you be to recommend the... to a friend or colleague using a scale from 1 to 10 where 1 is not at all likely and 10 is extremely likely? 13 2_1_1. [Saveonenergy Peaksaver PLUS Program] [Ease of participating] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 14 2_1_3. [Saveonenergy Peaksaver PLUS Program] [Amount of time it took after signing up to having the peaksaver equipment installed] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. Appendix C. Coupons Program Supplemental Appendices Page C 50

197 15 2_1_6. [Saveonenergy Peaksaver PLUS Program] [The amount and clarity of information you received about how to participate] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 16 2_1_7. [Saveonenergy Peaksaver PLUS Program] [Making the application process easy] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 17 2_1_8. [Saveonenergy Peaksaver PLUS Program] [That you received the benefits from participating that you were expecting] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 18 2_1_9. [Saveonenergy Peaksaver PLUS Program] [Courtesy of individuals that visited your home] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 19 2_1_10. [Saveonenergy Peaksaver PLUS Program] [Courtesy of individuals that you contacted to schedule an appointment] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 20 2_3_1. [Saveonenergy Heating and Cooling Incentive Program] [Ease of participating] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 21 2_3_2. [Saveonenergy Heating and Cooling Incentive Program] [Amount of cash rebate received as a result of participating] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 22 2_3_5. [Saveonenergy Heating and Cooling Incentive Program] [Amount of time it took to receive the rebate after the furnace or air conditioner was replaced] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 23 2_3_6. [Saveonenergy Heating and Cooling Incentive Program] [The amount and clarity of information you received about how to participate] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 24 2_3_7. [Saveonenergy Heating and Cooling Incentive Program] [Making the application process easy] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 25 2_3_8. [Saveonenergy Heating and Cooling Incentive Program] [That you received the benefits from participating that you were expecting] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. Appendix C. Coupons Program Supplemental Appendices Page C 51

198 26 2_4_1. [Saveonenergy Coupons Program] [Ease of participating] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 27 2_4_2. [Saveonenergy Coupons Program] [Amount of cash rebate received as a result of participating] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 28 2_4_8. [Saveonenergy Coupons Program] [That you received the benefits from participating that you were expecting] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 29 2_4_12. [Saveonenergy Coupons Program] [Being easy to find the items you bought in the store] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 30 2_1. [SUMMARY TOP4BOX (7,8,9,10)] [SAVEONENERGY PEAKSAVER PLUS PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 31 2_2. [SUMMARY TOP4BOX (7,8,9,10)] [SAVEONENERGY FRIDGE AND FREEZER PICK UP PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 32 2_3. [SUMMARY TOP4BOX (7,8,9,10)] [SAVEONENERGY HEATING AND COOLING INCENTIVE PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 33 2_4. [SUMMARY TOP4BOX (7,8,9,10)] [SAVEONENERGY COUPONS PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 34 2_1. [SUMMARY TOP3BOX (8,9,10)] [SAVEONENERGY PEAKSAVER PLUS PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 35 2_2. [SUMMARY TOP3BOX (8,9,10)] [SAVEONENERGY FRIDGE AND FREEZER PICK UP PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 36 2_3. [SUMMARY TOP3BOX (8,9,10)] [SAVEONENERGY HEATING AND COOLING INCENTIVE PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 37 2_4. [SUMMARY TOP3BOX (8,9,10)] [SAVEONENERGY COUPONS PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 38 2_1. [SUMMARY MID3BOX (5,6,7)] [SAVEONENERGY PEAKSAVER PLUS PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. Appendix C. Coupons Program Supplemental Appendices Page C 52

199 39 2_2. [SUMMARY MID3BOX (5,6,7)] [SAVEONENERGY FRIDGE AND FREEZER PICK UP PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 40 2_3. [SUMMARY MID3BOX (5,6,7)] [SAVEONENERGY HEATING AND COOLING INCENTIVE PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 41 2_4. [SUMMARY MID3BOX (5,6,7)] [SAVEONENERGY COUPONS PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 42 2_1. [SUMMARY LOW3BOX (1,2,3)] [SAVEONENERGY PEAKSAVER PLUS PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 43 2_2. [SUMMARY LOW3BOX (1,2,3)] [SAVEONENERGY FRIDGE AND FREEZER PICK UP PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 44 2_3. [SUMMARY LOW3BOX (1,2,3)] [SAVEONENERGY HEATING AND COOLING INCENTIVE PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 45 2_4. [SUMMARY LOW3BOX (1,2,3)] [SAVEONENERGY COUPONS PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 46 2_1. [SUMMARY LOW4BOX (1,2,3,4)] [SAVEONENERGY PEAKSAVER PLUS PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 47 2_2. [SUMMARY LOW4BOX (1,2,3,4)] [SAVEONENERGY FRIDGE AND FREEZER PICK UP PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 48 2_3. [SUMMARY LOW4BOX (1,2,3,4)] [SAVEONENERGY HEATING AND COOLING INCENTIVE PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 49 2_4. [SUMMARY LOW4BOX (1,2,3,4)] [SAVEONENERGY COUPONS PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 50 2_1. [SUMMARY MEAN] [SAVEONENERGY PEAKSAVER PLUS PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 51 2_2. [SUMMARY MEAN] [SAVEONENERGY FRIDGE AND FREEZER PICK UP PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. Appendix C. Coupons Program Supplemental Appendices Page C 53

200 52 2_3. [SUMMARY MEAN] [SAVEONENERGY HEATING AND COOLING INCENTIVE PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 53 2_4. [SUMMARY MEAN] [SAVEONENERGY COUPONS PROGRAM] Thinking about your participation in the..., rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. 54 3_1_3. [Saveonenergy Peaksaver PLUS Program] [The contractor that installed your peaksaver PLUS equipment] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 55 3_1_5. [Saveonenergy Peaksaver PLUS Program] [saveonenergy programs] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 56 3_3_2. [Saveonenergy Heating and Cooling Incentive Program] [The contractor that installed your new heating or cooling equipment] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 57 3_3_5. [Saveonenergy Heating and Cooling Incentive Program] [saveonenergy programs] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 58 3_4_1. [Saveonenergy Coupons Program] [The retailer where you redeemed your saveonenergy coupon] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 59 3_4_5. [Saveonenergy Coupons Program] [saveonenergy programs] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 60 3_1. [SUMMARY TOP4BOX (7,8,9,10)] [SAVEONENERGY PEAKSAVER PLUS PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 61 3_2. [SUMMARY TOP4BOX (7,8,9,10)] [SAVEONENERGY FRIDGE AND FREEZER PICK UP PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 62 3_3. [SUMMARY TOP4BOX (7,8,9,10)] [SAVEONENERGY HEATING AND COOLING INCENTIVE PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? Appendix C. Coupons Program Supplemental Appendices Page C 54

201 63 3_4. [SUMMARY TOP4BOX (7,8,9,10)] [SAVEONENERGY COUPONS PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 64 3_1. [SUMMARY TOP3BOX (8,9,10)] [SAVEONENERGY PEAKSAVER PLUS PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 65 3_2. [SUMMARY TOP3BOX (8,9,10)] [SAVEONENERGY FRIDGE AND FREEZER PICK UP PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 66 3_3. [SUMMARY TOP3BOX (8,9,10)] [SAVEONENERGY HEATING AND COOLING INCENTIVE PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 67 3_4. [SUMMARY TOP3BOX (8,9,10)] [SAVEONENERGY COUPONS PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 68 3_1. [SUMMARY MID3BOX (5,6,7)] [SAVEONENERGY PEAKSAVER PLUS PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 69 3_2. [SUMMARY MID3BOX (5,6,7)] [SAVEONENERGY FRIDGE AND FREEZER PICK UP PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 70 3_3. [SUMMARY MID3BOX (5,6,7)] [SAVEONENERGY HEATING AND COOLING INCENTIVE PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 71 3_4. [SUMMARY MID3BOX (5,6,7)] [SAVEONENERGY COUPONS PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 72 3_1. [SUMMARY LOW3BOX (1,2,3)] [SAVEONENERGY PEAKSAVER PLUS PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 73 3_2. [SUMMARY LOW3BOX (1,2,3)] [SAVEONENERGY FRIDGE AND FREEZER PICK UP PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 74 3_3. [SUMMARY LOW3BOX (1,2,3)] [SAVEONENERGY HEATING AND COOLING INCENTIVE PROGRAM] As a result of your participation in the..., what would you say is your impression Appendix C. Coupons Program Supplemental Appendices Page C 55

202 of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 75 3_4. [SUMMARY LOW3BOX (1,2,3)] [SAVEONENERGY COUPONS PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 76 3_1. [SUMMARY LOW4BOX (1,2,3,4)] [SAVEONENERGY PEAKSAVER PLUS PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 77 3_2. [SUMMARY LOW4BOX (1,2,3,4)] [SAVEONENERGY FRIDGE AND FREEZER PICK UP PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 78 3_3. [SUMMARY LOW4BOX (1,2,3,4)] [SAVEONENERGY HEATING AND COOLING INCENTIVE PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 79 3_4. [SUMMARY LOW4BOX (1,2,3,4)] [SAVEONENERGY COUPONS PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 80 3_1. [SUMMARY MEAN] [SAVEONENERGY PEAKSAVER PLUS PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 81 3_2. [SUMMARY MEAN] [SAVEONENERGY FRIDGE AND FREEZER PICK UP PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 82 3_3. [SUMMARY MEAN] [SAVEONENERGY HEATING AND COOLING INCENTIVE PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 83 3_4. [SUMMARY MEAN] [SAVEONENERGY COUPONS PROGRAM] As a result of your participation in the..., what would you say is your impression of the following are using a scale of 1 to 10 where 1 is extremely negative and 10 is extremely positive? 84 4_1. [Saveonenergy Peaksaver PLUS Program] Did your participation in the... meet your expectations, exceed your expectations or did not meet your expectations? 85 4_3. [Saveonenergy Heating and Cooling Incentive Program] Did your participation in the... meet your expectations, exceed your expectations or did not meet your expectations? 86 4_4. [Saveonenergy Coupons Program] Did your participation in the... meet your expectations, exceed your expectations or did not meet your expectations? 87 5_1. [Saveonenergy Peaksaver PLUS Program] Why would you say that your participation...? Appendix C. Coupons Program Supplemental Appendices Page C 56

203 88 5_3. [Saveonenergy Heating and Cooling Incentive Program] Why would you say that your participation...? 89 5_4. [Saveonenergy Coupons Program] Why would you say that your participation...? 90 5x_1. [Saveonenergy Peaksaver PLUS Program] Why would you say that your participation...? 91 5x_3. [Saveonenergy Heating and Cooling Incentive Program] Why would you say that your participation...? 92 5x_4. [Saveonenergy Coupons Program] Why would you say that your participation...? 93 5y_1. [Saveonenergy Peaksaver PLUS Program] Why would you say that your participation...? 94 5y_3. [Saveonenergy Heating and Cooling Incentive Program] Why would you say that your participation...? 95 5y_4. [Saveonenergy Coupons Program] Why would you say that your participation...? 96 5z_1. [Saveonenergy Peaksaver PLUS Program] Why would you say that your participation...? 97 5z_3. [Saveonenergy Heating and Cooling Incentive Program] Why would you say that your participation...? 98 5z_4. [Saveonenergy Coupons Program] Why would you say that your participation...? 99 6_1. [Saveonenergy Peaksaver PLUS Program] Did you experience any problems while participating in the... and if so, was this problem fully resolved? 100 6_3. [Saveonenergy Heating and Cooling Incentive Program] Did you experience any problems while participating in the... and if so, was this problem fully resolved? 101 6_4. [Saveonenergy Coupons Program] Did you experience any problems while participating in the... and if so, was this problem fully resolved? [SUMMARY YES] Did you experience any problems while participating in the... and if so, was this problem fully resolved? 103 7_1. [Saveonenergy Peaksaver PLUS Program] How does your experience participating in this program influence your interest in participating in future saveonenergy programs? Would you say you are _3. [Saveonenergy Heating and Cooling Incentive Program] How does your experience participating in this program influence your interest in participating in future saveonenergy programs? Would you say you are _4. [Saveonenergy Coupons Program] How does your experience participating in this program influence your interest in participating in future saveonenergy programs? Would you say you are [SUMMARY TOP2BOX (MORE LIKELY)] How does your experience participating in this program influence your interest in participating in future saveonenergy programs? Would you say you are [SUMMARY LOW2BOX (LESS LIKELY)] How does your experience participating in this program influence your interest in participating in future saveonenergy programs? Would you say you are... Appendix C. Coupons Program Supplemental Appendices Page C 57

204 108 8_1. [Saveonenergy Peaksaver PLUS Program] And how does your experience with this program influence your interest in making other energy efficiency improvements to your home which are not covered by a saveonenergy program? Would you say you are _3. [Saveonenergy Heating and Cooling Incentive Program] And how does your experience with this program influence your interest in making other energy efficiency improvements to your home which are not covered by a saveonenergy program? Would you say you are _4. [Saveonenergy Coupons Program] And how does your experience with this program influence your interest in making other energy efficiency improvements to your home which are not covered by a saveonenergy program? Would you say you are [SUMMARY TOP2BOX (MORE LIKELY)] And how does your experience with this program influence your interest in making other energy efficiency improvements to your home which are not covered by a saveonenergy program? Would you say you are [SUMMARY LOW2BOX (LESS LIKELY)] And how does your experience with this program influence your interest in making other energy efficiency improvements to your home which are not covered by a saveonenergy program? Would you say you are GENDER 114 AGE 115 EDUCATION 116 REGION 117 INCOME 118 HOUSEHOLD COMPOSITION 119 HHCMP1. How many people are living or staying at your current address? 120 EMPLOYMENT STATUS 121 USMAR2. What is your marital status? 122 US01PGS. How much of your household's grocery shopping do you, yourself, do? 123 CAETHN2. Sometimes people identify themselves with a specific ethnicity or cultural background which is different from their citizenship or nationality. Thinking about your own identity in ethnic or cultural terms, please select which group you most identify yourself with. Ipsos Longitudinal Mass Market Research Survey (LMMRS) 1 QYEAR/MONTH. Age 2 QRESP_GENDER. What is your gender? 3 QLDC. LDC 4 Qregion. Region 5 Qhiddenreg. Region 6 QS3. Are you the primary decision maker in your household? 7 QS4. Do you receive, review or pay the electricity bill? 8 QS5. Are you at least aware of your household's electricity bill? Appendix C. Coupons Program Supplemental Appendices Page C 58

205 9 10 QA0. There are a number of programs and rebate offers in place to help Ontarians reduce their electricity use in their home. Which of the following statements best describes your awareness of these programs and offers? QA1. Since the beginning of January, what programs or initiatives have you heard about that encourage consumers to conserve electricity in the home? 11 QA2. Have you ever heard of Save on Energy? 12 QA3. Save on Energy [...] provides Ontarians with a wide variety of incentive programs and tips to help manage electricity use in the home, save money and help the environment. Having read this description, QA4. Where did you learn about Save on Energy? 14 QA5. How trustworthy do you find information from Save on Energy? QA6. [SUMMARY TOP2BOX (Strongly/Somewhat Agree)] Based on your understanding of Save on Energy, please rate how strongly you agree or disagree with the following statements.following advice from Save on Energy and participating in Save on Energy programs... QA6. [SUMMARY MIDBOX (Neither Agree Nor Disagree)] Based on your understanding of Save on Energy, please rate how strongly you agree or disagree with the following statements.following advice from Save on Energy and participating in Save on Energy programs... QA6. [SUMMARY LOW2BOX (Somewhat/Strongly Disagree)] Based on your understanding of Save on Energy, please rate how strongly you agree or disagree with the following statements. Following advice from Save on Energy and participating in Save on Energy programs... QA6_1. [Save on Energy helps increase the energy efficiency of my home] Based on your understanding of Save on Energy, please rate how strongly you agree or disagree with the following statements. Following advice from Save on Energy and participating in Save on Energy programs... QA6_2. [Save on Energy benefits my community] Based on your understanding of Save on Energy, please rate how strongly you agree or disagree with the following statements. Following advice from Save on Energy and participating in Save on Energy programs... QA6_3. [Save on Energy increases the comfort and value of my home] Based on your understanding of Save on Energy, please rate how strongly you agree or disagree with the following statements. Following advice from Save on Energy and participating in Save on Energy programs... QA6_4. [Save on Energy helps me save on my energy bills] Based on your understanding of Save on Energy, please rate how strongly you agree or disagree with the following statements. Following advice from Save on Energy and participating in Save on Energy programs... Appendix C. Coupons Program Supplemental Appendices Page C 59

206 QA6_5. [Save on Energy makes me feel better] Based on your understanding of Save on Energy, please rate how strongly you agree or disagree with the following statements. Following advice from Save on Energy and participating in Save on Energy programs... QA6_6. [Save on Energy encourages people to make energy efficient choices for their homes] Based on your understanding of Save on Energy, please rate how strongly you agree or disagree with the following statements. Following advice from Save on Energy and participating in Save on Energy programs... QA6_7. [Save on Energy helps people and businesses in Ontario be more successful] Based on your understanding of Save on Energy, please rate how strongly you agree or disagree with the following statements. Following advice from Save on Energy and participating in Save on Energy programs... QA6_8. [Save on Energy encourages Ontarians to talk about energy efficiency] Based on your understanding of Save on Energy, please rate how strongly you agree or disagree with the following statements. Following advice from Save on Energy and participating in Save on Energy programs QA7. Which organization or organizations are responsible for Save on Energy? 27 QA8. Have you ever heard of Power What's Next? 28 QA9. To the best of your knowledge, is Power What's Next' different than Save on Energy? 29 QA10. Have you seen this commercial on television? Total 30 QA10. Have you seen this commercial on television? Recall Cupcake 31 QA10. Have you seen this commercial on television? Habenaro Nachos 32 QA10. Have you seen this commercial on television? Manifesto 33 QA10. Have you seen this commercial on television? Train 34 QA11. Have you seen this ads, either online, newspapers or other media? 35 QS8. Do you own or rent your current place of residence? 36 QS9. What type of home do you live in? 37 QS10. How old is your home? Your best estimate is fine. 38 QS12. What type of air conditioning, if any, do you have in your home? 39 QS13. What is the approximate age of your central air conditioning system? 40 QS14. What do you mainly use to heat your home? Please think about the method you use and not the fuel source such as natural gas, oil, etc. 41 QS15. How old is your furnace? QS16_1. [Replaced your existing furnace] Since January 1st, 2016 which of the following activities, if any, have you done? QS16_2. [Replaced your existing central air conditioner (CAC)] Since January 1st, 2016 which of the following activities, if any, have you done? QS17_1. [Replaced your existing furnace] Have you done any of these activities before January 1st, 2016? Appendix C. Coupons Program Supplemental Appendices Page C 60

207 45 QS17_2. [Replaced your existing central air conditioner (CAC)] Have you done any of these activities before January 1st, 2016? 46 QS18. How many years ago did you replace your furnace? 47 QS19. How many years ago did you replace your central air conditioner? 48 QB1. Have you heard of this program by either name or description? COUPON EVENT 49 QB2. Have you heard of this program by either name or description? HEATING & COOLING INCENTIVE 50 QC1. Where did you learn about the Save on Energy COUPON EVENT initiative? QC2. Please indicate which of the following products you have purchased since January 1st, 2016 using a Save on Energy coupon. A few examples of the coupons can be found below. QC3. In which month did you purchase a product using a Save on Energy coupon? If you purchased a product in more than one month please indicate the most recent and answer the following questions based on that most recent experience. QC4. Considering your experience, how likely would you be to recommend the Save on Energy Coupons Program to a friend or colleague using a scale from 1 to 10 where 1 is not at all likely and 10 is extremely likely? QC5. [SUMMARY MEAN] Thinking about your participation in the Save on Energy Coupons Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QC5. [SUMMARY TOP3BOX (Extremely satisfied (8 10))] Thinking about your participation in the Save on Energy Coupons Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. Thinking about your participation in the Save on Energy Coupons Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QC5. [SUMMARY MID4BOX (4 7)] Thinking about your participation in the Save on Energy Coupons Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. Thinking about your participation in the Save on Energy Coupons Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QC5. [SUMMARY LOW3BOX (Extremely dissatisfied (1 3))] Thinking about your participation in the Save on Energy Coupons Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. Thinking about your participation in the Save on Energy Coupons Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QC5_1. [Ease of participating] Thinking about your participation in the Save on Energy Coupons Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. Appendix C. Coupons Program Supplemental Appendices Page C 61

208 QC5_2. [Amount of cash discount received as a result of participating] Thinking about your participation in the Save on Energy Coupons Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QC5_3. [Being easy to find the items you bought in the store] Thinking about your participation in the Save on Energy Coupons Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QC5_4. [Being easy to use the coupon at the cash register] Thinking about your participation in the Save on Energy Coupons Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QC6. As a result of your participation in the Save on Energy Coupons Program what would you say is your impression of the retailer where you redeemed your coupon? QC7. Did your participation in the Save on Energy Coupons Program meet your expectations, exceed your expectations or did not meet your expectations? QC8. Did you experience any problems while participating in the Save on Energy Coupons Program and if so, was this problem fully resolved? 65 QC9. What was the problem you experienced? QC10. How does your experience participating in this program influence your interest in participating in future Save on Energy programs? QD1. Where did you learn about the Save on Energy HEATING & COOLING INCENTIVE initiative? QD2. How satisfied were you with the amount and clarity of information about the program you received from the contractor on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QD3. Did you or your contractor apply for the Save on Energy HEATING & COOLING INCENTIVE rebate? QD3a. In which month did you participate? If you participated in more than one month please indicate the most recent month that you participated and answer the following questions based on this most recent experience. QD4. Considering your experience, how likely would you be to recommend the Save on Energy Heating and Cooling Incentive Program to a friend or colleague using a scale from 1 to 10 where 1 is not at all likely and 10 is extremely likely? QD5. [SUMMARY MEAN] Thinking about your participation in the Save on Energy Heating and Cooling Incentive Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QD5. [SUMMARY TOP3BOX (Extremely satisfied (8 10))] Thinking about your participation in the Save on Energy Heating and Cooling Incentive Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. Appendix C. Coupons Program Supplemental Appendices Page C 62

209 QD5. [SUMMARY MID4BOX (4 7)] Thinking about your participation in the Save on Energy Heating and Cooling Incentive Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QD5. [SUMMARY LOW3BOX (Extremely dissatisfied (1 3))] Thinking about your participation in the Save on Energy Heating and Cooling Incentive Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QD5_1. [Ease of finding an eligible contractor] Thinking about your participation in the Save on Energy Heating and Cooling Incentive Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QD5_2. [Amount of cash rebate received as a result of participating] Thinking about your participation in the Save on Energy Heating and Cooling Incentive Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QD5_3. [Amount of time it took to receive the rebate after the furnace or air conditioner was replaced] Thinking about your participation in the Save on Energy Heating and Cooling Incentive Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QD5_4. [The amount and clarity of information about how to participate] Thinking about your participation in the Save on Energy Heating and Cooling Incentive Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QD5_5. [Making it easy to participate] Thinking about your participation in the Save on Energy Heating and Cooling Incentive Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QD5_6. [That you received the benefits from participating that you were expecting] Thinking about your participation in the Save on Energy Heating and Cooling Incentive Program, rate your satisfaction with the following on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied. QD6. As a result of your participation in the Save on Energy Heating and Cooling Incentive Program, what would you say is your impression of the contractor that installed your new equipment? QD7. [SUMMARY MEAN] How satisfied are you with the contractor in each of the following areas on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied? QD7. [SUMMARY TOP3BOX (Extremely satisfied (8 10))] How satisfied are you with the contractor in each of the following areas on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied? QD7. [SUMMARY MID4BOX (4 7)] How satisfied are you with the contractor in each of the following areas on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied? Appendix C. Coupons Program Supplemental Appendices Page C 63

210 QD7. [SUMMARY LOW3BOX (Extremely dissatisfied (1 3))] How satisfied are you with the contractor in each of the following areas on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied? QD7_1. [Keeping the appointment and arriving on time] How satisfied are you with the contractor in each of the following areas on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied? QD7_2. [Acting in a professional manner] How satisfied are you with the contractor in each of the following areas on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied? QD7_3. [Leaving the job site clean and in good shape] How satisfied are you with the contractor in each of the following areas on a scale of 1 to 10 where 1 is extremely dissatisfied and 10 is extremely satisfied? QD8. Did your participation in the Save on Energy Heating and Cooling Incentive Program meet your expectations, exceed your expectations or did not meet your expectations? QD9. Did you experience any problems while participating in the Save on Energy Heating and Cooling Incentive Program and if so, was this problem fully resolved? 92 QD10. What was the problem you experienced? 93 QD11. How does your experience participating in this program influence your interest in participating in future Save on Energy programs? 94 QCAEDU2 / F1. What is the highest degree or level of school you have completed? 95 QF2. Which of the following best describes your current job status? 96 QF3. Which of the following best describes your marital status? 97 QF4. How many people are living or staying at your current address? 98 QF4A. Are there any children 18 years or younger living in your household? QF5. For statistical purposes only, please tell me which of the following categories applies to your total household income for the year 2015? QF6. Do you or does anyone in your household receive any of the following types of benefits from any of the following government programs QF7. Have you received Utility Low Income Energy Assistance Program (LEAP) grant within the past 12 months? QF8. Have you qualified to participate in a natural gas low income Demand Site Management program during the past 12 months? 103 PlaylistAlloc Checking table for Banner2 104 Ethnicity Appendix C. Coupons Program Supplemental Appendices Page C 64

211 Heating and Cooling Program Supplemental Appendices PY2016 Heating and Cooling Program Demographics As in past years, over half of the participants households reported incomes over $80,000 per year (Figure 12). Additionally, as in PY2015, participants most frequently reported that their households earn over $120,000. This trend suggests that households with higher incomes are likely to participate in the program. Figure 12. Participant Household Income 70% 60% 58% 53% 52% 60% 50% 40% 30% 20% 10% 4% 18% 19% 29% 23% 23% 16% 11% 13% 7% 9% 4% 0% PY 2013 (n=139) PY 2014 (n=156) PY 2015 (n=303) PY 2016 (n=155) Under $30,000 $30,000 to under $50,000 $50,000 to under $80,000 $80,000 or more Source: Participant Survey Question H13. Which of the following categories applies to your total household income? Participants reported being well educated. As indicated in Figure 13, 77% of participants had either a college, university or postgraduate degree, while 36% of the population of Ontario had achieved a comparable level of education. Appendix D. Heating and Cooling Program Supplemental Appendices Page D 65

212 Figure 13. Participant and Ontario Population Education Level Source for program participants: Participant Survey Question H2. What is the highest level of education that you have completed? (n=155) Source for population of Ontario: Statistics Canada Census of Population. Population 15 years and over by highest degree, certificate or diploma, by province and territory (Quebec, Ontario, Manitoba, Saskatchewan). Last modified July 29, tableaux/sum som/l01/cst01/educ41b eng.htm Ninety percent of participants said that the primary language in their household is English. As illustrated in Figure 14, the remaining participants primarily speak French and other languages (including Chinese or Arabic). In Ontario, by comparison, 79% of households report that the language most frequently spoken at home is English. Figure 14. Participant and Ontario Population Primary Household Languages Source for program participants: Participant Survey Question A1. What is the primary language used in your household? (n=214) Source for population of Ontario: Statistics Canada Census of Population. Population by home language, by province and territory (Quebec, Ontario, Manitoba, Saskatchewan). Last modified February 13, tableaux/sumsom/l01/cst01/demo61b eng.htm Appendix D. Heating and Cooling Program Supplemental Appendices Page D 66

213 All but two respondents reported that they own their current residence (n=210) and 87% reported living in a single family home (Figure 15). Sixty percent of participants reported living in a two story home, 31% reported living in a single floor home and 9% reported living in a three story home (n=208). Figure 15. Participant Home Type Source: Participant Survey Question A6. What type of home do you live in? (n=210) While the age of respondents homes was fairly evenly distributed across a range from 12 to 67 years old, the most frequently reported age of the home was between 32 and 42 years (Figure 16). Figure 16. Participant Home Age Source: Participant Survey Question A10. How old is your home? (n=210) Ninety nine percent of participants reported owning their home, while in the population of Ontario 68% of participants own their home (Figure 17). Appendix D. Heating and Cooling Program Supplemental Appendices Page D 67

214 Figure 17. Participant and Ontario Population Home Ownership Source for program participants: Participant Survey Question A5. Do you own or rent your current place of residence? (n=210) Source for population of Ontario: Statistics Canada. Dwelling characteristics and household equipment, by province (Ontario). Last modified January 27, tableaux/sum som/l01/cst01/famil133g eng.htm Ninety three percent of program participants reported that they lived in homes with three or more bedrooms, while in the population of Ontario 64% of homes have three or more bedrooms (Figure 18). Additionally, 89% of program participants reported that they that their home had two or more bedrooms, while in in Ontario the 52% of homes have two or more bathrooms. Figure 18. Participant and Ontario Population Number of Bedrooms Source for program participants: Participant Survey Question A7. How many bedrooms are in your home? (n=209) Source for population of Ontario: Statistics Canada. Dwelling characteristics and household equipment, by province (Ontario). Last modified January 27, tableaux/sum som/l01/cst01/famil133g eng.htm Additionally, 91% of respondents reported having wireless internet at their home, while in Ontario overall only 19% of homes have wireless internet in PY2015. Appendix D. Heating and Cooling Program Supplemental Appendices Page D 68

215 PY2016 Heating and Cooling Program Participant Survey Survey Section Screening Awareness and Motivation Installation Experience and Rebound NTG Intention and Influence Questions Satisfaction Spillover Demographics Section Letter A B C D E F G Introduction Hi, my name is, and I m calling from Nielsen on behalf of the Save on Energy program sponsored by the Independent Electricity System Operator, or IESO. Our records show that your household received a $[TOTAL REBATE AMOUNT] rebate from the IESO for the new [IF FURNACEINSTALLEDINDCTR=1 AND ACINSTALLEDINDCTR=0] furnace [IF FURNACEINSTALLEDINDCTR=0 AND ACINSTALLEDINDCTR=1] central air conditioner [IF FURNACEINSTALLEDINDCTR=1 AND ACINSTALLEDINDCTR=1] furnace and central air conditioner that you had installed last year. INTRO 1: Are you the person in your household who is most knowledgeable about the rebate program your household participated in, which included installation of the new [IF FURNACEINSTALLEDINDCTR=1 AND ACINSTALLEDINDCTR=0] furnace [IF FURNACEINSTALLEDINDCTR=0 AND ACINSTALLEDINDCTR=1] central air conditioner [IF FURNACEINSTALLEDINDCTR=1 AND ACINSTALLEDINDCTR=1] furnace and central air conditioner? [DO NOT READ; SELECT FIRST RESPONSE] 1. Yes [GO TO INTRO 1.1] 2. No [GO TO INTRO 1.2] 3. Not a good time [GO TO INTRO 1.3] 4. No one at home is aware of program [GO TO INTRO 1.4] [IF NEEDED] I only need about minutes The Ontario Power Authority, or OPA, merged with the Independent Electricity System Operator a few years ago Heating and cooling appliances refer to your home s furnace and/or central air conditioner Nielsen adheres to the code of practise of the Marketing Research & Intelligence Association, and any opinions or comments you share with us will be reported only after all personally identifying information has been removed. Please be assured that no sales or promotional solicitation will occur during the interview or afterwards as a result of your participation. If you have questions or concerns about this study, you may call IESO s call centre directly with this phone number: or heatingandcoolingincentive@ieso.ca Appendix D. Heating and Cooling Program Supplemental Appendices Page D 69

216 INTRO 1.1. [ASK IF INTRO1=1] I d like to ask you some questions about your experience with the Heating and Cooling Rebate Program. Your feedback is very important to the IESO and will help inform future program offerings to help Ontario residents save energy and reduce energy bills. Would you be able to complete the survey now? 1. Yes [SKIP TO A1] 2. No [SKIP TO INTRO 1.3] INTRO 1.2. [IF INTRO1=2] May I please speak to the person who knows the most about the rebate program? [IF RESPONDENTS SAYS SOMETHING LIKE MY CONTRACTOR REALLY DID EVERYTHING YOU WOULD NEED TO TALK TO THEM, RESPOND WITH WE WANT TO ASK YOU SOME QUESTIONS ABOUT THE EXPERIENCE OF WORKING WITH YOUR CONTRACTOR CAN I PLEASE SPEAK WITH THE PERSON WHO WAS MOST INVOLVED WITH THE CONTRACTOR WHO INSTALLED IT? ] 1. Yes [RETURN TO INTRO 1 AND CONTINUE] 2. No [SKIP TO INTRO 1.3] INTRO 1.3. [IF INTRO1=3] [ASK FOR CALL BACK DAY/TIME] INTRO 1.4. [IF INTRO1=4] [THANK AND TERMINATE] [INSERT STANDARD CELL PHONE SCREENING TO CHECK IF RESPONDENT IS ON CELL AND TO MAKE SURE THEY ARE NOT DRIVING, BUT ARE SAFE TO COMPLETE THE SURVEY] A. Screening A1. Thank you. According to program records, you received a Save on Energy rebate for [IF FURNACEINSTALLEDINDCTR=1 AND ACINSTALLEDINDCTR=0] a furnace [IF FURNACEINSTALLEDINDCTR=0 AND ACINSTALLEDINDCTR=1] a central air conditioner [IF FURNACEINSTALLEDINDCTR=1 AND ACINSTALLEDINDCTR=1] both a furnace and a central air conditioner in Is that correct? 1. Yes, received a rebate for furnace only [ONLY SHOW IF FURNACEINSTALLEDINDCTR=1 AND ACINSTALLEDINDCTR=0] Qualifies for Furnace only complete 2. Yes, received a rebate for central air conditioning only [ONLY SHOW IF FURNACEINSTALLEDINDCTR=0 AND ACINSTALLEDINDCTR=1] Qualifies for CAC only complete 3. Yes, received rebate for both furnace and central air conditioning [ONLY SHOW IF FURNACEINSTALLEDINDCTR=1 AND ACINSTALLEDINDCTR=1] Qualifies for Furnace and CAC complete 4. No, only received a rebate for a furnace [DO NOT SHOW IF FURNACEINSTALLEDINDCTR=1 AND ACINSTALLEDINDCTR=0] Qualifies for Furnace only complete Appendix D. Heating and Cooling Program Supplemental Appendices Page D 70

217 5. No, only received a rebate for an air conditioner [DO NOT SHOW IF FURNACEINSTALLEDINDCTR=0 AND ACINSTALLEDINDCTR=1] Qualifies for CAC only complete 6. No, received rebate for both furnace and central air conditioning [DO NOT SHOW IF FURNACEINSTALLEDINDCTR=1 AND ACINSTALLEDINDCTR=1] Qualifies for Furnace and CAC complete 7. No, did not receive a rebate for either in 2016 [THANK AND TERMINATE] 8. Does not remember [THANK AND TERMINATE] COMPUTE QUOTA_TYPE BASED ON A1: FURNANCE ONLY = MAX 70 CAC ONLY = MAX 70 FURNANCE AND CAC = MAX 70 A2. Was the [IF A1=1 OR 4] furnace [IF A1=2 OR 5] central air conditioner [IF A1=3 OR 6] furnace and central air conditioner installed in an existing home or a new home that was being built? 1. Existing home 2. New home [SKIP TO Q9] 98. (Don t know) [THANK AND TERMINATE] 99. (Refused) [THANK AND TERMINATE] A3. [ASK IF A1=2, 3, 5 or 6] Did you replace an existing central or room air conditioning unit? [READ LIST ONLY IF NEEDED] 1. Yes home already had central air conditioning 2. Yes home already had room air conditioning 3. No did not previously have air conditioning 98. (Don t know) [THANK AND TERMINATE] 99. (Refused) [THANK AND TERMINATE] B. Awareness and Motivation B1. [ASK IF A1=1, 3, 4 OR 6 AND A2=1] Which of the following best describes the age of the heating system you were using before you installed your new furnace? [READ LIST] 1. One to five years old 2. Six to 10 years old to 15 years old years or older 98. (Don t know) 99. (Refused) Appendix D. Heating and Cooling Program Supplemental Appendices Page D 71

218 B2. [ASK IF A1=1, 3, 4 OR 6 AND A2=1] Which of the following best describes the condition of the previous heating system that you replaced? [READ LIST; SINGLE RESPONSE] 1. It was totally broken and no longer provided heating 2. It did not work properly or as good as it used to, but still provided heat 3. It was in good working condition 98. (Don t know) 99. (Refused) B3. [ASK IF A1=1, 3, 4 OR 6 AND A2=1] Why did you decide to replace your heating system? [PROBE: ANY OTHER REASONS? DO NOT READ; MULTIPLE RESPONSE] 1. Furnace was broken/malfunctioning and was unrepairable, had to be replaced 2. Furnace was broken/malfunctioning and was repairable, but replaced instead 3. Furnace was old and replaced before it broke 4. Wanted to improve comfort 5. Wanted to make home more energy efficient in order to save money on energy bills [PROBE TO CODE] 6. Wanted to make home more energy efficient in order to help the environment [PROBE TO CODE] 7. Contractor recommendation 8. Interested in rebates/incentives 9. I had planned to install a new air conditioning system and decided to also install a new furnace 10. It was part of a larger home remodeling project 11. Other [SPECIFY: ] 98. (Don t know) 99. (Refused) B4. [ASK IF A1=2, 3, 5 OR 6 AND A3=1 OR 2] Which of the following best describes the age of the air conditioner you were using before you installed your new air conditioner? [READ LIST] 1. One to five years old 2. Six to 10 years old to 15 years old years or older 98. (Don t know) 99. (Refused) Appendix D. Heating and Cooling Program Supplemental Appendices Page D 72

219 B5. [ASK IF A1=2, 3, 5 OR 6 AND A3=1 OR 2] Which of the following best describes the condition of the previous air conditioning system that you replaced? [READ LIST; SINGLE RESPONSE] 1. It was totally broken and no longer provided cool air 2. It did not work properly or as good as it used to, but still provided cool air 3. It was in good working condition 98. (Don t know) 99. (Refused) B6. [ASK IF A1=2, 3, 5 OR 6 AND A3=1 OR 2] Why did you decide to replace your air conditioning system? [PROBE: ANY OTHER REASONS? DO NOT READ; MULTIPLE RESPONSE] 1. AC was broken/malfunctioning and was unrepairable, had to be replaced 2. AC was broken/malfunctioning and was repairable, but replaced instead 3. AC was old and replaced before it broke 4. Wanted to improve comfort 5. Wanted to make home more energy efficient in order to save money on energy bills [PROBE TO CODE] 6. Wanted to make home more energy efficient in order to help the environment [PROBE TO CODE] 7. Contractor recommendation 8. Interested in rebates/incentives 9. Planned to install a new furnace and decided to also install an air conditioner 10. It was part of a larger home remodeling project 11. Previously had a different type of AC (such as window units) and wanted to upgrade to central AC 12. (Other) [SPECIFY: ] 98. (Don t know) 99. (Refused) B7. [ASK IF A1=3 OR 6] Were the furnace and central air conditioning units installed at the same time? [FOR INSTALLED AS PART OF A LARGER/SAME PROJECT, COUNTS AS YES] 1. Yes 2. No Furnace installed first 3. No AC installed first 98. (Don t know) 99. (Refused) [ASK IF B7=2 OR 3] Okay, for the next several questions, please focus on the time surrounding the first unit that you installed in your case, the [IF B7=2] furnace [IF B7=3] AC unit. I will let you know when we will resume asking you questions about both the furnace and air conditioning system. Appendix D. Heating and Cooling Program Supplemental Appendices Page D 73

220 B8. [ASK IF ((A1=3 OR 6) AND (B3<>1 OR 2) AND (B6<>1 OR 2)) OR ((A1=1 OR 4) AND (B3<>1 OR 2)) OR ((A1=2 OR 5) AND (B6<>1 OR 2))] Were you aware of the Save on Energy rebate prior to the decision to install a new system? [IF (B3=1 OR 2) OR (B6=1 OR 2)] Were you aware of the Save on Energy rebate prior to the system failure or malfunction? 1. Yes 2. No 3. Not applicable 98. (Don t know) 99. (Refused) B9. Did you know about the Save on Energy rebate offer before calling or researching contractors, or did a contractor or contractor advertisement inform you of the offer? 1. Knew about the offer beforehand (including general media/opa/ieso advertisement) 2. A contractor or contractor advertisement informed me of the offer 98. (Don t know) 99. (Refused) C. Installation Experience and Rebound C1. Did you have a previous working relationship with the contractor that installed your system? 1. Yes 2. No 98. (Don t know) 99. (Refused) C1a. How did you find your contractor? [DO NOT READ; PROBE: ANY OTHER WAYS? ; MULTIPLE RESPONSE] 3. Friend or family 4. Save on Energy website 5. Internet search 6. Phone book 7. Mail insert 8. Flyer 9. Other [SPECIFY: ] 98. (Don t know) 99. (Refused) [READ IF B7=2 OR 3] Okay, for the rest of the survey, we will focus on both the furnace and the air conditioning system you installed. Appendix D. Heating and Cooling Program Supplemental Appendices Page D 74

221 C2. [ASK IF B7=2 OR 3] This next question is about the thermostat controlling your [IF A1=1 OR 4] furnace [IF A1=2 OR 5] central air conditioner [IF A1=3 OR 6] furnace and central air conditioner. What type of thermostat do you use with your system? 1. Manual thermostat 2. Programmable thermostat 3. Smart thermostat (may be called learning thermostat) 98. (Don t know) 99. (Refused) C3. [ASK IF C2=1, 2 OR 3] Did you install this thermostat when you installed your [IF A1=1 OR 4] furnace [IF A1=2 OR 5] central air conditioner [IF A1=3 OR 6] furnace and central air conditioner? 1. Yes 2. No 98. (Don t know) 99. (Refused) C3a. [ASK IF C3=1] What type of thermostat did you have before installing the new thermostat? 3. Manual thermostat 4. Programmable thermostat 5. Smart thermostat (may be called learning thermostat) 98. (Don t know) 99. (Refused) C3b. [ASK IF C3A=1, 2 OR 3] Which option best represents how you most often used or interacted with your OLD thermostat? [READ LIST; SINGLE RESPONSE] 1. Kept thermostat(s) set at a constant temperature throughout each season 2. [OPTION APPEARS IF C3A=2] Relied on the programmed schedule of temperatures and never manually changed the temperature 3. [OPTION APPEARS IF C3A=2] Relied on the programmed schedule of temperatures, but sometimes manually changed the temperature 4. [OPTION APPEARS IF C3A=3] Allow smart thermostat to program itself, function as designed and never manually changed the temperature 5. [OPTION APPEARS IF C3A=3] Allow smart thermostat to program itself, function as designed, but sometimes manually changing the settings 6. Manually adjusted temperature using a regular schedule by changing the temperature for different times of the day or week 7. Manually adjusted temperature using no regular schedule 8. Some other way [SPECIFY: ] 98. (Don t know) 99. (Refused) Appendix D. Heating and Cooling Program Supplemental Appendices Page D 75

222 C3c. [ASK IF C2=1, 2 OR 3] Which option best represents how you use or interact with your CURRENT thermostat? [READ LIST; SINGLE RESPONSE] 1. Keep thermostat(s) set at a constant temperature throughout each season 2. [OPTION APPEARS IF C2=2] Rely on the programmed schedule of temperatures and never manually changed the temperature 3. [OPTION APPEARS IF C2=2] Rely on the programmed schedule of temperatures, but sometimes manually changed the temperature 4. [OPTION APPEARS IF C2=3] Allow smart thermostat to program itself, function as designed and never manually changed the temperature 5. [OPTION APPEARS IF C2=3] Allow smart thermostat to program itself, function as designed, but sometimes manually changing the settings 6. Manually adjust temperature using a regular schedule by changing the temperature for different times of the day or week 7. Manually adjust temperature using no regular schedule 8. Some other way [SPECIFY: ] 98. (Don t know) 99. (Refused) C3d. Have you changed the temperature settings in your home since installing the [IF A1=1 OR 4] furnace [IF A1=2 OR 5] central air conditioner [IF A1=3 OR 6] furnace and central air conditioner, or have you generally kept the same temperature settings? [SINGLE RESPONSE] 1. Yes, changed the settings 2. No, kept the setting the same 98. (Don t know) 99. (Refused) C3e. [ASK IF C3D=1] How did you change your settings? [RECORD ALL THAT APPLY; READ LIST IF NEEDED] 1. Increased temperature setting in the winter 2. Decreased temperature setting in the winter 3. Increased temperature setting in the summer 4. Decreased temperature setting in the summer 5. Changed the times of the settings to heat during more winter hours 6. Changes the times of the setting to heat during less winter hours 7. Changes the times of the setting to cool during more summer hours 8. Changed the times of the setting to cool during less summer hours 98. (Don t know) 99. (Refused) Appendix D. Heating and Cooling Program Supplemental Appendices Page D 76

223 C3f. [ASK IF C3E=1 OR 2] What were your thermostat temperature settings both before and after the change in the winter? [RECORD RESPONSE] 1. Before 2. After 98. (Don t know) 99. (Refused) C3g. [ASK IF C3E=3 OR 4] What were your thermostat temperature settings both before and after the change in the summer? [RECORD RESPONSE] 1. Before 2. After 98. (Don t know) 99. (Refused) C3h. [ASK IF C3E=5 OR 6] About how many hours per week are affected by your setting change in the winter? [RECORD RESPONSE] (Don t know) 99. (Refused) C3i. [ASK IF C3E=7 OR 8] About how many hours per week are affected by your setting change in the summer? [RECORD RESPONSE] (Don t know) 99. (Refused) C3j. [ASK IF C2=1, 2 OR 3] Which mode best describes how you operate your HVAC system fan, is it? [READ LIST] 1. Continuous, or on? 2. Circulation mode, which allows you to set the fan to run a certain amount of time per hour? 3. Auto or advanced fan controls? 98. (Don t know) 99. (Refused) C3k. [ASK IF C3J=1 OR 2] Can you estimate the average number of hours per day it is set to run each season? 1. Spring/fall [SPECIFY: 0 24] 2. Summer [SPECIFY: 0 24] 3. Winter [SPECIFY: 0 24] 98. (Don t know) 99. (Refused) Appendix D. Heating and Cooling Program Supplemental Appendices Page D 77

224 C3l. [ASK IF C2=1, 2 OR 3] Since installing the [IF A1=1 OR 4] furnace [IF A1=2 OR 5] central air conditioner [IF A1=3 OR 6] furnace and central air conditioner, would you say the amount of time your fan runs when not heating or cooling has 1. Increased? 2. Decreased? 3. Stayed the same? 98. (Don t know) 99. (Refused) C3m. [ASK IF C3L=1 OR 2] Can you estimate the hours of [INCREASE/DECREASE] for each season? 1. Spring/fall [SPECIFY: 0 24] 2. Summer [SPECIFY: 0 24] 3. Winter [SPECIFY: 0 24] 98. (Don t know) 99. (Refused) D. Net to Gross Intention and Influence Questions D1. Did your existing equipment require immediate replacement or could you have repaired your equipment or postponed the purchase of a new system for a year or more? 1. Immediate replacement 2. Could have repaired or postponed the purchase 98. (Don t know) 99. (Refused) D2. [ASK IF D1=1] Which of the following would you have most likely done if there were no heating and cooling program? [READ LIST] 1. Would have purchased less expensive or less efficient equipment 2. Would have purchased whatever equipment was recommended by my contractor 3. Would have purchased the exact same equipment and paid the full cost without getting a rebate 98. (Don t know) 99. (Refused) D3. [ASK IF D1=2] Which of the following would you have most likely done if there were no heating and cooling program? [READ LIST] 1. Would have stuck with or repaired the old equipment 2. Would have purchased less expensive or less efficient equipment within one year 3. Would have purchased whatever equipment was recommended by my contractor within one year Appendix D. Heating and Cooling Program Supplemental Appendices Page D 78

225 4. Would have purchased the exact same equipment and paid the full cost without the rebate within one year 98. (Don t know) 99. (Refused) D4. [ASK IF A1=1, 3, 4, OR 6] I m going to read a list of things that you may have considered when buying the energy efficient furnace. For each one, please indicate how important it was in your purchase decision. Please answer with a number from 1 to 4, where 1 means not at all important, 2 means a little important, 3 means somewhat important and 4 means very important. How important was the [READ STATEMENT; RANDOMIZE LIST; IF RESPONDENT SAYS I DIDN T DO/USE THAT, THEN FOLLOW UP WITH SO WOULD YOU SAY IT WAS NOT AT ALL IMPORTANT? AND PROBE TO CODE] a. Heating and cooling program rebate b. Contractor recommendation c. Marketing and/or advertising about the program d. Information from my utility or the IESO about energy efficiency in general 1. Not at all important 2. A little important 3. Somewhat important 4. Very important 98. (Don t know) D5. [ASK IF A1=2, 3, 5, OR 6] I m going to read a list of things that you may have considered when buying the energy efficient central air conditioner. For each one, please indicate how important it was in your purchase decision. Please answer with a number from 1 to 4, where 1 means not at all important, 2 means a little important, 3 means somewhat important and 4 means very important. How important was the [READ STATEMENT; RANDOMIZE LIST; IF RESPONDENT SAYS I DIDN T DO/USE THAT, THEN FOLLOW UP WITH SO WOULD YOU SAY IT WAS NOT AT ALL IMPORTANT? AND PROBE TO CODE] a. Heating and cooling program rebate b. Contractor recommendation c. Marketing and/or advertising about the program d. Information from my utility or the IESO about energy efficiency in general 1. Not at all important 2. A little important 3. Somewhat important 4. Very important 98. (Don t know) Appendix D. Heating and Cooling Program Supplemental Appendices Page D 79

226 E. Satisfaction E1. Next, I m going to ask about your satisfaction with the program. Please use a 4 point scale where 1 means not at all satisfied, 2 means a little satisfied, 3 means somewhat satisfied and 4 means very satisfied to rate how satisfied you were with.. [READ STATEMENT; RANDOMIZE LIST] a. The equipment for which you received a Save on Energy rebate b. The sales process with your contractor c. Your contractor s installation work d. Your electricity bill since installing the new equipment [IF RESPONDENT SAYS SOMETHING LIKE MY ELECTRICITY BILL ISN T AFFECTED BECAUSE IT IS A GAS FURNACE, SAY REBATED GAS FIRED FURNACES STILL SAVE ELECTRICITY BECAUSE OF THEIR EFFICIENT FAN MOTORS ] e. The time it took to receive your Save on Energy rebate f. [ANCHOR LAST STATEMENT] And finally, how would you rate your overall satisfaction with the rebate program 1. Not at all satisfied 2. A little satisfied 3. Somewhat satisfied 4. Very satisfied 5. (Not applicable) 98. (Don t know) E2. [ASK FOR ANY E1A F=1 OR 2] You indicated some dissatisfaction. What in particular were you dissatisfied with, and why? [RECORD RESPONSE] (Don t know) 99. (Refused) F. Spillover F1. Since installing your [IF A1=1 OR 4] furnace [IF A1=2 OR 5] central air conditioner [IF A1=3 OR 6] furnace and central air conditioner, have you made additional energy efficiency upgrades to your home? If so, what? [DO NOT READ LIST; MULTIPLE RESPONSE; PROBE: ANY OTHERS? ] 1. Additional heating or cooling equipment 2. New windows 3. New ENERGY STAR appliances 4. Weatherstripping, caulking, etc. 5. Energy efficient lighting products (LEDs, dimmers, timers/motion sensors, light fixtures, lighting control products, etc.) 6. Insulation 7. ENERGY STAR water heater 8. Seasonable LED lighting Appendix D. Heating and Cooling Program Supplemental Appendices Page D 80

227 9. ENERGY STAR television 10. Advanced power strips (Tier 1 or 2) 11. Outdoor timers 12. Clotheslines 13. Smart thermostat 14. Ceiling fans 15. Other 1 [SPECIFY: ] 16. Other 2 [SPECIFY: ] 17. Other 3 [SPECIFY: ] 18. None 98. (Don t know) 99. (Refused) F2. [ASK IF F1=1 14; SHOW ALL PRODUCTS SELECTED IN F1] For each of the additional energy efficiency upgrades you made to your home, please tell me if you used a Save on Energy coupon or received an incentive (or rebate) from Save on Energy or your local utility provider when performing the upgrade. [READ EACH ITEM MENTIONED AND CODE RESPONSE ACCORDINGLY; INCLUDE ONLY ONE RESPONSE FOR EACH OTHER TO MAXIMUM OF THREE OPEN ENDS] a. Additional heating or cooling equipment b. New windows c. New energy efficient appliances d. Weatherstripping, caulking, etc. e. Energy efficient lighting products (LEDs, CFLs, dimmers, timers/motion sensors, light fixtures, lighting control products, etc.) f. Insulation g. Water heater h. Seasonable LED lighting i. ENERGY STAR television j. Advanced power strips (Tier 1 or 2) k. Outdoor timers l. Clotheslines m. Smart thermostats n. Ceiling fans o. Other 1 p. Other 2: q. Other 3: 1. Yes 2. No 98. (Don t know) 99. (Refused) Appendix D. Heating and Cooling Program Supplemental Appendices Page D 81

228 F3. [ASK IF F1 18, 98 OR 99] How important was your experience with the Save on Energy heating and cooling incentive in your decision to install [THESE PRODUCTS/THIS PRODUCT]? Would you say it was [READ LIST] 1. Very important 2. Somewhat important 3. A little important 4. Not at all important 98. (Don t know) 99. (Refused) F4. [ASK IF F3=1 OR 2] Can you tell me how your experience with the Save on Energy heating and cooling incentive was important in your decision to install [THESE PRODUCTS/THIS PRODUCT]? [RECORD RESPONSE] (Don t know) 99. (Refused) [READ IF F1=1, 3, 5 OR 7] I would like to get a little more detail on the type of products or equipment you installed. F5. [IF F2A=1] Can you tell me what was the additional heating and cooling equipment you installed? [DO NOT READ; MULTIPLE RESPONSE; PROBE: ANY OTHERS? ] 1. Central forced air furnace (that used ducts to deliver heat) 2. Floor furnace (no ducts) 3. Ductless heat pump or ductless mini split (no ducts) 4. Ducted heat pump or ducted mini split 5. Baseboard heaters 6. Wall heaters or cadet heaters 7. Plug in space heaters 8. Heating stove 9. Fireplace 10. Central air conditioner (that uses ducts to deliver cool air) 11. Evaporative cooler, swamp cooler, desert cooler or wet air cooler 12. Window AC unit(s) 13. Portable air conditioning unit(s) 14. Other [SPECIFY: ] 98. (Don t know) 99. (Refused) Appendix D. Heating and Cooling Program Supplemental Appendices Page D 82

229 F6. [IF F2C=1] What new ENERGY STAR appliances did you install? [DO NOT READ; MULTIPLE RESPONSE; PROBE: ANY OTHERS? ] 1. High efficiency dishwasher 2. High efficiency clothes washer 3. High efficiency water heater 4. High efficiency refrigerator 5. Other [SPECIFY: ] 98. (Don t know) 99. (Refused) F7. [ASK IF F6=1, 2 OR 3] Do you have a gas or an electric water heater? [SINGLE RESPONSE] 1. Natural gas 2. Electric 3. Other [SPECIFY: ] 98. (Don t know) F8. [IF F1E=1] What types of energy efficient lighting or controls products did you install? [DO NOT READ; MULTIPLE RESPONSE; PROBE: ANY OTHERS?] 1. CFLs (compact fluorescent light bulbs or curly bulbs) 2. LEDs (light emitting diodes) 3. Motion sensors 4. Dimmers 5. Other 1 [SPECIFY: ] 6. Other 2 [SPECIFY: ] 7. Other 3 [SPECIFY: ] 98. (Don t know) 99. (Refused) F9. [ASK IF F8=1 7; SHOW ALL PRODUCTS SELECTED IN F8] How many of the following energy efficient lighting products did you install? Your best estimate is fine. [READ LIST AND CODE RESPONSE ACCORDINGLY; INCLUDE ONLY ONE RESPONSE FOR EACH OTHER BOX TO MAXIMUM OF THREE OPEN ENDS] a. CFLs (compact fluorescent light bulbs or curly bulbs) b. LEDs (light emitting diodes) c. Motion sensors d. Dimmers e. Other 1: f. Other 2: g. Other 3: (Don t know) 99. (Refused) Appendix D. Heating and Cooling Program Supplemental Appendices Page D 83

230 F10. [ASK IF F6=3] Is your new water heater gas or electric? [SINGLE RESPONSE] 1. Natural gas 2. Electric 3. Other [SPECIFY: ] 98. (Don t know) F11. [ASK IF F6=3; SKIP IF F10=98] Was your old water heater also [INSERT F10 RESPONSE]? [SINGLE RESPONSE] 1. Yes 2. No 98. (Don t know) F12. Have you, or would you, recommend the Save on Energy program to a friend or colleague? [IF YES, PROBE AND HAVE YOU ALREADY OR NOT YET? ] 1. Yes, I already have 2. Yes, but I have not yet 3. No I would not 98. (Don t know) 99. (Refused) F13. Thinking back over your whole experience with the Save on Energy heating and cooling incentive, do you have any other comments for us? [RECORD RESPONSE] No suggestions 98. (Don t know) 99. (Refused) G. Demographics G1. What is the primary language used in your household? 1. English 2. French 3. Chinese 4. Spanish 5. German 6. Italian 7. Arabic 8. Other [SPECIFY: ] 99. (Refused) Appendix D. Heating and Cooling Program Supplemental Appendices Page D 84

231 G2. What is the last level of education that you have completed? (Select one) 1. Grade school or less 2. Some high school 3. High school grad 4. Vocational/technical school 5. College 6. Some university 7. University grad 8. Postgraduate degree 99. (Refused) G3. How many people, including yourself, live in the household full time? (Select one) 1. One 2. Two 3. Three 4. Four 5. Five 6. Six 7. Seven or more 99. (Refused) G4. How many people, including yourself, live in the household part time? (Select one) 1. One 2. Two 3. Three 4. Four 5. Five 6. Six 7. Seven or more 99. (Refused) G5. Do you own or rent your current place of residence? 1. Own 2. Rent 3. Occupy rent free 99. (Refused) Appendix D. Heating and Cooling Program Supplemental Appendices Page D 85

232 G6. What type of home do you live in? 1. Single family detached house 2. Single family semi detached 3. Townhouse or rowhouse 4. Duplex, triplex or fourplex 5. Condominium 6. Apartment 7. Mobile home 8. Other [SPECIFY: ] 99. (Refused) G7. How many bedrooms are in your home? 1. 1 bedroom 2. 2 bedrooms 3. 3 bedrooms 4. 4 bedrooms or more 99. (Refused) G8. How many bathrooms are in your home? 1. 1 bathroom bathrooms 3. 2 bathrooms or more 99. (Refused) G9. How many stories is your home? 1. One 2. Two 3. Three or more 99. (Refused) G10. How old is your home? (estimates are fine) 1. Less than two years old (built in 2015 or after) 2. Two to less than seven years old (built between 2010 and 2015) 3. Seven to less than 12 (built between 2005 and 2009) to less than 17 (built between 2000 and 2004) to less than 22 (built between 1995 and 1999) to less than 27 (built between 1990 and 1994) to less than 32 (built between 1985 and 1989) to less than 42 (built between 1975 and 1984) to less than 52 (built between 1965 and 1974) Appendix D. Heating and Cooling Program Supplemental Appendices Page D 86

233 to less than 67 (built between 1950 and 1964) to less than 92 (built between 1925 and 1949) years or more (built in 1924 or earlier) 99. (Refused) G11. How many square feet is your home? 1. <1,000 square feet 2. 1,000 to 1,999 square feet 3. 2,000 to 2,999 square feet 4. 3,000 to 4,999 square feet 5. >5,000 sq. ft. 99. (Refused) G12. Do you have wireless internet in your home? 1. Yes 2. No 98. (Don t know) 99. (Refused) G13. Please tell me which of the following categories applies to your total household income in 2016? 1. Under $20, $20,000 to under $30, $30,000 to under $40, $40,000 to under $50, $50,000 to under $60, $60,000 to under $80, $80,000 to under $100, $100,000 to under $120, $120,000 or more 98. (Don t know) 99. (Refused) Appendix D. Heating and Cooling Program Supplemental Appendices Page D 87

234 PY2016 Heating and Cooling Program Savings Permutations for High Efficiency Furnace with Electronically Commutated Motor HVAC Modes Existing Heat and Cool Heat Only New Heat and Cool Heat Only Matched Fan Setting Table 5. Non Electric Furnace Replacing Non Electric Furnace Base kwh Efficient kwh AHRI Derating kwh Savings Summer kw Winter kw Natural Gas Penalty (m 3 ) Matched Continuous 4,827 3, Matched Non Continuous 2,146 1, Matched Continuous to Non Cont. 4,827 1, , Unmatched Continuous 4,827 3, , Unmatched Non Continuous 2,146 1, Unmatched Continuous to Non Cont. 4,827 1, , Unmatched Continuous 2,146 1, Unmatched Non Continuous Unmatched Continuous to Non Cont. 2, , Matched Continuous 5,140 4, , Matched Non Continuous 2,226 1, Matched Continuous to Non Cont. 5,140 1, , Unmatched Continuous 5,140 4, , Unmatched Non Continuous 2,226 1, Unmatched Continuous to Non Cont. 5,140 1, , Unmatched Continuous 2,008 1, Unmatched Non Continuous Unmatched Continuous to Non Cont. 2, , Appendix D. Heating and Cooling Program Supplemental Appendices Page D 88

235 Heat and Cool Existing Heat and Cool Heat Only New Heat and Cool Heat Only AHRI Status Fan Setting Base kwh Table 6. Non Electric Furnace Replacing Electric Furnace Efficient kwh AHRI Derating kwh Savings Summer kw Winter kw Natural Gas Penalty (m 3 ) Matched Continuous 33,390 3, , ,129 Matched Non Continuous 32,414 1, , ,003 Matched Continuous to Non Cont. 33,395 1, , ,187 Unmatched Continuous 33,472 3, , ,945 Unmatched Non Continuous 32,695 1, , ,129 Unmatched Continuous to Non Cont. 33,472 1, , ,003 Unmatched Continuous 30,791 1, , ,187 Unmatched Non Continuous 31, , ,945 Unmatched Continuous to Non Cont. 30, , ,129 Matched Continuous 23,074 4, , ,074 Matched Non Continuous 21,453 1, , ,943 Matched Continuous to Non Cont. 23,094 1, , ,106 Unmatched Continuous 23,182 4, , ,911 Unmatched Non Continuous 21,954 1, , ,074 Unmatched Continuous to Non Cont. 23,182 1, , ,943 Unmatched Continuous 20,050 1, , ,106 Unmatched Non Continuous 20, , ,911 Unmatched Continuous to Non Cont. 20, , ,074 Appendix D. Heating and Cooling Program Supplemental Appendices Page D 89

236 Heat and Cool Existing Heat and Cool Heat Only New Heat and Cool Heat Only AHRI Status Fan Setting Base kwh Table 7. Electric Furnace Replacing Non Electric Furnace Efficient kwh AHRI Derating kwh Savings Summer kw Winter kw Natural Gas Penalty (m 3 ) Matched Continuous 33,390 34, Matched Non Continuous 32,414 31, Matched Continuous to Non Cont. 33,395 32, , Unmatched Continuous 33,472 34, Unmatched Non Continuous 32,695 32, Unmatched Continuous to Non Cont. 33,472 32, , Unmatched Continuous 30,791 32, , Unmatched Non Continuous 31,353 31, Unmatched Continuous to Non Cont. 30,791 31, Matched Continuous 23,074 23, Matched Non Continuous 21,453 21, Matched Continuous to Non Cont. 23,094 21, , Unmatched Continuous 23,182 23, Unmatched Non Continuous 21,954 21, Unmatched Continuous to Non Cont. 23,182 21, , Unmatched Continuous 20,050 21, , Unmatched Non Continuous 20,243 20, Unmatched Continuous to Non Cont. 20,050 20, Appendix D. Heating and Cooling Program Supplemental Appendices Page D 90

237 Residential New Construction Program Supplemental Appendices PY2016 Residential New Construction Program Stakeholder Interview Guide Researchable Questions Identify roles and responsibilities, estimate direct job impacts Gather insights into program design and delivery and determine if program is being implemented as planned Identify areas working well Identify areas that could be improved Identify and assess program goals Assess builders and delivery agent satisfaction Assess awareness and education Review communication protocol Follow up on previous evaluation recommendations Question Sections C, E Sections D, E Sections E, G, I Sections E, I Section D Section H Section F Section G Section E Audience: IESO program lead, LDCs and implementer staff. Purpose: Identify key roles and responsibilities, document design process and delivery, assess market actor satisfaction, determine what is working well and areas where challenges exist and assess ways to improve evaluability. Econoler conducted these in depth interviews, scheduled in advance of the call via e mail. The interviews took between 45 and 60 minutes. This information was included in the invitation. The interview guide was not read verbatim, but was used to guide the conservation. Target: Up to six interviews with IESO, active LDCs and implementer staff. Specific questions and sections for IESO staff are identified below. General Instructions Interviewer instructions are in green [LIKE THIS]. Skip pattern instructions are in red [LIKE THIS]. Invitation To: From: Subject: [ ] [YOUR ] Evaluation Interview about the Residential New Construction Program Hello [NAME], As part of the 2016 Consumer Evaluation, the Cadmus team (Cadmus, Apex and Econoler) is conducting in depth interviews with key local distribution companies (LDCs), implementer and Independent Electric System Operator (IESO) staff. As such, I would like to set up a time to speak with you regarding the Residential New Construction Program. Appendix E. Residential New Construction Program Supplemental Appendices Page E 91

238 The purpose of these interviews is to make sure we have a thorough understanding of the Residential New Construction Program. We ll also get your perspective on things that are working well or any challenging areas. Please let me know if you are available during any of the following times [LIST OPTIONS]. I expect this interview to take about minutes, but we can always schedule a follow up if we need more time. If these times and dates do not work well, let me know what availability you have next week and I ll schedule a time for us to speak then. I appreciate your time and help with this. I look forward to speaking with you. Calendar Invite To: From: Subject: [ ] [YOUR ] Interview about the Residential New Construction Program Hello [NAME], Thank you for agreeing to speak with me about the Residential New Construction Program. Our call is scheduled for [INSERT TIME AND DATE OF CALL] and will take about 45 to 60 minutes. Thank you. I look forward to speaking with you. A. Introduction Thank you for making the time to speak with me. The Cadmus team is conducting in depth interviews with LDCs, implementer staff and IESO staff as part of our assessment of the Residential New Construction Program. The purpose of these interviews is to make sure we have a thorough understanding of the program, data sources and what you are looking forward to learning through the evaluation. We ll also get your perspective on things that are working well or any challenging areas. We will use the information you provide to inform our understanding of the program, so that we can provide well rounded and balanced observations and recommendations. A1. To start, please tell me about your role and main responsibilities working on the Residential New Construction Program. A2. How many full time or part time staff from your organization work to support the RNC Program in 2016? A3. I would like to know who is involved in the RNC Program. Can you tell me who his involved and their main roles, including the IESO, the LDCs and implementers? A4. Are there any delivery agents involved in the program? If so, who are they? Appendix E. Residential New Construction Program Supplemental Appendices Page E 92

239 A5. [IF DELIVERY AGENTS INVOLVED] How long have the delivery agents been involved with the program? A6. [IF DELIVERY AGENTS INVOLVED] How were the delivery agents selected? B. Program Goals [ASK B1 B6 TO IESO PROGRAM LEAD] I d like to talk about the goals and objectives of the RNC Program. B1. What were the program goals for 2016 in terms of: 1. Savings? 2. Participation? B2. Are there specific target savings per LDC? B3. Did the program reach its goals for 2016? 1. Savings? If not, why not? 2. Participation? If not, why not? B4. It looks like the number of projects and participating LDCs have declined in What can explain this change? [PROBE: 25 LDCS IN 2015 COMPARED TO EIGHT IN 2016] 2. Are you considering any program design changes that could help increase participation? B5. Did the program have any process or non impact goals for 2016? [PROBE: INCREASED AWARENESS, MARKET TRANSFORMATION, SPILLOVER MEASURES SUCH AS DUCT SEALING OR INSULATION] B6. In your opinion, how has the program performed in 2016 (in terms of both process and savings/participation goals)? Why do you think this is? [ASK B7 B11 TO LDCS] B7. [IF B2=YES] I d like to talk about the goals and objectives of the RNC Program. B8. What were the specific targets for your network in 2016 in terms of: 1. Savings? 2. Participation? B9. Did you reach your targets for 2016? 1. Savings? If not, why not? 2. Participation? If not, why not? B10. Has participation in the RNC Program declined or increased in 2016? 1. What can explain this change? 2. What do you think would help increase participation? Appendix E. Residential New Construction Program Supplemental Appendices Page E 93

240 B11. Can you provide us with the following information: 1. [IF APPLICABLE] The number of houses involved per prescriptive measures? 2. [IF APPLICABLE] The measures implemented in the ENERGY STAR certified homes? C. Program Design and Implementation Now, I d like to talk with you about design and implementation of the RNC Program. C1. The program is offered to residential home builders. Other similar programs also allow homeowners to apply as well. Why is the program design aimed at builders and not homeowners? C2. Can you briefly walk me through a high level overview of how the RNC is administered and delivered, beginning with how builders are identified all the way to documenting the results. [PROBE FOR STEPS IN THE PROCESS AND ROLES AND RESPONSIBILITIES OF THE IESO VS. LDCS VS. IMPLEMENTER (WHO DOES WHAT)] C2m. How is the program marketed? [IF AT ALL, BY WHOM, WHAT MEDIUMS] C2n. How do the builders enter the program? C2o. How do builders select which program track to participate in and which measures to implement? C2p. For the Performance and Custom tracks, is there an audit conducted up to code by an Energy Advisor using the building plans? C2q. For the Performance and Custom tracks, is the audit conducted after the house is built used only to determine the final EnerGuide rating? C3. Has the program changed over the last year? [PROBE: INCENTIVE CHANGES, CHANGES TO THE MEASURE MIX, CHANGES TO PROGRAM DELIVERY, BASELINE CHANGES] 1. [IF YES] Why were these changes implemented? C4. In your opinion, should any additional measures be considered in future years? If so, which ones? C5. Who would you say are the primary drivers of program participation? [PROBE: LDCS OR DELIVERY AGENTS, IF ANY] C6. What specific tools, protocols or processes do IESO and LDC staff use to closely monitor program performance on an ongoing basis? C6a. Are there any challenges with this and, if yes, what are the challenges? C7. On which basis are the results reported to the IESO? [WHAT DOES A PROJECT IN THE PROJECT LIST CONSISTS OF?] C8. Are there any site verification or quality assurance protocols? [IF NO, WHY NOT? IF YES, PLEASE DESCRIBE. PROBE FOR SOME OR ALL HOMES AND HOW IT IS PERFORMED AND BY WHOM.] Appendix E. Residential New Construction Program Supplemental Appendices Page E 94

241 [ASK C9 C11 TO IESO LEAD ONLY] C9. From a preliminary data review, it appears that Hydro Ones Brampton, Hydro One Networks and Milton Hydro Distribution contribute over 87% of the RNC savings. C9a. Do you know why these LDCs are the primary conduits for RNC? C9b. Are you aware why the other LDCs have not reported significant participation? C10. One recommendation from the 2015 evaluation was that the IESO develop a consistent methodology across the province for capturing key data points for savings calculations. Was this implemented? [IF NO] Do you know why not? C11. Where there some other recommendations from the 2015 evaluation implemented so far? D. Awareness and Education D1. Is there any education component in the RNC Program? D1a. [IF YES] Can you describe how the education component is delivered? D2. Are there any documents left with builders to provide to the future homeowners? [PROBE: HOME ENERGY ASSESSMENT REPORT, ENERGUIDE RATING] E. Communication Processes The following questions are about your typical communication processes as they relate to the program. E1. What types of communication regarding the program exist between the IESO and individual LDCs? [PROBE: COMMUNICATION CHANNELS, FREQUENCY, SATISFACTION, CHALLENGES] E2. Do you have suggestions on how communications between the IESO and LDCs could be improved? E3. [IF DELIVERY AGENTS INVOLVED] Do you communicate directly with the delivery agents? E3a. [IF YES] How frequently and regarding what types of matters? E3a. Do you have any suggestions on how communications with the delivery agents could be improved? F. Customer and Contractor Satisfaction Now, I d like to talk about any experience you have with participant and delivery agent program satisfaction. F1. Have you heard any feedback from builders regarding the program? F2. [IF DELIVERY AGENTS INVOLVED] What about the delivery agents? Have you heard any feedback from them? Please describe. F3. What do you think are the main challenges with the program for participants? Appendix E. Residential New Construction Program Supplemental Appendices Page E 95

242 F4. [IF DELIVERY AGENTS INVOLVED] What about the main challenges for delivery agents? F5. Do you know if there are previous participant builders that did not participate in 2016? F5a. [IF YES] Can you provide us with their contact information? We would like to get in touch with them for evaluation purposes. G. Successes, Challenges and Future Planning Please think about the program overall for the next set of questions. G1. What would you say worked particularly well in 2016? G1a. Why do you think that is? G2. Conversely, what did not work as well as anticipated? G2b. Why do you think that is? G3. Overall, do you have any suggestions for how to improve the program? G4. According to you, what is the impact of the RNC Program on the residential market? G5. In your opinion, what are some of the current trends in the residential market in the province? Can you provide insight on how you see the current building practises? [PROBE FOR TENDENCIES AMONG BUILDERS TO BUILD ABOVE CODE OR NOT] G6. [Ask of IESO LEAD ONLY] Other than [ANSWERS TO B4.2], are you considering any changes to improve the program? H. Closing We are almost finished. I have a few final questions. H1. Thinking about the 2016 evaluation, what are you interested in learning from this evaluation? [PROBE FOR ANY FEEDBACK WE SHOULD GET FROM EITHER PARTICIPANTS OR DELIVERY AGENTS] H2. Is there anything else you would like to cover that we did not discuss? H3. [IF DELIVERY AGENTS INVOLVED] We would like to talk with delivery agents. Can you provide us with contact information? [IF YES, REQUEST CONTACT INFORMATION] H3a. [IF NO] Who should I request this contact list from? [DOCUMENT NAME AND CONTACT INFORMATION] H4. [ASK OF IESO LEAD ONLY] We would also like to talk with LDCs involved in the program. Can you provide us with contact information? [IF YES, REQUEST CONTACT INFORMATION] H4a. [IF NO] Who should I request this contact list from? [DOCUMENT NAME AND CONTACT INFORMATION] Thank you for your input. We appreciate your time. Have a nice day. Appendix E. Residential New Construction Program Supplemental Appendices Page E 96

243 PY2016 Residential New Construction Program Nonparticipating LDC Interview Guide Determine participation and barriers Review communication protocol Researchable Questions Gather insights into program design and delivery Identify areas working well Identify areas that could be improved Collect contacts of nonparticipating builders Question Section D Section E Section F Section E, F Section E, F Section G Audience: Nonparticipating LDCs who did not participate in RNC in PY2016, but did participate in previous years. Purpose: Identify areas where challenges exist, reasons for not participating in 2016 and ways to improve the program. These in depth interviews were conducted by Econoler staff. Interviews were scheduled in advance of the call via e mail. Each interview took about 10 minutes. The interview guide was not read verbatim, but was used to guide the conservation. Target: Up to two interviews with past participant LDCs. General Instructions Interviewer instructions are in green [LIKE THIS]. Skip patterns and programming instructions are in red [LIKE THIS]. General Instructions A. E mail Invitation Subject: Evaluation Interview about the Residential New Construction Program Hello [NAME], As part of the PY2016 Consumer Evaluation, the Cadmus team (Cadmus, Apex and Econoler) is conducting in depth interviews with key Local Distribution Companies (LDCs), implementer and Independent Electric System Operator (IESO) staff. As such, I would like to set up a time to speak with you regarding the Residential New Construction Program. The purpose of this interview is to understand why [LDC NAME] did not offer the program in 2016 and get your perspective on things that are working well or any challenging areas based on [LDC NAME] prior experience with RNC. Appendix E. Residential New Construction Program Supplemental Appendices Page E 97

244 Please let me know if you are available during any of the following times [LIST OPTIONS]. I expect this interview to take about 10 minutes. If these times and dates do not work well, let me know what availability you have next week, and I ll schedule a time for us to speak then. I appreciate your time and help with this. I look forward to speaking with you. B. Calendar Invite Subject: Interview about the Residential New Construction Program Hello [NAME], Thank you for agreeing to speak with me about the Residential New Construction Program. Our call is scheduled for [INSERT TIME AND DATE OF CALL] and will take about 10 minutes. Thank you. I look forward to speaking with you. C. Introduction Thank you for making the time to speak with me. The Cadmus team is conducting in depth interviews with LDCs, implementer staff and IESO staff as part of the Residential New Construction Program. The purpose of this interview is to understand why you did not participate in the program in 2016 and to get your perspective on things that are working well or on any challenging areas. We will use the information you provide to inform our understanding of the program, so that we can provide wellrounded and balanced observations and recommendations. D. Participation According to our record, your LDC offered the Residential New Construction (RNC) Program in 2015, but not in D1. Does this mean there were no applications from builders in your area even though you offered the program or did you actually decide not to offer the program in 2016? D2. [IF A1=NOT TO OFFER THE PROGRAM IN 2016] What are the main reasons why [LDC UTILITY NAME] did not offer the RNC program in 2016? D3. [IF A1=NO PROJECTS SUBMITTED] According to you, what can explain why there were no projects submitted under the RNC Program in 2016 in your area? Appendix E. Residential New Construction Program Supplemental Appendices Page E 98

245 E. Communication Processes The following questions are about communication processes related to the program. E1. What types of communication regarding the program exist between the IESO and individual LDCs? [PROBE: COMMUNICATION CHANNELS, FREQUENCY, SATISFACTION, CHALLENGES, ETC.] E2. Are there any challenges with these communications? 1. [IF YES] WHAT ARE THEY? 2. Do you have suggestions on how communications between the IESO and LDCs could be improved? F. Program Design and Implementation Now, I d like to talk with you about design and implementation of the RNC Program. F1. Can you briefly give me your perspective on how the RNC is administered and delivered? [PROBE: PERSPECTIVE ON THE NEW ROLES LDCS HAVE BEEN GIVEN UNDER THE NEW FRAMEWORK] F2. In your opinion, should any additional measures be considered in future years? If so, which ones? F3. Who would you say are the primary drivers of program participation? [PROBE LDCS OR DELIVERY AGENTS, IF ANY] F4. Is there any reference documentation/information that you would like to see provided to LDCs about the RNC Program? F5. Overall, do you have any suggestions for how to improve the RNC Program? G. Closing We are almost finished. I have a few final questions. G1. Is there anything else you would like to cover that we did not discuss? G2. We would like to talk with additional builders who did not participate in the RNC Program in Do you have a contact list of builders that you can provide us with? [IF YES, REQUEST CONTACT INFORMATION] Thank you for your input. We appreciate your time. Have a nice day. Appendix E. Residential New Construction Program Supplemental Appendices Page E 99

246 PY2016 Residential New Construction Program Builder Interview Guide Research Objectives Question Screening B1 B8 Builders Experience and Barriers to Participation C1 C13 Free Ridership D1 D9 Spillover E1 E5 Existing Practises (Baseline definition) F1 F6 Firmographics G1 G4 Audience: Participating builders who received RNC Program incentives in 2016 Target Quota: 18 RNC participating builders (n=22 participating builders, including builders with multiple offices) Purpose: Assess builders general views of the program and obtain feedback on their experience and satisfaction with the program, barriers to participation, free ridership, spillover and baseline activities. Econoler scheduled and conducted these interviews. The interviews each took around 30 minutes. General Instructions Interviewer instructions are in green [LIKE THIS]. Skip patterns and programming instructions are in red [LIKE THIS]. A. Introduction A1. Hello, I m [INSERT NAME], calling on behalf of the Independent Electricity System Operator or IESO. We are conducting research on the Residential New Construction Program. Our records show that your company [INSERT COMPANY NAME] participated in the program and received incentives from the IESO for energy efficient houses built in Are you the person in your company who is most knowledgeable about participation in the program? 1. Yes 2. No, person is able to come to phone [START AGAIN] 3. No, person is not able to come to phone [GET NAME AND PHONE NUMBER, SCHEDULE CALL BACK] 98. (Don t know) [ASK TO SPEAK WITH SOMEONE WHO KNOWS AND BEGIN AGAIN] 99. (Refused) [THANK AND TERMINATE] Appendix E. Residential New Construction Program Supplemental Appendices Page E 100

247 A2. I d like to ask you some questions about your experience with the Residential New Construction Program. Your feedback is very important to the IESO and will help inform future program offerings. Would you be able to complete the interview now? 1. Yes 2. No [GET NAME AND PHONE NUMBER, SCHEDULE CALL BACK] 98. (Don t know) [GET NAME AND PHONE NUMBER, SCHEDULE CALL BACK] 99. (Refused) [THANK AND TERMINATE] [IF NEEDED] I only need about 30 minutes. This survey is for research purposes only and this is not a marketing call. Your perspectives help the IESO decide about their energy efficiency program offerings. B. Screening B1. Thank you. According to program records, your firm participated in the Residential New Construction (RNC) Program in 2016, meaning that your firm received incentives for houses built in Is this correct? 1. Yes 2. No [USE THE NONPARTICIPATING BUILDER INTERVIEW GUIDE] 98. (Don t know) [ASK TO SPEAK WITH SOMEONE WHO WOULD KNOW AND START AGAIN. IF NO ONE, THEN THANK AND TERMINATE] 99. (Refused) [THANK AND TERMINATE] B2. Has your firm participated in the Residential New Construction Program before 2016? If yes, for how many years have you participated in the program? 1. Yes [RECORD NUMBER OF YEARS] 2. No 98. (Don t know) 99. (Refused) B3. As part of the Residential New Construction Program, incentives are available for eligible measures installed in new homes and for homes achieving ENERGY STAR certification or a final EnerGuide rating of 83 or greater. Which categories of incentives did your firm received in 2016? [READ, CHECK ALL THAT APPLY] 1. Incentives for eligible individual measures (Prescriptive track) 2. Incentives for ENERGY STAR certified homes 3. Incentives for EnerGuide homes 98. (Don t know) 99. (Refused) Appendix E. Residential New Construction Program Supplemental Appendices Page E 101

248 B4. [ASK FOR EACH B3 ANSWER] Why did you choose to participate in this category of incentives? [RECORD RESPONSE] B5. Why did you choose NOT to participate in the other categories of incentives? [RECORD RESPONSE] B6. Approximately how many new houses including single units, semi detached units and row houses did your firm built in 2016 in Ontario? If you built semi detached units, please count each as two units. 1. [ENTER NUMBER] 98. (Don t know) [ASK TO SPEAK WITH SOMEONE WHO WOULD KNOW AND START AGAIN. IF NO ONE, THEN THANK AND TERMINATE] 99. (Refused) [THANK AND TERMINATE] B7. Among the new houses that your firm built in 2016 in Ontario, for how many houses did your firm receive incentives from the Residential New Construction Program? [ENTER NUMBER; ANSWER TO B7 CANNOT BE HIGHER THAN ANSWER TO B6] B8. [IF MORE THAN ONE B3 ANSWER] Among these [ENTER VALUE FROM B7] house(s), how many have received [READ AND ENTER NUMBER] 1. Incentives for eligible individual measures: 2. Incentives for being ENERGY STAR certified: 3. Incentives for EnerGuide: 98. (Don t know) 99. (Refused) C. Builders Experience and Barriers to Participation C1. Why did you decide to participate in the Residential New Construction Program? [PROBE: PROGRAM FACTOR (INCENTIVE, SELLING POINT) VERSUS NON PROGRAM FACTOR] C2. Did the program provide you or someone of your firm with training or information about energyefficient building practises? [RECORD RESPONSE] C3. Is your company certified by NRCan to build ENERGY STAR or EnerGuide homes in Ontario? [IF YES] What was the influence of the Residential New Construction Program in your decision to obtain this certification? 1. (Very important) 2. (Somewhat important) 3. (A little important) 4. (Not at all important) 98. (Don t know) 99. (Refused) Appendix E. Residential New Construction Program Supplemental Appendices Page E 102

249 C4. [ASK IF C3=1 OR 2] Why was the program [RESPONSE FROM C3] in your decision? 1. [RECORD RESPONSE] 98. (Don t know) 99. (Refused) Now, I d like to ask you a series of questions regarding your satisfaction with various elements of the program. Each question will use the same rating scale, using very satisfied, somewhat satisfied, a little satisfied or not at all satisfied. C5. How satisfied are you with the process of submitting application to the Residential New Construction Program? 1. (Very satisfied) 2. (Somewhat satisfied) 3. (A little satisfied) 4. (Not at all satisfied) 98. (Don t know) 99. (Refused) C6. [ASK IF C5=3 OR 4] Why are you [RESPONSE FROM C5] with the application process? 1. [RECORD RESPONSE] 98. (Don t know) 99. (Refused) C7. How satisfied are you with the incentive amount offered by the Residential New Construction Program? 1. (Very satisfied) 2. (Somewhat satisfied) 3. (A little satisfied) 4. (Not at all satisfied) 98. (Don t know) 99. (Refused) C8. [ASK IF C7=3 OR 4] Why are you [RESPONSE FROM C7] with the incentive amount offered? 1. [RECORD RESPONSE] 98. (Don t know) 99. (Refused) Appendix E. Residential New Construction Program Supplemental Appendices Page E 103

250 C9. How satisfied are you with the Residential New Construction Program overall? 1. (Very satisfied) 2. (Somewhat satisfied) 3. (A little satisfied) 4. (Not at all satisfied) 98. (Don t know) 99. (Refused) C10. [ASK IF C9=3 OR 4] Why are you [RESPONSE FROM C9] with the program? 1. [RECORD RESPONSE] 98. (Don t know) 99. (Refused) C11. Among all homebuilder companies working in Ontario, what proportion would you say is aware of the Residential New Construction Program? 1. [RECORD RESPONSE] 98. (Don t know) 99. (Refused) C12. In your opinion, why would a builder that knows about the program choose not to participate? 1. [RECORD RESPONSE] 98. (Don t know) 99. (Refused) C13. If you could offer one suggestion to improve the Residential New Construction Program next year, what would you recommend? 1. [RECORD RESPONSE] 2. (Nothing) 98. (Don t know) 99. (Refused) D. Free Ridership [FREE RIDERSHIP SECTION IS DIVIDED IN THREE TRACKS, AS PER ANSWER TO B3. ASK FREE RIDERSHIP QUESTIONS ONLY ONCE. PRIORITIZE B3=3] Now, I would like to understand the factors that influenced your decision to equip houses with energyefficient measures eligible through the Residential New Construction Program. Appendix E. Residential New Construction Program Supplemental Appendices Page E 104

251 D1. Please think about the houses you built in Ontario in 2016 that include incented measures. If you had not received incentives and assistance from the Residential New Construction Program: a. Would you have implemented these measures? [READ] 1. In the same number of homes 2. In most homes 3. In close to half of the homes 4. In a few homes only 5. In no home at all b. Without the RNC Program, would you have implemented? [READ] 1. All of the measures that were included in the houses you built in Most of the measures 3. Some of the measures 4. A few measures only 5. None of these measures D2. I m going to read a list of factors that may have influenced your decision to implement energyefficient measures eligible through the program in the houses built in For each one, please indicate how important it was in your decision to implement these measures please answer with a number from 1 to 4, where 4 means not at all important, 3 means a little important, 2 means somewhat important and 1 means very important. How important was the [READ] a. Program incentives 1. (Very important) 2. (Somewhat important) 3. (A little important) 4. (Not at all important) 98. (Don t know) b. Program marketing and advertising about energy efficiency in new homes 1. (Very important) 2. (Somewhat important) 3. (A little important) 4. (Not at all important) 98. (Don t know) c. [ASK IF C2=YES] Education assistance offered by the program (in your decision to implement these measures) 1. (Very important) 2. (Somewhat important) 3. (A little important) 4. (Not at all important) 98. (Don t know) Appendix E. Residential New Construction Program Supplemental Appendices Page E 105

252 [ASK D3 AND D4 IF B3=2] Now, I would like to understand the factors that influenced your decision to build houses achieving ENERGY STAR certification. D3. Please think about the houses you built in Ontario in 2016 that achieved ENERGY STAR certification and for which you received an incentive. If you had not received incentives and assistance from the Residential New Construction Program: a. Would you have built to ENERGY STAR standards? [READ] 1. The same number of homes 2. Most homes 3. Close to half homes 4. A few homes only 5. No homes at all b. And, without the program, would you have adopted? [READ] 6. All of the measures required to meet ENERGY STAR certification 7. Most of the measures required to meet ENERGY STAR certification 8. Some of the measures required to meet ENERGY STAR certification 9. A few measures only 10. None of these measures D4. I m going to read a list of factors that may have influenced your decision to build houses achieving ENERGY STAR certification. For each one, please indicate how important it was in your decision to build to this standard please answer with a number from 1 to 4, where 4 means not at all important, 3 means a little important, 2 means somewhat important and 1 means very important. a. Program incentives 1. (Very important) 2. (Somewhat important) 3. (A little important) 4. (Not at all important) 98. (Don t know) b. Program marketing and advertising about energy efficiency in new homes 1. (Very important) 2. (Somewhat important) 3. (A little important) 4. (Not at all important) 98. (Don t know) Appendix E. Residential New Construction Program Supplemental Appendices Page E 106

253 c. [ASK IF C2=YES] Education assistance offered by the program (in your decision to implement these measures) 1. (Very important) 2. (Somewhat important) 3. (A little important) 4. (Not at all important) 98. (Don t know) [ASK D5 AND D6 IF B3=3] Now, I would like to understand the factors that influenced your decision to build houses achieving EnerGuide rating of 83 or greater. D5. Please think about the houses you built in 2016 that achieved an EnerGuide rating of 83 or greater and for which you received an incentive. If you had not received incentives and assistance from the Residential New Construction Program: a. Would you have reach this EnerGuide rating for? [READ] 1. The same number of homes 2. Most homes 3. Close to half of the homes 4. A few homes only 5. No homes at all b. And, without the program, would you have adopted? [READ] 6. All of the measures required to reach this EnerGuide rating 7. Most of the measures required to reach this EnerGuide rating 8. Some of the measures required to reach this EnerGuide rating 9. A few measures only 10. None of these measures D6. I m going to read a list of factors that may have influenced your decision to build houses achieving an EnerGuide rating of 83 or greater. For each one, please indicate how important it was in your decision to build to this standards. Please answer with a number from 1 to 4, where 4 means not at all important, 3 means a little important, 2 means somewhat important and 1 means very important. a. Program incentives 1. (Very important) 2. (Somewhat important) 3. (A little important) 4. (Not at all important) 98. (Don t know) Appendix E. Residential New Construction Program Supplemental Appendices Page E 107

254 b. Program marketing and advertising about energy efficiency in new homes 1. (Very important) 2. (Somewhat important) 3. (A little important) 4. (Not at all important) 98. (Don t know) c. [ASK IF C2=YES] Education assistance offered by the program (in your decision to implement these measures) 1. (Very important) 2. (Somewhat important) 3. (A little important) 4. (Not at all important) 98. (Don t know) D7. What leads you to build high efficiency homes? [PROBE: CUSTOMER DEMAND, DECLINING COST OF EFFICIENT MEASURES, BUILDING CODE] 1. [RECORD VERBATIM] D8. Are you aware of the upcoming changes to the building code later this year? 1. Yes 2. No 98. (Don t know) 99. (Refused) D9. [ASK IF D8=1] Have you already started making changes to your building practises in response to this upcoming code change? E. Spillover [ASK THIS SECTION ONLY IF B6>B7 We will now talk about the houses you built in 2016, but without receiving any incentives from the Residential New Construction Program. E1. You told me that you or your firm built approximately [ENTER VALUE FROM B6] houses including single units, semi detached units and row houses in 2016 in Ontario. Among these houses, you told me that roughly [ENTER VALUE FROM B7] received incentives as part of the Residential New Construction Program. It means that you or your firm built [B6 B7] house(s) in 2016 in Ontario without receiving any incentives from the Residential New Construction Program. Is that correct? 1. Yes 2. No [READ INSTRUCTION BELOW] 98. (Don t know) 99. (Refused) Appendix E. Residential New Construction Program Supplemental Appendices Page E 108

255 [INSTRUCTION:] Can you please tell me which answer(s) need(s) to be corrected between the number of new houses you or your firm built in 2016 and the number of houses that received incentives from the Residential New Construction Program? [CHANGE ANSWER TO B3 AND/OR B7 BASED ON ANSWER FROM RESPONDENT] E2. Among the [B6 B7] house(s) your firm built in 2016 without receiving any incentives from the Residential New Construction Program, how many would have been eligible for an incentive as part of the Residential New Construction Program? 1. [RECORD NUMBER; CANNOT BE HIGHER THAN B6 B7)] 98. (Don t know) 99. (Refused) E3. [DO NOT ASK IF E2=0] Now I would like to know which incentives these houses would have been eligible for. How many of the [ENTER VALUE FROM E2] houses would have? [READ; IF NEEDED, SAY YOUR BEST ESTIMATE IS FINE ] Eligible Performance Measures Met the ENERGY STAR for New Homes Guideline Obtained a final EnerGuide rating of 83 or 84 Obtained a final EnerGuide rating of 85 or higher Eligible Prescriptive Measures ENERGY STAR qualified indoor light fixtures (hard wired) with one or two sockets ENERGY STAR qualified indoor light fixtures (hard wired) with three or more sockets Number of houses (should not exceed E2) If don t know, ask for proportion (%) On average, number of units per house E4. [DO NOT ASK IF E2=0] How important was your experience with the Residential New Construction Program in your decision to implement those energy efficiency measures or build houses to that level of energy efficiency? Would you say it was [READ] 1. Very important 2. Somewhat important 3. A little important 4. Not at all important 98. (Don t know) 99. (Refused) E5. [DO NOT ASK IF E2=0] Why did you not take part in the Residential New Construction Program for houses that would have been eligible for an incentive? [RECORD VERBATIM] Appendix E. Residential New Construction Program Supplemental Appendices Page E 109

256 F. Existing Practises I would now like to ask you questions about your residential building practises and the prevailing residential building practises in Ontario. F1. Do you usually build custom houses (client build) or tract houses? [RECORD VERBATIM] F2. [ASK IF F1=CUSTOM HOUSES] [RECORD VERBATIM] 1. Do you typically propose energy efficient measures or labels to your client builds? If so, which ones? 2. Do your client builds typically ask for some energy efficient measures or labels? If so, which ones? [IF F1=TRACT HOUSES] [RECORD VERBATIM] 3. Do you typically include energy efficient measures in your tract houses? If so, which ones? If no, why? F3. How, if at all, do your building practises regarding energy efficiency in homes that do not participate in the program vary from the other residential builders in Ontario? [RECORD VERBATIM] F4. I now have some questions about the prevailing residential building practises in Ontario regarding energy efficiency. a. Speaking specifically about the building envelope, would you say 1. Building practises are ahead of the current Part 9 of the Ontario Building Code 2. Building practises are similar to the current Part 9 of the Ontario Building Code 3. Building practises are not yet up to the current Part 9 of the Ontario Building Code b. Speaking specifically about windows, would you say 1. Building practises are ahead of the current Part 9 of the Ontario Building Code 2. Building practises are similar to the current Part 9 of the Ontario Building Code 3. Building practises are not yet up to the current Part 9 of the Ontario Building Code c. Speaking specifically about HVAC equipment, would you say 1. Building practises are ahead of the current Part 9 of the Ontario Building Code 2. Building practises are similar to the current Part 9 of the Ontario Building Code 3. Building practises are not yet up to the current Part 9 of the Ontario Building Code F5. [IF F4A, B OR C=1] What are the reasons for these building practises being at that level? [PROBE: PROGRAM FACTOR (INCENTIVE, SELLING POINT) VERSUS NON PROGRAM FACTOR (BUILDING CODE CHANGE COMING IN 2017/2018)] [RECORD VERBATIM] Appendix E. Residential New Construction Program Supplemental Appendices Page E 110

257 F6. We have a few additional questions on the type of material and equipment that you usually install in new houses. Would you prefer me to send them to you by or fax? [ASK FOR CONTACT INFO WHERE TO SEND FORM] G. Firmographics G1. How many offices does your company have in Ontario? 1. [RECORD NUMBER] 98. (Don t know) 99. (Refused) G2. Approximately how many full time equivalent workers are employed by your company within Ontario? 1. Fewer than five 2. Five to nine to to to to or more 98. (Don t know) 99. (Refused) G3. And about how many full time equivalent workers are employed in projects participating in the RNC Program? [RECORD NUMBER] G4. Does your company also participate in other green home programs, like LEED Canada or Green Seal/Built Green? 1. Yes, LEED Canada 2. Yes, Built Green/Green Seal 3. No, neither This completes our questions. As discussed earlier, we will send the form on the equipment and building material to the contact information you sent us. This needs to be completed and send back by [ONE WEEK AFTER SURVEY DATE]. We appreciate your participation and thank you for your time. Have a good [EVENING/DAY]. Appendix E. Residential New Construction Program Supplemental Appendices Page E 111

258 PY2016 Residential New Construction Program Nonparticipating Builder Interview Guide Screening Research Objectives Builders Experience and Barriers to Participation Spillover Existing Practises (Baseline definition) Firmographics Question B1, B3 B2, B4, C1 C11 D1 E5 F1 F6 F1 F3 Audience: Nonparticipating builders (these are builders who did NOT participate in the RNC Program in 2016) Target Quota: 10 builders (extracted from TARION Ontario Builder Directory) Purpose: Assess builders general awareness of the program, views of the program, feedback on past experience and satisfaction with the program where applicable, barriers to participation, spillover and baseline analysis. Econoler scheduled and conducted these interviews. The interview each took around 30 minutes. General Instructions Interviewer instructions are in green [LIKE THIS]. Skip pattern instructions are in red [LIKE THIS]. A. Introduction A1. Hello, I m [INSERT NAME] calling on behalf of the Independent Electricity System Operator or IESO. We are conducting research on the Residential New Construction Program. The program offers incentives to Ontario builders who install energy efficient measures in houses. This is NOT a marketing or sales call. Your perspectives are important to help the IESO decide about their energy efficiency program offerings. Are you the person in your company who is most knowledgeable about energy efficient home design and construction? 1. Yes 2. No, person is able to come to phone [START AGAIN] 3. No, person is not able to come to phone [GET NAME AND PHONE NUMBER, SCHEDULE CALL BACK] 98. (Don t know) [ASK TO SPEAK WITH SOMEONE WHO KNOWS AND BEGIN AGAIN] 99. (Refused) [THANK AND TERMINATE] Appendix E. Residential New Construction Program Supplemental Appendices Page E 112

259 A2. I d like to ask you some questions about your awareness of the Residential New Construction Program and your building practises. Your feedback is very important to the IESO and will help inform future program offerings. Would you be able to complete the interview now? 1. Yes 2. No [GET NAME AND PHONE NUMBER, SCHEDULE CALL BACK] 98. (Don t know) [GET NAME AND PHONE NUMBER, SCHEDULE CALL BACK] 99. (Refused) [THANK AND TERMINATE] [IF NEEDED] I only need about 30 minutes. As I mentioned before, this interview is for research purposes only, and this is not a marketing neither a sales call. Your perspectives help the IESO decide about their energy efficiency program offerings. B. Screening B1. Thank you. Approximately how many new houses including single units, semi detached units and row houses did your firm build in 2016 in Ontario? If you built semi detached units, please count each as two units. 1. [ENTER NUMBER] [IF NO HOUSE, THANK AND TERMINATE] 98. (Don t know) [ASK IF SOMEONE KNOW OR PROBE FOR BEST APPROXIMATION] 99. (Refused) [THANK AND TERMINATE] B2. Before today, had you ever heard of the Residential New Construction Program offered by IESO specifically to Ontario builders? [IF NEEDED] As mentioned, the program offers incentive amounts to builders who install energy efficient measures in houses built in Ontario. 1. Yes 2. No [SKIP TO SECTION F] 98. (Don t know) [ASK TO SPEAK WITH SOMEONE WHO WOULD KNOW AND START AGAIN. IF NO ONE, THEN THANK AND TERMINATE] 99. (Refused) [THANK AND TERMINATE] B3. Has your firm ever participated in the Residential New Construction Program? If yes, which years has your firm participated in the program? [RECORD ALL YEARS OF PARTICIPATION] [USE THE PARTICIPATING BUILDER INTERVIEW GUIDE] Appendix E. Residential New Construction Program Supplemental Appendices Page E 113

260 Never 98. (Don t know) [ASK TO SPEAK WITH SOMEONE WHO WOULD KNOW AND START AGAIN. IF NO ONE, THEN THANK AND TERMINATE] 99. (Refused) [THANK AND TERMINATE] B4. Is your company certified by NRCan to build ENERGY STAR or EnerGuide homes in Ontario? 1. [IF YES] For which brand are you certified? C. Builders Experience and Barriers to Participation C1. [ASK IF B3=8] Why has your firm NEVER participated in the Residential New Construction Program? [ASK IF B3=1 7] Why did your firm NOT participate in the Residential New Construction Program in 2016? [PROBE: PROGRAM FACTOR (INCENTIVE AMOUNT, ELIGIBILITY CONCERNS, PROCESS CONCERNS, SATISFACTION) VERSUS NON PROGRAM FACTOR (CONSUMERS INTEREST)] C2. Did the program provide you or someone at your firm 1. With training or information about energy efficient building practises? [IF YES, PROBE FOR DETAILS] 2. With training or information about the program, such as the criteria/eligibility and how to participate? [IF YES, PROBE FOR DETAILS] C3. Among all homebuilder companies working in Ontario, what proportion would you say is aware of the Residential New Construction Program? 1. [ENTER %] 98. (Don t know) 99. (Refused) D. Spillover We will now talk about the houses you built in Among the [INSERT VALUE FROM B1] house(s) your firm built in 2016 in Ontario, I would like to determine with you how many would have been eligible for an incentive as part of the Residential New Construction Program. D1. How many of the [INSERT VALUE FROM B1] houses would have? [READ; IF NEEDED, SAY YOUR BEST ESTIMATE IS FINE ] Appendix E. Residential New Construction Program Supplemental Appendices Page E 114

261 Eligible Performance Measures Met the ENERGY STAR for New Homes Guideline Obtained a final EnerGuide Rating of 83 or 84 Obtained a final EnerGuide Rating of 85 or higher Eligible Prescriptive Measures ENERGY STAR qualified indoor light fixtures (hard wired) with one or two sockets ENERGY STAR qualified indoor light fixtures (hard wired) with three or more sockets ENERGY STAR qualified LED Number of houses (should not exceed B1) If don t know, ask for proportion (%) On average, number of units per house D2. [DO NOT ASK IF D1=0] What do you base this estimated number of houses on? [PROBE: WHAT MEASURES IMPLEMENTED IN THOSE HOUSES MAKE THEM THINK THEY WOULD HAVE REACHED THE REQUIRED ENERGY EFFICIENCY LEVEL?] 1. [RECORD VERBATIM] 98. (Don t know) 99. (Refused) D3. [DO NOT ASK IF D1=0] How important was your awareness or your experience with the Residential New Construction Program in your decision to implement those energy efficiency measures or to build houses with that level of energy efficiency? Would you say it was [READ] 1. Very important 2. Somewhat important 3. A little important 4. Not at all important 98. (Don t know) 99. (Refused) D4. [DO NOT ASK IF D1=0] Why did you not take part in the Residential New Construction Program for houses that would have been eligible for an incentive? [RECORD VERBATIM] E. Existing Practises I now would like to ask you questions about your residential building practises and the prevailing residential building practises in Ontario. E1. Do you usually build custom houses (client build) or tract houses? [RECORD VERBATIM] Appendix E. Residential New Construction Program Supplemental Appendices Page E 115

262 E2. [IF F1=CUSTOM HOUSES] [RECORD VERBATIM] 1. Do you typically propose energy efficient measures or labels to your client builds? If so, which ones? 2. Do your clients typically ask for some energy efficient measures or labels? If so, which ones? [IF F1=TRACT HOUSES] [RECORD VERBATIM] 3. Do you typically include energy efficient measures in your tract houses? If so, which ones? If no, why? E3. Talking about your building practises regarding energy efficiency, what is the approximate percentage of houses you build that that are certified by ENERGY STAR or EnerGuide with a rating of 83 or more? [RECORD VERBATIM] E4. I now have some questions about the prevailing residential building practises in Ontario regarding energy efficiency. a. Speaking specifically about the building envelope, would you say 1. Building practises are ahead of the current Part 9 of the Ontario Building Code 2. Building practises are similar to the current Part 9 of the Ontario Building Code 3. Building practises are not yet up to the current Part 9 of the Ontario Building Code b. Speaking specifically about windows, would you say 1. Building practises are ahead of the current Part 9 of the Ontario Building Code 2. Building practises are similar to the current Part 9 of the Ontario Building Code 3. Building practises are not yet up to the current Part 9 of the Ontario Building Code c. Speaking specifically about HVAC equipment, would you say 1. Building practises are ahead of the current Part 9 of the Ontario Building Code 2. Building practises are similar to the current Part 9 of the Ontario Building Code 3. Building practises are not yet up to the current Part 9 of the Ontario Building Code E5. [IF F4A,B OR C=1] What are the reasons for these building practises being at that level? [PROBE: PROGRAM FACTOR (INCENTIVE, SELLING POINT) VERSUS NON PROGRAM FACTOR (BUILDING CODE CHANGE COMING IN 2017/2018)] [RECORD VERBATIM] E6. [IF E1=TRACT HOUSES] We have a few additional questions on the type of material and equipment that you usually install in new standard houses. Would you prefer me to send them to you by e mail or fax? [ASK FOR CONTACT INFO WHERE TO SEND FORM] Appendix E. Residential New Construction Program Supplemental Appendices Page E 116

263 F. Firmographics We are almost done. F1. How many offices does your company have in Ontario? 1. [RECORD NUMBER] 98. (Don t know) 99. (Refused) F2. Approximately how many full time equivalent workers are employed by your company within Ontario? 1. Fewer than five 2. Five to nine to to to to or More 98. (Don t know) 99. (Refused) F3. Does your company also participate in other green home programs, like LEED Canada or Green Seal/Built Green? 1. Yes, LEED Canada 2. Yes, Built Green/Green Seal 3. No, neither This completes our questions. As discussed earlier, we will send the form on the equipment and building material to the contact information you provided. This needs to be completed and send back by [ONE WEEK AFTER SURVEY DATE]. We appreciate your participation and thank you for your time. Have a good [EVENING/DAY]. Y2016 Residential New Construction Program Common Residential Building Practices Survey General Information This survey is being conducted as part of an assessment of the energy efficiency practices in the residential Ontario market for Save on Energy. We invite you to answer the following questions to the best of your ability. Your answers will remain strictly confidential. Appendix E. Residential New Construction Program Supplemental Appendices Page E 117

264 While completing the questionnaire: Think about the STANDARD NEW HOUSES that you build the most often DO NOT think about ENERGY STAR EnerGuide certified houses that you may built This questionnaire: Will help us understand current practices in the residential market Focuses on certain elements of particular interest to Save on Energy and does not cover all residential building practices and components (on purpose) IDENTIFICATION (Correct the information below if necessary) First and last name Company name City Title Number of locations in Ontario Years and months in business years months Telephone number Other telephone number E mail address # # Questionnaire Fiberglass or Fiberock Batts Sprayed or Blown Cellulose Polyurethane Spray Foam 1. Attic Ceilings Check and describe the insulation material that you install the most often. Insulation Material (x) Thickness (in inches) R Value Other: Appendix E. Residential New Construction Program Supplemental Appendices Page E 118

265 2. Cathedral Ceilings Check and describe the options that you install the most often. Truss Structure Standard Scissor Frame Layer (x) Insulation Material Thickness (inches) R Value Insulation Between the Frame Continuous Insulation Layer Frame 2 x 6 Other: Frame Spacing (Select): Other: Fiberglass or Fiberock Batts Sprayed or Blown Cellulose Polyurethane Spray Foam Other: Expanded Rigid Board (EPS) Extruded Rigid Board (XTPS) Polyurethane Spray Foam Other: 3. Walls Check and describe the options that you install the most often. Layer (x) Type of Material Thickness (inches) R Value Insulation Between the Frame Continuous Insulation Layer 2 x 6 Other: Frame Spacing (Select): Other: Fiberglass or Fiberock Batts Sprayed or Blown Cellulose Polyurethane Spray Foam Other: Expanded Rigid Board (EPS) Extruded Rigid Board (XTPS) Isocyanurate Board Polyurethane Spray Foam Other: Appendix E. Residential New Construction Program Supplemental Appendices Page E 119

266 4. Foundations Check and describe the options that you install the most often. Layer (x) Type of Material Thickness (inches) R Value Floor Insulation Frame Insulation Between the Frame Continuous Insulation Layer Continuous Exterior Insulation Layer No Insulation Foundation Floor Expanded Rigid Board (EPS) Extruded Rigid Board (XTPS) Isocyanurate Board Other: Percentage of Floor Insulated: 2 x 6 Interior Insulation Foundation Walls Other: Percentage Frame Spacing (Select): Fiberglass or Fiberock Batts Sprayed or Blown Cellulose Polyurethane Spray Foam Other: Percentage of Wall Height Insulated: Expanded Rigid Board (EPS) Extruded Rigid Board (XTPS) Isocyanurate Board Polyurethane Spray Foam Other: Percentage of Wall Height Insulated: Exterior Insulation Foundation Walls Expanded Rigid Board (EPS) Extruded Rigid Board (XTPS) Isocyanurate Board Polyurethane Spray Foam Other: Percentage of Wall Height Insulated: Other: Appendix E. Residential New Construction Program Supplemental Appendices Page E 120

267 5. Exposed Floors (typically above a garage) Check and describe the options that you install the most often. Layer (x) Type of Material Thickness (inches) R Value Frame Insulation Between the Frame Continuous Insulation Layer Energy Star Certified Windows? 2 x 6 Other: Frame Spacing (Select): Other: Expanded Rigid Board (EPS) Extruded Rigid Board (XTPS) Isocyanurate Board Other: Expanded Rigid Board (EPS) Extruded Rigid Board (XTPS) Isocyanurate Board Polyurethane Spray Foam Other: 6. Windows Check and describe the type of window you install the most often. Options No Yes Don t know What is the manufacturer and model number of the window you install the most often? Do you know the ER value of this window? If yes, please provide. What ventilation system do you usually install? 7. Ventilation Check and describe the ventilation system that you install the most often. Options No central ventilation system Air exchanger with no heat recovery (x) Air exchanger with heat recovery (HRV) Air exchanger with energy recovery (ERV) What is the manufacturer and model number of the ventilation system you install the most often? What cooling system do you usually install? 8. Cooling System Check and describe the cooling system that you install the most often. Options Air conditioning (AC) Heat pump (HP) What is the AHRI/HSPF of the AC/HP you install the most? What is the SEER of the AC/HP you install the most often? Ductless Central (x) (x) Appendix E. Residential New Construction Program Supplemental Appendices Page E 121

268 Fuel Type 9. Primary Heating System Check and describe the heating system that you install the most often. Options (x) Specify Efficiency (%) Gas Forced Air Furnace Gas Boiler Electric Forced Air Furnace Electric Baseboard Heat Pump Ductless Central Other: With an ECM (Electrically commutated motors)? No Yes Type of Hot Water Heater Hot Water Heater Fuel Type 10. Domestic Hot Water (DHW) Check and describe the DHW system that you install the most often. Options (x) Average Volume and Efficiency (%) Tankless Tank Natural Gas Electric Other What is the manufacturer and model number of the hot water heater you install the most often? With tank wrap (insulating blanket)? No Yes Do you usually install any other water savings devices? Specify: LED or CFL lighting/fixtures? 11. Lighting Check and describe the lighting you install the most often. Options No Yes, less than 25% CFL or LED Yes, 50% to 75% CFL or LED Yes, more than 75% CFL or LED Example EnerGuide 80 Model (x) Specify: The house is a typical two storey brick and vinyl house with an attached garage. The HVAC system has a condensing 96% AFUE natural gas furnace, a 14 SEER air conditioner, and an HVI certified heat recovery ventilator. The DHW is a direct vent (sealed) 189L. The windows are ENERGY STAR certified hinged windows and there is a patio door at the back of the house. The doors are steel polyurethane core. The foundation is insulated with R 12 batt insulation with R 5 insulation under the slab. The exterior walls are 2X6 studs with R 20 batt insulation and 1" EPS continuous insulation on the exterior. The attic space is insulated with R 51 blown mineral fibre and the cathedral ceiling is insulated with R 32 batt insulation between the trusses. Appendix E. Residential New Construction Program Supplemental Appendices Page E 122

269 Page 1 Energy Efficiency Evaluation Report,, Ontario K0A 1A1 File Number: 7S74N99999 Year Built: 2017 Date of Evaluation: Mar. 07, 2017 Builder Name: Congratulations on the purchase of your new house! This report contains information on how your new house will consume energy. Anyone can claim that a house is energy efficient, but the EnerGuide label and this report shows how efficient your house is. Any energy efficiency upgrades that you and your builder decided to include in your house will reduce energy consumption for years to come and will help protect our environment. Your House's Energy Efficiency Rating A qualified energy advisor has assessed the energy efficiency of your house by using Natural Resources Canada's EnerGuide Rating System procedures. Based on this evaluation, your house has an energy efficiency rating of Level of energy efficiency Energy Efficiency Target The EnerGuide scale ranges from 0 to 100. It accommodates millions of houses across Canada - from older houses in need of renovation to newer, more energy-efficient ones. A "0" on the scale would represent an uncomfortable house that has major air leakage, no insulation and extremely high-energy consumption. At the other end of the scale, "100" represents a house that is very well insulated, airtight yet well ventilated, and heated by renewable energy sources, such as wind or solar power. Several factors, such as the size of a home's windows and the direction they face, can affect the rating. Even if two houses appear identical, their ratings can be very different if they have different levels of insulation, types of heating equipment, etc. For many older houses, meeting 65 or higher on the scale would be quite an achievement. New houses typically receive a rating of 65 or higher, simply because of improvements in building standards and practices over the years. Relatively few houses achieve a rating of 80 or higher, and those that do represent the most energy-efficient houses on the market. Therefore, the EnerGuide rating scale shown ranges from 65 to 100. Typical Energy Efficiency Ratings Typical Rating New house built to building code standards New house with some energy-efficiency improvements Energy-efficient new house House requiring little or no purchased energy

270 Page 2 Estimated Annual Energy Consumption Below, you will find the estimated annual energy consumption of electricity, natural gas, propane or oil for your house. These estimates are based on a number of standard assumptions, such as a family of four living in the home, specific thermostat settings, and usage rates for hot water, lighting and appliances. These assumptions may not reflect your lifestyle but, since they are the same for all houses, they allow you to compare your house's rating with similar-sized houses built in similar regions. The number of occupants and their day-to-day habits and overall lifestyle may significantly influence your house's actual energy consumption and your future savings. This house, as currently rated, has an estimated annual energy consumption of 98 GJ*. * One GJ is the amount of energy that would be consumed by a 100-Watt light bulb lit continuously for four months. Table 1. Estimated Annual Energy Consumption Electricity (kilowatt-hours) Natural Gas (cubic metres) Oil (litres) Propane (litres) Total (gigajoules) Current estimate Did you know? Today, 17 percent of all energy used in Canada goes toward running our homes. By using less energy in your home, you can help reduce the production of greenhouse gas (GHG) emissions that contribute to climate change and harm the environment. Your house produces 3.3 tonnes per year less GHGs than a similar house built to minimum building code requirements.

271 Page 3 Estimated Energy Consumption by End Use All houses lose heat to the outdoors during the heating season through air leakage, ventilation (e.g. exhaust fans in bathrooms and kitchen) and the transfer of heat through the basement, walls, roof, windows and doors. Lost heat must be replaced by your main heating device (furnace, boiler, fireplace, etc.). This is called space heating. Generally, space heating, domestic hot water, and lights and appliances make up most of the energy consumption in a house. Figure 1 shows the breakdown of space heating, domestic hot water, and lights and appliances for your house. Figure 1. Energy Consumption Estimates by End Use Figure 2 shows how the energy for the "space heating" segment shown in Figure 1 actually gets used in the heating of your home. A long bar indicates where your house will lose more heat; a short bar indicates where it will lose less heat. If you decide to renovate your home, look at the long bars on this graph and consider improving those areas. Figure 2. Estimated Heat Loss Maintenance tip The building envelope is made up of a house's exterior walls, basement, exposed floors, ceilings, windows, roofs and doors. As houses age in Canada's severe climate, tiny cracks open in the building envelope. Any future renovations you undertake may also affect your house's building envelope. Since increased air leakage through cracks or holes decreases your house's energy efficiency and the comfort of the occupants, keep this in mind over time so that you can protect your investment.