2015 Nonresidential Downstream Custom ESPI Lighting Research Plan

Transcription

1 2015 Nonresidential Downstream Custom ESPI Lighting Research Plan Prepared for California Public Utilities Commission Itron, Inc High Bluff Drive, Suite 210 San Diego, California (858) June 7, 2016

2 Table of Contents Nonresidential Custom ESPI Lighting Research Plan Goals and Objectives Evaluation Approach and Research Objectives Data Sources Program Tracking Data and Participant Applications On-Site Audits Time of Use Lighting Loggers Participant Phone Survey On-Site Sample Design Participant Phone Survey Sample Design Work Plan Task 1: Project Management Task 2: Develop and Update Research Plan Task 3: Tracking System Management and Sampling Task 4: Data Collection Instrument and Tool Development Task 5: Gross Impact Evaluation Task 6: Net Impact Evaluation Task 7: Reporting Budget List of Tables Table 1-1: Summary of 2015 Nonresidential Custom Lighting Gross Ex Ante Lifecycle Savings Table 1-2: 2015 On-Site Sample Design for Custom Lighting Projects Itron, Inc. i Table of Contents

3 Nonresidential Custom ESPI Lighting Research Plan This document presents the research plan for the Nonresidential Downstream Custom Lighting Impact Evaluation of the 2015 investor owned utilities (IOU) energy efficiency programs. The overall goal of this study is to address the need for ex post evaluation for customer measures as outlined in the Efficiency Savings and Performance Incentive (ESPI) decision 1. This research plan includes the goals and objectives of the evaluation, the researchable issues, data sources that will be used, the approach to sampling and how coordination and communication between other related evaluation studies will be met to provide timely feedback to the program administrators (PAs). All of these issues, along with the methodology that will be used to determine gross and net impacts from custom lighting projects, are similar to those that were employed for both the 2013 and 2014 Nonresidential Downstream Custom ESPI Impact Evaluations. 1.1 Goals and Objectives As mentioned, the overall goal of this evaluation is to address the needs for ex post evaluation for custom measures as outlined in the ESPI decision. As discussed in Appendix 2 of the decision, for custom projects, all components of the projects will be subject to review. An evaluation based estimate of the savings claim for custom projects in the defined program year will be applied to the custom ex ante claim to adjust gross savings. Net to gross ratios will also be estimated for the projects based on ex post analysis. Therefore, all nonresidential lighting measures that are considered to be custom (non-deemed) are the focus of this study. While the approach to this impact evaluation is not measure-specific, a random sampling of projects will be selected that will likely cover a variety of nonresidential downstream lighting measures. 1.2 Evaluation Approach and Research Objectives Based on the study goal, the primary research issues for this evaluation center around determining net and gross ex post impacts. For this evaluation, a gross realization rate (GRR) 1 D , Decision Adopting Efficiency Savings and Performance Incentive Mechanism. Itron, Inc Custom Lighting Research Plan

4 approach will be employed, where site-specific gross ex post impacts are estimated from a sample of participants. These site-specific gross ex post impacts will then be compared to the ex ante impacts garnered from the tracking data to develop a ratio between the ex post and ex ante impacts. GRRs will be developed for each of participant segments and applied to the entire population of participants to generate population level estimates of ex post gross savings. This approach is consistent with that employed for custom lighting measures under the , 2013 and 2014 Nonresidential Downstream Lighting Impact Evaluations. A separate net-to-gross (NTG) analysis will also be performed using a self-report analysis based on participant phone survey data. The evaluation team will develop a set of NTGRs that can be applied to the population s gross savings in order to develop net savings estimates. In order to implement this approach in meeting the overall study goal, several research objectives have been identified. The approach is site-specific, so these objectives are predicated on the types of projects that are being evaluated: Confirm installations (verification). This included on-site verification of measure installation to confirm the installations reported by the PAs. Estimate baseline (both pre-retrofit and code/isp based) and replacement (post-retrofit) equipment wattages, operating hours, and use shapes to support the estimate of energy savings values and 8,760 impact load shapes. Collect facility-wide and area square footage estimates (where applicable) in order to calculate baseline savings using the Lighting Power Density area method. Estimate participant free-ridership to support the development of net-to-gross ratios and net savings values. Estimate remaining useful life values for measures that are subject to a dual baseline, and update effective useful life estimates based on ex post operating hours. Based on the above, estimate first year and lifetime gross and net ex post impacts (kwh, kw). Based on the ex post savings values, develop gross and net realization rates (GRRs and NRRs) that can be applied to the entire nonresidential downstream custom lighting population to estimate population level estimates of ex post gross and net savings, both first year and lifecycle. 1.3 Data Sources There are multiple primary and secondary sources of data that will utilized in order to meet the goals and objectives of this evaluation. Itron, Inc Custom Lighting Research Plan

5 1.3.1 Program Tracking Data and Participant Applications Program tracking data were provided and uploaded by each of the PAs onto a centralized server. These separate data sets were analyzed, cleaned, re-categorized, reformatted, and merged into one program tracking database. The sample of projects will be drawn from these data. For each randomly sampled project (and for all projects in the census categories), participant applications will be requested. These applications include detailed information regarding when the project was completed, detailed invoices, project specifications and the methods/calculations by which the ex ante savings for each project were estimated. Key data from these applications and calculation workbooks will be entered into an evaluation database to help inform the on-site verification process. As part of the project documentation data request, the evaluation team will also request contact and decision-maker information from PA account managers (where applicable) and updated customer contact information to help facilitate the recruitment of projects for both the on-site audits as well as the participant phone interview On-Site Audits On-site audits will be utilized to collect important site-specific data that will be used to estimate the ex post gross energy and demand impacts. The disposition of each measure will be collected in order to develop installation rates along with storage, removal and burn-out rates Equipment manufacturer and model numbers will also be collected in order to perform lookups that provide information on the wattage of installed and replaced equipment (where available) to support the estimate of pre- and post-retrofit wattages. For some on-sites, spot watt measurements may also be collected to estimate post-installation wattage. Self-report data on the wattage of pre-existing equipment will be collected when actual equipment replaced is not on site and project applications do not sufficiently document pre-wattages, to help support the estimate of preretrofit wattages. Facility-wide and activity-area square footage estimates will be collected for New Construction projects or retrofits that utilized a Lighting Power Density area method for the second baseline. Finally, self-report data will also be gathered on lighting equipment usage schedules to aid in the development of pre- and post-retrofit load shapes where monitoring is not done and/or the equipment is not on an EMS schedule Time of Use Lighting Loggers As part of the on-site audit, installed lighting equipment will be monitored (where feasible and when allowed) to gather time-of-use data to support the development of operating hours. Lighting loggers that employ optical sensors are the predominant type of monitoring equipment used for this study. However, when lighting is not accessible for optical sensors, logging will be done at the electrical panel by collecting circuit amperage. Itron, Inc Custom Lighting Research Plan

6 1.3.4 Participant Phone Survey A computer assisted telephone interview (CATI) phone survey will be conducted to recruit customers for the on-site visit, as well as to collect data useful for the NTG analysis and various other components of the evaluation. For larger sites that fall within the census categories, the same phone survey instrument will be implemented, but it will be administered as a professional interview. One other key use of the phone survey data is to help identify if customer installations are early replacement (ER) or replacement on burnout (ROB). The decision-maker is asked questions about the age and condition of the pre-existing equipment during the phone interview and this information is also collected during the on-site verification process. These data are used to verify the ER claim provided in the customer s application documentation. 1.4 On-Site Sample Design The overall objective of the sample design is to allow for the development of net first-year and lifecycle realization rates at a reasonable level of relative precision, while considering the budget allocated for this activity. This objective is based on the fact that the ESPI incentive mechanism is based on net lifecycle savings. Separate realization rates will be developed by PA and, since PG&E and SCE have significantly larger savings claims, more resources will be dedicated to evaluating net lifecycle savings for those PAs. Table 1-1 summarizes the absolute lifecycle energy savings from nonresidential downstream custom lighting measures, the percentage of each PA s total lifecycle portfolio savings as well as the statewide totals and percentage of statewide savings 2. In 2015, energy savings from nonresidential downstream custom lighting measures represent roughly 9% of the total portfolio of kwh savings and 6% of overall ex ante gross kw savings. Table 1-1: Summary of 2015 Nonresidential Custom Lighting Gross Ex Ante Lifecycle Savings PA Total Savings Savings as a % of Portfolio MW GWh kw kwh PG&E % 7% SCE % 11% SDG&E % 5% Statewide 179 1,479 6% 9% To improve the statistical precision of the PA-specific realization rates, the sample will be stratified by project size (represented by ex ante first year gross kwh savings) in order to develop 2 The percentage of portfolio savings do not include savings from Codes and Standards. Itron, Inc Custom Lighting Research Plan

7 population level estimates of net lifecycle savings for each of the three PAs. A sample size of 60 projects has been chosen (25 for PG&E, 21 for SCE and 14 for SDG&E). A similar sample design with similar sample sizes was developed for the 2013 and 2014 ESPI studies. Variation between ex ante and ex post savings values decreased significantly from 2013 to 2014, with the relative precision on net lifecycle kwh savings measured at the 90% confidence level decreasing from 21% in 2013 to only 7% in 2014 at the statewide level. In 2014, the relative precisions for net lifecycle kwh savings for PG&E, SCE and SDG&E were 10%, 14% and 13%, respectively. The resulting relative precisions are related to the correlation between the ex ante and ex post savings values. The more correlated the ex ante and ex post savings estimates, the more precise the resulting realization rates. We may not find in 2015 the same level of correlation, and therefore the same high level of precision in the realization rates. However, given the similar sample design, we might expect to see relative precisions for the net lifecycle savings values in the neighborhood of 90/20 or better for PG&E and SCE, and 90/30 or better for SDG&E, unless there is a drop in correlation between ex ante and ex post savings value similar to Table 1-2 presents the sample design for each PA along with statewide totals. The cut points for project size strata are based on an examination of how the population distribution of savings for each PA is clustered. The only exception to this is the very large category in PG&E and SCE, the large category in SDG&E and the UC category in PG&E. For the large and very large categories, they are limited to a small number of sites in the population and the cut points are estimated where there is a drop off in savings from one site to the next incremental site. These strata will represent the census strata. For example, the smallest site-project in the PG&E very large segment represents 11.2 million lifecycle kwh whereas the largest site-project in the large category represents 6.8 million kwh of savings. Along with these segments, the evaluation team discovered 4 projects in PG&E that represent one unique contact and also represent a significant amount of ex ante kwh savings and determined that these projects could be evaluated independently from the sample of other projects within their respective strata. For SCE, projects clustered in the 1,600 mwh to 4,500 mwh range represent roughly 33% of lifecycle energy savings (at the PA level), so the resulting segments in SCE were stratified in order to develop similar populations of total savings. Itron, Inc Custom Lighting Research Plan

8 Table 1-2: 2015 On-Site Sample Design for Custom Lighting Projects PA PG&E Project Size Project Size (LC MWh) Population # of Projects 2015 Ex Ante LC MWh Savings Percent Savings w/in PA Target Sample Size Very Large >11, ,975 13% 4 Large 2,600 to 11, ,367 28% 6 Medium 800 to 2, ,374 26% 6 Small < ,148 26% 6 UC Census 1,580 to 13, ,385 7% 4 PG&E Total , % 26 SCE Very Large >20, ,036 7% 3 Large 4,500 to 20, ,387 33% 6 Medium 1,600 to 4, ,486 33% 6 Small <1, ,842 27% 6 SCE Total , % 21 SDG&E Large >7, ,024 48% 2 Medium 1,200 to 7, ,887 41% 4 Small <1, ,789 11% 5 SDG&E Total 24 43, % 11 Statewide 1,887 1,478, Participant Phone Survey Sample Design One of the key objectives of the phone survey is to develop NTGRs for each PA. This analysis will be done based solely on the participant phone survey responses. The phone survey will also be administered as part of the recruitment for the onsite audits. Therefore, the same stratification scheme will be used for sampling the telephone surveys (PA and project size). The precision objective for the phone surveys will be to estimate the NTGRs at a relative precision of approximately 10% for each PA at the 90% confidence level. This is based on a COV estimate of 0.35 obtained from the 2013 and 2014 Custom Lighting ESPI evaluations. As discussed above, the evaluation team will request assistance from the PAs and their account representatives to recruit customers for the phone survey and on-sites. However, for the smaller sites that do not have an account representative, recruitment will be done solely through the phone survey. For these smaller customers the evaluation team expects to get a recruitment rate around 50%. Therefore, we expect to have more participants contacted for surveys than for onsite audits in the medium and small strata categories. Itron, Inc Custom Lighting Research Plan

9 1.6 Work Plan This section summarizes the activities, approaches and budget for conducting an energy efficiency impact evaluation for nonresidential downstream custom lighting measures Task 1: Project Management The objective of this task is to provide all necessary coordination and communication across the project team, CPUC staff and their consultants and other related work order teams. Itron will ensure that the CPUC staff and their consultants stay fully informed of the project status through a combination of regularly scheduled telephone calls and meetings and written project status reports to document status and results. Monthly progress reports will be linked to invoices showing percentages of work complete. Monthly PCG meetings will also be held to discuss status with CPUC staff, their consultants and IOU staff. Cross-Evaluation group meetings and communications will be scheduled as needed to promote the sharing of information and to leverage data collection and information relevant to multiple contract groups. These meetings will also allow the development of consistent approaches for metering, monitoring, and evaluation methodologies Task 2: Develop and Update Research Plan This task includes the development of this research plan and future updates to this plan. One of the primary inputs for supporting the development of this plan was the 2015 program tracking data. This plan will be thoroughly reviewed and updated throughout the year as tracking data resubmissions are filed Task 3: Tracking System Management and Sampling The objective of this task is to synthesize and analyze information from the PA tracking systems for the projects, measures and programs in the nonresidential custom lighting category. Any delays in receiving complete and accurate tracking data may result in delays in reporting. Of particular interest for informing sampling development is the distribution of reported savings. In this task, we will prepare the necessary data requests and collect and analyze all program data necessary to support this evaluation. This includes receiving copies of applications for all participant sites that will be included in the sample frame and entering key information from the applications into the evaluation database. The sample design for on-site and telephone survey points documented in this evaluation plan will be implemented as planned. The sample design will be updated (if necessary) with data received from PA re-submissions throughout the year. Itron, Inc Custom Lighting Research Plan

10 1.6.4 Task 4: Data Collection Instrument and Tool Development This task is comprised of developing all data collection instruments for on-sites and phone surveys. In addition, this task includes the development and maintenance of the on-site databases that will be used to input all data collected from the on-site verification process Task 5: Gross Impact Evaluation This task will involve on-site data collection, monitoring, and analysis of a representative sample of nonresidential downstream custom lighting projects in order to a) independently verify reported measure installation records, b) develop ex post estimates of the energy savings, and c) apply those findings to the full participant population in order to obtain a complete estimate of program impacts for nonresidential custom lighting measures Task 6: Net Impact Evaluation The evaluation team will conduct net-to-gross surveys of a statistically valid sample of participating customers that represent the participant population. The purpose of these interviews is to obtain data needed for the calculation of the net-to-gross ratio. The net-to-gross participant sample will overlap significantly with the gross impact sample Task 7: Reporting The final report will document study assumptions, data, methods, results, and recommendations. All material will be presented in a clear, concise fashion. The Executive Summary will emphasize important findings and recommendations and provide a general overview. The draft report will be provided to the CPUC project manager and management team, and the PAs for review and comments. We will incorporate any necessary changes and finalize the report. The final report will include: Results of the gross impact analysis based on the on-site data collection efforts. Results of the net-to-gross estimation analysis. Recommendations for research activities to further improve the ex ante savings estimates at the PA level and, where feasible, at the program and/or measure level Budget The cost estimate for this impact evaluation is approximately $800,000. Itron, Inc Custom Lighting Research Plan