NSTAR Electric. Small Business Solutions Program Program Year 2002 Impact Evaluation. Final Report. November Prepared by

Transcription

1 Small Business Solutions Program Program Year 2002 Impact Evaluation Final Report November 2003 Prepared by 179 Main Street, 3rd Floor Middletown, CT 06457

2 Acknowledgments The authors wish to thank all of the people at NSTAR Electric who took the time to support and help with this study. Regrettably, we cannot thank everyone individually, but we do want to acknowledge the contributions made by Susan Haselhorst, Lisa Shea, and Ralph Prahl. The data, insight, and support provided by these individuals helped to establish the foundation for this report. Of course, we assume sole responsibility for any errors or omissions.

3 NSTAR Electric 2002 Small Business Solutions Program Impact Evaluation Table of Contents EXECUTIVE SUMMARY...I 2002 SMALL BUSINESS SOLUTIONS PROGRAM SUMMARY... I EVALUATION APPROACH SUMMARY...II SAVINGS SUMMARY...II ADDITIONAL RESEARCH FINDINGS... III INTRODUCTION STUDY OVERVIEW AND PURPOSE THE ANALYSIS FRAMEWORK: SAMPLE DESIGN AND SELECTION POPULATION CHARACTERISTICS STATISTICAL SAMPLING METHODOLOGY ON-SITE SAMPLE DESIGN SUPPLEMENTAL NET EFFECTS SURVEY SAMPLE DESIGN EXPECTED FINAL NET PRECISION SITE LEVEL ENGINEERING ASSESSMENT FILE REVIEWS FOR DOCUMENTATION ERRORS SITE DATA COLLECTION BILLING METER MATCH SURVEY COOLER CONTROL MEASURE SAVINGS ESTIMATION LIGHTING MEASURE SAVINGS ESTIMATION OTHER MEASURE SAVINGS ESTIMATION ADJUSTED GROSS ANNUAL SAVINGS FOR ON-SITE SAMPLE PROGRAM LEVEL ADJUSTED GROSS ENERGY SAVINGS STRATIFIED RATIO ESTIMATION ANALYSIS REPORTED VERSUS MEASURED OPERATING HOURS ADJUSTED GROSS ANNUAL ENERGY SAVINGS IMPLEMENTATION VENDOR RESULTS ADJUSTED GROSS LIFETIME ENERGY SAVINGS CONNECTED, SUMMER PEAK, AND WINTER PEAK DEMAND DETERMINING NET ENERGY SAVINGS ANALYTICAL APPROACH METHODOLOGICAL SURVEY APPROACH PARTICIPANT LIKE SPILLOVER NON-PARTICIPANT SPILLOVER PARTICIPANT FREE RIDERSHIP NET ANNUAL ENERGY SAVINGS NET LIFETIME ENERGY SAVINGS NSTAR REPORTING TEMPLATES TOTAL PROGRAM RESULTS COOLER CONTROL MEASURES LIGHTING MEASURES RLW Analytics, Inc. Page i November 2003

4 6.4 OTHER END-USE OTHER SURVEY RESULTS BILLING METER MATCH SURVEY PROGRAM SATISFACTION ALTERNATIVE ON-SITE NTG APPROACH NON-PARTICIPANT SPILLOVER VIA ELECTRICAL SUBCONTRACTORS LIGHTING HVAC INTERACTIVE EFFECTS CONCLUSIONS SUMMARY OF PROGRAM IMPACTS LESSONS LEARNED APPENDIX A: GROSS ON-SITE RESULTS APPENDIX B: ON-SITE DATA COLLECTION FORM List of Tables Table I: Program Population Summary... I Table II: 2002 SBS Total Reporting Template... III Table III: 2002 SBS Savings Summary By End Use... III Table 1: Program Population Summary... 2 Table 2: On-Site Sample Design Parameters... 3 Table 3: On-Site Sample Coverage and Expected Gross Precision by Measure Category... 3 Table 4: Cooler Control On-Site Sample Design... 4 Table 5: Lighting On-Site Sample Design... 4 Table 6: On-Site Sample Coverage and Expected Gross Precision by Vendor... 4 Table 7: NTG Sample Design Parameters... 5 Table 8: Lighting NTG Sample Design... 5 Table 9: Cooler Control NTG Sample Design... 6 Table 10: Cooler Control Savings by Detailed Measure Type...9 Table 11: Gross Cooler Control Savings Approach... 9 Table 12: Lighting Measure Savings by Technology... 9 Table 13: Gross Lighting Savings Calculations Table 14: Other Measure Savings Estimation Table 15: Comparison of Tracking to Adjusted Gross for On-Site Sample Table 16: Summary of Adjusted Gross Annual Energy Savings Table 17: Adjusted Gross Annual Energy Savings by Vendor Table 18: Summary of Adjusted Gross Lifetime Energy Savings Table 19: Completion Rate for Participant Telephone Surveys Table 20: Telephone Survey Accounting Table 21: Participants Identified with Spillover Table 22: Participant Spillover Annual Energy Impacts by End Use Table 23: Summary of Non-Participant Spillover Reponses Table 24: Non-Participant Spillover Confirmation Questions (Vendor #1) Table 25: Non-Participant Spillover Annual Energy Impacts by End Use Table 26: Participants Identified with Free Ridership Table 27: Free Ridership Annual Energy Impacts by End Use Table 28: Summary of Net Annual Energy Savings Table 29: Summary of Net Lifetime Energy Savings Table 30: Legend for Template Adjustment Factor Denominators Table 31: 2002 SBS Program Reporting Template Table 32: 2002 SBS Savings Summary By End Use RLW Analytics, Inc. Page ii November 2003

5 Table 33: Cooler Control Reporting Template Table 34: Lighting Reporting Template Table 35: Other Reporting Template Table 36: Meter Match Reliability Assignments Table 37: Billing Meter Map Reliability Table 38: Program Satisfaction Table 39: Program Dissatisfaction Table 40: Comparison of NTG Results by Approach Table 41: Lighting HVAC Interaction Table 42: 2002 SBS Program Reporting Template List of Figures Figure 1: Tracking System vs. Gross On-Site Scatterplot Figure 2: Reported Versus Measured Operating Hours Figure 3: Pooling On-Site and NTG Studies RLW Analytics, Inc. Page iii November 2003

6

7 Executive Summary NSTAR Electric 2002 Small Business Solutions Program Impact Evaluation This report summarizes an impact evaluation performed by RLW Analytics, Inc. (RLW) during 2003 to quantify adjusted gross and net energy savings associated with the 2002 NSTAR Electric Small Business Solutions Program. The Small Business Solutions Program ( SBS Program, previously the Small C&I Retrofit Program) identifies cost-effective efficiency retrofit opportunities and provides direct installation, financial incentives, and other strategies to encourage the early replacement of existing equipment with high efficiency alternatives, as well as the installation of new equipment. All energy-using systems are eligible for improvements including lighting, refrigeration controls, domestic hot water, and HVAC tune-ups. The SBS Program is a turnkey offering that provides customers with a single source for information, technical assistance, and financial incentives. NSTAR has offered the SBS Program since mid-1999; it is presently administered through several prime contractors who are responsible for the following elements: 1) Marketing to customers; 2) Performing site visits and collecting all equipment and energy data, identifying efficiency opportunities, performing analysis, and making recommendations to the customer; 3) Presentation of the recommendations to the customer and obtaining customer agreement to proceed with the installation; 4) Preparation and submission of completed customer rebate applications; 5) Installation of eligible measures; 6) Tracking program activity; and 7) Tracking and setting aside all equipment for proper disposal Small Business Solutions Program Summary In 2002, the SBS Program served 1,424 participants, achieving approximately 24,655 MWh in energy savings. Approximately 82% of program savings were in the Lighting end use category. Seventeen percent of the energy savings were Cooler Control measures, while just 1% of the program savings were in Other end uses such as HVAC and Refrigeration. Cooler Control measures include walk-in cooler economizers, door heater controls, evaporator fan controls and vendor case shutdown. Measure Category Tracking Energy Savings (kwh) Customers Measures Cooling 4,309, Lighting 20,119,315 1,248 16,059 Other 226, Total 24,654,701 1,424 16,359 Table I: Program Population Summary RLW Analytics, Inc. Page I November 2003

8 Evaluation Approach Summary NSTAR structured this impact evaluation to pursue Cooling and Lighting results via primary data collection. Given the small relative impact of Other measures, it was agreed that NSTAR would supply realization rates and estimated precision for the Other measures. The 2002 SBS evaluation work included the following core activities: 1. An analysis plan for selection of a representative sample of participants for on-site surveys and an independent sample for telephone surveys. The participants were segmented by lighting and cooler control measure type. 2. On-site visits at 50 Lighting sites and 5 Cooler Control sites with engineering reanalysis of annualized adjusted gross impacts. The lighting hours of operation relied upon customer interviews and did not include loggers. The five cooler control sites utilized logged data. The five cooler control sites visits were completed under a separate contract by a separate contractor (Select Energy Services, Inc., or SESI ). 3. Participant free-ridership and spillover telephone surveys of an independent sample of 35 Lighting sites and 46 Cooler Control sites to inform net-to-gross savings estimates. 4. Non-participant spillover surveys of the 8 program implementation vendors and 35 electrical subcontractors. 5. Expansion of the site and the telephone surveys to program level savings and ancillary analysis. Savings Summary Table II presents the final reporting template for total SBS program savings. The net realization rate for the 2002 NSTAR Small Business Solutions program is 85.8% ± 12.0% at the NSTAR program level. Most of the gross adjustments were due to a 12.4% overstatement of operating hours. This adjustment was primarily a function of a reported to measured hours of operation realization rate determined in a logger study completed in and findings at sites with cooler control measures. The net-to-gross ratio was particularly high at 99.0%, with 2.7% free-ridership almost entirely offset by spillover effects. 1 NSTAR Electric Logger Study of the Small Commercial & Industrial Retrofit Program, RLW Analytics, Inc., October RLW Analytics, Inc. Page II November 2003

9 TOTAL 2002 SBS PROGRAM ENERGY SUMMER DEMAND WINTER DEMAND Reporting Parameter kwh Factor Summer kw Factor Winter kw Factor A - Gross Tracking Savings 24,654,701 N/A 4,736 N/A 4,736 N/A B - Documentation Errors (177,003) -0.7% 1, % 1, % C - Revised Gross Savings 24,477, % 6, % 6, % D - Technology Adjustment (44,273) -0.2% % % E - Quantity Adjustment (28,640) -0.1% % % F - Hours of Operation Adjustment (3,036,971) -12.4% N/A N/A N/A N/A G - Coincident Adjustment N/A N/A (1,255.2) -19.9% (2,930.5) -46.4% H - Adjusted Gross Savings 21,367, % 5, % 3, % I - Participant Spillover Adj. 296, % % % J - Non-Participant Spillover Adj. 56, % % % K - Free Ridership Adjustment (575,630) -2.7% (120.4) -2.3% (89.0) -2.6% L - Net Savings 21,144, % 5, % 3, % M - Net Realization Rate 85.8% 107.9% 72.5% N - Measure Lifetime 15.0 O - Precision, 90% Confidence ±12.0% Table II: 2002 SBS Total Reporting Template Individual reporting templates for Cooling, Lighting, and Other end use categories are presented later in this report. Table III presents a broad overview of annual energy results by end use. Adjusted gross realization rates ranged from a low of 70.4% for Cooling measures to a high of 90.2% for Lighting measures. In sum, net-to-gross effects were extremely limited, with a low of 96.9% for Cooling measures. End Use Tracking Adj Gross Gross Real. Net Real. Net-to-Gross Net kwh kwh kwh Rate Rate Ratio Cooling 4,309,015 3,032, % 2,937, % 96.9% Lighting 20,119,315 18,150, % 18,022, % 99.3% Other 226, , % 184, % 100.0% Total 24,654,701 21,367, % 21,144, % 99.0% Table III: 2002 SBS Savings Summary By End Use Additional Research Findings In conjunction with the primary task of determining net impacts, additional research was conducted to measure customer satisfaction, examine net effects methodologies, provide field data regarding future billing analysis and quantify a lighting-cooling interactive effect. In summary, the findings are: Program Participants were highly satisfied with the program, citing reduced energy bills, financial incentives and improved equipment as the key benefits of the program. About 40% of the participants could not identify any program improvements. It was hypothesized that the electrical subcontractors employed by the program implementation vendors might induce non-participant spillover. However, non-participant spillover interviews of all 35 of these contractors yielded no spillover. It was hypothesized that on-site vs. telephone surveys may yield different free rider and spillover results. RLW administered the telephone surveys on-site and found the results were not statistically different from those surveys administered by telephone. It was hypothesized that multi-meter sites might make a billing analysis difficult for this program. A billing meter mapping on-site survey found that at two-thirds of sites, the billing meters mapped well to the energy efficiency measures. However they did not for one-third of the sites. RLW Analytics, Inc. Page III November 2003

10 Additionally, the sites with the largest savings experienced the most uncertainty in mapping billing meter to the installed measures. Evaluators concluded that 2002 SBS lighting retrofits yielded an additional 2.1% of savings due to interactive heating and cooling effects. An electric heating penalty of -3.0% was offset by a cooling benefit of 5.1%. The interactive effect is not included in the reported program estimates of savings. RLW Analytics, Inc. Page IV November 2003

11 Introduction 1 Study Overview and Purpose NSTAR Electric 2002 Small Business Solutions Program Impact Evaluation This report summarizes an impact evaluation performed by RLW Analytics, Inc. (RLW) during 2003 to quantify adjusted gross and net energy savings associated with the 2002 NSTAR Electric Small Business Solutions Program. The Small Business Solutions Program ( SBS Program, previously the Small C&I Retrofit Program) identifies cost-effective efficiency retrofit opportunities and provides direct installation, financial incentives, and other strategies to encourage the early replacement of existing equipment with high efficiency alternatives, as well as the installation of new equipment. All energy-using systems are eligible for improvements including lighting, refrigeration controls, domestic hot water, and HVAC tune-ups. The SBS Program is a turnkey offering that provides customers with a single source for information, technical assistance, and financial incentives. NSTAR has offered the SBS Program since mid-1999; it is presently administered through several prime contractors who are responsible for the following elements: 1) Marketing to customers; 2) Performing site visits and collecting all equipment and energy data, identifying efficiency opportunities, performing analysis, and making recommendations to the customer; 3) Presentation of the recommendations to the customer and obtaining customer agreement to proceed with the installation; 4) Preparation and submission of completed customer rebate applications; 5) Installation of eligible measures; 6) Tracking program activity; and 7) Tracking and setting aside all equipment for proper disposal. NSTAR Electric is committed to determining the ongoing effectiveness of its energy efficiency programs. As part of this commitment, NSTAR sponsors process and impact evaluations. These activities are focused on improving the delivery of the programs and the quantification of electric savings achieved through their implementation. This impact evaluation examined projects completed in 2002 under NSTAR s Small Business Solutions program. The results are presented by electrical end use and are tabulated for both adjusted gross and net energy savings. Impact Evaluation Objective Determine the adjusted gross and net energy impacts associated with 2002 SBS measure installations using on-site engineering assessments and standardized methods for freeridership and spillover. RLW Analytics, Inc. Page 1 November 2003

12 2 The Analysis Framework: Sample Design and Selection 2.1 Population Characteristics In 2002, the SBS Program served 1,424 participants, achieving approximately 24,655 MWh in energy savings. Approximately 82% of program savings were in the Lighting end use category. Seventeen percent of the energy savings were Cooler Control measures, while just 1% of the program savings were in Other end uses such as HVAC and Refrigeration. Cooler Control measures include walk-in cooler economizers, door heater controls, evaporator fan controls and vendor case shutdown. Measure Category Tracking Energy Savings (kwh) Customers 1 Measures 2 Cooling 4,309, Lighting 20,119,315 1,248 16,059 Other 226, Total 24,654,701 1,424 16,359 Table 1: Program Population Summary NSTAR structured this impact evaluation to pursue Cooling and Lighting results via primary data collection. Given the small relative impact of Other measures, it was agreed that NSTAR would supply realization rates and estimated precision for the Other measures. RLW would assist in this task, review them for appropriateness, and pool the results from these three segments into program-level results. 2.2 Statistical Sampling Methodology A central impact evaluation challenge is to develop statistically reliable results. RLW used statistical sampling principles to select a sample of customers for analysis. The tracking information and sampling plan provided the basis for extrapolating the results found in the sample back to the target populations. We used the methods and principles of survey sampling and load research to get unbiased estimates of the total impacts that would have been found by conducting the analysis for the entire population of program participants. The tracking system was used to develop a sampling frame and sampling plan for selecting the projects for analysis. Moreover, an error bound or confidence interval was associated with this estimate of total impact that reflects the sensitivity of the results to the sample that is actually analyzed. RLW employed Model-Based Statistical Sampling (MBSS) methodologies for the design of the impact evaluation sample. MBSS offers proven methods for stratifying the populations of interest. Typically, the MBSS methodology begins by sorting the sample population by magnitude of estimated annual kwh savings. As the sample was designed to emphasize the portion of participants with larger impacts, MBSS provided an excellent framework for conducting the analysis and the subsequent stratified ratio estimation. Our MBSS methodology was used to target ±10% relative precision at the 90% confidence interval for net energy savings at the NSTAR level. 1 In this table, a Customer is a unique NSTAR account number with energy savings in the NEEDS tracking system database for this program year. In actuality, there are 1,426 unique account numbers in the database, but two entries have zero savings. Since a customer can have savings across multiple measure categories, 1,424 customers is not the total of the preceding rows. 2 In this table, a Measure is a single record entry with energy savings in the NEEDS tracking system database for this program year. The database has 17,235 unique records, but 866 have blank savings and another 10 have zero savings. RLW Analytics, Inc. Page 2 November 2003

13 2.3 On-Site Sample Design At the project kickoff meeting, the evaluation team decided that the on-site sample design would be based upon a sample size of 55 on-sites: 50 Lighting sites and 5 Cooler Control sites. It was also decided that both Active and Inactive (about 6% of the participants) accounts would be included in the sample. Upon receipt of the 2002 SBS Program database, RLW modeled the expected SBS program results given these preliminary sample sizes for several purposes. First, RLW was interested in assessing if the sample of 55 is sufficient to achieve ±10% relative precision at the program level. Second, NSTAR wanted to examine the potential for achieving the required program precision with fewer sample points and hence lower evaluation cost. Finally, we thought it prudent to consider the expected precision within each of the measure categories, since NSTAR requested that they be on separate reporting templates. Table 2 presents the sample design parameters that were used as inputs in the statistical model. The error ratio is a measure of the variability in the relationship between the measured savings and the tracking estimate of savings. At the evaluation kickoff meeting, the project team agreed upon an assumed error ratio of 0.50 for the Cooler Control measures. RLW reran the statistical model from our earlier lighting logger study of this program to conclude an error ratio of 0.39 for Lighting sites, designing this sample based upon past performance of similar evaluations. The true gamma - a measure of the degree of heteroscedasticity in the relationship between measured savings and the tracking estimate of savings - was assumed to be 0.8, which is typical for DSM applications involving on-site surveys. Measure Error True Category Ratio Gamma Cooler Control Lighting Table 2: On-Site Sample Design Parameters In Table 3, we see the coverage rate and expected relative precision by measure category. This sample design was expected to meet the impact evaluation goals by achieving ±9.8% precision at the NSTAR level. At the end-use level, a sample of 50 Lighting sites should yield excellent precision at ±9.2%, and a sample of 5 Cooler Control sites was expected to result in ±36.9% relative precision 3. At this stage, we arbitrarily estimated that the precision of the non-sampled Other results will be ±20.0%, although the small savings in this category would result in little impact on the Total NSTAR precision estimate. Measure Population On-Site Sample Gross Category Size (N) Sample (n) Coverage Rel. Prec. Cooler Control % ±36.9% Lighting 1, % ±9.2% Other % ±20.0% Total 1, % ±9.8% Table 3: On-Site Sample Coverage and Expected Gross Precision by Measure Category Table 4 presents the stratified sample design for customers with Cooler Control measures. A five strata design was selected which resulted in the allocation of two sample points per stratum. There were 69 customers with Cooler Control savings less than or equal to 14,150 kwh. In total, this stratum accounts for 691,896 kwh of savings in this measure category. One customer was 3 NSTAR has elected to utilize the small sample size in the Cooler Control measure category despite the high estimate of relative precision in this sub-segment. This decision is inconsequential to the ±10% precision goal of the evaluation and was made for cost-containment purposes. RLW Analytics, Inc. Page 3 November 2003

14 selected at random from this stratum to receive an on-site engineering assessment from Select Energy Services to evaluate Cooler Control measures. Cooler Strata Population Total On-Site Control Cut-point Size Savings Sample Weight Stratum (kwh) (N) (kwh) (n) (N/n) 1 14, , , , , , , , , ,066, Total 184 4,309,015 5 Table 4: Cooler Control On-Site Sample Design Likewise, Table 5 presents the stratified sample design for customers with Lighting measures. A five strata design was selected which resulted in the allocation of ten sample points per stratum. There were 694 customers with Lighting savings less than or equal to 10,128 kwh. In total, this stratum accounts for 2,892,887 kwh of savings in this measure category. Ten customers were selected at random from this stratum to receive on-site engineering assessments from RLW Analytics, Inc. to evaluate Lighting measures. Strata Population Total On-Site Lighting Cut-point Size Savings Sample Weight Stratum (kwh) (N) (kwh) (n) (N/n) 1 10, ,892, , ,502, , ,979, , ,478, , ,265, Total 1,248 20,119, Table 5: Lighting On-Site Sample Design Fundamentally, any program parameter that is influential on program performance is a valid consideration for a sampling dimension. It was posed that the implementation contractor or vendor may be of significance. By analyzing the sample design by this sector variable, Table 6 outlines the sample coverage rates and expected precision by vendor. Population On-Site Sample Gross Vendor Size (N) Sample (n) Coverage Rel. Prec. Aarow % ±20.5% Ameresco % ±30.2% Coastal % N/A Harris % ±13.4% NRM % ±36.9% Prism % N/A Rise % ±17.3% Synergy % N/A Total 1, % ±9.8% Table 6: On-Site Sample Coverage and Expected Gross Precision by Vendor RLW Analytics, Inc. Page 4 November 2003

15 2.4 Supplemental Net Effects Survey Sample Design At the kickoff meeting, the evaluation team decided to assess net program effects via an independent telephone sample of participants. Table 2 presents the sample design parameters that were used as inputs in the statistical model. Measure Error True Category Ratio Gamma Cooler Control Lighting Table 7: NTG Sample Design Parameters Originally, the evaluation team considered computing the error ratio as a proportional problem using results from past Net-to-Gross surveys. NTG results from recent studies suggest an error ratio of 0.25 to 0.30 for either measure category. However, since this evaluation will be the first time NSTAR employs a new NTG approach from the Joint Utility Study, the NSTAR evaluation manager felt it was appropriate to select more conservative estimates of error ratio. Evaluators elected to employ the same error ratios as the on-site work as a hedge against uncertain survey performance. While for on-site work we employ a gamma parameter of 0.8 to place greater emphasis on sites with larger savings, a gamma of 1.0 is used for net-to-gross surveys to treat all participants equally. This is based upon the premise that the probability of a free-rider determination is not related to the amount of savings or size of customer. Using these sampling parameters, MBSS computed a sample size of 50 Lighting and 47 Cooler Control net-to-gross surveys to achieve the target relative precision of ±10%. The expected relative precision for the NTG surveys is ±9.8% and ±9.4% in the Lighting and Cooler Control categories, respectively. Table 8 shows the sample design based upon the Lighting population of 1,248 sites. We constructed five strata according the tracking estimate of the savings for each site. We randomly selected the sample points within each stratum employing a random number generator, eliminating any opportunity for selection bias. For example, in Stratum 1 we drew 10 sites from a population of 793 sites, yielding a case weight of = Strata Population Total Survey Lighting Cut-point Size Savings Sample Weight Stratum (kwh) (N) (kwh) (n) (N/n) 1 12, ,017, , ,004, , ,012, , ,020, , ,063, Total 1,248 20,119, Table 8: Lighting NTG Sample Design Similarly, Table 9 shows the sample design based upon the Cooler Control population of 184 sites. We constructed five strata according the tracking estimate of the savings for each site. We randomly selected the sample points within each stratum employing a random number generator, eliminating any opportunity for selection bias. RLW Analytics, Inc. Page 5 November 2003

16 Cooler Strata Population Total Survey Control Cut-point Size Savings Sample Weight Stratum (kwh) (N) (kwh) (n) (N/n) 1 16, , , , , , , ,005, , , Total 184 4,309, Table 9: Cooler Control NTG Sample Design Armed with this information, NSTAR elected to proceed with a round number of 100 telephone NTG surveys, 50 in each end-use. 2.5 Expected Final Net Precision In order to ascertain the relative precision for net savings, one must pool the expected precision for the Gross on-site estimates and the Net-To-Gross surveys. For two independent statistical studies, the pooled precision is represented by the square-root of the sum of the relative precisions squared, or: ( RP ) ( ) 2 gro RP RP net = + 2 NTG Therefore, the analytical expectation is for ±12.4% precision on the program estimate of net savings. At the time of the sample design, evaluators expected the final net precision to be closer to the study objective of ±10% since the cooler control and lighting error ratios had been estimated conservatively. In the end, however, the calculated prediction of ±12.4% was very close to the final achieved precision of ±12.0%. RLW Analytics, Inc. Page 6 November 2003

17 3 Site Level Engineering Assessment Field data necessary to support the computation of Program impacts and other analysis was collected via on-site engineering assessments. The on-site work was assigned to two different engineering firms according to electrical measure type category. Select Energy Services, Inc. (SESI) collected data on cooler control measures and RLW Analytics was responsible for all lighting measures as well as the overall impact evaluation. The following sections detail the methods employed while on-site; results are presented later in the report. The SESI site surveys were part of an independently contracted effort with separate reporting, although the selection of the five cooler control sites and the expansion of site level results to the program level were included in this study. At the time of this report, the separate cooler control study is still in process and will include fifteen additional sites beyond the five included here. This study will be completed the end of File Reviews for Documentation Errors RLW and SESI performed comprehensive file reviews for each project in the on-site sample. This process identified a category of discrepancies called Documentation Errors which included, in order of significance a truing up of demand savings, a significant discrepancy at one site and rounding errors. At the onset of the evaluation, it was known that the NSTAR tracking system only tracked connected kw demand savings and contained a considerable number of missing values. As part of the file review process, RLW/SESI determined what the tracking demand values should have been for missing demands at the 55 visited sites. SESI identified one documentation error for annual savings at Cooler Control site The tracking system showed annual savings as 20,022 kwh, while NRM project documentation indicates that the savings should have been 16,315 kwh, due to changes after the application was filed. RLW replicated the savings calculations documented in paper project files with spreadsheet calculations for Lighting sites, frequently identifying rounding errors (<0.1%) in total kwh savings Within the on-site samples, there was a total Lighting rounding discrepancy of just +43 kwh between the tracking and revised estimates. As mentioned, on the Cooling side there was one negative documentation adjustment of -3,707 kwh. These adjustments are for the sample points only, prior to stratified ratio estimation. The extrapolated impacts on the program population are presented in the Documentation Error column of Table 16 in Section 4.3: Adjusted Gross Annual Energy Savings. 3.2 Site Data Collection The fundamental data collection activity associated with this project was the on-site visit. These visits were comprised of three primary activities: inventorying and assessment of the installed energy efficiency measures (EEMs), performing a billing meter mapping assessment, and administering a survey with the decision-maker. The cooler control sites also included a measurement activity. These data collection activities performed at each site visit were to be used to satisfy the following evaluation conditions: Verify the tracking EEM data including quantity and technology (including ballast type); Identify if and when EEMs were removed; RLW Analytics, Inc. Page 7 November 2003

18 Collect current reported hours of operation from the participant for the EEM; Collect select business characteristics, such as business type and seasonal operation; Identify the timing and extent of any changes to the facility affecting energy consumption since the service date; For cooler control sites: conduct spot measurements, short term monitoring and longer long term logging; for lighting sites: no logging or measurements were taken; Conduct data collection for the billing meter mapping assessment; and Ask a limited set of customer satisfaction questions; Survey the customer as required to support the net effects research (for the purpose of comparing on-site vs. telephone net surveys although only the telephone survey results were used in the reported results). RLW produced an on-site data collection form for use during the visit. This form is provided in Appendix A of this report. This instrument was created to guide the on-site survey process and enforce a standard data collection procedure for all of the research activities listed above. The form was used to collect information about the site for use in the savings analysis, including building characteristics, operating schedules, HVAC systems (for interactive effects), and other key parameters. The on-site data collection form was reviewed and approved by NSTAR prior to implementation. 3.3 Billing Meter Match Survey The objective for this task was to assess how well the utility meter maps to the customer site and end-uses within the premise. NSTAR is considering the use of billing analysis for future impact evaluations of the SBS Program. A billing analysis was attempted unsuccessfully in It was hypothesized at the time that one reason the analysis may have failed was a frequent mismatch between the billing meter and the actual area receiving the energy efficiency services. This survey was designed to assess how frequently this may be an issue. Through a reasonable process of inspection and querying the owner, on-site engineers noted the identification numbers of all utility meters that appeared to serve each customer site. The scope of work did not include physically tracing wires. A meter match component was included in the on-site survey presented in Appendix A. Using this survey, RLW developed a simple assessment of the reliability of the meter map. 3.4 Cooler Control Measure Savings Estimation SESI conducted the on-site surveys and all of the savings analysis for the five cooler control sites. A full reporting of the methodology and findings for cooler control measures will be separately published in a report to be issued at the end of Cooler control measures primarily target walk-in coolers at liquor and convenience stores. The measures include: Economizer cooling: Outdoor air is brought in when outdoor air temperatures drop below the cooler set point (typically below 36ºF). Evaporator fan control: Control evaporator fans to cycle in conjunction with the compressor rather than continuously. Door heater control: Control anti-sweat door heaters to operate only when humidity conditions warrant it. Vendor case shutdown: Turn novelty cases (typically soda) off during unoccupied periods. RLW Analytics, Inc. Page 8 November 2003

19 Preliminary findings indicate issues with the economizer measures, inspections protocols and some of the algorithm assumptions. Findings for individual cooler control measures are detailed below in Table 10. Annual kwh Savings Savings Adjustments Realiz. Cooler Control Measure Gross Adj Gross Disconnect Base Cond. Quantity Operation Total Rate Economizer 14,510 2, ,281-12,281 15% Fan Control 58,878 47, ,608-11,608 80% Cooler Door Heater 30,414 8,081-11,109-10, ,333 27% Freezer Door Heater 10,717 10, % Vendor Case Night Shut Down 20,060 17, , ,837 86% Air Des N/A TOTAL 134,579 84,202-11,109-10,436-3,708-25,125-50,378 63% % of Tracking 63% -8% -8% -3% -19% -37% Table 10: Cooler Control Savings by Detailed Measure Type The approach to estimating energy savings for the cooler control measures is summarized below: Task Measurement Analysis Measure Economizer Fan Control Door Heater Case Shutdown Estimation Approach Verify through inspection end-to-end continuity of controls. Spot kw and short term kw monitoring to confirm connected kw load. Control logs to determine long term cycling (up to 3 month s interval data, one year s totalized data by month). Regress logger cycle-on time against outdoor temperature. Use manufacturer's data to determine kw at load point. Use NOAA bin data to annualize. Savings Estimation Base case assumes compressor operation determined via equation from regressing cycle time against OAT. Post case based on measured cycle time. Base case assumes continuous operation. Post case based on measured cycle time. Base case assumes continuous operation. Post case assumes door operation determined via equation from regressing cycle time against wet-bulb. Base case assumes twenty-four hour operation at the observed cycle rate. Post case based on measured cycle time. Table 11: Gross Cooler Control Savings Approach 3.5 Lighting Measure Savings Estimation The lighting measures installed in the 2002 Small Business Solutions Program include the following distribution of lighting technologies: Percent of Lighting Technology Tracking Savings CFL fixtures 20% CFL bulbs 0% T8 fixtures 68% HID fixtures 6% LED exists 6% Controls 0% Table 12: Lighting Measure Savings by Technology The fundamental task of the site visit was to collect data to inform estimates of adjusted impacts. As short-term metering and time-of-use logging were outside of the scope of this project, all operational characteristics were obtained through available non-measurement means. Such methods included: discussion of operating schedules with facility management; interviews of RLW Analytics, Inc. Page 9 November 2003

20 occupants or maintenance staff; review of time clock, EMS, or other automated system controls; and any other resource for informing operational characteristics. The engineering estimation of savings was calculated on a line-by-line basis in a spreadsheet which cascades and disaggregates the influences of installed measure type, quantity, diversity, interactive effects and operating hours for the derivation of adjustment factors. The fundamental algorithm and source of each factor is described in the table below. The Table also indicates how discrepancies in a parameter affected the reporting adjustment factors. Pre Quantity Pre Watts Factor Source Adjustment Factors Technology & count provided by installation vendor. Watts per fixture from NSTAR Fixture Wattage Table Doc Error: Watts in tracking differs from Table. Post Quantity Verified by count on site Quantity: Tracking < or > than found in field Post Watts Technology verified on site. Watts per fixture from NSTAR Fixture Wattage Table Doc Error: Watts in tracking differs from Table Tech: Tracking technology differs from field Document: Watts in tracking differs from Table Hours of Operation RLW interviews of customers. Operation: Tracking hours differs from interview hours Interactive Adjustment % RLW determined if lights are in a cooled or heated space and applied factor appropriate for cooling system. Reported separately, not incorporated into final results. Applied to energy only and not to reported demand. Reported Adjustment % -8.1% factor from 2001 Logger Study Applied at a program level; site level estimates do not include this adjustment. % on at peak RLW interviews of customers. NA Connected kw = (Pre Qty) x (Pre Watts) - (Post Qty) x (Post Watts) Peak kw = (Connected kw) x (% on peak) Energy kwh = (Connected kw) x (Hours of Operation) x (Interactive Adjustment %) x (Reported Adjustment %) Table 13: Gross Lighting Savings Calculations 3.6 Other Measure Savings Estimation Primary data collection was not performed for measures in the Other end use category. Given the small relative impact of Other measures, it was agreed at project inception that NSTAR would supply realization rates and estimated precision for the Other measures. RLW assisted in this task and helped NSTAR find secondary sources of measure performance in this category. Together, RLW and NSTAR estimated an adjusted gross realization rate of 81.7% for annual energy savings. The computations were fairly comprehensive and reasonable, considering secondary research findings for specific measures such as refrigeration, HVAC tune-ups, EMS, and Lighting Controls. RLW reviewed these estimates for appropriateness, and helped NSTAR choose a reasonable associated precision of ±30.0% for this aggregated realization rate. A number of the data sources informing the realization rate were not statistical studies, so ±30.0% is simply an intuitive assessment of the expected error bound associated with the combined estimate. While arbitrary, the selected relative precision for the Other end use had little impact on the precision of the overall estimate across all end uses. RLW Analytics, Inc. Page 10 November 2003

21 Table 14 summarizes the sources for the gross adjusted savings estimates by measure. Number of Tracking Adjusted Gross Realiz. Measures Measures kwh kw kwh kw Rate Domestic Refrigerator , , % Energy Management System , , % Lighting Occupancy Sensors 6 5 2, , % Stripheat Cutoff Thermostat 1 5, , % Oven Shutdown Timer 1 1, , % Other Refrig Vendor Case 7 1 2, , % HVAC TuneUp , , % Other Total , , Table 14: Other Measure Savings Estimation 3.7 Adjusted Gross Annual Savings for On-Site Sample Table 15 summarizes the gross estimates of energy savings developed during the on-site engineering assessments. Specifically, it lists the annual tracking and adjusted gross energy savings for each sites in the on-site sample. Recall from Section 2 that we sampled five Cooler Control sites and 50 Lighting sites. The table shows a site identification number, the end-use category, the tracking system estimate of annual savings, the adjusted gross on-site estimate of annual savings, and the percent difference between the two savings estimates. A negative sign on the percent difference indicates that the gross on-site estimate of savings was lower than the tracking estimate of savings. Four of the five (80%) Cooler Control sites and 23 of the 50 (46%) Lighting sites of the thirty-nine applications had on-site savings that were lower than their tracking system counterparts. For the entire sample, the gross estimate of savings is 6% lower than the tracking system. This does not represent the overall program difference, however, as the sample was stratified by project savings and thus weighted toward the larger sites. Table 15 does not include the Reported to Metered Hours Adjustment factor of -8.1% or net free ridership and spillover adjustments. More savings detail for each site is listed in Appendix A of this report. 4 Single Family EnergyWise Program, prepared for National Grid by Quantec LLC, July 10, C&I Retrofit Program Program Year 2001, Impact Evaluation Final Report, RLW Analytics, Inc., April Factor used by Northeast Utilities direct install program. 7 Assumed conservative 60% realization rate. 8 NSTAR Small C&I HVAC Tune Up M&V Verifications Results for Tune-Up, Xenergy, Inc., December 12, RLW Analytics, Inc. Page 11 November 2003

22 Site ID End Use Tracking kwh Adj. Gross kwh Percent Diff. Site ID End Use Tracking kwh Adj. Gross kwh Percent Diff. C1 COOL 10,214 7,036-31% L24 LIGHT 95, ,038 29% C2 COOL 34,932 26,516-24% L25 LIGHT 33,563 11,704-65% C3 COOL 48,469 20,855-57% L26 LIGHT 22,006 29,209 33% C4 COOL 24,649 10,891-56% L27 LIGHT 17,507 15,572-11% C5 COOL 20,022 21,953 10% L28 LIGHT 31,035 33,644 8% COOL Sample Total 138,286 87,251-37% L29 LIGHT 14,334 16,556 16% L1 LIGHT 45,659 35,606-22% L30 LIGHT % L2 LIGHT 66,718 81,319 22% L31 LIGHT 15,975 16,012 0% L3 LIGHT 2,283 1,173-49% L32 LIGHT 13,550 11,796-13% L4 LIGHT 31,791 35,392 11% L33 LIGHT 152,005 72,579-52% L5 LIGHT % L34 LIGHT 20,311 19,666-3% L6 LIGHT % L35 LIGHT 39,893 38,422-4% L7 LIGHT 16,408 16,478 0% L36 LIGHT 62,903 55,324-12% L8 LIGHT 44,967 72,812 62% L37 LIGHT 41,459 30,930-25% L9 LIGHT 45,544 46,204 1% L38 LIGHT 12,910 12,916 0% L10 LIGHT % L39 LIGHT 78,410 68,573-13% L11 LIGHT 6,273 5,435-13% L40 LIGHT 7,410 15, % L12 LIGHT 17,668 16,271-8% L41 LIGHT 68,701 61,683-10% L13 LIGHT 38,085 25,451-33% L42 LIGHT 45,854 37,175-19% L14 LIGHT 6,840 5,086-26% L43 LIGHT 32,037 23,302-27% L15 LIGHT 57,511 59,482 3% L44 LIGHT 56,369 42,017-25% L16 LIGHT 1,577 1,704 8% L45 LIGHT 26,589 26,252-1% L17 LIGHT 164, ,935 2% L46 LIGHT 104, ,245 14% L18 LIGHT 34,944 46,045 32% L47 LIGHT 114, ,925 10% L19 LIGHT 17,920 18,243 2% L48 LIGHT 35,981 36,246 1% L20 LIGHT 25,843 25,194-3% L49 LIGHT 25,878 20,530-21% L21 LIGHT 82,260 87,470 6% L50 LIGHT 2, % L22 LIGHT 14,883 17,602 18% LIGHT Sample Total 1,906,552 1,842,377-3% L23 LIGHT 11,220 11,253 0% TOTAL Sample Total 2,044,838 1,929,628-6% Table 15: Comparison of Tracking to Adjusted Gross for On-Site Sample RLW Analytics, Inc. Page 12 November 2003

23 4 Program Level Adjusted Gross Energy Savings 4.1 Stratified Ratio Estimation Analysis The on-site gross savings estimates were combined in a stratified ratio estimation (SRE) analysis framework. Statistical weights were developed and applied to each sample participant to develop the total gross estimates of savings by end-use. These same weights were utilized to compute the program level demand, diversified demand and adjustment factors reported in the templates in Section 6. As outlined above, a final modification was made to adjust results for measured operating hours instead of reported hours of use. Prior to this adjustment, auditor reported hours were 2.0% less than expressed in the tracking system. Adjusted gross savings from the 2002 on-site work was pooled with the additional 8.1% reduction based upon 2001 measurement to yield an overall adjustment factor of -9.9% for operating hours. Relative precision was pooled between the two studies to accurately express the statistical confidence of the combined estimate. Annual Energy Savings Analysis Tracking vs. Gross On-Site 180, ,000 Gross Annual Savings (kwh) 140, , ,000 80,000 60,000 40,000 Lighting Realization Rate = 90.2% Cooling Realization Rate = 70.4% 20, ,000 40,000 60,000 80, , , , , ,000 Tracking Annual Savings (kwh) Cooling Sites Cooling RR Lighting Sites Lighting RR Reference Line Figure 1: Tracking System vs. Gross On-Site Scatterplot Figure 1 presents a scatter plot of the tracking system estimates of savings versus the adjusted gross, on-site engineering estimates of savings for the sampled participants. A one-to-one reference line is plotted as a dashed line on the diagonal. In addition, gross realization rates for Cooling and Lighting measures are plotted as solid lines reflecting the average savings-weighted realization rate of the sample points 9. For Cooler Control measures, the adjusted gross realization rate was calculated to be 70.4% with a statistical precision of ±24.1%. Lighting measures 9 This is the stratified ratio estimate of the realization rate in the population. It is calculated using the equation n i= 1 w i y i n i= 1 w i x i in which y is the measured savings of each project in the sample, and x is the tracking estimate of savings. The case weight w is the ratio between the total number of projects in the population in each stratum, divided by the total number of projects in the sample in that stratum. RLW Analytics, Inc. Page 13 November 2003

24 performed better with an adjusted gross realization rate of 90.2% and a statistical precision of ±12.7%. 4.2 Reported Versus Measured Operating Hours The scope of work for this evaluation was limited to the collection of Lighting hours of use information through interview means only and did not include a logger component. A comprehensive lighting logger study performed in the evaluation of the prior program year yielded results on adjustment factors for operating hours that NSTAR considers appropriate to apply to the 2002 SBS Program. Specifically, the study concluded that auditor reported hours overstate lighting operation by 8.1% compared to direct time-of-use measurement 10. RLW was instructed to integrate this adjustment factor and associated precision with this evaluation. Tracked Hours -2% Adjustment, based on customer interviews during on-sites Auditor-Reported Hours -8.1% Adjustment, from 2002 Logger Study, applied at Program Level Adjusted Gross Hours Figure 2: Reported Versus Measured Operating Hours Auditor reported hours were gathered through interviews of facility staff and used to create weekday, weekend and holiday operating profiles for each unique lighting line item. The profiles were annualized to include adjustments for seasonal operation. 4.3 Adjusted Gross Annual Energy Savings Table 16 summarizes the results for adjusted gross annual savings for the 2002 Small Business Solutions Program. At the NSTAR system level, total adjusted gross annual savings for all measures was found to be 21,368 MWh. The corresponding tracking estimate from Table 1 was 24,655 MWh, for a realization rate of 86.7%. Overall, the relative precision of the total adjusted gross estimate was ±11.3%. Following project reporting requirements, adjustment factors were computed to quantify the savings impacts of different types of discrepancies. NSTAR defined the three fundamental adjustment factors as Technology, Quantity, and Operation. Technology adjustments account for disparities between tracking system and field-observed technologies, such as equipment type, efficiency, power rating, etc. Quantity adjustments account for disparities in the number of measures documented in the tracking system and observed in the field. Operational adjustments account for all other operational disparities between tracking system and field-observations, such as annual hours and other measure dynamics. Having identified several minor documentation errors, RLW saw it fit to include these in a separate adjustment factor called Documentation Errors. For the total of all end use categories, all adjustment factors were negative, imparting a downward adjustment to the tracking system estimate of savings. On an absolute percentage 10 NSTAR Electric Logger Study of the Small Commercial & Industrial Retrofit Program, June 2003, Tables 9 and 13. RLW Analytics, Inc. Page 14 November 2003