Draft Evaluation Plan. California Statewide Codes and Standards (C&S) Program

Transcription

1 Draft Evaluation Plan California Statewide Codes and Standards (C&S) Program February 17, 2012

2 Prepared by: Allen Lee Dan Groshans February 17, 2012 Date Signature M. Sami Khawaja, Ph.D. Vice President The Cadmus Group, Inc.

3 Table of Contents Contact List List of Acronyms Executive Summary Background on C&S Programs Description of California Statewide Codes and Standards Program Objective and Mission of Codes and Standards Program Program Implementation Projected Energy Savings During C&S Evaluation Protocol with Recommended Revisions Overview of Evaluation Approach Evaluation Objectives Planning Assumptions Researchable Questions Revisions to Integrated Standards and Savings Model Treatment of Whole Building Performance Time Dependent Valuation Effects on Savings Multifamily Building Data Feasibility of Including Renovations in Evaluation Analysis of Title 24 Composite for Remainder (CfR) Cost Effectiveness and PEB LGP Reach Codes Evaluation Methodologies Building Codes (Title 24) Appliance Standards (Title 20 and Federal)...43 Mgmt/Admin...53 Evaluation QA/QC Reach Codes Including Local Government Programs (LGPs) Compliance Enhancement Program (CEP) Evaluation Coordination Efforts Coordination with Other Impact Evaluation Workplans Data Collection Approach Data Requests to the IOUs Feedback to the IOUs on Key Findings KEMA and The Cadmus Group i

4 6. Evaluation Timeline and Work Plan Evaluation Budget Appendix A: Title 24 Building Codes to Be Evaluated Appendix B: Title 20 and Federal Appliance Standards to be Evaluated Appendix C: Summary of CPUC Policy Decisions KEMA and The Cadmus Group ii

5 Contact List Organization Name Address Phone Energy Division, California Public Utilities Commission Ayat Osman, PhD Energy Division, CPUC 505 Van Ness Avenue San Francisco, CA Ken Keating Ken Keating, PhD 6902 SW 14th Ave Portland, OR (415) (503) DMQC: Nikhil Gandhi Strategic Energy Technologies, Inc. 17 Willis Holden Drive Acton, MA KEMA John Stoops, PhD 155 Grand Avenue, Suite 500 Oakland, CA (978) (510) The Cadmus Group Allen Lee, PhD 720 SW Washington, Suite 400 Portland, OR (503) The Cadmus Group Dan Groshans 720 SW Washington, Suite 400 Portland, OR (503) KEMA and The Cadmus Group 1

6 1. List of Acronyms Acronyms A/C (AC) ACCA ACM ACP ADM AEC AERS AHP ARI ASHRAE BEA Bldg C&I C&S CASE CATI CBEE CEC CEP CFL CF1-R CfR CG CHEERS CIEE CMFNH CMMHP CPUC CRCA CTZ CV CZ DEER DfC DHW DRET DSA ECM ED EE EEGA EM&V EER EUL FLA GWh Air Conditioning Air Conditioning Contractors of America Alternative Calculation Method Air Care Plus ADM Associates Architectural Energy Cooperation Automated Energy Review for Schools Analytic Hierarchy Process Air Conditioning and Refrigeration Institute American Society of Heating, Refrigerating and Air-Conditioning Engineers Building Efficiency Analysis Building Commercial Codes & Standards Codes and Standards Enhancement Initiative Computer Assisted Telephone Interviewing California Board of Energy Efficiency California Energy Commission Compliance Enhancement Program Compact Fluorescent Lamp Title 24 Residential Compliance Form Composite for Remainder Contract Group California Home Energy Efficiency Rating Services California Institute for Energy Efficiency California Multifamily New Homes Program Comprehensive Manufactured-Mobile Home Program California Public Utilities Commission Computerized Refrigerant Charge & Airflow Climate Thermal Zone Coefficient of Variation Climate Zone Database for Energy Efficiency Resources Designed for Comfort Domestic Hot Water Demand Response Emerging Technologies Division of the State Architect Energy Conservation Measure Energy Division Energy Efficiency Energy Efficiency Groupware Application Evaluation, Measurement, and Verification Energy Efficiency Rating Economic Useful Life Full Load Amps Gigawatt Hours KEMA and The Cadmus Group 2

7 Acronyms HERS HIM HMG HUD HVAC ICF IDEEA InDEE IOU IPMVP ISSM ITD kbtu kw kwh LADWP LBNL LEED LGP LPD M&V MECT MF MHRA Mil MS Mtherms n NAC NC NCCS NOMAD NOSAD NP NRNC NTG NTGR NTP P PG&E PIER PTAC PY Q2 Q3 Q4 QA QC Home Energy Rating System High Impact Measure Heschong-Mahone Group Housing & Urban Development Heating, Ventilation & Air Conditioning ICF International Innovative Designs for Energy Efficiency Applications Innovative Design for Energy Efficiency Investor Owned Utility International Performance Measurement and Verification Protocol Integrated Standards Savings Model Installed To Date Thousand Btu Kilowatt Kilowatt Hour Los Angeles Department of Water & Power Lawrence Berkeley National Laboratory Leadership in Energy and Environmental Design Local Government Programs Lighting Power Density Measurement & Verification Master Evaluation Contractor Team Multifamily Manufactured Housing Research Alliance Million Microsoft Million therms; also MTherms Sample Size Normalized Annual Consumption New Construction New Construction/Codes & Standards Naturally Occurring Market Adoption Normally Occurring Standards Adoption Non Participant Non Residential New Construction Net to Gross Net to Gross Ratio Notice to Proceed Participant Pacific Gas & Electric Public Interest Energy Research Packaged Terminal Air Conditioner Project Year Second Quarter Third Quarter Fourth Quarter Quality Assurance Quality Control KEMA and The Cadmus Group 3

8 Acronyms QII RCA Res RFP RH RLA RMSE RNC ROB RP SAS SBD SCE SCG SCP SDG&E SDGE SEER SES SF sf SFA SHGC SoCalGas SOW sqft T20 T24 TBD TDV TXV UES VFD VSD VSP W/SF WH Quality Insulation Installation Refrigerant Charge and Airflow Residential Request for Proposal Relative Humidity Rated Load Amps Root Mean Square Error Residential New Construction Replace on Burnout Relative Precision Statistical Analysis Software Savings By Design Southern California Edison Southern California Gas Sustainable Communities Program San Diego Gas & Electric San Diego Gas & Electric Seasonal Energy Efficiency Rating Savings Estimate Spreadsheet Single Family Square Foot Single Family Attached Solar Heat Gain Coefficient Southern California Gas Statement of Work Square Foot Title 20 Appliance Efficiency Standards Title 24 Building Energy Efficiency Standards To Be Determined Time-Dependent Valuation Thermostatic Expansion Valve Unit Energy Savings Variable Frequency Drive Variable Speed Drive Verification Service Providers Watts per square foot Water Heater KEMA and The Cadmus Group 4

9 2. Executive Summary This evaluation plan describes the activities that the evaluation team will undertake to verify the savings projections of the statewide Codes and Standards (C&S) program during program years 2010, 2011, and This program is implemented jointly by the four California investorowned utilities (IOUs): Pacific Gas and Electric (PG&E), Southern California Edison (SCE), Southern California Gas (SCG), and San Diego Gas and Electric (SDG&E). This evaluation plan mirrors the C&S program as implemented by the IOUs. Since the IOUs implement the program through four subprograms, this evaluation plan focuses on those same four components: 1. Building Codes, Title Appliance Standards, Title 20 and Federal Standards 3. Compliance Enhancement Program (CEP) 4. Reach Codes As expected, the IOU record of program projections shows that nearly all Program savings are produced by Title 24 building codes, Title 20 appliance standards, and Federal appliance standards. Although the Reach Codes (including Local Government Programs, or LGP) subprogram is expected to produce some savings, the current forecast is for this to be less than 0.5% of the overall C&S total. The IOUs have classified CEP as a non resource program for the evaluation period. In this plan, evaluation activities and resources are focused on the C&S that are expected to produce the largest energy savings. We have also taken into account the relative uncertainty associated with specific parameters although this is difficult to quantify. In particular, we have identified compliance with Title 24 non-residential building codes as an area where there are both large projected savings and significant uncertainties that can be resolved only through extensive field data collection. This evaluation plan is based on a total budget of $3.0 million. This was determined by ED and was informed by the evaluators analysis of the C&S within the scope of this effort. We estimated the resources required to increase coverage of recently adopted C&S and also to improve the confidence and precision of the results. The plan recognizes that the prior evaluation (of program impacts) faced a number of challenges involving data collection to assess code compliance for non-residential buildings. These challenges included the absence of a defined baseline, limited access to building permit records, and limited access to buildings. Due to this experience, there are uncertainties about what can be accomplished in the planned effort to determine non-residential code compliance. The evaluation plan has been designed to take into account the lessons learned from this past experience. In addition, the contractor will hold most of the budget for evaluating non-residential building compliance to Title 24 codes in a contingency fund. After completion of compliance analysis on approximately 20 non-residential buildings, the Contractor team will check in with the ED project manager with a report on what has been learned and a recommendation for KEMA and The Cadmus Group 5

10 proceeding. At that time, the ED project manager will determine how to re-direct or release some or all of the contingency funds. This rest of the evaluation plan document is organized into these five chapters: 3. Background on C&S Programs 4. Overview of Evaluation Approach 5. Evaluation Coordination Efforts 6. Evaluation Timeline and Work Plan 7. Evaluation Budget Chapter 3 provides background on the C&S program including a program description, a summary of projected energy savings, the evaluation protocol with modifications, and relevant CPUC policy decisions. Chapter 4 describes the core evaluation activities, beginning with our objectives and planning assumptions. We then include a survey of the researchable questions to be addressed by the evaluation and our approach to each. Finally, the methodologies to be used to evaluate each factor defined in the C&S protocol are discussed along with the specific plans for evaluating some or all of the C&S that fall within the project scope. Chapter 5 details our efforts to coordinate with other impact evaluations. Our goal here is to identify data that are needed by this evaluation and any other study. In such cases, our goal is to avoid redundancy by collecting these data only once in such a way that it is useful to both projects. This chapter also describes our general approach to data collection and data requests we have made to the IOUs. Chapter 6 includes the project timeline and Chapter 7 provides the project budget. Planned Updates to the Project Management Team This evaluation plan including the task descriptions, timeline, and budget reflect the level of detail included in the project planning to date. Additional detail that describes the activities, deliverables, and timeline has been added to the following sections in this update: Section Building Codes (Title 24) Title 24 Potential Standards Energy Savings Title 24 Gross Standards Energy Savings Title 24 Net Standards Energy Savings Title 24 Attribution Section Appliance Standards (Title 20 and Federal) Title 20 & Federal Potential Energy Savings Title 20 & Federal Gross Standards Energy Savings Title 20 & Federal Net Standards Energy Savings Title 20 & Federal Attribution KEMA and The Cadmus Group 6

11 Section Reach Codes including Local Government Programs Section Compliance Enhancement Program To address uncertainties, to incorporate additional planning, and to provide transparency to the project management team (and other stakeholders) in an orderly way, the contractors (KEMA and Cadmus) will provide periodic updates to the project management team. We plan to deliver comprehensive updates for each task at roughly three month intervals. These updates will be integrated and a summary provided to the project management team. Each summary will include the status of each task relative to its timeline, deliverables, and budget. KEMA and The Cadmus Group 7

12 3. Background on C&S Programs 3.1 Description of California Statewide Codes and Standards Program Efficiency standards set minimum efficiency levels that new appliances and buildings must meet or exceed. Because they eliminate low-efficiency products from the market (at least in theory), standards are an important component of reducing energy consumption. In the 1970s, states began establishing regulatory frameworks for developing, adopting, and implementing efficiency standards. In California, the California Energy Commission (CEC) was created, with a regulatory role to adopt building and appliance efficiency standards. The California building standards are referred to as Title 24 standards and the appliance standards are referred to as Title 20 standards based on their respective locations in the California Administrative Code. Both the federal government and individual states have continued developing and upgrading their efficiency standards over the past 40 years. Starting in the late 1990s, California utilities began having a significant role in researching, proposing, and promoting efficiency standards through what has become the statewide utility C&S program. Like other statewide programs, each IOU has a C&S program. These individual programs provide a place within each utility for funding the program activities and claiming savings in the IOU portfolio. The individual IOU programs that will be addressed by the evaluation of program years 2010, 2011, and 2012 are shown in Table 1. Table 1. Utility Program Components of the Statewide Program Utility PG&E SCE SCG SDG&E Program Name PGE2107 SCESW00008 SCG SW Codes & Standards Program SDG&E SW Codes and Standards Program Objective and Mission of Codes and Standards Program The mission of the Codes and Standards (C&S) program to is save energy on behalf of ratepayers by directly influencing standards and code-setting bodies to strengthen energy efficiency regulations, by improving compliance with existing codes and standards, and working with local governments to develop ordinances that exceed statewide minimum requirements. ED approved Statewide program performance metrics (PPMs) for the Statewide program are shown in Table 2.. KEMA and The Cadmus Group 8

13 Table 2. Statewide Codes and Standards Program PPMs C&S Subprogram Buildings Standards Advocacy Appliance Standard Advocacy Compliance Enhancement Reach Codes PPM Number of Residential and Commercial CASE studies, as defined in Building Standards Objectives 1 & 2 for which adoption by the CEC is anticipated by the IOUs, targeting efficient technologies practices and design in each of the following areas: lighting; HVAC; envelope; water heating; and cross-cutting measures in support of the following: (a) Integrated Design, including data management and automated diagnostic systems, with emphasis on HVAC aspects of Whole Building, (b) ZNE technologies, practices, and design in Residential Sector, (c) Peak efficient technologies including plug loads and HVAC technologies, (d) Advanced Lighting Technologies Number of draft CASE Studies, as defined in Appliance Standards Objective 1, developed as mutually agreed upon by the CEC and IOUs in support of plug loads, refrigeration, advanced lighting, and/or other technologies that are adopted by the CEC, within authorized budget. Number of role-based, Title 24, training sessions delivered. Number of jurisdictions in IOU Service territories with CEC approved Reach Codes in residential and/or commercial sectors as a result of the RC sub-program activities. The program has a strategic design to overcome various barriers. The design is focused around barriers in each of the four statewide subgroups of Codes and Standards. Table 3 features a summary for overcoming barriers that was adopted by all program utilities. Table 3. C&S Program Strategy to Overcome Barriers from PPMs Subprogram Building Codes Appliance Standards Compliance Enhancement (CE) Reach Codes (RC) Strategy for Overcoming Barriers Codes and Standards advocacy; Developing baselines for building advocacy; Improving rate-of-compliance with Title 24 and Title 20 Codes and Standards advocacy; Developing baselines for appliance advocacy Promote energy efficiency to local governments; Provide T-20 and T-24 outreach and support; Coordinate and Support Enhancement Efforts with the CEC and other Organizations Develop and/or support the development of reach codes, or locally adopted ordinances, that exceed statewide minimum requirements Table 4 shows the Codes and Standards Core approved program budget for with the totals for each utility. KEMA and The Cadmus Group 9

14 Utility Table 4. C&S Program Budgets for Administrative Cost Direct Implementation Total Budget PGE $2,392,963 $17,249,322 $19,642,285 SCE $1,014,866 $5,751,852 $6,766,717 SCG $233,672 $1,568,863 $1,802,535 SDGE $348,840 $1,855,451 $2,204, Program Implementation One of the major program activities is funding third party research into specific code changes. The resulting research reports have generally been named Codes and Standards Enhancement Initiatives and are commonly referred to as CASE reports. The CEC process for proposing and approving changes to the California appliance and building standards has come to rely on these CASE reports to a large extent. Savings from the C&S program are difficult to estimate and attribute to utility actions. Still, utilities and the California Public Utilities Commission (CPUC) are highly motivated to overcome the challenges to recognize savings from the C&S program as another asset in the demand-side management (DSM) portfolio. Over time, the IOUs have developed the C&S program as an integral building block of the portfolio of statewide DSM programs. Since codes and standards are one significant way to procedurally establish market transformation, the IOUs view the C&S program as one link in the process ranging from research and emerging technology programs, to incentive and information programs, to adoption of specific requirements into codes and standards. The CPUC and IOUs have worked to establish rigorous procedures for attributing credit to the C&S program. To this end, efforts have been made to estimate savings from the C&S program and establish a regulatory process to account for them. In 2006, California adopted an evaluation protocol that included a methodology for evaluating C&S programs [CPUC 2006]. That methodology was fully implemented for the first time during the evaluation cycle. The protocol and the prior evaluation have now established a basis for evaluating and verifying savings from the program to be used in the regulatory process. A notable part of the prior evaluation was the development of the Integrated Standards Savings Model (ISSM), a flexible Microsoft Excel - based model for calculating energy and demand savings that may be credited to the California IOUs. As described later, the CPUC has established rules for how C&S program savings can be counted toward achievement of the Minimum Performance Standard (MPS), the CPUC s savings goal threshold. In this cycle the IOUs, for the first time, also will be able to count verified savings from the C&S program in the calculation of the Performance Earnings Basis (PEB) and in the cost-effectiveness of the portfolio. 1 The PEB may determine the amount of shareholder incentives that the utility will be eligible to collect from electric distribution or gas transportation rates through the Risk/Reward Incentive Mechanism (RRIM). Because the claimed savings from 1 pp See also Attachment 10 to D KEMA and The Cadmus Group 10

15 the C&S program are likely to be very large compared to the rest of the portfolio, the rigor and thoroughness of the C&S evaluation takes on special importance. 3.2 Projected Energy Savings During Planning for this program evaluation and securing the appropriate resources to fund it requires some estimate of the projected savings to allow decision makers to compare this program to others in the IOUs portfolios. To obtain an estimate of the projected savings from the C&S program, we submitted a data request as described in Section 5.3. Although the IOUs provided two separate and distinct savings models, following a request from the CPUC ED they agreed that the evaluation planning could be based on a single savings model delivered as EEGA 1465 et al_attachment 2_Total CS Program Savings.xls and referred to as the PG&E model. As noted, these events are described in more detail below. The model was revised in the areas shown in Table 5. Among these changes were the removal of interactive effects from the PG&E model 2 (for consistency and to make gas savings visible), the valuation of federal advocacy based on savings that occur within the state of California only (described below as the CPUC ED method), and the removal of the 5% compliance improvement assumed for many of the pre-2006 standards. Table 5. Summary of Changes Made to Produce Evaluation Plan Scenario Scenario IOU Model Original (from Data Response) Evaluation Plan Scenario Interactive Effects PG&E PG&E Yes SCE SCG SDG&E PG&E SCE SCG SDG&E SCE / Sempra PG&E No No Federal Advocacy Per IOU Submission Valued per CPUC ED method Comment Model includes interactive effects as negative gas impact for some electric measures. SCE / Sempra Model is same spreadsheet tool as PG&E but no interactive values were included by these IOUs. - Includes verified savings for Pre 2006 C&S - Include net savings for Pre 2006 standards - No improvement in compliance included Note: Lists of the specific C&S to be evaluated are included in the following sections. Appendix A: Title 24 Building Codes to be evaluated Appendix B: Title 20 and Federal Appliance Standards to be evaluated These revisions produced the projected savings shown in Table 6 that have been used to develop the evaluation plan. These estimates use the IOU-provided values for unit energy savings (UES) and Expected Useful Life (EUL). As described in sections and below, the evaluators will compare these values to ED-approved values (when available) as documented in DEER, to values from other ED-approved documentation, or to verified values based on recent research. 2 Interactive effects count in the calculation of PEB consistent with CPUC policy. The impact of these effects will be evaluated and will be added back to the final impact totals and CE calculations. KEMA and The Cadmus Group 11

16 Generally the IOUs projected savings from standards that went into effect prior to January 2006 ( Pre Jan 2006 ) increased as these are now projected by applying the PG&E valuation approach, based on the results of the prior evaluation, to all the IOUs. Also savings from later standards ( Post Jan 2006 ) decreased as savings from federal standards were reduced as a result of applying the ED methodology. Gas savings increased as interactive effects were removed to report savings consistently across the IOUs. Interactive effects (based on ED-approved values) will be included for all IOUs in the final evaluated savings. Table 6. Comparison of Original and Revised First Year Projected Savings C&S Original Revised Estimate Category Total GWh Total MW Total MTherms Category Total GWh Total MW Total MTherms Pre Jan , Pre Jan , Post Jan , Post Jan , Total Gross 3, Total Gross 5, Total Net 1, Total Net 2, Net as Proportion of Gross 32% 28% 0% Net as Proportion of Gross 43% 41% 54% Total Portfolio Total Portfolio Gross excl C&S 11,719 2, Gross excl C&S 11,719 2, Gross incl C&S 15,355 2, Gross incl C&S 16,830 3, C&S % Total 24% 23% 8% C&S % Total 30% 29% 30% Net excl C&S 7,852 1, Net excl C&S 7,852 1, Net incl C&S 9,002 1, Net incl C&S 10,029 1, C&S % Total 13% 11% 0% C&S % Total 22% 20% 25% Additional details on the original IOU savings data and the revised IOU Savings Model with more detail on the projected savings is provided in the Excel spreadsheet inserted here. IOU Savings Model Sept C&S Evaluation Protocol with Recommended Revisions The overall purpose of the C&S impact evaluation is to determine the energy and demand savings that can be attributed to the IOUs C&S program activities. Savings continue to accrue for C&S evaluated in the last cycle as new buildings and appliances enter the market each year. KEMA and The Cadmus Group 12

17 Since then, new codes and standards have been adopted and gone into effect that were influenced by C&S program advocacy occurring since the beginning of 2006 and before The energy and demand savings that we will estimate during 2010 through 2012 will be counted towards the energy savings goals (MPS). As noted earlier, this evaluation will, for the first time, produce data that could be used in the PEB process. The specifics are guided by CPUC policies and decisions summarized in the next section. In addition to the basic evaluation of program advocacy activities, the evaluation will also address two new program components Compliance Enhancement and Reach Codes as discussed at the end of this plan. In the market, energy-efficiency levels are distributed over a range of values: various items lie above and below the average efficiency value. Codes and standards for new buildings and appliances make it illegal to construct buildings and market items produced after the effective dates that are below a specified efficiency level. In effect, these standards alter the distribution of market commodities to eliminate those that use more than a certain amount of energy. This outcome not only reduces the maximum energy consumption allowed, it should also reduce the average consumption due to the altered distribution of efficiency levels. This program differs from other energy-efficiency programs in several ways that affect how it is evaluated, including that there are no customers who actively sign up to be participants. For most resource acquisition programs, there are participants individuals or businesses that receive incentives for purchasing and/or installing energy-efficient measures. By tracking the participants, the IOUs are able to monitor program projections continuously and to produce reports at regular intervals. When an energy code or standard is changed, all products in the market are to meet or exceed the new efficiency requirements so, ideally, individuals and/or businesses can purchase only complying products. These purchasers are equivalent to participants in conventional efficiency programs, but their participation is passive and the IOUs do not have mechanisms in place for tracking them. Consequently, one of the challenges of the evaluation is estimating the number of buildings constructed and products sold under the new C&S. In addition, it is essential to determine whether these new buildings and products actually meet the efficiency requirements and this must be analyzed through compliance studies. Several factors complicate these results and the assessment of energy savings from C&S: For the highest level of accuracy, savings should be estimated based on pre and post averages; however, sufficiently detailed efficiency data are rarely available to use in accurately estimating average consumption. A new standard may cause complicated impacts to the market and may alter the shape of the distribution, instead of just eliminating a portion of the distribution. Not all new products and buildings will comply with the relevant standards. Building standards pose special complexities because they permit compliance using performance or tradeoff approaches. 3 As of the date of this document, all standards for which the IOUs have documented savings were adopted before For planning purposes, we are regarding this as the complete set to be evaluated. KEMA and The Cadmus Group 13

18 Some efficiency improvements would likely occur even without adoption of the standards. Other market actors positively affect the adoption of standards, so all of the resulting savings cannot be attributed to the IOUs program efforts. For the prior C&S program evaluation, the evaluation started with the California Evaluation Protocols and the evaluation team then modified them, as appropriate, during the evaluation process. The final C&S evaluation process used in the cycle is shown in Figure 1. Figure 1. C&S Advocacy Program Evaluation Protocol Market Baseline Naturally Occurring Market Adoption (NOMAD) Attribution to C&S Program Allocation Energy Use Baseline Potential Standards Energy Savings Gross Standards Energy Savings Net Standards Energy Savings Net C&S Program Savings Savings by Utility Unit Energy Savings Compliance Rate The potential energy savings attributable to the C&S are based on the estimated unit energy savings and the number of those units (measures or appliances) entering the market each year. These potential savings are adjusted by the compliance rate to derive gross energy savings. Net savings result from adjusting the gross savings by the naturally occurring market adoption of measures or appliances (NOMAD) that would have occurred in absence of the code or standard. The resulting savings that are credited to the C&S program are determined by calculating the attribution adjustment and allocating the net program savings to individual IOUs. The analysis is implemented using an Integrated Standards Savings Model (ISSM) developed by the evaluators specifically for the prior C&S evaluation and to be modified for this evaluation that incorporates all the input data from the EM&V activities. Appendix C provides a summary of CPUC Policy Decisions relevant to the treatment of C&S. KEMA and The Cadmus Group 14

19 4. Overview of Evaluation Approach As expected, the IOU s projected energy savings show that nearly all program savings are produced by the Title 24 building codes, the Title 20 appliance standards, and the Federal standards that will be studied according to the plan described in this section. The Reach Codes subprogram is expected to produce some savings but the current forecast is for this to be less than 0.5% of the overall C&S total. The IOUs have classified the Compliance Enhancement Program (CEP) as a non-resource program for the evaluation period. Accordingly, nearly all evaluation activity is focused on the C&S that are expected to produce the bulk of the savings. Some effort is dedicated to research on the impact of Reach Codes and also on the process used to implement the CEP. This effort is consistent with ED and other authorities direction. Evaluation of each building code or appliance standard follows the protocol shown in Figure 1 above. As shown in the figure and also described in the section, C&S Evaluation Protocol with Recommended Revisions, the evaluators must estimate a value for each factor shown as a block on the diagram. Once values have been found for each factor, the ISSM model will be used to integrate the data and calculate values for each of the stages of the protocol. Descriptions of the major tasks to be completed in the evaluation are provided in the sections below. These descriptions have been kept to a brief, high level to keep the plan to an appropriate overall length. Anyone seeking more detailed information about the C&S evaluation tasks and process is referred to the evaluation report 4 since most of these tasks were also conducted for the prior evaluation. Note on scope and organization: As may be apparent from earlier parts of this plan, savings are usually organized by category (building / appliance), advocacy, and effective date. For example, one group of C&S resulted from pre January 2006 advocacy. Of this group, most were effective prior to All of these were included in the PY evaluation. For the codes in this group that were effective between 2006 and 2008, some were included in the prior evaluation and some were specifically excluded. All the rest of the California C&S were the result of post January 2006 advocacy. In this evaluation, all these codes that became effective prior to the end of 2012 will be studied, though the level of analysis may vary by code or standard.. As noted above, a list of Title 24 Building Codes being considered for evaluation is included in Appendix A and a list of Title 20 and Federal Appliance Standards being considered for evaluation is included in Appendix B. 4.1 Evaluation Objectives As noted above, the primary objective of this evaluation is to verify the savings projections of the statewide Codes and Standards (C&S) program during program years 2010, 2011, and This chapter describes the approach, assumptions, and discussion of a number of researchable questions. The next chapter describes the methods and specific steps to be followed to evaluate 4 Codes & Standards (C&S) Programs Impact Evaluation California Investor Owned Utilities Codes and Standards Program Evaluation for Program Years , 4/9/2010. This report can be found on the CALMAC database ( by searching on code 'CPU ' KEMA and The Cadmus Group 15

20 each of the four subprograms listed at the beginning of Chapter 2. In this plan, we have prioritized activities based on the magnitude of savings associated with each code and also on the uncertainty of the values used to compute savings. In short, we are focusing activities with large savings and/or significant uncertainty. A secondary objective of this evaluation is to provide specific feedback on the evaluation protocol itself and to make recommendations for its further refinement. Resources for this purpose are not specifically identified but we expect that the evaluators and the ED will collect and document these items in the course of the work. 4.2 Planning Assumptions This evaluation plan was developed using the assumptions summarized in Table 7. Table 7. Assumptions Assumptions Specific C&S to be considered will depend on IOU documentation of savings achieved and the potential to address uncertainties. The evaluation will attempt to develop more accurate counts of multifamily buildings and include low and high rise multifamily buildings in the compliance analysis (no low-rise and only three high-rise buildings were included in the prior compliance study) The evaluation will assess the feasibility of determining savings from remodel/renovation projects (which was evaluated on only in a very limited scale in PY) The evaluation will study federal standards savings that resulted in part from California IOU advocacy. The evaluation will provide support for the cost-effectiveness analysis. Additional Comments It is necessary to prioritize standards to optimize the use of limited evaluation resources. The evaluation of savings from multifamily buildings in the PY evaluation was limited due to incomplete reported data on multifamily buildings. Improving the accuracy of the counts and conducting compliance assessments will significantly improve the accuracy of savings estimates for multifamily residential buildings. Savings projections resulting from remodel/renovation projects reflect a larger portion of the Title 24 total in PY than it did in the period covered in the PY evaluation. Evaluators will determine the appropriate method and values for calculating attribution. It is necessary to determine the cost effectiveness of codes and standards to allow comparison of the program and associated savings to other energy efficiency programs. 4.3 Researchable Questions Each of the research topics described in this chapter addresses a critical need for the current evaluation. Each topic also furthers the development of evaluation methods for the C&S program Revisions to Integrated Standards and Savings Model The model revision is needed to support analysis of several of the researchable questions described in the sections to follow. The ISSM is a flexible Microsoft Excel -based model for calculating energy and demand savings that may be credited to the California IOUs for their efforts promoting the adoption of energy-efficiency C&S. The model has a dynamic user interface to allow standards to be analyzed by group. It also contains a built-in Monte Carlo simulation tool to allow for KEMA and The Cadmus Group 16

21 uncertainty analysis on forecasted inputs, such as per unit savings, attribution scores, future compliance, and naturally occurring market adoption. The evaluation team developed ISSM for the evaluation. Using this as a starting point, the IOU s C&S team developed a version that estimates savings based on more recent California Title 20 and Title 24, federal appliance standards, and other user inputs. As noted earlier, the IOUs provided two of these modified ISSM models in response to the ED data request. For the current evaluation, the evaluators will review the IOU s most recent version of the model to catalog and understand the changes. We will discuss new model requirements with the project stakeholders and integrate agreed-upon features into a new ISSM model. The new model requirements are expected to include: Addition of all post-2006 C&S; Support for modeling of interactive effects such as increased gas consumption for heating due to reductions in electric lighting energy; Application of independent savings calculations for federal standards; and Support of residential and nonresidential whole building efficiency modeling for a subset of newly constructed or remodeled buildings These enhancements will supplement the existing model inputs, which can be separated into static inputs, time dependent inputs, NOMAD inputs, and attribution inputs. Static inputs do not vary by year, only by standard. These include: C&S implementation month/year Effect of utility programs 5 Effective measure life First year potential energy savings per unit First year potential demand savings per unit First year potential gas savings per unit Time dependent inputs vary by year and include: Annual compliance rate Volume of units in marketplace To forecast naturally occurring market adoption, the model requires: NOMAD start year 5 For some standards, utility programs encouraged use of energy efficient measures prior to the adoption of the regulation. In such cases, the observed market is larger than the true natural market. Adjusting the observed market for the (measure volume) effects of IOU programs produces a more accurate estimate of the true natural market. KEMA and The Cadmus Group 17

22 Maximum achievable penetration Leading behavior coefficient Following behavior coefficient Finally, the model incorporates the attribution score inputs for each standard. The attribution score is a combination of the weights assigned to three factors for each standard plus a measure of the credit the C&S program receives for its contribution toward adoption of the standard. Based on these inputs, the model calculates the net C&S program electric energy and demand and gas savings for a single standard or a group of standards (depending on the user selection). First, the model calculates the total potential savings of a standard by multiplying the volume of new units in the marketplace in each year by the per unit potential energy savings. After this, the model makes a series of adjustments to account for non-compliance, naturally occurring market adoption, the effect of utility programs on market adoption, measure life, and the attribution credit for the IOU C&S program. A basic savings analysis can be conducted using expected values for each input. However, there is some degree of uncertainty in the value for most inputs. To account for this, the model has a built-in Monte Carlo simulator to allow for conducting sensitivity analysis on the program savings. Probability distributions can be input into the model for each standard s per unit savings, volume of units in the marketplace, attribution score, annual compliance rate, and NOMAD parameters (including the maximum penetration as well as the shape of the Bass curve). The user decides which of the input variables to model as being uncertain. The model outputs are in the form of tables and graphs that show confidence intervals and probability distributions for savings. KEMA and The Cadmus Group 18

23 Table 8 summarizes the inputs and outputs for each stage defined by the C&S evaluation protocol. The inputs/parameters are factors to be determined or verified by the evaluators and are a concise summary of what is required to determine C&S energy and demand impacts. KEMA and The Cadmus Group 19

24 Table 8: ISSM Inputs and Outputs Protocol Stage Inputs / Parameters Outputs Unit energy savings (kwh, kw, Therms) Potential standards energy savings (kwh, kw, therms) Annual installations (volume of units in marketplace) Potential Energy Interactive effect (kwh, kw, therms) Start date (month, year when standard takes effect) Compliance NOMAD Attribution Allocation Measure life Potential standards energy savings (kwh, kw, therms) Annual compliance rate Gross standards energy savings (kwh, kw, therms) NOMAD start year (product introduction) Maximum market penetration P value (for leading behavior) Q value (for lagging behavior) Alternative: linear market adoption duration Utility program effects (units or percentage) Net standards energy savings (kwh, kw, therms) Weighted attribution score Net program energy savings (kwh, kw, therms) IOU share of CA market (electricity, gas) Gross standards energy savings (kwh, kw, therms) Net standards energy savings (kwh, kw, therms) Net program energy savings (kwh, kw, therms) Net IOU energy savings (kwh, kw, therms) Treatment of Whole Building Performance The researchable question here is the impact on energy consumption of the whole building performance path (non-residential) to compliance with the energy code. Energy savings from new codes are determined relative to the baseline, i.e., what would have been built if the new code had not been adopted. If baseline construction practices are known, then the expected savings from the new code are the difference between the energy use of a building built to the new code and the same building if it were built using baseline practices. If the baseline practices have not been determined (through comprehensive field data collection), then the baseline can be assumed to be the initial building code. The prior evaluation introduced the approach of determining Title 24 energy savings by assessing whole-building energy consumption. This was motivated by the fact that Title 24 permits compliance to be demonstrated, not only by installing a set of prescriptive measures, but also based on whole-building performance through which measures can be traded off as long as the energy consumption is no more than a building that meets all the prescriptive requirements. This approach avoids the problem of trying to determine Title 24 energy savings accurately when a measure that was adopted in the code is not incorporated in a building, but another measure that provides equivalent savings is. KEMA and The Cadmus Group 20

25 In the program evaluation, the evaluators used this whole-building performance approach for single-family residential buildings. This was feasible because a software tool to conduct the necessary analysis was available and a comprehensive baseline study of residential buildings was conducted. Since neither the comparable software nor data were available for nonresidential buildings, the Title 24 nonresidential evaluation did not include a whole-building performance analysis for nonresidential buildings. To evaluate energy savings at the whole building level, it is necessary to have a process for comparing the energy consumption of buildings as built with their energy use if they were built to the new code, as well as built to baseline conditions. The standard compliance software for Title 24 is designed to compare an as-built building to the same building if built to just meet the code. For the program evaluation, a version of Micropas was created that also allowed comparing the as-built building to the same building if built to meet the prior 2001 Title 24. The results from these three runs as-built, 2005 Title 24, and 2001 Title 24 permitted determination of compliance with the 2005 Title 24 as well as estimation of the savings margin relative to the expected savings in going from the 2001 to 2005 version of Title 24. For this evaluation, we will subcontract with a firm to develop a version of EnergyPro that can perform these three analyses (as-built, designed to 2005 Title 24, and designed to 2008 Title 24) for nonresidential buildings. This tool and the characteristics data collected through nonresidential building site visits will permit an assessment of whole building compliance for the nonresidential Title 24. The use of the software (described in the next subsection) will require the collection of characteristics data for a sufficiently large sample of nonresidential buildings. The latter effort is described in the description of the Title 24 nonresidential code compliance analysis Time Dependent Valuation Effects on Savings This work is being done to answer questions that remain regarding the impact on energy and demand savings of TDV methods currently in use in California. TDV is a method of valuing energy savings in terms of the real (economic) resources used in energy production. This calculation procedure is embedded in software used to determine the compliance of residential and nonresidential buildings with Title 24. The IOUs claim that this calculation method leads to the installation of building measures that reduce peak demand and energy consumption. Although the IOUs and the CEC have made significant commitments to incorporating Time Dependent Valuation (TDV) in Title 24, the evaluators were unable during the previous evaluation to confirm impacts associated with TDV. Specific findings from that report 6 include the following: For residential construction, higher efficiency air conditioners were becoming fairly common but there was no evidence that this was because of the TDV requirement. 6 CPUC New Construction/Codes & Standards Codes and Standards Program Evaluation, Appendix J, KEMA, Cadmus, et al KEMA and The Cadmus Group 21

26 For the nonresidential sector, no documentation was provided that described how the savings were estimated for nonresidential buildings. The most significant concern was that the way the TDV methodology is implemented it does not appear to require significant peak savings for compliance, even though the underlying methodology provides more credit for on-peak energy savings. Given these findings, the evaluation team could not find sufficient evidence that the approach used to estimate savings was consistent with the impacts of the TDV requirements. Consequently, we did not have a sufficient basis for crediting the TDV standards with energy or demand savings. The evaluation will review the IOUs savings claims from residential and nonresidential TDV for the upcoming evaluation cycles. To evaluate the TDV energy and demand savings projections, the evaluators will use the following approach: 1. Request documentations from IOUs about their savings claims, including the theory, assumptions, and calculations; 2. Study the hypothesized links between TDV and energy and demand savings, and develop a method for verifying the savings claims; 3. Collect the data required to verify the savings claims; 4. Perform the verification analysis and estimate TDV potential and gross savings, if any; and 5. Perform additional evaluation steps (NOMAD, attribution, etc.) to estimate the net TDV standard savings from residential and nonresidential utility programs Multifamily Building Data This work will answer questions about the energy impact of codes and standards on multifamily buildings. The prior evaluation did not include certain types of multifamily residential buildings and building renovations to focus resources on larger, verifiable savings in other segments, such as new construction. With the new construction industries operating in a depressed market in the current years, these multifamily and renovation segments have relatively grown. It is essential to understand whether it is possible to evaluate these categories of construction, and if not, to identify barriers that prevent evaluation. Multifamily buildings have unique evaluation issues for two reasons: (1) buildings that have three stories or less are covered by the residential version of Title 24, whereas buildings that have four stories or more are covered by the nonresidential version of Title 24; and (2) the sources of data on building construction start and completion dates typically distinguish multifamily buildings by their number of units, not by their number of stories. For the previous C&S program evaluation, the information IOUs used to estimate savings did not accurately quantify savings from multifamily buildings as covered by the two versions of Title 24. The evaluation attempted to differentiate the savings affected by the residential and nonresidential Title 24, but was limited by the lack of reliable estimates on the number of units in buildings, number of stories, and by the lack of compliance data for multifamily buildings. KEMA and The Cadmus Group 22

27 Available construction data indicate that the proportion of new residential construction that is multifamily buildings has increased significantly in the last five years. This trend increases the importance of having better information and savings estimates on multifamily buildings for the C&S evaluation. Key factors include the number of units covered by the residential and nonresidential versions of Title 24, and determination of compliance levels. Our plan to evaluate multifamily building savings includes the following steps: 1. Review the approach used in the prior evaluation. 2. Review multifamily savings estimates used in IOU analysis and savings claims. 3. Review additional California studies on multifamily buildings. 4. Develop and apply an improved method for estimating the quantities of multifamily buildings covered by residential and nonresidential versions of Title Develop an approach for assessing multifamily building compliance. 6. Document the approach and apply it during the Title 24 compliance analysis Feasibility of Including Renovations in Evaluation This work will determine whether sufficient data exist to evaluate the impact of codes and standards on remodels and renovations (also referred to as additions and alterations). Much of this activity is subject to Title 24 requirements; however, information on the amount of construction covered by Title 24 that is not new buildings is limited. Methods to determine the energy impact of C&S on these types of construction are also less developed than the methods used for new buildings. The purpose of this task is to investigate whether it is feasible to evaluate the Title 24 impact on remodels and renovations of nonresidential buildings, and to develop an approach to address the issues outlined above. The analysis would include the following steps: 1. Review how renovations were examined in the previous evaluation, and review other studies that have addressed this issue. 2. Identify and review sources of data on nonresidential renovations. 3. Develop a method to estimate the quantity of nonresidential renovations and a plan for assessing Title 24 compliance. 4. Determine possible sources of information on residential retrofits/renovations that trigger the need for compliance with Title Based on step 4, determine the feasibility of collecting and analyzing the number and type of residential retrofits/renovations covered by Title 24, along with their compliance levels. 6. Propose a method to assess residential retrofits/renovations based on findings of step 5. Given the uncertainty of data for residential retrofits/renovations, the eventual level of evaluation effort for this task is difficult to determine. The proposed budget for this task is based on the best KEMA and The Cadmus Group 23

28 information currently available, but may require adjustment as more information is available during the research phase Analysis of Title 24 Composite for Remainder (CfR) What is the energy impact that can be attributed to the IOUs from C&S that were adopted without the preparation of specific CASE reports? Because it is unique and questions remain about how to address it in this evaluation, treatment of the CfR is discussed as a research topic separately from the other Title 24 measures. As noted earlier, the IOUs have defined the CfR as the Title 24 codes that were implemented without the benefit of a C&S program CASE studies. For the cycle evaluation, the evaluators applied the same steps to evaluating the CfR that they did for the other Title 24 measures. However, additional research and effort was required because: 1. There was no CASE study fully documenting the CfR. 2. The CfR consisted of many measures, some residential and some nonresidential. 3. The relationship between the C&S program and adoption of the CfR measures was indirect and not well documented. Although the last evaluation provided verified savings estimates for the CfR, the estimates were probably the least certain of all the measures for the reasons above. However, since the CfR was the single largest contributor to both gross and net electricity savings for nonresidential buildings, it was very important in the verification of the savings projections from this category.. For the CfR presents a similar situation in terms of its contribution to the nonresidential savings. It represents the largest single contributor to nonresidential gross and net estimated savings. To date, minimal information has been provided by the IOUs to support these savings estimates. For this evaluation, we will not reexamining the evaluated CfR savings for the cycle since it is out of scope for this evaluation cycle. For evaluating the cycle savings, we plan to follow the same set of steps outlined in Figure 1 that will be applied to the other C&S. They are described here, rather than as part of the Title 24 discussion in Section 4.4.1, because of the uniqueness of the CfR. The analysis will include the following steps: 1. Obtain documentations from C&S program clarifying what codes/measures are included in the CfR for this cycle. 2. Identify the codes/measures that comprise at least 30% of the savings for the CfR. 3. Contact IOUs for related documentation on program activities related to the CfR. 4. Complete all steps in the evaluation process for the selected codes/measures. 5. Apply the findings for selected codes/measures to the full set of CfR codes/measures Cost Effectiveness and PEB As noted earlier, the current CPUC policy calls for the inclusion of the net savings attributable to the C&S program in the cost effectiveness and earnings analysis process (PEB) starting in the cycle. Many of the data required by these analyses will be provided by the evaluation KEMA and The Cadmus Group 24

29 steps described earlier, but the quality of the information will be subject to the caveats associated with each level of analysis delineated in the preceding chapters. At this stage, it is not possible to identify exactly what additional tasks will be required to provide any additional information needed for the PEB analysis. We will amend this plan, as needed, when more details become available LGP Reach Codes What is the impact of Reach Codes implemented through Local Government Programs (LGP)? For the C&S Reach Codes sub-program, the IOUs provide support to the development of locally adopted ordinances that exceed statewide minimum requirements. These Reach Codes, in addition to saving energy within local jurisdictions, provide a means of testing new codes, as well as testing the efficacy of increasing the stringency of existing codes at a local level prior to disseminating the code on a statewide basis. Each jurisdiction's experience with local codes can be used to inform the state's process by documenting both the successes and barriers faced for both adoption and implementation. Many local governments have passed energy efficiency policies or goals, some of which meet the CPUC s definition of reach codes (approved and enforceable under CEC procedures) (see Section above). While the intention is good, the volume and variety of codes have resulted in an inconsistent codes structure in California. To address these problems, the IOUs have implemented a coordinated development approach under the C&S Reach Codes sub-program. It is hoped that this approach has reduced the wasted effort and cost resulting from duplicated efforts. In addition, coordinated development is expected to provide better staging for statewide adoption, to leverage local governments to encourage adoption, and to increase the likelihood of adoption and compliance. Following the steps detailed in Section 4.4.3, we will conduct a pilot evaluation study of this program to study both the IOU process overall and the program impact in a small subset of the jurisdictions that have adopted reach codes. 4.4 Evaluation Methodologies This section includes subsections that correspond to the four IOU subprograms: Building Codes (Title 24), Appliance Standards (Title 20 and Federal), Reach Codes, and Compliance Enhancement. Each subsection then includes a description of the general methods and specific steps to be followed in each stage of the evaluation protocol (as shown in Figure 1) Building Codes (Title 24) Title 24 Potential Standards Energy Savings General Methods We will review primary sources that provided the basis for evaluating the estimated energy savings impact from individual building code change proposals that were successfully adopted. Based on the experience gained in the prior evaluation, these include: 1. The Total CS Program Savings spreadsheet, May This spreadsheet contains firstyear estimated energy impacts for each measure for which program credit is claimed. KEMA and The Cadmus Group 25

30 Prepared by Heschong Mahone Group for Statewide Code Group. This file was received in response to the ED data request: EEGA 1465 et al_attachment 2_Total CS Program Savings_ PG&E.4381.xlsx 2. Volume III Codes & Standards (C&S) Programs Impact Evaluation California Investor Owned Utilities Codes and Standards Program Evaluation for Program Years CALMAC Study ID: CPU , KEMA, Inc., The Cadmus Group, Inc., Itron, Inc., and Nexus Market Research, Inc Impact Analysis, 2008 Update to the California Energy Efficiency Standards for Residential and Nonresidential Buildings, Architectural Energy Corporation, Prepared for California Energy Commission, November 7, The Impact Analysis of 2005 Update to the Building Energy Efficiency Standards, June This report is an analysis of the statewide impact of the entire Codes and Standards program, including components contributed by the California Energy Commission and its consultant, the IOUs, and other organizations that created and put forth code change proposals. Prepared by Eley Associates (Eley) and managed by the California Energy Commission. This document is referred to as the Eley report here. 5. Codes and Standards Enhancements Initiatives for Title 24. This is the final contractor s report for the Codes and Standards work commissioned by PG&E and prepared by Heschong Mahone Group. We refer to these documents as CASE reports or CASE studies. 6. Individual CASE Reports. These are the original consultant reports prepared to justify and defend the code change proposal. Links to these reports are provided at the end of the methodology description. The savings listed in the Total CS Program Savings spreadsheet were typically derived from either the Impact Analysis report (reference 3) or the Evaluation report (reference 2). Validating First-Year Unit Energy Savings To validate the estimated savings, we will assess the underlying assumptions for the first-year savings calculations and apply more current and accurate information when available. The evaluator will thoroughly review the CASE reports to identify the savings calculation method. Our team will use this information to determine whether or not an alternative method would be more appropriate. Validating Potential Energy Savings The most important factors in potential energy savings verification are measure unit energy savings and measure quantities, which rely heavily on construction estimates. These values are used to calculate the potential energy savings for the first year for each measure. Potential savings in year 1 = Energy savings per measure * number of measures Any remaining gaps in CASE reports (e.g., assumptions not stated in the report) and all additional questions will be sent to the CASE report author. After the CASE report author responds, we will interpret these results for accuracy. In parallel with checking the information in the CASE report, we will search for other sources of information on energy savings and market KEMA and The Cadmus Group 26

31 volume. Based on the information obtained from the CASE report and any other more recent reports, we will then adjust first-year and forecast potential energy savings estimates. New Building Completions Building completions for purposes of calculating potential savings are expressed in terms of the number of units for residential, which includes single family and multifamily, and square footage for non residential. Residential building starts will be based on permit data, representing the number of units from the Construction Industry Research Board (CIRB) covering the state of California. These data will be analyzed for the years 2010 through We determined lags between typical building permit date and completed construction date from previous interviews with code officials. Based on these interviews, we are initially assuming the following construction lags: Single family: six months Multifamily: eight months Nonresidential: nine months This evaluation will collect additional data on the length of time between a permit date and issuance of a certificate of occupancy. Analysis of these data will be used to modify the lag duration when the data are sufficient to support a change to the lag model. To determine nonresidential building construction in square footage requires a couple of steps. Two different industry data sources will be analyzed to derive floor area estimates: 1. CIRB: nonresidential permit data in construction dollars 2. Reed Construction Data: contract award data in construction dollars and square footage From the Reed data we will estimate average construction dollars per square foot. The average values will then be used with the CIRB permit data in construction dollars to derive nonresidential building permits in square footage by construction year as follows: Non Residential square footage = construction dollars per square foot * construction dollars Specific Steps Notes: This task is secondary research that consists of evaluating a number of primary and secondary sources of information. Parameters to be determined by this effort are shown in KEMA and The Cadmus Group 27

32 Table 8 in Section above. The potential energy savings is the technical potential for a building code, and is based on the energy savings from the code and the potential number of new buildings each year. Standards & Building Types Determine Applicability of Standards to Building Types. Review the 17 standards of 2008 T-24, effective Jan. 1, 2010 (or later). Categorize standards as applicable to residential buildings, nonresidential building or both. Description of Standards Obtain Full Descriptions of Each Standard Understand full extent of each standard. CASE Reports Review CASE and Research Reports Review CASE reports and research papers that document the derivation of unit energy savings per standard in the ISSM spreadsheet. Document any discrepancies between the analyses and the numbers used for the 17 standards. Determine unit energy savings for each standard. Annual Installations The total potential energy savings is the sum of the product of unit energy savings per standard times the number of installations of that standard. The number of installations can be found from construction activity provided by third party sources. Analyze data provided by third party sources. Determine the number of installations applicable to each standard. These tasks, the associated deliverables, expected timeline, and estimated cost are shown in KEMA and The Cadmus Group 28

33 Table 9 below. KEMA and The Cadmus Group 29

34 Table 9. Title 24 Potential Energy Savings Task Deliverable Timeline Standards & Building Types Summary of 2008 T-24 standards by building type Jan 2012 Feb 2012 Description of Standards Summary of unit energy savings per standard Jan 2012 Feb 2012 CASE Reports Documentation of analysis and discrepancies in reports. Evaluated unit energy savings. Jan 2012 May 2012 Annual Installations Evaluated number of installations applicable to each standard Mar 2012 Jun 2012 Management, QA/QC N/A Jan 2012 Jun 2012 Title 24 Gross Standards Energy Savings The plan recognizes that the prior evaluation (of program impacts) faced a number of challenges involving data collection to determine the compliance rate for savings resulting from code changes for non-residential buildings. These challenges included the absence of a defined baseline, limited access to building permit records, and limited access to buildings. Due to this experience, there are uncertainties about what can be accomplished in the planned effort to determine the current non-residential code compliance rate. The evaluation plan has been designed to take into account the lessons learned from this past experience. In addition, the contractor will hold most of the budget for evaluating non-residential building compliance to Title 24 codes in a contingency fund. After completion of compliance analysis on approximately 20 non-residential buildings, the Contractor team will check in with the ED project manager with a report on what has been learned and a recommendation for proceeding. At that time, the ED project manager will determine how to re-direct or release some or all of the contingency funds. General Methods Gross savings under Title 24 are determined by adjusting the potential savings by a compliance rate. The previous evaluation collected the residential data through a separate residential baseline task. The evaluators did not collect baseline data for nonresidential buildings, but collected as-built nonresidential building data from building departments and building site visits. However, the number of nonresidential buildings that could be included within the project scope was too small to develop accurate compliance rate estimates, and the software to determine the nonresidential whole building compliance rate was not fully developed and tested in time for use in the analysis. The evaluation recommended using the whole building compliance rate analysis approach because Title 24 allows compliance to be achieved through whole building performance that allows for tradeoffs among measures without requiring each measure to comply with prescriptive requirements. This approach provides a more accurate analysis of gross savings because it links directly to an analysis of energy use and savings. For both residential and nonresidential analysis, a sufficiently large and representative sample of buildings is required to accurately estimate compliance. For both building categories, representativeness depends on the coverage of a sufficient number of climate zones. For residential buildings, representativeness necessitates the inclusion of single family and low-rise multifamily buildings. For nonresidential buildings, buildings covering a sufficient diversity of KEMA and The Cadmus Group 30

35 building types (e.g., offices, warehouses, high-rise multifamily) are required to accurately estimate compliance effects. The evaluation team will collect compliance data from building departments and building site visits. Only buildings that are required to meet the current Title 24 will be included (i.e., the permit application occurred on or after the effective date of the standard). Although the focus of data collection is on new measures required in the current Title 24, a comprehensive data collection of building characteristics is needed to assess compliance, which is dependent on all the features and measures in a building that are affected by Title 24. Specific Steps At present the evaluation team assumes for purposes of preparing this plan that the assessment of energy savings being realized through the code changes regarding the construction and renovation of non-residential buildings and multifamily buildings is a high priority since very limited analyses have been conducted in the past so data are limited and significant uncertainties remain. For single-family residential, there is a body of compliance data from the prior C&S evaluation and other new construction evaluations. For this reason, assessing compliance of this sector is a lower priority and fewer resources are dedicated to it. The proposed residential analysis relies on acquiring estimates of building starts and reviewing available compliance literature; it includes no primary compliance data collection. Under this scenario, the evaluation will assume that compliance rates are comparable to those rates we determined during our prior evaluation. However, because the prior compliance rates were based on the previous Title 24 requirements, this approach will require us to carefully develop a calculation procedure to ensure that the adjustment is as accurate as possible given the updates to the code. For (single-family and low-rise multifamily) residential buildings, the steps include: 1. Review previous compliance analysis and develop a detailed method for consistently using results for this study. 2. Obtain single-family residential construction data. 3. Obtain low-rise, multifamily residential construction data. 4. Apply compliance estimates to single-family and multifamily buildings and estimate an energy savings adjustment. a. Further details regarding these steps will be provided at a later date. For nonresidential buildings, the analysis will be enhanced to include primary compliance data collection on a sample of buildings. Data will be collected to support analysis of compliance for specific prescriptive measures and also to allow building simulation and the evaluation of whole building performance. The target sample size for this evaluation is 272 buildings. The steps and deliverables planned are discussed in more detail below. Simulation Software The evaluation will use a custom version of the simulation software EnergyPro by EnergySoft, llc. The evaluation team will construct an As-Built version of the building based on the KEMA and The Cadmus Group 31

36 information collected at the building official s office and at the building. The software will then construct two more models of the building, one that complies perfectly with the 2008 Title 24 Code and one that complies with the 2005 Title 24 Code. Construction Population The statewide construction population will be developed from data provided by McGraw-Hill Construction. The data will show the total area and total value constructed by zip code for the past three years. Assuming that the budget is able to support other database purchases, other sources such as the Construction Industry Research Board (CIRB) will be used to corroborate information on the volume and type of construction across the state. Statewide Sampling Strategy The following assumptions will be used when developing the statewide sampling strategy: The population analyzed for this study will be constrained by the electric Investor Owned Utility s (IOU) geographic territories. The sample targets will be determined based on recent construction activity. The following McGraw-Hill non-residential building type designations will be excluded from the population: Parking Garages and Automotive Services. Exclusion of these structures reduces the chance of misallocating the sample targets. Sample points will be proportionally divided between renovations and new construction. The final achieved sample is expected to be the result of multiple sampling stages: Stage 1 Sample points will be proportionally divided among a number of climate regions that cover the entire state. The target sample points will be allocated based on the reported total recent square feet constructed within each region that occurred within the IOU territories. Stage 2 Within each climate region, municipalities will be randomly selected and ordered using a proportion to size method. The building official s offices will then be contacted for recruitment based on their position in the list. Stage 3 Once recruited, a staff member will visit each building department to construct a population of sample points that were permitted under the 2008 Title 24 building code. It is assumed that the building department will be able to provide or construct a list of recent permitting activity. The evaluation team will explore the possibility of supplementing this list with data on new accounts from the IOUs as well as individual records provided by McGraw-Hill. Once the jurisdiction s permit list has been constructed, projects will be randomly selected for recruitment until the sample targets are achieved with sufficient backups. This phased approach using proportionate to size sampling will provide an efficient sample set that covers the entire geographic area of the state. No preference to specific building types will be used. It is assumed that a statistically significant number of data points will occur for the most frequently constructed buildings. This sample design does attempt to eliminate bias by reducing the burden that will be required by each participating building department. However, it is expected that some building departments and building owners will refuse to participate and that some officials will influence the final KEMA and The Cadmus Group 32

37 sample by choosing what information to provide. The evaluation team will attempt to mitigate these biases by teaming with its stakeholders to determine if an incentive for participation can be provided. Any bias resulting from the choice of the building official or building owner is outside the control of this study. The evaluation team will provide the CPUC and the IOUs with the list of building departments that have been sampled for participation, both primary and back up. This list will include the target sample points per jurisdiction. Any assistance the CPUC or the IOUs can provide to encourage participation will be helpful and appreciated. The evaluation team will draft a letter for the CPUC to send to all building departments in the state notifying them of this study and its researchers. The evaluation team will finalize the draft sampling plan with initial sample targets and submit them to the CPUC before the pilot phase recruitment begins. The final sampling plan and targets will be drafted at the completion of the pilot phase. Non-Residential Data Collection Pilot Phase Two of the sampled jurisdictions will be used for the pilot phase of the project. Both are expected to be in PG&E s territory, one in CZ 3 and the second in CZ 12. Both climate zones are expected to include multiple sample points based on an initial analysis of the McGraw Hill data and both can be accessed easily by the available staff while still having different climates and community profiles. The evaluation team will select the jurisdictions for this effort based primarily on overall sampling targets. Additional factors to be considered include: Availability of online permit data records since this is expected to simplify the development of a construction population within each sampled jurisdiction. Willingness to participate based on an existing relationship with an IOU (PG&E in this case). Due to the extremely tight schedule for this evaluation, it is important to test the building compliance audit process as soon as possible. For this reason, we are inclined to use a cooperative building department in the pilot phase while recognizing that the results may not be entirely representative. In their final report, Impact Analysis 2008 Update to the California Energy Efficiency Standards for Residential and Nonresidential Buildings (11/7/2007), Architecture Energy Corporation estimates that approximately 60% of the savings in the non-residential sector due to the code change will be realized through renovation projects. Further analysis is required to understand how these estimated savings are claimed by the IOUs. For the purpose of the pilot phase of this project, half of the sites visited will be new construction projects and half will be renovation projects. The pilot phase goal is therefore to visit and analyze 10 new construction projects and 10 renovation projects. If the jurisdictions sampled for the pilot phase are unable or not required by the sample to provide 10 projects each, then additional jurisdictions will be sampled until the goal is achieved. Using the jurisdictional population, pilot phase sites will be sampled for analysis. Ideally, these sites will be randomly sampled from the population but complete random selection at this level is not expected to be possible throughout the entire sample or projects. Individual projects may be KEMA and The Cadmus Group 33

38 selected or rejected based on the documentation available and the expectation of the team s ability to visit the site. The sampled projects will be checked against IOU program participant records. In the pilot phase, the project will not be removed if it participated in the Savings by Design program, but rejection due to participation will be used in the full study. Participants are expected to represent the best possible situation for documentation and customer willingness to participate in this evaluation. Inclusion in the early part of the pilot phase will allow the team to demonstrate its data collection methods. The best customer contact will be determined for each sampled project. Customer contact information is expected to be shown in the permit documents and be available from the customer s utility. Each site will be visited by members of the evaluation team to collect as-built data. The level of rigor for the data collection will be weighted towards the code changes that are expected to provide the most savings across the IOU s territories. Each site will be analyzed to determine its individual compliance rate. The evaluation team will provide the CPUC with the data collection tools to be used during the pilot phase of this project before visits to the building departments begin. The evaluation team will provide the CPUC a summary of savings projections, results, and recommendations based on the pilot phase of this project. Based on this summary, the CPUC will then determine the funding that will be made available to complete the project. If successful in securing additional funding, the evaluation team will modify the project design based on the result of the pilot phase and the funding available. Non-Residential Data Collection Full Sample This phase will follow the same outline of tasks for the pilot phase of the project. However, this phase represents a full order of magnitude increase in the effort required. It is expected that the experiences of the pilot phase will inform the recruitment processes used in this phase as the team recruits building departments and completed projects throughout the state. Any desired alterations in the research plan due to the pilot phase will be discussed at the conclusion of the pilot. The evaluation team will provide the CPUC with a final sampling plan and final data collection tools before visits to building departments begin. The evaluation team will provide the CPUC with regular updates on the progress of recruitment and scheduling. Non-Residential Analysis Each building will be analyzed in three different cases: As Built, Meeting the 2008 Code, and Meeting the 2005 Code. The compliance rate for the building will be calculated as the ratio of the difference in energy consumption between the actual building and the building meeting the 2005 code to the difference in energy consumption between the building meeting the 2008 code and the building meeting the 2005 code. The overall non-residential compliance rate for the state will be determined from site level results and the sampling strategy used. New Construction: The data collected from the building permit office and the data collected on site will be used to create an EnergyPro model of each new construction building. KEMA and The Cadmus Group 34

39 Renovations: The data collected from the building permit office and data collected on site will be used to develop either spreadsheet models or EnergyPro models of the renovated space. The model for each site will be determined based on the size and complexity of each renovation. Reporting At the conclusion of this project, the evaluation team will provide the CPUC with a report discussing all activities attempted and completed as part of this task. Results associated with specific buildings or building departments are considered confidential. No building department, individual or building will be referenced in the report. The following table shows the actions and deliverables associated with this project. The following timeline represents an aggressive goal that will require support and participation by all stakeholders to achieve. Table 10: Title 24 Compliance Task Actions/Deliverable Date Non-Residential Sampling Summary of Initial Sampling Strategy 2/17/12 Building Official Visit Pilot Data Collection Forms 3/2/12 Building Site Visit Pilot Data Collection Forms 3/2/12 Pilot: Non-Residential Data Collection, 20 sites Building Official Recruitment & Scheduling 3/5/12 3/9/12 Building Department Visits 3/19/12 3/23/12 Site Recruitment 3/21/12 3/30/12 Site Visits On Site Data Collection 4/2/12 4/20/12 Site Specific Modeling 4/9/12 4/27/12 Summary of Pilot Results 5/4/12 CPUC Review & Presentation Period 5/7/12 5/18/12 Non-Residential Sampling Summary of Final Sampling Strategy 5/25/12 Building Official Visit Final Data Collection Forms 6/1/12 Building Site Visit Final Data Collection Forms 6/1/12 Non-Residential Data Collection: All Remaining Sites Building Official Recruitment & Scheduling 6/4/12 6/29/12 Building Department Visits 6/18/12 7/20/12 Site Recruitment 6/20/12 8/31/12 Site Visits On Site Data Collection 7/5/12 9/30/12 Site Specific Modeling 7/16/12 10/15/12 Summary of Compliance Results 10/31/12 Non-Residential Analysis T24 Compliance Task Report, Non-Residential 12/31/12 Residential Analysis T24 Compliance Task Report, Residential TBD KEMA and The Cadmus Group 35

40 Title 24 Net Standards Energy Savings The method for estimating natural market adoption is identical for building codes and appliance standards. Since this is the case, the following descriptions are not repeated. Instead, the reader is referred to this section. General Methods Definition Net standards energy savings from C&S are determined by adjusting the gross standards energy savings to account for the level of NOMAD. This adjustment is directly analogous to the freerider concept applied in many energy resource program evaluations. In both cases, the adjustment is made to separate out the effect of the program from what would have happened in the program s absence. For resource programs, evaluators use a variety of methods to identify freeriders, who are defined as participants that would have installed the same energy-efficiency measure(s) without the program. The goal for new C&S is essentially the same: the NOMAD research will enable the evaluation to determine what portion of the regulated market would have used the energy-efficient measure in the absence of the regulation. Methodology For each of the energy-efficient measures and appliances studied, the evaluation will develop a curve representing a best estimate of the market adoption of efficient measures/appliances that would have occurred without adoption of the C&S. Using the same general method as the prior evaluation, the evaluation will draw upon literature, market data, and expert opinion to define a Bass market adoption curve for each regulated measure/appliance. We will establish a market introduction date from literature and information from market experts, which will provide the required starting point for each market adoption curve. For building energy codes, where compliance can be achieved through a variety of approaches including whole building simulation, the prior evaluation estimated the market adoption in terms of improvements in whole building efficiency. If analysis is conducted at the whole building level, we will estimate whole building efficiency trends. If analysis is conducted at the measure level, we will need estimates of trends in efficiency at the measure level. This evaluation may include additional research into this topic, depending on the magnitude of building energy savings claimed. We will accomplish data acquisition using a market adoption tool, through which experts will provide and explain NOMAD estimates for each measure/appliance. The evaluation team developed this tool for the prior evaluation, so only a small effort should be required to configure the tool to the new set of C&S and to provide ongoing technical support for the duration of the evaluation project. Role of Experts The evaluation will apply the lessons learned from the prior evaluation to obtain expert input as efficiently as possible. We will again offer incentives to experts for participation, and will direct our staff to help the experts use the online tool (as this improves the response rate). We will target obtaining participation from five to seven experts for each measure/appliance, with some experts providing input on more than one item. KEMA and The Cadmus Group 36

41 We will recruit experts from the pool identified in the CEC and IOU documentation of the C&S process, and also from industry trade groups and publications. We will consult CPUC ED and other stakeholders as we define expert recruitment lists and methods. Delphi Process In the first round, each expert gives their input and explanatory comments independently. After all initial inputs are collected, we will develop an average market adoption curve. To implement a Delphi approach, experts will be asked to review the average curve and the initial inputs from the other experts (although identities are not revealed). They will then have an opportunity to modify their initial input and to provide additional comments. The interaction between experts is the key to the Delphi approach as the experts build a common base of knowledge about the measure/building market. Specific Steps Notes: This task is primary research that collects expert input on the market for each regulated product or practice. Parameters to be determined by this effort are shown in KEMA and The Cadmus Group 37

42 Table 8 in Section above. The NOMAD tasks involve three major subtasks (not including evaluation plan, QAQC, and final reporting): 1. Updating the online data collection tool 2. Identification, recruitment and input of independent industry experts for each C&S 3. Development of Bass curves to model NOMAD for each code and standard synthesizing information provided by industry experts Updating Online Collection Tool Cadmus will update the online collection tool used in the evaluation for the 14 appliance standards, 14 building codes, and seven federal standards to be evaluated in the 2012 evaluation. Each code or standard has a designated page in the online tool on which each independent industry expert can provide their informed estimate regarding the naturallyoccurring market adoption for that code or standard. Cadmus does not anticipate making significant changes to the online tool. Deliverables Online Tool with pages for 35 codes/standards in 2012 evaluation Summary descriptions on each code s/standard s page in the online tool Industry Experts The NOMAD task uses a modified Delphi approach to collect the information necessary to build the bass curves for each code and standard. Cadmus will identify individuals for each evaluated code and standard able to provide an informed estimate of naturally-occurring market adoption of that code or standard. We will recruit these individuals to provide their informed estimates via the online tool. After all experts have weighed in, each individual will have the opportunity to modify his or her estimate, after reviewing those offered by their peers, or commit to their original estimate. The online tool offers experts to the ability provide not only their estimates, but also the supporting reasons behind their estimate, in text format. Deliverables List of industry experts identified for each code and standard. The goal is to have ten experts for each C&S representing manufacturers, trade associations, industry consultants, IOUs, and regulators. List of industry experts successfully recruited to provide estimates for each code and standard. The target is to have at least five experts provide inputs for each C&S. This work will be focused on the C&S that have the highest projected savings. Round 1 estimates provided by experts Round 2 estimates provided by experts Development of NOMAD Estimates for Each Code and Standard KEMA and The Cadmus Group 38

43 After receiving the final input in the online tool from each expert, Cadmus will review the data provided, removing any outliers and estimates appearing to exhibit bias. We will analyze the remaining estimates for each code and standard to provide a bass curve and its parameters that is the synthesis of those estimates. Deliverables Final estimates to be analyzed for each code and standard Final bass curve parameters for each code and standard These tasks, the associated deliverables, expected timeline, and estimated cost are shown in Table 11 below. Table 11. Title 24 NOMAD Task Deliverable Timeframe Update for 14 building codes evaluated in 2012 Jan Feb 2012 T-24 - Updating Online Tool Updated code and standard descriptions Jan Feb 2012 Identify experts for each code Apr 2012 May 2012 T-24 Industry Experts List of experts successfully recruited for each code Apr 2012 May 2012 Round 1 estimates Apr 2012 May 2012 Round 2 estimates Apr 2012 May 2012 T-24 Development of Final estimates used in analysis Apr 2012 May 2012 NOMAD estimates Final bass curve parameters Jun 2012 Aug 2012 Title 24 Attribution Notes: This task consists of three steps that are done in preparation for the final scoring step. These first three steps collect information from a number of primary and secondary sources. The final step, attribution scoring is primary research that depends on the work done in the first three steps as well as the personal knowledge of the expert panel. Parameters to be determined by this effort are shown in KEMA and The Cadmus Group 39

44 Table 8 in Section above. The evaluation will estimate the contributions of the California IOU C&S program to the development of California energy building codes that went into effect in The analysis will result in an attribution score for each building code representing the percentage contribution of the C&S program to the development of the code. The product of the attribution score and net standard savings from the code will result in an estimate of the energy savings attributable to the C&S program. To estimate the contributions, we will follow the attribution methodology developed for the previous ( ) evaluation cycle. 7 The attribution tasks will determine the credit the Codes and Standards Program will receive for its efforts in supporting the development of California building codes and appliance standards and federal appliance standards. Attribution will result in a percentage score indicating the relative contribution of the IOUs to the development of each code or standard. The attribution tasks involve four main subtasks (not including writing an evaluation plan and QA/QC): 1. Development of attribution guidelines for federal appliance standards and review and revision (if necessary) of existing attribution guidelines for T24 and T20 2. Collection of data about contributions of IOUs and other stakeholders to codes and standards development 3. Analysis of IOU and other stakeholder contributions 4. C & S Program attribution scoring by an independent panel of experts Attribution Guidelines Review and Development Cadmus will review the attribution guidelines used in the evaluation and identify potential improvements. If potential improvements are identified, Cadmus would initiate a process involving stakeholders for making the changes. As federal appliance standards are developed by the U.S. Department of Energy, Cadmus will develop separate attribution guidelines for federal appliance standards. Cadmus will research the federal rulemaking process by interviewing stakeholders and reviewing academic studies to understand the IOUs and other stakeholders roles. We expect the federal guidelines to have a structure similar to the California guidelines, with the IOUs attribution score being a function of the IOUs contributions to different areas of the rulemaking process. Cadmus will give stakeholders an opportunity to comment on the guidelines and revise them based on feedback as necessary. Deliverables: Draft federal appliance standards attribution guidelines Final federal appliance standards guidelines 7 The Cadmus Group. March 9, The Proposed Cadmus Attribution Methodology (Revised). This document is available at KEMA and The Cadmus Group 40

45 Data Collection Cadmus will collect data about IOU and other stakeholder contributions from transcripts of CEC meetings, public comments about the proposed code and standards changes, IOU CASE reports and code change theories, and interviews with stakeholders representing different perspectives about the rulemaking process. Cadmus will collect the data and catalog the information about stakeholder contributions for future reference in estimating the attribution scores. In addition, Cadmus will survey stakeholders about the relative importance of different stakeholder activities in the development of particular codes and standards. A limited number of the surveys may be conducted with CEC staff regarding the CfR since advocacy for these codes and standards was not led by the IOUs. The survey responses will be used to develop attribution factor weights. 8 Deliverables: Draft stakeholder survey instrument Data Analysis Final stakeholder survey instrument Cadmus will analyze the information about stakeholder contributions obtained from stakeholder interviews, public documents, and other data sources. We will summarize the IOU and other stakeholder contributions to each area of code development (factor areas).9 Cadmus will then develop a preliminary estimate of the percentage contribution of the IOUs to the development of each code or standard factor area. Cadmus will also estimate weights for each factor score and code or standard. Deliverable: Slide presentation for each C&S summarizing Program contributions Codes and Standards Attribution Scoring It is critical throughout this process to maintain objectivity and independence from the C&S program and any stakeholders. To reduce the potential for bias or perceived bias on the part of the CADMUS (as the lead researcher into attribution questions), we plan to have KEMA convene a panel of independent and knowledgeable experts to evaluate the IOUs contributions to the development of each code and standard.. The panel will evaluate their contributions to each code or standard in each factor area based on evidence provided to the panel. Deliverables: Memo reporting attribution scores These tasks, the associated deliverables, expected timeline, and estimated cost are shown in 8 The Cadmus Group. March 9, The Proposed Cadmus Attribution Methodology (Revised). This document is available at 9 The evaluation areas are: 1. Development of compliance determination and other analytical methods; development of code language and analysis of energy savings and cost effectiveness; 3. Demonstrating feasibility of new codes and standards and stakeholder outreach. KEMA and The Cadmus Group 41

46 Table 12 below. KEMA and The Cadmus Group 42

47 Table 12. Title 24 Attribution Task Deliverable Timeline Data Collection Stakeholder Survey Instrument Mar Aug 2012 Data Analysis C&S Program Contributions for each code and standard Mar Aug 2012 C&S Attribution Scoring Memo Reporting Attribution Scores Aug Oct 2012 Management & QA/QC N/A Ongoing Title 24 Savings by Utility (Allocation) All the components described will be combined to calculate the Net Program Savings. These savings then will be allocated to the individual utilities. The existing ISSM model makes this allocation based on utility sales, and this has been acceptable to the utilities and the CPUC in the past. We will use the same approach in this analysis, based on updated utility sales data. Title 24 Integration and Analysis Deliverables TBD, Timeline TBD The following description applies to both Title 24 and Appliance Standards. Each of the major tasks to be completed under the protocol produces a set of data for a number of specific building codes and appliance standards. As described previously, the ISSM tool once updated will provide a vehicle specifically designed to support integration of these various task-specific data sets. This task provides for the labor needed to perform the integration, to communicate issues back to the task teams, and to do the iterative integration that leads to a solid well-integrated final result. Results from all of the evaluation tasks (described above) will be integrated and analyzed using the Integrated Standards Savings Model (ISSM). Analysis will include calculation of savings using the Monte Carlo method to estimate the uncertainty associated with outputs, based on the distributions assumed for key inputs. Reporting also represents a significant effort as work in each area will be summarized and presented in a concise and understandable form. Documentation will also include a summary of the integration and analysis that produced the final results of the evaluation. The evaluation will include integration and analysis for 17 Title 24 standards, 17 Title 20 Appliance standards, and 7 Federal appliance standards for which the IOUs report savings Appliance Standards (Title 20 and Federal) These activities will focus on appliance standards that fall into two categories: those adopted by the state of California into the Title 20 regulations and those adopted by the U. S. Department of Energy (DOE). Most of the evaluation work on appliance standards will be focused on those that became effective after In a few cases, the evaluation will investigate some aspects of standards from the prior cycle (adopted before January 2006). This includes standards with large energy savings that were found to have low compliance, and those for which IOUs did not claim KEMA and The Cadmus Group 43

48 savings in the prior evaluation because they indicated advocacy occurred after January 2006 (e.g., incandescent bulbs Tier 2). The standards covered in this evaluation by each analysis step are indicated in Table 13. Table 13. Appliance Standard Coverage in Evaluation Standard ID Standard Name Potential (Gross) Compliance (Gross) NOMAD (Net) Attribution (Net) Std28a Televisions - Tier 1 Std27 General Purpose Lighting and 40 watt Std28b Televisions - Tier 2 Std25 Std26 Std29 Std22b Std12a Std11b Std9 Std22a Std4 General Purpose Lighting watt General Purpose Lighting watt Battery charger - consumer - Tier 1 BR, ER and R20 Incandescent Reflector Lamps: Commercial Pulse Start Metal Halide HID Luminaires, Tier 1 General Service Incandescent Lamps, Tier 2 Residential Pool Pumps, 2- speed Motors, Tier 2 BR, ER and R20 Incandescent Reflector Lamps: Residential Walk-In Refrigerators / Freezers Std24 Portable Lighting Fixtures Std23 Metal Halide Fixtures Std32 Std31 Fed6 Fed7 Battery Charger - Large - Tier 2 Incremental Battery Charger - Large - Tier 1 Incandescent Reflector Lamps General Service Fluorescent Lamps KEMA and The Cadmus Group 44

49 Standard ID Fed5 Fed4 Standard Name Potential (Gross) Compliance (Gross) NOMAD (Net) Attribution (Net) Residential Electric & Gas Ranges ASHRAE Products (Commercial boilers) Fed1 Electric Motors 1-200HP Fed2 Refrigerated Beverage Vending Machines Fed3 Commercial Refrigeration Total Coverage Appliance Standards Potential Standards Energy Savings The first step consists of a thorough review of the CASE report estimate for each standard. The report will be assessed to determine whether the reported potential savings can be replicated and the estimation method is valid. Per Unit Savings Validating the per-unit savings involves assessing two specific components: baseline energy consumption (the energy consumption of a representative unit before the standard was implemented) and energy consumption for units that meet the standard. Baseline Consumption The appliance baseline consumption is the basis for all energy savings estimates. We will not conduct any primary data collection to analyze baseline consumption, but will be guided by the method used by the C&S program to establish baseline consumption and will either verify application of that method and/or rely on other secondary data. Baseline consumption is generally calculated using one or more of the following methods: 1. Standardized energy-efficiency calculation based on the energy consumption of a specific appliance just meeting an existing standard 2. Equipment metered data to determine an overall average baseline consumption 3. Engineering equations involving a number of assumptions to arrive at baseline energy consumption. The standardized calculation approach typically involves applying a federal standard efficiency level to calculate baseline daily or annual energy consumption of a particular appliance. With some basic assumptions (e.g., days of use in a year), a total baseline annual consumption can be derived. If this method was used in the CASE report, we will verify the equations and assumptions with other sources and repeat the calculations to confirm the baseline energy consumption estimated in the report. Any differences between the evaluation team s estimates KEMA and The Cadmus Group 45

50 and those in the CASE report will be noted. When differences or questions occur, we will contact the report author for clarification. The second method involves equipment metering. Typically, CASE report authors did not meter equipment, but instead relied on secondary sources for metered values. If the data are currently available, we will review the source document to assess the metering methodology. Any concerns about the results of the metering will be noted, investigated, revised (if appropriate), and documented. The third method uses engineering equations. When using engineering equations several assumptions must be made such as operating hours, typical wattages, and equipment performance characteristics. All the initial assumptions will be thoroughly evaluated for accuracy. Initially, we will use the stated assumptions and any concerns or discrepancies will be documented for subsequent research and revision to determine the revised unit energy consumption. Unit Energy Savings The next step involves verifying the energy consumption that is consistent with the standard and/or energy savings from the standard. Some appliance standards are based on a percentage savings over a baseline model; others involve moving to a third-party Tier 1 or Tier 2 standard; and some involve specified improvements in components that lead to the whole unit consuming less energy. Again, the analysis or approach used to estimate the unit energy savings will be thoroughly investigated for this evaluation. Each CASE report will be comprehensively reviewed to identify the proposed savings calculation method and determine if an alternative method would be more appropriate. As noted earlier, we will review DEER and any other ED-approved estimates to determine whether they are applicable. One key item evaluated will be consistency in the method used for baseline consumption and proposed standard consumption calculations. As an example: if a federal calculation equation is in place to establish the maximum allowable energy consumption under an existing standard and there is a maximum energy consumption equation for the proposed standard, we will examine whether the same equations were used for both baseline consumption and the proposed standard baseline consumption. If a different methodology is used, the team will review the finding with the CASE report author(s) to determine why. If a percentage savings method is used for a standard that addresses multiple components of an appliance, we will perform a general review to verify that the savings for the components, including any interactions, equate to the appliance (unit) percentage savings for the proposed standard. Again, the team will attempt to replicate the initial CASE report savings estimates, check whether they were consistent, and attempt to resolve any discrepancies. Review Current Sales Data Once the unit baseline and unit energy savings are evaluated, we will review the initial unit sales per year starting with the effective date of the standard. At first, this involves using the data sources identified in the CASE reports or HMG savings estimate report. These sources will be assessed to determine if the methodology in the secondary or primary source is accurate and to KEMA and The Cadmus Group 46

51 verify that the analysis was done properly. Errors will be documented and corrected to reestimate the potential energy savings in year one. Validate Potential Energy Savings The team will use all the previously mentioned factors to verify the total potential energy savings expected for the first year of introduction of appliances that meet the standard. The most important factors are the unit sales and energy savings. These values are used to derive the first year potential energy savings as follows: Potential energy savings in year 1 = unit energy savings * unit sales Knowing how the potential energy savings are calculated, we will identify any remaining CASE report gaps (assumptions not stated in the CASE reports) or areas needing clarification and forward them to the CASE report author for response. Based on the CASE report author s response, the results will be reviewed for accuracy; and any changes to update the potential energy savings in the first year of installation will be made in the final step of the potential energy savings evaluation. Review More Recent Information; Update; Revise Potential Energy Savings Once we verify the first-year potential energy savings and correct all the calculations from the CASE reports, as needed, the most recent studies and sales data will be evaluated. Since most of the work on the Title 20 standards was completed some years earlier, the team expects better estimates or more recent actual sales data will be available. If we find better data, we will use the new information in the analysis of the potential savings and document the change. Specific Steps The potential energy savings is the technical potential for an appliance standard, and is based on the energy savings from the standard and the potential number of new appliances each year. This task is secondary research that consists of evaluating a number of primary and secondary sources of information. Parameters to be determined by this effort are shown in KEMA and The Cadmus Group 47

52 Table 8 in Section above. The process to determine potential energy savings includes the following steps: 1. Review potential savings in the Total CS Program Savings file and the CASE reports for each appliance standard. Review references cited in the CASE reports and search for other possible sources including cross-cutting studies conducted for this evaluation cycle. Note: For Federal standards, we will follow a similar process using any documentation provided by the California IOUs and selected documents available from the U.S. Department Of Energy (DOE). a. Review per unit savings estimates. b. Review and verify sales data sources identified in the CASE reports or savings estimate reports. c. Review the calculations for consistency between the baseline consumption and standard consumption methodologies. 2. Discuss any discrepancies or proposed changes to savings calculation methodology with the CASE report authors. 3. Revise potential savings estimates when more accurate methodologies or updated data are available. 4. Conduct market research to assess the market potential and NOMAD. Revise potential savings estimates when discrepancies or errors are found. These tasks, the associated deliverables, expected timeline, and estimated cost are shown in Table 14 below. Table 14. Title 20 & Federal Potential Energy Savings Task Deliverable Timeline Review CA Standards Q1 Progress Report, March Jan 2012 Oct 2012 Review Federal Standards Q2 Progress Report, June Planning and coordination Q3 Progress Report, Sept Ongoing Report results Memo Oct 2012 Final Report Jun 2013 Appliance Standards Gross Energy Savings General Methods The evaluation team s approach for assessing Title 20 compliance is based on the methodology used in the prior CPUC Codes and Standards program evaluation, 10 the Khawaja, M.S., A. Lee, and M. Levy Statewide Codes and Standards Market Adoption and Noncompliance Rates, SCE , Prepared for Southern California Edison. KEMA and The Cadmus Group 48

53 compliance evaluation, 11 and the protocol presented in the California evaluation protocol document. 12 The scope of these prior studies has been adjusted to include the majority of the energy savings. 13 In addition to the site visits being conducted for the five California Title 20 incandescent lighting standards, and the one federal incandescent lighting standard as part of the lighting shelf survey through another work order, onsite data will also be collected for the four Title 20 standards with the highest potential gross energy savings, and the four other federal nonincandescent lighting standards with claimed first year gross energy savings. Our approach will include site visits to assess market compliance rates for the top energy saving standards. Several different methods of data collection will be utilized to maximize the data we are able to obtain for the budget available. These methods include vendor site visits to determine market compliance following the methodology established in the prior evaluations. In addition to this step, data will also be collected through other work orders and the Title 24 evaluation effort. This will allow the evaluation team to obtain data on both what vendors are distributing, and the efficiency of measures actually being installed in the market. Table 15 and Table 16 show the data collection methods for the high energy saving standards that will be assessed for compliance. Additional detail and specific steps included in our approach are provided in the subsequent sections KEMA, Inc., The Cadmus Group, Inc., Itron, Inc., and Nexus Market Research, Inc Codes & Standards (C&S) Programs Impact Evaluation: Final Evaluation Report. The TecMarket Works Team California Energy Efficiency Evaluation Protocols: Technical, Methodological, and Reporting Requirements for Evaluation Professionals. Prepared for the California Public Utilities Commission. The study scope does not permit detailed compliance analysis of every standard, so the research focuses on the standards providing the greatest energy savings. KEMA and The Cadmus Group 49

54 Table 15. Data Collection Methods for Title 20 Standards ID Standards 1st Year Gross Savings (GWH) Start Year Robust Budget Std28a Televisions - Tier Jan-11 SV Std25 General Purpose Lighting watt Jan-11 WO13 Std26 General Purpose Lighting watt Jan-12 WO13 Std29 Battery charger - consumer - Tier Sep-12 SV Std22b BR, ER and R20 Incandescent Reflector Lamps: Commercial Jan-08 WO13 Std12a Pulse Start Metal Halide HID Luminaires, Tier Jan-06 T24 Std11b General Service Incandescent Lamps, Tier Jan-08 WO13 Std9 Residential Pool Pumps, 2-speed Motors, Tier Jan-08 T24 Std22a BR, ER and R20 Incandescent Reflector Lamps: Residential 67.3 Jan-08 WO13 Std4 Walk-In Refrigerators / Freezers 63.4 Jan-06 T24 Std24 Portable Lighting Fixtures 43.3 Jan-10 SV Std23 Metal Halide Fixtures 38.1 Jan-10 SV, T24 Std8 Residential Pool Pumps, High Eff Motor, Tier Jan-06 T24 Std12b Pulse Start Metal Halide HID Luminaires, Tier Jan-08 T24 SV = Site Visit; Data Collected Through Title 24 Visits; WO13 = Data Collected Under WO13 Table 16. Data Collection Methods for Federal Standards* ID Standards 1st Year Gross Savings (GWH) Start Year Robust Budget Fed6 Incandescent Reflector Lamps Jul-12 S WO13 Fed7 General Service Fluorescent Lamps Jul-12 SV Fed5 Residential Electric & Gas Ranges 7.56** Apr-12 SV Fed4 ASHRAE Products (Commercial boilers) 5.95** Mar-12 SV Fed1 Electric Motors 1-200HP 5.14 Dec-10 SV SV = Site Visit; WO13 = Data Collected Under WO13 * Federal claimed savings values were multiplied by 12.1% to be comparable to gross California standard values. ** Savings values were converted from therms to kwh to be comparable to other standard values. KEMA and The Cadmus Group 50

55 Specific Steps The analysis approach is based on methodology developed in the prior evaluation efforts, along with several overall enhancements, and the budget amount available for this study. Data on the energy-efficiency specifications of individual appliance models available in the market will be collected through site visits to vendor locations. Due to budget constraints, data on products available in the market will also be collected by phone for several appliance types whose distribution channels make this possible. This approach will include these twelve tasks: 1. Obtain a list of complying appliances compiled in the CEC online database. 2. Prioritize the evaluation activities on high impact energy saving standards based on the ex ante estimates (see Table 15 and Table 16 for projected evaluation activities for high energy saving standards). 3. Conduct in-depth review of energy-efficiency requirements of the high energy saving appliances. 4. Obtain a contact list for vendors that sell or distribute the products selected for data collection. 5. Stratify the list of vendors by business size and three geographic regions: southern, central, and northern. This list may be supplemented with online research where necessary. 6. Call distributors to solicit data over the phone for appropriate measures. 7. Visit distributor sites to collect product specifications. 8. Populate a spreadsheet database with product details collected from distributors such as model number, specifications etc., and compare the details with the CEC database (if list is available) to determine whether the model is listed, indicating that it is in compliance with Title Determine whether unlisted models being sold comply with the technical requirements of the standard. 10. Determine which models are non-compliant. 11. Extrapolate the individual distributor data to the statewide level. 12. Calculate the aggregate compliance rate. In addition to collecting on site vendor data for the highest energy saving standards that went into effect after 2008, data will also be collected for the next highest energy saving standards in the 2006 to 2008 code cycle that were not previously assessed for compliance: portable lighting fixtures and metal halide fixtures. Additional data will be collected through coordination with the Title 24 site visits for six of the standards as well to allow for more accurate compliance estimates. These enhancements will result in compliance estimates for more standards and greater confidence in the results overall. In addition to the site visits being conducted for the five California Title 20 incandescent lighting standards, and the one federal incandescent lighting standard as part of the lighting shelf survey, data will be collected at distributor locations for four Title 20 and four federal standards, along with collecting data for six title 20 standards through the Title 24 visits. In the evaluation, the evaluation team realized that on site data are difficult to collect for several appliance types. These measures included walk-in refrigeration and metal halide KEMA and The Cadmus Group 51

56 luminaires. Store visits and calls to manufacturers in the evaluation confirmed that the majority of walk-in refrigeration is custom made. To address this issue, the evaluation team coordinated with the Title 24 site visit effort to collect data on walk-in refrigerators and freezers at several of the commercial new construction locations visited and assessed for Title 24 compliance. Measurements necessary to determine compliance with the energy efficiency requirements of the walk-in refrigeration standard were collected and analyzed for compliance. Through lessons learned in the prior evaluation, we also know that metal halide luminaires are often ordered per customer specifications, so it may be difficult to find accessible distributors to visit for this appliance type. We plan to collect data on site for walk-in refrigeration, metal halide luminaires, and metal halide fixtures whenever possible in conjunction with the non-residential Title 24 site visit effort. If possible, data will be collected for a maximum of 20 walk-in refrigeration units. For metal halide luminaires and fixtures, data will be collected at a maximum of 20 sites, for an 80/20 confidence interval and precision rate. The approach for the Title 24 data collection coordination will include the following standards: a. Walk-In Refrigerators / Freezers b. Pulse Start Metal Halide HID Luminaires, Tier 1 c. Pulse Start Metal Halide HID Luminaires, Tier 2 d. Metal Halide Fixtures Updated Plan: Deliverables, Timeline, and Estimated Cost Notes: This task includes both primary and secondary research activities. Parameters to be determined by this effort are shown in KEMA and The Cadmus Group 52

57 Table 8 in Section above. In addition to the descriptions provided above, the Title 20 compliance assessment is broken down into the following steps in order to describe the work with associated deliverables, timeline, and cost. Mgmt/Admin Evaluation QA/QC Define Standards & Develop Data Collection Tools Define Sample & Strata Schedule and Conduct Site Visits and Enter Data Coordination with Other Work Orders and Data Collection Efforts Product Specification Research Analyze Data Reporting Define Standards & Develop Data Collection Tools Fully understanding the energy efficiency requirements of the standards is a critical first step in compliance assessment. To ensure all data necessary to determine compliance is collected on site, the efficiency requirements must be analyzed to identify the key product specifications necessary to assess whether each product is compliant. This will consist of an in depth engineering review of the standard language, summarizing the standard language into a reference sheet, and developing on site data collection instruments. Define Sample & Strata This task will include purchasing a vendor contact list, defining strata criteria, and selecting a site visit sample. Based on our experiences in the previous C&S impact evaluations, we expect to supplement the purchased sample list with online research. Schedule and Conduct Site Visits and Enter Data Scheduling and conducting site visits is the core data collection task occurring for the Title 20 compliance assessment. Based on our previous experiences, we will not attempt to schedule site visits for certain appliance types since this is not a preferred approach for many vendors. As part of this task we will contact some vendors before visiting their location to solicit data sharing over the phone or through . Once all calls have been made and a site visit route has been established, experienced Cadmus research analysts will conduct the site visits. Data will be analyzed after each trip to ensure accuracy of data collection and that all information necessary to determine compliance is being collected. If data is found to be unobtainable in the field, this will be identified early on in this process. Coordination with Other Work Orders There will be ongoing coordination for lighting site visits being conducted through WO 13 and metal halide and walk-in refrigeration visits being conducted in conjunction with the Title 24 KEMA and The Cadmus Group 53

58 compliance assessment. The Cadmus task lead will review the data collected early on and work with the field technicians to make adjustments to the data collection methods if necessary. Since there is some uncertainty regarding the data that we will be able to collect through the Title 24 site visits, we anticipate this task will require a high level of coordination. Product Specification Research Certain product specifications that are necessary to determine a product s compliance with the energy efficiency requirements of the standard are often not available in the field. To address this, research will need to be done on the back end once data has been collected to collect the missing information. This effort will consist of calls to manufacturers and online research to supplement the missing field data. Analyze Data This task will consist of analyzing all data collected as part of the appliance compliance assessment. The methodology developed in the previous codes and standards evaluation will be used and each individual model will be assessed for whether it is 1) listed on the CEC compliant model list (if available for the product type) and 2) whether the product s specifications quality with the energy efficiency requirements of the standard. After compliance has been determined for each model and appliance type, statewide compliance estimates will be developed by extrapolating the results to the statewide level. KEMA and The Cadmus Group 54

59 Evaluation QA/QC To ensure a quality product, quality assurance and control will be implemented at crucial stages of the evaluation. Quality assurance will be an ongoing process that will be ensured through multiple levels of internal review and project manager approval of work products. Table 17. Title 20 & Federal Standards Compliance Task Deliverable Timeline Define Standards & Develop Standard details document Mar 2012 Data Collection Tools Data collection tools April 2012 Define Sample & Strata Sample list May June 2012 Schedule and Conduct Site Jul 2012 Sep 2012 Site visit data Visits and Enter Data (Data supplied at end of contract) Coordination with Other WOs Site visit data Jul 2012 Sep 2012 (Data supplied at end of contract) Product Specification Research Site visit and research supplemented data Jul 2012 Sep 2012 (Data supplied at end of contract) Analyze Data Data analysis Sep 2012 Oct 2012 Interim Findings Memo Oct 2012 Reporting Final Report Jun 2013 Mgmt/Admin N/A Ongoing Evaluation QA/QC N/A Ongoing Appliance Standards Net Energy Savings Notes: This task is primary research that collects expert input on the market for each regulated product or practice. Parameters to be determined by this effort are shown in KEMA and The Cadmus Group 55

60 Table 8 in Section above. The description of the process and tasks for estimating NOMAD is the same for both Title 24 and Appliance Standards. Please refer to the section Title 24 Net Standards Energy Savings above. The NOMAD tasks involve three major subtasks (not including evaluation plan, QAQC, and final reporting): 1. Updating the online data collection tool 2. Identification, recruitment and input of independent industry experts for each C&S 3. Development of Bass curves to model NOMAD for each code and standard synthesizing information provided by industry experts These tasks, the associated deliverables, expected timeline, and estimated cost are shown in Table 18. Table 18. Title 20 & Federal NOMAD Task Deliverable Timeframe Update for 14 CA and 7 Fed stds evaluated in 2012 Jan Feb 2012 Updating Online Tool Updated standard descriptions Jan Feb 2012 Identify experts for each standard Apr 2012 May 2012 Industry Experts List of experts successfully recruited for each std Apr 2012 May 2012 Round 1 estimates Apr 2012 May 2012 Round 2 estimates Apr 2012 May 2012 Development of Final estimates used in analysis Apr 2012 May 2012 NOMAD estimates Final bass curve parameters Jun 2012 Aug 2012 Appliance Standards Attribution Notes: This task consists of three steps that are done in preparation for the final scoring step. These first three steps collect information from a number of primary and secondary sources. The final step, attribution scoring is primary research that depends on the work done in the first three steps as well as the personal knowledge of the expert panel. Parameters to be determined by this effort are shown in KEMA and The Cadmus Group 56

61 Table 8 in Section above. The evaluation will estimate the contributions of the C&S program to the development of California and federal energy appliance standards that went into effect between 2008 and The IOUs are claiming savings for 14 new state standards and seven new federal standards. For the California Title 20 appliance standards, the evaluation will follow the plan and procedures used in the attribution of building code savings (see the building code evaluation plan above): The attribution of federal appliance standard savings will follow a similar approach; however, as federal appliance standards are developed at the federal level using a different rulemaking process, the evaluation will develop an attribution methodology for federal appliance standards based on the methodology for California codes and standards. We expect the federal standards methodology to be similar to the California one in the sense that an attribution score would be determined based on the IOUs contributions to different areas of federal standards development. Development of Attribution Methodology for Federal Standards Development of a methodology for evaluating the contributions of the California IOUs to federal standards development will require understanding of the federal energy standards rulemaking process. The evaluation will first research the federal rulemaking process and the IOUs role in it. We will interview C & S program staff, USDOE staff, and other federal stakeholders, and review academic studies and IOU reports to understand the process. Attribution of savings from federal appliance standard savings will follow the plan and procedures for evaluating IOU contributions to California building code development, including data collection, analysis, and evaluation. Attribution will be determined relative to all stakeholders that influenced adoption of the federal standards and the savings attributed to the IOU efforts will consist of only those savings occurring in the IOU service territories. These tasks, the associated deliverables, expected timeline, and estimated cost are shown in Table 19 below. Table 19. Title 20 & Federal Attribution Task Deliverable Timeline Evaluation Plan Evaluation Plan Jan 2012 Mar 2012 Attribution Guidelines Review and Development Federal Appliance Standards Guidelines Feb 2012 May 2012 Data Collection Stakeholder Survey Instrument Mar 2012 August 2012 Data Analysis C&S Program Contributions for each code and standard Mar 2012 August 2012 Codes and Standards Attribution Scoring Memo Reporting Attribution Scores Aug 2012 Oct 2012 Management & QA/QC N/A Ongoing Appliance Standards Savings by Utility (Allocation) All the components described will be combined to calculate the Net Program Savings. These savings then will be allocated to the individual utilities. The existing ISSM model makes this allocation based on utility sales, and this has been acceptable to the utilities and the CPUC. We will use the same approach in this analysis, based on updated utility sales data. KEMA and The Cadmus Group 57

62 Appliance Standards Integration and Analysis The description of analysis and reporting is the same for both Title 24 and Appliance Standards. Please refer to the section Title 24 Integration and Analysis above Reach Codes Including Local Government Programs (LGPs) Note: This task depends on a mix of primary and secondary research. The IOU Reach Code subprogram provides technical assistance to support the development of locally adopted ordinances that exceed statewide minimum requirements. Only those reach codes that are approved and enforceable under CEC procedures qualify for consideration. In addition to reach codes implemented in conjunction with the IOUs subprogram, some local jurisdictions are putting reach codes in place with less IOU intervention. In this case, the reach code is described as a Local Government Program (LGP). This evaluation will assess both the IOU Reach Code subprogram and the LGP reach code efforts. The IOUs have included energy savings from the Reach Code subprogram in their data request response. This evaluation will analyze program processes and determine impact in terms of energy savings. To do this, the evaluators will review program documentation and interview key stakeholders (such as implementers, local government officials, and IOU program staff). The evaluation will attempt to interview officials from all participating jurisdictions to achieve a census. We expect that these interviews will collect data for both the process and impact evaluations. The process evaluation will analyze program processes and determine if the IOUs are being successful in providing the support needed by the jurisdiction to adopt and pursue reach codes. To do this, the evaluators will review program documentation and interview key stakeholders (such as implementers, local government officials, and IOU program staff). The evaluation will attempt to interview officials from all participating jurisdictions to achieve a census. We expect that these interviews will collect data for both the process and impact evaluations. The impact evaluation component will determine how closely the method used by the IOUs to determine the savings generated by the Reach Code subprogram follows the codes and standards evaluation protocol, and will make recommendations of how the protocol could be enhanced to address any challenges of applying to the Reach Codes subprogram. For most information required by the protocol, the evaluation will rely on review of the program records and documentation. The one area where we expect to collect primary data and to use it in the impact evaluation is attribution (the extent to which the IOUs were responsible for local adoption of reach codes). For attribution, we will analyze documentation and use the interviews to support an estimate using methods similar to the weighted factors used for all other attribution analyses. Evaluability Assessment Develop detailed task plan to outline deliverables and estimated budget absorption. Document holistic components, information needed to evaluate, information available, and prepare an evaluability assessment. Deliverables: Task Plan, Evaluability Assessment Evaluation Plan KEMA and The Cadmus Group 58

63 Review key processes including: permitting, inspection, tracking and internal communications and review available tools and resources and how they are currently used. Deliverable: Data Collection Evaluation Plan Conduct interviews with subprogram implementers and participants and conduct process evaluation. Analyze the data and prepare evaluation report. Deliverable: Impact and Process Evaluation Report These tasks, the associated deliverables, expected timeline, and estimated cost are shown in Table 20 below. Table 20. Reach Codes Task Deliverable Timeframe Evaluability Assessment Task Plan Evaluability Assessment Jan 2012 Feb 2012 Evaluation Plan Evaluation Plan Feb 2012 Data Collection and Analysis Interview guide, Data summary Feb 2012 TBD Reach Code Evaluation Report Impact and Process Evaluation Report Oct / Nov 2012 KEMA and The Cadmus Group 59

64 4.4.4 Compliance Enhancement Program (CEP) Note: This task depends on a mix of primary research and secondary research. The CEP is subprogram to develop building department best practices for enhancing code compliance. The program includes working with approximately twelve building departments to develop process improvement interventions. The CEP subprogram activities might include, but are not limited to: conducting a comprehensive needs assessment/gap analysis through staff interviews: reviewing of permitting and inspection processes, tracking, documentation, and internal communications as projects move toward completion: and reviewing available tools and resources and how they are currently used. From this research the program goal is to identify opportunities and create tools to help optimize the existing processes and simplify enforcement. For the present program cycle, the ED was directed to conduct a pilot evaluation on CEP.. The evaluation is expected to include the following steps: Evaluability Assessment Develop detailed task plan to outline deliverables and estimated budget absorption. Document holistic components, information needed to evaluate, information available, and prepare an evaluability assessment. Deliverables: Task Plan, Evaluability Assessment Evaluation Plan Conduct a comprehensive needs assessment/gap analysis through staff interviews: reviewing of permitting and inspection processes, tracking, documentation, and internal communications as projects move toward completion: and reviewing available tools and resources and how they are currently used. Deliverable: Data Collection Evaluation Plan Conduct interviews with subprogram implementers and participants and conduct process evaluation. Analyze the data and prepare evaluation report. Deliverable: Process Evaluation Report These tasks, the associated deliverables, expected timeline, and estimated cost are shown in Table 21 below. Table 21. CEP Task Deliverable Timeline Evaluability Assessment Task Plan Evaluability Assessment Jan 2012 Feb 2012 Evaluation Plan Evaluation Plan Feb 2012 Mar 2012 Data Collection Interview guide, (Analysis and Reporting) Data summary Feb 2012 May 2012 CEP Process Evaluation Report Evaluation Report TBD KEMA and The Cadmus Group 60

65 KEMA and The Cadmus Group 61

66 5. Evaluation Coordination Efforts 5.1 Coordination with Other Impact Evaluation Workplans The evaluation team is working through the Impact Evaluation Working Group to identify opportunities to leverage work being done on other impact studies that can improve the quality or decrease the cost (or both) of the C&S evaluation. At present, our coordination effort has identified several other projects whose activities could help this evaluation. We have worked most closely with the project staff on Work Order 13 (WO013) since they will conduct a retail store shelf survey as part of the residential lighting study process evaluation. The lighting shelf stocking data collected will be used to determine compliance for several of the Title 20 lighting standards. Other work orders with the potential to help this evaluation have also been identified. These are shown in Table 22. Table 22. CPUC Work Orders Work Order WO 013 WO 017 WO 021 WO 023 WO 024 WO 028 WO 032 WO 033 WO 034 WO 042 WO 046 WO 047 Name Lighting Programs Process Evaluation and Market Characterization Impact Evaluation: Televisions Residential On-Site/Metering Survey Residential Market Share Tracking Commercial Saturation Survey Impact Evaluation: Residential/Advanced/Upstream Lighting Impact Evaluation: Residential and Small Com HVAC Impact Evaluation: Custom Measures Measure Cost Evaluation Impact Evaluation: Other Measures Impact Evaluation: Res Whole Building - Retrofit Impact Evaluation: Res Whole Building - New Construction 5.2 Data Collection Approach Evaluation of C&S requires the collection and integration of data from many sources. As discussed throughout this document, we must leverage data collected by other researchers in many areas to allow the available resources to cover the many codes and standards that contribute significantly to the total forecasted savings. Still, there are a number of areas where we have to do primary research since the data required is not available from any other source. These include the collection of compliance data for non-residential buildings and for several types of appliances. Also, we will collect information from industry and C&S process experts to define the natural market for each of the new C&S and to determine the extent to which code adoption can be attributed to the IOUs efforts. The evaluation team gained considerable experience in each of these areas during the prior evaluation. We expect to apply the lessons learned to collect accurate information as efficiently as possible. KEMA and The Cadmus Group 62

67 5.3 Data Requests to the IOUs The evaluation team will request data periodically from the IOUs throughout the evaluation. The first of these requests, identified with a separate Requestor Data Request number for each of the IOUs (PG&E EEGA 1465, SCE EEGA 1466, SDGE EEGA 1467, and SDGE EEGA 1468) was submitted on April 25, This request asked for the following: 1. Annual energy and demand savings the IOUs are claiming from the Statewide C&S program 2. Annual and cumulative gross energy and demand savings 3. Annual net energy savings 4. Savings claimed for pre versus post January 1, 2006 program advocacy activities 5. Are IOUs able to provide new construction hook up data for residential and non residential buildings 6. Are savings being claimed for any C&S for which the IOUs did not prepare CASE studies? 7. Are IOUs claiming savings for federal standards and if so, how they are estimated? 8. Information about each of the C&S subprograms The IOUs response dated May 13, 2011 provided substantial information in each of the areas requested. With regard to our request for savings information, the IOUs provided E Excel spreadsheets that included detail behind their summary tables. This detail reflected the C&S protocol and is very helpful to the evaluators. Two models (in Excel spreadsheet form) were provided, nearly identical in structure but significantly different in parameters used. Through further discussion with the IOUs, the evaluators have learned some of the reasons for this. In response to our request for a single statewide model, the IOUs have suggested that we use the Excel spreadsheet model provided as part of their response named EEGA 1465 et al_attachment 2_Total CS Program Savings_ PGE.4381 and known as the PG&E model. For this plan then, all savings values are based on the PG&E model. It should be noted that discussion of the specific meaning of the savings values provides has taken significant time and effort. Our request was for savings claims (to evaluate) but the IOUs have told us that these numbers are not their official claims. They have also indicated that these values can be described as the savings values for evaluation planning purposes. We expect to submit another data request in the first half of September Among the data to be requested is IOU documentation of the C&S process known as Code Change Theories and additional information on savings associated with Reach Codes and Local Government Programs (LGP). Additional requests for data from the IOUs will be made by ED, as needed, throughout this evaluation. KEMA and The Cadmus Group 63

68 5.4 Feedback to the IOUs on Key Findings ED and its advisors meet periodically usually quarterly with the IOU managers of the Statewide C&S program in a forum known as the Program Coordination Group meeting. Through this forum and other informal channels, the evaluation team has had frequent communication with the IOU program staff. This close relationship is viewed as beneficial to the program and the evaluation process due to the specialized nature of the C&S program and its evaluation protocol. This meeting has provided opportunities for the evaluation team to discuss issues such as the savings-related issues discussed in the preceding section. These discussions have been productive and the program managers have helped the evaluation to move forward. We expect that the evaluation team will continue to provide feedback to the program managers throughout the evaluation. We will also maintain objectivity and independence appropriate to evaluators. KEMA and The Cadmus Group 64

69 California Public Utilities Commission September 6, Evaluation Timeline and Work Plan Aug-11 Sep-11 Oct-11 Nov-11 Dec-11 Key Meetings and Deliverables Project Workplan 2010, 2011 Draft and Final D F Project Workplan 2012 Update F Quarterly Status Meetings Preliminary Report Draft and Final: Savings in 2010, 2011 D F Final Report Draft and Final: Savings in 2010, 2011, 2012 D F Jan-12 Feb-12 Mar-12 Apr-12 May-12 Jun-12 Jul-12 Aug-12 Sep-12 Oct-12 Nov-12 Dec-12 Jan-13 Feb-13 Mar-13 Apr-13 May-13 Jun-13 Update Integrated Standards Savings Model (ISSM) Evaluation of Savings from Building Codes, Appliance Standards (Subprograms 1, 2) Building Codes / Title 24 Potential Standards Energy Savings (unit savings, market data) Gross Standards Energy Savings (Compliance) Net Standards Energy Savings (NOMAD) Net C&S Program Savings (Attribution) Savings by Utility (Allocation) Analysis and Reporting Appliance Standards / Title 20 Potential Standards Energy Savings (unit savings, market data) Gross Standards Energy Savings (Compliance) Net Standards Energy Savings (NOMAD) Net C&S Program Savings (Attribution) Savings by Utility (Allocation) Analysis and Reporting Evaluation of Savings from the Compliance Enhancement Program (CEP)(Subprogram 3) Evaluability Assessment Data collection: interviews, research Analysis and Reporting For CEP, this schedule assumes that evaluation is primarily a process study. Evaluation of Savings from Reach Codes (Subprogram 4) Evaluability Assessment Data collection: interviews, research Analysis and Reporting The Cadmus Group, Inc. / Energy Services 65

70 California Public Utilities Commission September 6, Evaluation Budget This evaluation plan has been developed based on a total budget of $3.0M. This budget includes all of the activities to be performed by the project team. As noted above, this evaluation is planning to leverage some data collection activities that are being done for other impact evaluation projects. In these cases, this budget assumes that those activities are paid for by the other project through its work orders. Figure 2 shows the initial allocation of the budget to major evaluation activities. As noted in the preceding sections, initial funding for the Title 24 compliance task is provided for an initial small-scale effort to demonstrate the non-residential Title 24 compliance analysis approach. Funds for a larger effort have been placed in a contingency fund pending the outcome of the initial effort. Funds from other project tasks have also been placed in the contingency fund such that it represents 34% of the total budget as shown in the figure. Figure 2: Budget Detail The Cadmus Group, Inc. / Energy Services 66