Evaluation-Ready A practical approach for no savings left on the table. Lorene Flaming, PECI Keith Forsman, PG&E

Transcription

1 Evaluation-Ready A practical approach for no savings left on the table Lorene Flaming, PECI Keith Forsman, PG&E October 16, 2012

2 Introduction Genesis of the Statewide Commercial HVAC Quality Maintenance Program 2

3 3 A story from Afghanistan

4 People Matter Skills, Experience, Interest, Culture PG&E Utility s Product Manager and EM&V Specialist Energy Division and EM&V Contractor PUC PECI Contractor s Program Designer and EM&V Specialists 4

5 Ways Programs Forfeit Savings Missing i customer data Missing program theory Shrinking Savings Evaluator interviews wrong customer representative 5

6 The Solution Performance Management Plan 6

7 Why Create a Performance Management Plan? Early agreement on metrics Promote clarity and alignment. Create a richer vision. Capture critical data in real time. Forecasting Track progress against targets. and dfeedback Support continuous improvement. A sense of shared responsibility Achieve buy-in for documenting and achieving results. 7

8 Checklist: Program Design Phase Research market and regulatory context Create logic model Identify metrics (indicators & targets) Identify data and EM&V requirements Performance Management Plan drafted and vetted 8

9 Market and Regulatory Context Strategic Plan Vision for HVAC Market: Quality installation at and maintenance become the industry and market norm. EE Program Focus: Less on short-term EE activities. More on long-term market transformation activities. Standard 180 HVAC Maintenance Plans Inspection and maintenance tasks Equipment inventory Performance objectives Condition indicators Customer Reports 9

10 10

11 Example: Program Logic Activity Output Outcomes HVAC workforce development* HVAC contractors enrolled and trained. <1 year Improved skills and knowledge. 3-5 years QI/QM increases. >5 years QI/QM becomes the norm*. 11

12 Example: Program Metrics Indicators and Targets Market Effect Process Impact Indicator Indicator Indicator Number of individuals trained (technical and operational) % of leads from Utility Reps that result in a signed Maintenance Agreement. Annual gross kwh saved. 12

13 Vetting & Streamlining Metrics In (What Counts?) Short Term 1. Service Agreements and HVAC units. 2. Enrolled HVAC contractors. 3. Training courses and trainees. 4. Leads from Utility Reps that result in a Service Agreement. 5. Treatments (measures). 6. Incentives ($). 7. kwh, kw, therms saved. 8. Customers satisfied. 9. Applications processed within 1 week. Mid Term 10. Customers adopt premium measures (pursue deeper savings). Long Term 11. Customers renew a Service Agreement without incentives ( graduate ). Out Too Difficult to Track Change in unplanned service calls. Change in repair costs. Importance Unclear Funds leveraged from other sources. 13

14 Example: Data Requirements Who, What, When, Where Indicator % of leads from PG&E Reps that result in a signed Service Agreement. Data Collection Reps will provide a list of leads. Implementer will assign to Contractors through program portal, based on guidelines. Tracked in Implementer s database. Reporting Implementer s Progress Reports 14

15 Checklist: Contracting Phase Resources committed for performance management Utility: specify in RFP Bidders: identify risks and mitigation measures Bidders: budget adequate resources 15

16 Checklist: Implementation Phase Performance Management Plan (PMP) executed Familiarize team with PMP and incorporate in work plan Conduct internal reviews and early evaluation Correct and enhance design to achieve results 16

17 Example: Feedback Loops Stakeholder forums: contractor forums and Western HVAC Performance Alliance. Data QA/QC: Q trend analysis, triangulation, spot checks, instincts. Periodic internal evaluations: team meetings and data quality assessments. Early 3rd party evaluations: market effects study (Spring 2012) and impact evaluation (Fall 2012). 17

18 Example: Program Improvements Training modified in response to consistent errors. Application process streamlined. Enrollment criteria modified. New measures for heat pumps and split systems added. Questionnaire streamlined. 18

19 Lessons Learned Streamline. Start simple. Requires resources: staff time, database. Some tasks require M&V expertise. Most do not. Buy-in critical. It takes a team. Organizational values matter: transparency accountability evidence-based decision making continuous learning and improvement results etc. 19

20 Additional Information PG&E s Commercial HVAC QM Program info@commercialhvacqm.com Phone: Web: Evaluation Ready Approach Lorene Flaming Sr. Program Designer, PECI lflaming@peci.org Kerstin Rock Associate Director, PECI krock@peci.org PG&E refers to Pacific Gas and Electric Company, a subsidiary of PG&E Corporation Pacific Gas and Electric Company. All rights reserved.

21 Save the Dates Jan , 2013 Apr. 29 May 1, 2013 AESP s 23 rd National Conference & Expo Orlando, FL AESP s Spring Conference Dallas, TX For more information