Test of Several Whole-Building Baseline Models

Transcription

1 Test of Several Whole-Building Baseline Models Phillip Price Jessica Granderson Lawrence Berkeley National Laboratory September

2 Preliminary Baseline Evaluation Study Premise: statistical performance metrics can be used to evaluate automated whole-building baseline methods To determine and compare accuracy of both proprietary and open methods Objective performance assessment methodology can provide a win/win Allow vendors to retain proprietary IP underlying the algorithms Allow users to gauge performance of the tool/approach Give industry confidence needed for scaled deployment, widespread adoption 2

3 Motivation: Industry Need To sustain and increase savings, utilities must move to a whole-building focus Small savings are not sufficient; must show savings in WB site energy use Industry is not yet able to deliver cost effective whole-building M&V at scale, with accuracy and rigor IMPVP, ASHRAE Guideline 14 not specific enough, still require experts to implement Not all practitioners have the expertise, those that do produce different estimates EMIS hold promise to automate and reduce M&V costs, vendors do not yet commonly disclose the error/accuracy in baselines used to determine WB savings Site-level M&V is a common step in both gross and net savings, and important to owners 3

4 Motivation: Automated M&V Baseline model types in commercial tools Regression, bin, neural nets, weighted averaging Advantages: streamline, reduce M&V time, costs Disadvantages Lack of transparency in proprietary methods How to make adjustments if needed Standard performance metrics are commonly used Manufacturing: mean time to failure, Cpk process quality control Building systems: SEER, COP, W/sf, lumens/w Why not for energy performance tracking tools? 4

5 Goals Compare performance of Pulse Adaptive Model (PAM) to that of typical models currently in use for M&V, for a variety of buildings. (Study was funded by Pulse Energy). To the extent possible, determine whether the PAM complies with ASHRAE Guideline 14, which lists requirements for models used for M&V: No net determination bias when applied to existing data. Maximum nrmse of 0.20 = 20% for (monthly) energy use when less than 12 months of post-retrofit data are available. nrmse = normalized Root Mean Squared Error nrmse = 0.20 means the prediction for a typical month is wrong by about 20% of the average month s energy consumption 5

6 Models Compared 1. Pulse Adaptive Model (PAM) 2. Mean Week = every week is identical to the average week (MW) 3. Change-point = Temperature is the only explanatory variable (CP) 4. Day, Time, and Temperature = daily, hour-of-day, and temperature effect (DTT) 5. LBNL Regression Model = hour-of-week and temperature effects (LBNL) All of the comparison models (2-5) meet Guideline 14 s bias requirement. 6

7 Situations Considered 1. Use 6, 9, or 12 months of data to train the models 2. Predict daily, weekly, or monthly energy consumption 3. Compare results using several metrics, including: Normalized Root-Mean-Squared Error (nrmse) Median Absolute Relative Total Error Correlation of Predictions to Data

8 Example: Pulse Adaptive Model, 6 months training data, predict daily energy use. Top: black shows hourly power, red is daily average power, orange is forecast Bottom: hourly data not shown, red is daily average power, orange is forecast

9 Example 2: Pulse Adaptive Model, 6 months training data, predict daily energy use. Top: black shows hourly power, red is daily average power, orange is forecast Bottom: hourly data not shown, red is daily average power, orange is forecast

10 Comparing Models: Example Compare Pulse model to Change-point model: 9 months of training data Predict weekly energy consumption (integrated load) Plot normalized root mean squared error from each model, for each building Each building generates one data point on this plot Pulse model performs better than change-point model in all but a few buildings

11 Building-by-building model comparison PAM nrmse (y-axis) compared to LBNL regression model (x-axis) PAM was better (LBNL regression model was worse), for buildings plotted in lower right of each plot. LBNL regression was better (PAM was worse) for buildings plotted in upper left of each plot. 6, 9, or 12 months training (columns) Predicting Days, Weeks, or Months (rows) Bottom row shows relative total error for the entire predicted period. This is related to Guideline 14 compliance.

12 Summarize Model Performance Summarize the goodness-of-fit for each model (PAM, MW, CP, ), each predicted quantity (daily, weekly, monthly), and each training period Boil it down to one number for each combination of (model, quantity, training period): Example: what was the median nrmse over all of the buildings, for a given model and predicted quantity? nrmse = normalized Root Mean Squared Error nrmse = 0.20 means the prediction for a typical month is wrong by about 20% of the average month s energy consumption 12

13 Median nrmse by training period and predicted quantity, for each model

14 Observations About Model Results No single model worked best for all training periods and all buildings. Usually PAM, LBNL, and DTT were best Usually MW and CP were worst Small quantitative differences don t mean much: with a slightly different dataset the numbers, and the order of the models, might change a bit. 14

15 Study Validity Black dots show buildings where Pulse could have cheated (i.e. they had the data for the prediction period). Blue dots show buildings where Pulse could not have cheated. PAM performance was very similar to LBNL regression model for both types of buildings. There is no evidence that Pulse artificially improved their predictions for cases for which they had data.

16 PAM compliance with ASHRAE Guideline 14 LBNL cannot independently guarantee compliance. However: 1. When applied to data that are not used to fit the model, PAM shows no tendency towards bias in either direction. Also, the pattern of errors is very similar to the LBNL regression model, which is Guideline 14 compliant. 2. Even when applied to data that are not used to fit the model, PAM achieves the Guideline 14 accuracy requirement in most buildings, even though the accuracy requirement applies to fitting data that are used in the model. The bias requirement for Guideline 14 is very strict --- less than 0.005% net determination bias --- so it is possible that PAM fails to meet this requirement even though, with these data, PAM was not biased at a practically important level. Based on this sample of buildings, it is likely that PAM easily meets the Guideline 14 accuracy requirement in most buildings.

17 Conclusions The Pulse Adaptive Model, LBNL regression model, and Day-Time-Temperature models all performed roughly equally when applied to the available data. The Change-point-only model performed very badly compared to the others. The PAM model appears to be compliant or nearly compliant with ASHRAE guideline 14 (we cannot independently rule out a small bias). 12 months of training data aren t always needed: good models can be created with as little as 6 months of data (or even less) for some buildings. More months of data don t always lead to better models. An LBNL report with detailed results has been released: Evaluation of the Predictive Accuracy of Five Whole-Building Baseline Models Granderson J and Price PN, LBNL. August LBNL-5886E