Review of AEMO s 2013 National Electricity Forecasts

Transcription

1 Review of AEMO s 2013 National Electricity Forecasts A REPORT PREPARED FOR THE AUSTRALIAN ENERGY MARKET OPERATOR November 2013 Frontier Economics Pty. Ltd., Australia.

2

3 i Frontier Economics November 2013 Review of AEMO s 2013 National Electricity Forecasts Executive summary 1 Introduction Background information Scope of our review Review process 4 2 Electricity consumption forecasting Approaches to forecasting electricity consumption Desirable elements of an electricity consumption forecasting model 8 3 Review of AEMO s electricity consumption forecasting process Review of preliminary econometric models Review of revised econometric models Review of final econometric models High level review of additional modelling steps 32 4 Findings and recommendations for AEMO s electricity consumption models Overall assessment Main findings and recommendations Additional recommendations 40 5 Maximum demand forecasting Approaches to forecasting maximum demand Desirable elements of a maximum demand forecasting model 47 6 Review of maximum demand forecasting methodology Background Estimation model Forecasting model High level review of additional modelling steps 59 v Contents

4 ii Frontier Economics November Findings and recommendations for AEMO s maximum demand models Overall assessment Main recommendations for the estimation model Main recommendations for the forecasting model Additional recommendations Computational implications 65 8 AEMO s forecasting methodology compared to international best practice Summary of findings Review of forecasting procedures 72 Appendix: Glossary 81 Contents FINAL

5 November 2013 Frontier Economics iii Review of AEMO s 2013 National Electricity Forecasts Figures Figure 1: Native electricity consumption components 2 Figure 2: Scope of Frontier s review 4 Figure 3: General form of AEMO s model 12 Figure 4: Historical QLD non-large industrial energy consumption, by season 16 Figure 5: Historical NSW non-large industrial energy consumption, by season 17 Figure 6: Historical VIC non-large industrial energy consumption, by season 17 Figure 7: Historical SA non-large industrial energy consumption, by season 18 Figure 8: Historical TAS non-large industrial energy consumption, by season 18 Figure 9: Historical real income per capita 19 Figure 10: Historical residential electricity price index 20 Figure 11: Impulse response function for a shock to per capita income 25 Figure 12: Impulse response function for a shock to the electricity price 26 Figure 13: Per capita NLI energy consumption (actual and forecast) QLD 27 Figure 14: Per capita NLI energy consumption (actual and forecast) NSW 27 Figure 15: Per capita NLI energy consumption (actual and forecast) VIC 28 Figure 16: Per capita NLI energy consumption (actual and forecast) SA 28 Figure 17: Per capita NLI energy consumption (actual and forecast) TAS 29 Figure 18: General algebraic form of the Monash estimation model 53 Figure 19: Adjusted demand form of the Monash estimation model 54 Figure 20: General algebraic form of the Monash forecasting model 56 Figure 21: Economic Index 74 FINAL Tables and figures

6 iv Frontier Economics November 2013 Tables Table ES-1: Outcome of the 1 st stage review Table ES-2: Outcome of the 2 nd stage review Table ES-3: Review of the final electricity consumption models main findings and recommendations ix Table ES-4: Additional recommendations for electricity consumption models x Table ES-5: Review of the final maximum demand models main findings and recommendations xii Table ES-6: Additional recommendations for maximum demand models Table ES-7: Comparison of forecasting methodologies vi vii xiv xiv Table 1: Approaches to forecasting energy consumption 7 Table 2: Outcome of the 1 st stage review 13 Table 3: Outcomes of the 2 nd stage review 14 Table 4: Long-run model specification 21 Table 5: Results for QLD DOLS models with different numbers of lags and leads 22 Table 6: Short-run model specification 23 Table 7: Results summary 24 Table 8: Structural break test for long-run models 30 Table 9: Variance inflation check for QLD long-run model 30 Table 10: Variance inflation check for NSW long-run model 31 Table 11: Variance inflation check for VIC long-run model 31 Table 12: Variance inflation check for SA long-run model 31 Table 13: Variance inflation check for TAS long-run model 31 Table 14: Correlation between income and price 32 Table 15: Approaches to forecasting maximum demand 46 Table 16: Scaling factors applied by Monash to obtain adjusted price elasticities 59 Tables and figures FINAL

7 November 2013 Frontier Economics v Executive summary Frontier Economics (Frontier) was engaged by the Australian Energy Market Operator (AEMO) to provide an independent review of AEMO s electricity consumption and maximum demand forecasting methodologies. Forecasts produced using these methodologies were published in the 2013 National Electricity Forecasting Report (NEFR). Forecasting electricity consumption and maximum demand involves completing a number of interlinked analytical steps (e.g. data collection/preparation, model development, post-modelling adjustments). The scope of our engagement does not involve an in-depth review of all the analytical steps undertaken by AEMO and its consultants. Rather, the main focus of our review is on the methodological issues related to the selection, development and implementation of the econometric and statistical models for forecasting non-large industrial (NLI) energy consumption and maximum demand. 1 We also undertook a high level review of post-modelling adjustments. Econometric models for forecasting NLI energy consumption for each National Electricity Market (NEM) region were developed by AEMO and these models were shared with Frontier. 2 AEMO engaged the Business and Economics Forecasting Unit at Monash University (Monash) to produce the maximum demand forecasts for each region. Our review is based on the reports prepared by Monash for AEMO, as well as the description of Monash s modelling methodology in a number of published and working papers. It was outside the scope of our engagement to review computer code and data files used by Monash to produce the forecasts. Frontier undertook the review in three stages, with the first two stages running concurrently with AEMO developing the models. This arrangement allowed Frontier to review preliminary models and forecasts, and to assess whether there were any major issues that AEMO and Monash should address for the 2013 NEFR. Our findings and recommendations from these initial review stages were communicated to AEMO and Monash in a series of issue notes and meetings. These meetings provided an opportunity for the AEMO and Monash modelling teams to clarify aspects of their modelling approaches, and to discuss Frontier s suggestions and how to implement them. Some of Frontier s recommendations were able to be implemented for the 2013 NEFR, including our urgent concerns; 1 Non-large industrial (NLI) electricity consumption is defined as general mass market sales of electricity plus estimated rooftop PV generation and includes network losses, but not auxiliary loads. When modelling maximum demand, the definition of NLI demand is expanded to include auxiliary loads. 2 AEMO retained Woodhall Investment Research to assist in developing the econometric models. FINAL Executive summary

8 vi Frontier Economics November 2013 others were left for future consideration as part of AEMO s commitment to ongoing improvement of the models. The third stage of our engagement entailed reviewing the final models and forecasts for the 2013 NEFR. Review of electricity consumption models Description of electricity consumption models and review process The econometric models developed for all the NEM regions adopt a modified version of Engle and Granger s two-step approach to modelling non-stationary data. In the first step a long-run cointegrating equilibrium equation is estimated using the dynamic ordinary least squares (DOLS) approach. 3 In the second step, the lagged residuals from the DOLS equation are included in an error correction (EC) model. Electricity consumption is estimated as a function of income per capita, average retail electricity price, cooling and heating degree days (CDD and HDD), and seasonal dummy variables. 4 Forecasts are derived by substituting the projected values for the driver variables into the forecasting equations. Projections for the socio-economic drivers were provided by the National Institute of Economic and Industry Research (NIEIR). Frontier received AEMO s preliminary NLI energy consumption models for all states (except Tasmania) at the end of February Frontier s main issue with the models was that they were producing very large estimates of the short-run price elasticities, which moreover fluctuated from negative to positive across different lagged price variables. We considered those short-run price elasticity estimates to be counterintuitive, suggesting that there was something amiss with the data used in the forecasting models and/or with the structure of the forecasting models. Table ES-1: Outcome of the 1 st stage review Frontier s main finding Econometric models are likely confounding the seasonal and the price effects AEMO s response Develop seasonal error correction models Note: Based on Frontier s review of Stage 1 QLD, NSW, VIC, and SA models. 3 The DOLS approach has been shown to have better statistical performance in small samples than Engle and Granger s original approach; see Stock, J. and M. Watson (1993), A simple estimator of cointegrating vectors in higher order integrated systems, Econometrica, 61, pp AEMO s preliminary models were estimated using state final demand per capita as the income variable and the residential retail electricity price as the price variable, except for the Queensland model where the total electricity price was used as the price variable. Executive summary FINAL

9 November 2013 Frontier Economics vii In response to Frontier s review, AEMO, in consultation with Woodhall Investment Research, chose to revise its econometric models by adopting a special case of the seasonal error correction model discussed in Osborn (1993). 5 The revised models for NSW and VIC were provided to Frontier in March AEMO refers to its revised models as Integrated Dynamic Models (IDMs). Although Frontier considered the IDMs to be an improvement on AEMO s original modelling approach, we identified several issues that warranted further investigation and communicated them to AEMO. The issues identified by Frontier, and AEMO s response to them, are summarised in Table ES-2. The issues listed in this table as For the 2013 NEFR were addressed by AEMO in its final models. Table ES-2: Outcome of the 2 nd stage review Frontier s main findings Some long-run and short-run elasticity estimates seem counterintuitive and differ significantly between the two states Coefficients on some EC terms are positive Statistical tests indicate a structural break in the NSW model. This could potentially have an effect on the forecasts Price and income variables are highly correlated in the NSW model suggesting that elasticities may be estimated imprecisely. Imprecisely estimated elasticities will not have much impact on the forecasts if the correlation between income and price observed in the past continues into the future. However, if income and price move along different trends in the future, imprecisely estimated coefficients may lead to biased forecasts AEMO s responses For the 2013 NEFR, review historical load data and remove loads inadvertently included For the 2013 NEFR, refine seasonal error correction models by removing some EC terms For future NEFRs, undertake further investigation to determine the nature of the structural break, what effect the structural break has on the forecasts, and whether and how the model could be modified to account for the structural break For future NEFRs, investigate what effect the high correlation between income and price variables has on the forecasts, and whether and how the models could be modified to account for this issue Note: Based on Frontier s review of Stage 2 NSW and VIC models. 5 Osborn, D. (1993), Seasonal cointegration, Journal of Econometrics, 55, pp FINAL Executive summary

10 viii Frontier Economics November 2013 In the third stage of this review, Frontier reviewed the final electricity consumption models for all the NEM regions. 6 Table ES-3 summarises our main findings and recommendations in regard to the final model. In addition to the recommendations previously provided to AEMO (i.e. to investigate the materiality of structural breaks and the high correlation between the income and price variables for the forecasts), Frontier suggests that AEMO investigate alternative approaches to accounting for energy efficiency savings. Overall assessment The AEMO electricity consumption models possess many of the desirable elements of electricity consumption models. Of particular note is AEMO s commitment to transparency. To our knowledge, no other electricity organisation internationally, with similar responsibilities, provides as much data and detail on forecasting methodology as AEMO. There are, of course, different types of data sources available in different jurisdictions and different levels and types of available resources that affect the modelling done in a particular jurisdiction. In this context, the econometric modelling undertaken by AEMO puts it within the ranks of the more sophisticated organisations of similar type internationally. That does not mean that AEMO s models are the best possible and cannot be improved. There are always lessons to be learnt and opportunities to improve. Our comments and recommendations should be seen against this background. They are offered to assist AEMO s ongoing commitment to enhancing its forecasting methodology and transparency. Recommendations AEMO s current forecasting approach is to use an econometric model to forecast NLI electricity consumption, and then adjust the econometric forecasts for incremental energy efficiency savings which have not been captured by the econometric model. As energy efficiency savings continue to grow, it will become increasingly difficult to determine how much of the energy efficiency saving is captured by the econometric models, and how much post-modelling adjustment needs to be made. This is an issue that is challenging energy forecasters around the world, and there is no obvious solution to the overcoming this difficulty. One alternative approach that could be investigated is to model econometrically the demand for energy services rather than electricity consumption. This would involve adding estimated energy efficiency savings over the estimation period to native NLI electricity consumption prior to undertaking the econometric estimation. This should produce elasticity estimates more closely aligned with 6 As previously stated, Frontier received AEMO s preliminary NLI energy consumption models at the end of February 2013, and the revised models in March Frontier received the final models at the end of May Executive summary FINAL

11 November 2013 Frontier Economics ix economic reasoning than the present approach, since the economic drivers in the model drive the demand for energy services rather than electricity per se. We acknowledge, however, that whether or not this approach will produce more satisfactory econometric models and forecasts remains an empirical question. It was agreed that AEMO would investigate alternative approaches to account for energy efficiency savings for future NEFRs. We advise that AEMO give high priority to addressing our main recommendations summarised in Table ES-2 and Table ES-3. We recognise, however, that addressing some of these recommendations may involve significant effort in terms of the analysis and investigations that would need to be undertaken. Hence, the implementation of these recommendations may have to be phased in over time. Table ES-3: Review of the final electricity consumption models main findings and recommendations Frontier s main findings Statistical tests indicate structural breaks in several of the regional models. This could potentially have an effect on the forecasts Price and income variables are highly correlated in some other regions in addition to NSW (see Table ES-2). Hence the recommendation in Table ES-2 applies more broadly Temperature data from a single weather station in each NEM region is used to calculate the CDD and HDD variables (the explanatory variables in the econometric models). This may not adequately capture weather drivers for a region As energy efficiency savings continue to grow, it will become increasingly difficult to determine how much of the energy efficiency savings is captured by the econometric models, and how much postmodelling adjustment needs to be made Frontier s main recommendations Investigate constructing a measure of economic activity that would be better than GSP or SFD at capturing structural changes in the economy that affect electricity consumption Investigate how sensitive the forecasts are to the selection of weather stations Investigate alternative approaches to account for energy efficiency savings 1 (1) Frontier recognises that this recommendation will require significant effort in investigating alternative approaches and that the outcome is uncertain. Hence we understand that it will not be possible to implement this recommendation in the 2014 NEFR, and that its implementation in future NEFR reports will depend on the results of investigations to be undertaken in Note: Based on Frontier s review of final models for all regions. We also make a number of additional recommendations relating to model selection, estimation and assessment; and to forecasting practices. These additional recommendations are summarised in Table ES-4. We would advise AEMO to give due consideration to implementing these additional recommendations for future NEFRs. FINAL Executive summary

12 x Frontier Economics November 2013 Table ES-4: Additional recommendations for electricity consumption models Frontier s additional recommendations Provide more information on the selection of drivers tested for inclusion in the models Consider using annual data instead of quarterly data in the long-run models Investigate whether the log-log functional form is appropriate for modelling regional energy consumption or whether an alternative functional form should be used Consider estimating all the regional models together as a system of equations using the so-called seemingly unrelated regression method Consider asking large industrial users to provide forecasts under a base case scenario, and then develop a range of alternative scenarios based on independent assessments Adopt the methodologies in the 2011 Forecasting Accuracy Report and decompose past forecasting errors into contributions due to errors in projecting economic drivers and errors due to the model Note: Based on Frontier s review of final models for all regions. Review of maximum demand models Background AEMO engaged the Business and Economics Forecasting Unit at Monash University (Monash) to develop its maximum demand models. Monash s models have been developed over a number of years, and for the 2013 NEFR Monash had already undertaken to incorporate a number of improvements to its model as part of its ongoing process of model development. Hence, although we were provided with some preliminary reports on Monash s modelling for the 2013 NEFR, time constraints did not allow for any of Frontier s interim views to be incorporated in the 2013 final models. Consequently, our review of the maximum demand model did not proceed in stages, as with the electricity consumption model, but is essentially a review of Monash s final regional models. Description of maximum demand models The Monash NLI maximum demand model developed for AEMO consists of two sets of half-hourly models one set of 48 models for each half hour of a summer day; and another set of 48 models for each half hour of a winter day. Half-hourly demands are adjusted by dividing them by the average half-hourly demand for the season. 7 The logarithm of adjusted demand for a particular half hour of the day is then modelled in terms of a large number of temperature 7 Although the Monash team refers to this as adjusted demand, it is more accurately described as relative demand, since it is a ratio the ratio of actual half-hourly demand relative to average halfhourly demand. Executive summary FINAL

13 November 2013 Frontier Economics xi variables, as well as a day of the week effect, holiday effects, and a day of season effect. Each of the temperature variables and the day of season effect is modelled as a cubic regression spline. The day of the week and holiday effects are captured using dummy variables. The splines are then combined additively in the model. 8 In a standard econometric model, forecasts are derived by substituting projected future values for the drivers in the model, and then calculating the corresponding values of the dependent variable. The Monash maximum demand forecasting model is more sophisticated than that. Instead of providing a point forecast for peak demand for each year in the forecast horizon, it provides the full density function of future peak demands. Bootstrap simulation techniques are used to produce numerous different synthetic future half-hourly demand trajectories for each forecast year by varying temperature conditions and PV loads, and adding some additional random noise to the temperature profile. The results of these simulations can be used to forecast maximum demand at different probability of exceedance (POE) levels. Overall assessment The Monash peak demand model possesses many of the desirable elements of a peak demand model. The modelling of the weather variables is very detailed and sophisticated. The modelling of the calendar effects also seems appropriate. Further, we believe that the Monash team s approach to calculating the distributions of maximum demand and probability of exceedance (POE) levels is statistically sound. We are not aware of any other energy organisation in Australia that has forecasting models that are technically as sophisticated as AEMO s maximum demand models, and internationally this level of sophistication in maximum demand modelling is rare. Against this background, our recommendations for the maximum demand model should therefore be seen as part of the continuing model improvement process that Monash is engaged in. Recommendations Our main recommendations are summarised in Table ES-5. We advise that Monash give high priority to addressing these recommendations and attempt to incorporate them in the 2014 NEFR. We recognise, however, that some the 8 A model of this type is called semi-parametric, because the use of cubic splines allows each temperature variable to have a very flexible nonlinear impact on maximum demand. Despite their name, such models are, in fact, highly parameterised in the sense that they contain many parameters that need to be estimated. FINAL Executive summary

14 xii Frontier Economics November 2013 recommendations may involve significant effort in terms of analysis and investigations, and may have to be implemented over a longer period. 9 Table ES-5: Review of the final maximum demand models main findings and recommendations Frontier s main findings Efforts should be made to overcome the need for post-modelling bias correction for extreme temperatures Graphs in Monash s load factor report show evidence of a change in the load factors around 2006 to 2007 Historical average demands used in calculating the adjusted demands (i.e. the dependent variable in the semiparametric models) are not temperature corrected There are some undesirable consequences of Monash s current approach to adjusting price elasticities Models used to underpin elasticity adjustments are different to AEMO s annual electricity consumption models and do not appear to address possible non-stationarity in the data Frontier s main recommendations Investigate alternative specification of cubic splines to better capture maximum demand at extreme temperatures to overcome need for post-modelling bias correction Undertake formal statistical tests to investigate structural breaks in the regional maximum demand models and their effect on the modelling results Investigate temperature-correcting the average demand used as a base value for calculating the adjusted demands to ensure that these base values are comparable across years with respect to temperature Develop an explicit statistical/economic specification to provide a coherent framework for how price elasticities vary across different time periods and different levels of demand. This framework should also allow for longterm changes in load factors Revise methodology used to make elasticity adjustments to bring it in line with annual electricity consumption models Note: Based on Frontier s review of final maximum demand models for all regions. We elaborate on each of these recommendations. Monash s estimated semi-parametric models for different regions appear to have some bias at extreme high or low temperatures. 10 The current approach for dealing with this bias is to make a post-modelling adjustment to remove the bias. An alternative approach would be to re-visit the way Monash 9 In response to a draft version of this report, Monash has agreed to take on board many of the issues raised in our report for the next round of modelling. 10 See Figure 14 in Monash s regional reports. Executive summary FINAL

15 November 2013 Frontier Economics xiii specifies the cubic splines in its semi-parametric models and give more weight to the fit of these splines at extreme temperatures. We recommend that Monash investigate whether this approach can reduce or eliminate the need for the post-modelling adjustment. Our review of AEMO s electricity consumption models provided strong evidence of structural breaks in four of the NEM regions around 2006 and The graphs in Monash s load factor report similarly show evidence of a change in the load factors at around the same time. We recommend that for the future NEFRs Monash undertake formal statistical tests to investigate structural breaks in the regional maximum demand models and their effect on the modelling results. The average demands used by Monash to calculate the adjusted demands (i.e. the dependent variable in its semi-parametric models) are not temperature corrected. We recommend that Monash investigate whether temperature correcting the average demands improves the performance of its forecasting models. When calculating the impact of future price changes on future demand, Monash makes an adjustment to the price elasticities in the annual electricity consumption models to allow for the fact that price response at high levels of demand may be different to price response at average demand levels. While we commend Monash s efforts to incorporate different price elasticities in its forecasts for peak demand and average demand, the current approach has some undesirable consequences resulting from the fact that the scaling factor is applied to the average seasonal demand component of the model, rather than to high demand levels per se. Two implications of Monash s conclusion that price elasticities differ according to the level of demand are: (1) the convenient decomposition of half-hourly demand into an average demand component driven by economic variables, and a short-term component driven by temperature and calendar variables is no longer tenable; and (2) the price elasticity is endogenous. We recommend that in order to guide the further development of the elasticity adjustment, Monash develop an explicit statistical/economic specification to provide a coherent framework for how price elasticities vary across different time periods and different levels of demand. Monash s current approach to incorporating different price elasticities in its forecasts for peak demand and average demand, involves estimating average demand models that: (1) are different to those developed by AEMO for forecasting annual electricity consumption, and (2) do not pay as much attention to possible non-stationarity in the data. We recommend that the models used by Monash to underpin its elasticity adjustments be brought in line with the annual demand models. FINAL Executive summary

16 xiv Frontier Economics November 2013 We make a number of additional recommendations that Monash could explore, summarised in Table ES-6. Table ES-6: Additional recommendations for maximum demand models Frontier s additional recommendations Augment the model evaluations by calculating within-sample point backcasts (rather than POEs) using the actual values for all the drivers including temperatures Provide regression results from the estimated models (i.e. estimated coefficients for all variables included in the models) Investigate methods to incorporate causes of uncertainty other than temperature (e.g. uncertainty in price and economic activity) Develop whole-year annual POEs across seasons, in addition to POEs by season Note: Based on Frontier s review of final maximum demand models for all regions. Comparison of forecasting methodologies As part of this engagement Frontier compared AEMO s forecasting methodology to forecasting methodologies used by organisations with similar responsibilities as AEMO, including: PJM Interconnection (United States), Alberta Electric System Operator (Canada), Transpower (New Zealand), and National Grid Electricity Transmission (United Kingdom). Due to a lack of information made publicly available by those organisations, it was not possible to undertake a thorough comparison of forecasting methodologies (including inputs, assumptions, and modelling techniques). Our review was therefore limited to a high level comparison of specific analytical steps. Our finding are summarised in Table ES-7. Table ES-7: Comparison of forecasting methodologies Elements of forecasting methodology Transparency Forecasting horizon Electricity consumption modelling Frontier s findings AEMO is leading the way in terms of transparency of its forecasting process AEMO produces electricity consumption and maximum demand forecasts for a 20-year horizon, which is within the range of forecasting horizons of other organisations The dominant methodology for forecasting electricity consumption is the top down econometric (i.e. regression) approach with economic variables as the main drivers. This is the approach used by AEMO Executive summary FINAL

17 November 2013 Frontier Economics xv Elements of forecasting methodology Maximum demand modelling Economic activity driver Handling of uncertainty Model validation and testing Projections for economic and demographic drivers Frontier s findings AEMO s modelling approach is most similar to that used by PJM Interconnection, a regional transmission organisation in the United States. Both use sophisticated models which fall into the category of frequency distribution techniques, with future synthetic weather scenarios created through Monte Carlo simulations of past weather conditions In its regional electricity consumption models, AEMO defines economic activity in terms of GSP per capita or SFD per capita. Of the reviewed organisations, PJM uses the most elaborate economic variable, namely, a weighted combination of U.S. GDP, Gross Metropolitan Product, personal income, population, households, and non-manufacturing employment, with the shares of electricity sales by customer class used as weights AEMO handles economic uncertainty in its electricity consumption forecasts by considering three different economic scenarios. Uncertainty in future maximum demand is handled by combining the economic scenario analysis with Monte Carlo simulations of future weather scenarios. These are widely accepted approaches to handling uncertainty; one or both of these approaches are used by most of the reviewed organisations and can be regarded as standard industry practice The reviewed organisations do not provide enough technical information on their methodology and forecasting accuracy to make a detailed comparison with AEMO AEMO currently sources all historical and projected economic and demographic data from a single source. For the 2013 NEFR, AEMO assessed projections for regional electricity prices and GSP/SFD for reasonableness against forecasts from a number of independent sources. Many of the reviewed organisations validate the economic projections used in their forecasts against data from other public or commercial sources, and/or source such data from multiple data providers FINAL Executive summary

18

19 November 2013 Frontier Economics 1 1 Introduction Frontier Economics (Frontier) was engaged by the Australian Energy Market Operator (AEMO) to provide an independent review of AEMO s electricity consumption and maximum demand forecasting methodologies. 11 The methodologies are used to produce annual electricity consumption and maximum demand forecasts published in the 2013 National Electricity Forecasting Report (NEFR). The main objectives of this independent review process are to: identify practical ways in which the econometric and statistical models for forecasting energy consumption and maximum demand could be improved; and determine whether the forecasting procedures employed are in accordance with best international practice. 1.1 Background information There are several definitions of electricity demand and customer segments that are relevant for understanding AEMO s modelling approach and the scope of Frontier s engagement. Native demand refers to the total electricity generated in a region by local scheduled, semi-scheduled, and non-scheduled generating units plus the imports to the region. This can be defined at different levels of the electricity network. As generated demand is the energy measured at the terminals of generating units, sent out demand is the energy measured at the generator connection points to the grid, and as consumed demand is the energy measured at the distribution connection points. A simplified schematic representation of native demand is presented in Figure 1. The white boxes indicate the different components of native demand which AEMO models separately in order to isolate their underlying trends and main drivers. In the 2013 NEFR, electricity consumption forecasts are presented on a sent out basis, while maximum demand forecasts are presented on an as generated basis. The majority (about 79 per cent) of the end-use consumption supplied from the network is accounted for by mass market customers, which include residential, 11 The terms electricity consumption and energy consumption are used interchangeably when discussing annual or quarterly electricity energy consumption. Economists often refer to this as electricity or energy demand. However, we will reserve the term demand for the maximum demand models when discussing half-hourly demand and maximum demand. FINAL Introduction

20 2 Frontier Economics November 2013 commercial and small industrial customers. 12 Increasingly this customer segment has been substituting energy supplied from the network with rooftop photovoltaic generation (PV) and other non-traditional sources. AEMO postulates that mass market customers demand for energy supplied from the network and from PVs is driven by the same economic factors (i.e. electricity price and income) and hence models them together. AEMO refers to general mass market sales of electricity plus estimated PV generation (and including distribution losses) as non-large industrial (NLI) electricity consumption, and models it using its econometric models. When modelling maximum demand, AEMO expands the definition of NLI demand to include transmission losses and auxiliary loads. Figure 1: Native electricity consumption components Note: Mass market sales include distribution losses. Source: Frontier Economics, adapted from AEMO (2012) Regional Demand Definition. Econometric models for forecasting NLI electricity consumption for each National Electricity Market (NEM) region were developed by AEMO with the assistance of Woodhall Investment Research, AEMO s energy forecasting external advisor. These models, and the data inputs, were shared with Frontier. To produce the maximum demand forecasts for each of the regions, AEMO engaged the Business and Economics Forecasting Unit at Monash University (Monash). Our review of the maximum demand models is based on the reports prepared by Monash for AEMO, as well as the description of Monash s modelling methodology in a number of published and working papers. It was outside the scope of our engagement to review the computer code and data files used by Monash to produce the forecasts. 12 The mass market consumption share of 79 per cent is for the National Electricity Market (NEM) as a whole, and was calculated over for the period to Introduction FINAL

21 November 2013 Frontier Economics Scope of our review Forecasting energy consumption and maximum demand involves completing a number of successive and interlinked analytical steps. A simplified schematic representation of the analytical steps involved in developing AEMO s forecasts is presented in Figure 2. Since the required analytical steps are interlinked, both the chosen methodology and the execution of any one of the steps will have an effect on the final forecasts. The scope of our engagement does not involve an in-depth review of all the analytical steps undertaken by AEMO and its consultants. Rather, the main focus of our review is on the methodological issues related to the selection, development and implementation of the econometric and statistical models for forecasting NLI energy consumption and maximum demand. As part of this review, we also undertook a high level review of: AEMO s methodology for forecasting small non-scheduled generation and large industrial loads AEMO s methodology for adjusting the econometric/statistical energy and maximum demand forecasts to reflect factors that could not be effectively captured in the models, such as energy efficiency the reasonableness of the projections for the impact of energy efficiency policies and PV generation. Outside the scope of this review are: AEMO s methodology for collecting load and weather data, and preparing those data for the analysis (e.g. ensuring that NLI electricity consumption was defined in a consistent manner throughout the modelling period; or ensuring that weather stations used to model electricity consumption in a particular NEM region are representative of weather conditions in that entire NEM region) the methodology for projecting the key drivers of electricity consumption, i.e. electricity prices, and Gross State Product (GSP) or State Final Demand (SFD), and the projections themselves. 13 AEMO sources these projections from the National Institute of Economic and Industry Research (NIEIR) the methodologies used by AEMO to calculate and project the impact of energy efficiency policies and PV generation. 13 AEMO s electricity consumption models for some of the regions have GSP as the economic activity driver, while for other regions they have SFD as the economic activity driver. FINAL Introduction

22 4 Frontier Economics November 2013 Figure 2: Scope of Frontier s review Note: When modelling electricity consumption, non-large industrial (NLI) energy consumption is defined as general mass market sales of electricity plus estimated rooftop PV generation, and includes distribution losses. When modelling maximum demand, the definition of NLI demand is expanded to include transmission losses and auxiliary loads. Source: Frontier Economics. 1.3 Review process Frontier undertook the review in three stages, with the first two stages running concurrently with AEMO s development of the models. 14 In the first stage, we conducted a preliminary review of the draft modelling approaches and forecasts to assess whether there were any major issues that should be addressed for the 2013 NEFR in regard to the AEMO and Monash models, as well as the postmodelling adjustments. Our findings and recommendations from this initial review stage were communicated to AEMO and Monash in a series of issue notes and in-person meetings. In the second stage, we reviewed the revised models, which addressed some of Frontier s main findings and recommendations, and provided additional suggestions for model improvement. The third stage of our engagement entailed reviewing the final models and forecasts, and providing recommendations for further model improvement. The remainder of the report is organised as follows: In section 2, we provide some background information on modelling electricity consumption In section 3, we review AEMO s electricity consumption models 14 Throughout the review process, AEMO organised meetings at which AEMO s modellers, Woodhall Investment Research and the Monash team, provided additional information and clarifications to Frontier. Introduction FINAL

23 November 2013 Frontier Economics 5 In section 4, we discuss our major finding and provide recommendations for improving AEMO s electricity consumption models In section 5, we provide some background information on modelling maximum demand In section 6, we review of Monash s maximum demand models In section 7, we discuss our major findings and provide recommendations for improving the maximum demand models In section 8, we review the forecasting methodologies used by AEMOequivalent organisations in a number of countries to provide an international context for AEMO s methodology. In the Appendix we provide a glossary of terms used in the report. FINAL Introduction

24

25 November 2013 Frontier Economics 7 2 Electricity consumption forecasting 2.1 Approaches to forecasting electricity consumption There are a number of approaches to electricity consumption forecasting that vary in their analytical sophistication and data requirements. They fall into several broad categories, which we summarise in Table 1. The delineation between the approaches is not always clear-cut, and many forecasting processes combine two or more of the approaches and/or use models that are hybrids of two or more of the approaches. Furthermore, the final forecasts obtained from the models are typically fine-tuned using post-modelling adjustments to incorporate expert knowledge or insight about factors that could not been captured in the modelling process. Table 1: Approaches to forecasting energy consumption Approaches Top-level econometric models Bottom-up models Statistical (time series) forecasting methods Description Use economic, price, socio-demographic and weather variables to forecast electricity consumption Employ data from end-use studies at the customer and appliance level, sometimes also incorporating economic, price, socio-demographic and weather variables Rely on the trends and dynamics in the historical consumption data to forecast electricity consumption Where explanatory driver variables are used in the forecasting process, the following four steps are typically part of the process: 1) Develop one or more forecasting equations that specify the relationships between the variable(s) to be forecast (energy consumption in the present case) and driver variables (e.g. economic activity, price, temperature, technological change) that influence the variable(s) to be forecast. The driver variables are also referred to as inputs, the variables to be forecast as outputs. 15 2) Provide projections for the driver variables. 15 Other names used for the input variables are independent variables or covariates. The output variables are also known as dependent variables. Electricity consumption forecasting

26 8 Frontier Economics November ) Substitute the projected values for the driver variables into the forecasting equation(s) to calculate the forecasts. 4) Make post-modelling adjustments (sometimes called overlays) for any factors that could not be included directly in the forecasting equation(s), such as government policies affecting energy efficiency, emerging technologies, etc. 2.2 Desirable elements of an electricity consumption forecasting model The dominant approach to forecasting electricity consumption for the purpose of regulatory reviews is the top-level econometric approach using annual or quarterly data. This is the approach used by AEMO to forecast electricity consumption by non-large industrial (NLI) customers. In this subsection we discuss the desirable elements of such a model. Dependent variable The dependent variable in the model should capture as closely as possible the electricity usage measure that the customer makes decisions about. Economic drivers Econometric electricity consumption models typically include the following economic drivers: Economic activity a variable that captures the level of economic activity, such as production, total disposable income, number of customers, number of households, and gross state product (GSP) Own price a price variable that is as close as possible to the marginal cost of electricity Price of substitute product a price or price index for substitutes to electricity, e.g. natural gas. All these drivers should be linked as closely as possible to the customer segment whose electricity consumption is being modelled. Other independent variables Weather is an important non-economic driver of electricity consumption. The most commonly used variables to capture weather conditions are cooling degree days (CDD) for summer, and heating degree days (HDD) for winter. The reference temperatures for the CDD and HDD variables may vary from jurisdiction to jurisdiction, but are typically between 16 C and 18.5 C for HDD, and between 18.5 C and 22 C for CDD. Electricity consumption forecasting

27 November 2013 Frontier Economics 9 Other non-economic variables, such as household size or dwelling size for the residential sector, are sometimes also included as drivers. Such variables are often highly correlated with other drivers of consumption, and some judgement is typically required when including them in a model to avoid multicollinearity issues. In addition, dummy variables might be appropriate if unusual events have taken place during the estimation period whose impact is not captured by other drivers in the model; for example, to capture the impact of hosting the Olympics. Dynamics Economists recognise that consumers do not respond instantly to changes in the driver variables. For example, a household may wait several years after it becomes cost-effective before switching from an electric water heater to a gas one. Hence, at any point in time, the relationship between electricity consumption and its drivers is in disequilibrium. Dynamic models attempt to capture the constant adjustment to a shifting equilibrium relationship, enabling the analyst to distinguish between short and long-run responses to a change in an independent variable. Non-stationarity Over the past two decades, it has become standard practice in the econometric modelling of time series data to test whether the variables in the model are stationary. The use of ordinary least squares (OLS) regression with nonstationary variables can produce invalid inferences, since the usual probability values associated with t-tests and F-tests are incorrect. OLS estimation of models involving non-stationary variables could produce spurious regressions with high R 2 values and high t-statistics but with no economic meaning. If some of the variables in the model are non-stationary, one should test whether the variables are cointegrated. If the variables are cointegrated, it is common to estimate both the cointegrating long-run equilibrium relationship and a so-called error correction (EC) model showing how electricity consumption returns to equilibrium over time. If there is no cointegrating relationship, the common approach is to include the quarter-to-quarter or year-to-year differences of any non-stationary variables in the model, rather than the actual values themselves. Scenario analysis A forecasting model should also allow alternative scenarios to be investigated; scenario analysis leads to an appreciation of the uncertainties involved in the Electricity consumption forecasting

28 10 Frontier Economics November 2013 forecasts. This implies that the model should include any drivers for which one may wish to undertake a scenario analysis. 16 Post-modelling adjustments Any econometric or statistical forecasting model can only capture relationships that have been observed historically. The impact of recent and upcoming policy initiatives and technological changes likely to influence demand during the forecast horizon, will not be captured in the estimated model. It is common to adjust the forecasts produced by formal models to take into account such expected changes. Such post-modelling adjustments should be based on explicit assumptions and calculations rather than mere guesses, so that when new data become available, the cause of any discrepancy between forecast and actual adjustments can be identified. 16 Sometimes, the driver that one wants to use in a scenario analysis does not have a statistically significant impact on demand during the estimation period. In that case, one may need to give the coefficient of the driver a value based on expert opinion or a review of the literature. Electricity consumption forecasting

29 November 2013 Frontier Economics 11 3 Review of AEMO s electricity consumption forecasting process As explained in section 1, AEMO models different components of native electricity consumption separately in order to isolate their underlying trends and main drivers. The main focus of our review is on AEMO s econometric models for forecasting energy consumption by NLI customers at the sent-out level in each of the NEM regions. Frontier undertook the review of AEMO s econometric models in three stages, with the first two stages running concurrently with AEMO developing the forecasts. This staged approached allowed AEMO to implement a number of Frontier s recommendations in the 2013 NEFR. This section is organised as follows. In section 3.1, we describe AEMO s initial econometric models and the main issue raised by Frontier. In section 3.2, we explain what revisions were undertaken by AEMO in response to Frontier s concerns and describe the specification of the revised econometric models. We keep these two sections brief, and discuss many of the salient modelling issues in section 3.3, where we provide a detailed review of AEMO s final econometric models. In section 3.4, we provide a high level review of some other modelling steps undertaken by AEMO, namely, post-modelling adjustments to the econometric forecasts, forecasting of small non-scheduled generation, and forecasting of rooftop PV generation. 3.1 Review of preliminary econometric models Frontier received AEMO s preliminary NLI models for all NEM regions (except Tasmania) at the end of February Although the aim of the models is to provide forecasts for annual electricity consumption, the econometric models were estimated using quarterly data for the period 2000Q1 to 2012Q4. 17 Using quarterly data increases the sample size available for estimating the model; the trade-off being that one has to explain the seasonal pattern in electricity consumption. Annual electricity consumption forecasts can be obtained from a quarterly model by summing up the forecasts for the four quarters in the year. The econometric software package used for the estimation was EViews, one of the most widely used packages for econometric time series analysis. Each model was specified in a cointegrating framework with a Dynamic Ordinary Least Squares (DOLS) specification for the long-run relationship, combined with 17 For Tasmania, the dataset starts at 2002Q1. Review of AEMO s electricity consumption forecasting process

30 12 Frontier Economics November 2013 an Error Correction (EC) model to capture short-run dynamics. 18 This approach produces both the long-term equilibrium relationship between electricity consumption and its drivers, and the short-term adjustment process to move from a short-term disequilibrium towards the long-term equilibrium. The general form of the model is presented in Figure 3. If the long-term relationship is estimated using DOLS, then additional variables are added to the long-run equation, namely, the changes in the drivers from period to period, and lags and leads in these changes. 19 Figure 3: General form of AEMO s model Long-run equilibrium relationship Error correction model with adjustment to equilibrium where,,,, and are coefficients to be estimated, and and are random error term. The estimated coefficients on the differenced variables (denoted with the operator) represent short-run elasticities, while the estimated coefficients on the variables in levels (the parameters in the parentheses) represent long-run elasticities. Source: Frontier Economics. Frontier s main issue with AEMO s preliminary 2013 models was that for some regions they produced very large estimates of the short-run price elasticities, which moreover fluctuated between negative and positive across different lags. We considered those short-run elasticity estimates to be counterintuitive, suggesting that there was something amiss with the data used and/or with the structure of the forecasting models. 18 We elaborate on DOLS/EC modelling in sections to A discussion of alternative econometric modelling approaches is provided in AEMO s 2012 Forecasting Methodology Information Paper. For the 2012 NEFR, AEMO was not able to develop satisfactory DOLS/EC models for QLD and VIC, and for these regions AEMO developed Autoregressive Distributed Lag (ARDL) models instead. For further reading on econometric time series modelling see Franses, P. (1998), Time Series Models for Business and Economic Forecasting, Cambridge, and Maddala, G. and I-M Kim (1998), Unit Roots, Cointegration and Structural Change, Cambridge. 19 For a review of methods for estimating the cointegrating relationship, some of which do not involve an EC model, see Lim, G. and V. Martin (1995), Regression-based cointegration estimators with applications. Journal of Economic Studies, 20(1), pp Review of AEMO s electricity consumption forecasting process

31 November 2013 Frontier Economics 13 Frontier s findings prompted AEMO to modify its econometric models with more emphasis given to capturing the seasonality in the data. Table 2: Outcome of the 1 st stage review Frontier s main finding Econometric models seem to be confounding seasonal and short-run price effects AEMO s responses Develop seasonal error correction models Note: Based on Frontier s review of Stage 1 QLD, NSW, VIC, and SA models. 3.2 Review of revised econometric models Frontier was provided with revised models for NSW and VIC at the end of March The models addressed Frontier s main concern with the preliminary models. In the revised models, AEMO (in consultation with Woodhall Investment Research) adopted a special case of the seasonal error correction model (SECM) discussed in Osborn (1993). 20 Whereas in the general SECM, the long-run equilibrium relationship between the variables can vary by season, in AEMO s model the long-run relationship is the same for all seasons. In the short-run part of the model, AEMO included the error correction term lagged 1, 2, 3 and 4 quarters; that is, the coefficient on the EC term differed by season. AEMO refers to its models as Integrated Dynamic Models (IDMs). We note that modelling seasonal data in an error correction framework is considerably more complex than modelling non-seasonal data. According to one of the leaders in the field: Cointegration for seasonal time series processes remains a somewhat perplexing issue. 21 One of the alternative ways of modelling cointegrated seasonal data is the periodic error correction model (PECM) discussed in Franses & Kloek (1995). 22 The PECM imposes the same long-run elasticities for all quarters, as in AEMO s long-run model; however, instead of including 4 lags of the EC term in the shortrun model, it includes residual term from the previous year, EC(-4), and allows the adjustment coefficients to vary by season. We suggested that this alternative 20 Osborn, D. (1993), Seasonal cointegration, Journal of Econometrics, 55, pp Osborn, D. (2002), Cointegration for seasonal time series processes, Working Paper, School of Economic Studies, University of Manchester. 22 Franses, P. & T. Kloek (1995), A periodic cointegration model of quarterly consumption, Applied Stochastic Models and Data Analysis, 11, pp Review of AEMO s electricity consumption forecasting process

32 14 Frontier Economics November 2013 specification be one of the variants explored in future development of AEMO s forecasting methodology. Although Frontier considered the IDM models to be an improvement on AEMO s original modelling approach, we identified several issues that warranted further investigation (acknowledging that not all of the issues raised could be addressed in the 2013 NEFR). Based on our review and further internal data investigations, AEMO revised its econometric models. The outcome of the second review stage is summarised in Table 3. Table 3: Outcomes of the 2 nd stage review Frontier s main findings The magnitude of the estimated long-run price and income elasticities differed considerably between the two states. Although we would expect some difference between the states due to variations in the mix of residential, commercial and small industrial (which compose the so called NLI sector), it was puzzling that the estimated long-run income elasticity for NSW was twice the size of that for VIC (0.56 compared to 0.29) The small size (-0.02) of the short-run price elasticity for VIC was questionable, indicating minimal response to price changes in the short-run. For comparison, the estimated short-run price elasticity for NSW was To investigate the reasonableness of these estimated parameters, we estimated impulse response function for both states. The results provided some assurance that the estimated short-run effects, although somewhat erratic and smaller than expected, did not lead to extreme impacts. The impulse response functions indicated that after a shock, electricity consumption would revert to a long run state within a reasonably short time period of up to about 10 quarters Coefficients on two of the EC terms in the NSW model, and on one EC term in the VIC model were positive. This is counterintuitive, since a positive coefficient would take consumption away from the long-run level. While the VIC coefficient, and one of the NSW coefficients were not statistically significant, the EC term for the lag in the AEMO s responses AEMO reviewed historical load data for the 2013 NEFR to ensure consistency in allocating customers to the NLI sector across the estimation period. AEMO will review and monitor estimated elasticities for each of the states for all future NEFRs and assess whether the elasticities are reasonable Modelling changes made in response to the 1 st stage review addressed the seasonality in the data. As part of this process, impulse response functions were also developed to track the speed at which energy consumption would revert to a long-run state. The estimated short-run elasticities differed between states. AEMO will monitor and track these estimates for future NEFRs to ensure they are reasonable AEMO refined the seasonal error correction models for the 2013 NEFR by removing some EC terms Review of AEMO s electricity consumption forecasting process

33 November 2013 Frontier Economics 15 Frontier s main findings NSW model was both statistically significant and of reasonable size Results of a Quandt-Andrews (Q-A) test for unknown structural break indicated that the NSW model has a statistically significant break point at 2007Q2 The NSW model was estimated using real average residential electricity price and real state final demand per capita. Price and income variables were highly correlated in the NSW model suggesting that elasticities may be estimated imprecisely. Imprecisely estimated elasticities will not have much impact on the forecasts if the correlation between income and price observed in the past continues into the future. However, if income and price move along different trends in the future, imprecisely estimated coefficients could lead to biased forecasts There was a computer coding error and a sample period definition issue (i.e. data for the first quarter in the sample period was inadvertently excluded from the analysis) AEMO s responses AEMO agreed to undertake further investigations for future NEFRs to determine the nature of the structural break, what effect the structural break has on the forecasts, and whether and how the model could be modified to account for the structural break For future NEFRs, AEMO agreed to investigate what effect the high correlation between income and price variables has on the forecasts, and whether and how the models could be modified to account for this issue AEMO corrected the coding and sample period definition errors Note: Based on Frontier s review of Stage 2 NSW and VIC models. 3.3 Review of final econometric models Frontier received AEMO s final econometric models for all NEM regions at the end of May 2013, and supporting materials throughout June Basic checks There are a number of basic checks we have undertaken as part of our quality assurance process The same basic checks were undertaken when reviewing AEMO s models in the first and second stages of the review process. For brevity, we only discuss our review of the final models. Review of AEMO s electricity consumption forecasting process

34 Non-large industrial energy consumption per capita (in kwh) 16 Frontier Economics November 2013 Input data We have checked that the socio-economic input data used in EViews for model development and forecasting corresponds to the NIEIR data provided to us in Excel files. The socio-economic data are taken from the qmc05 spreadsheet in the respective NIEIR Excel file for each state, which contains the quarterly data for the base case future scenario. The population and income data used in the EViews models match exactly the data provided by NIEIR. Annual electricity prices are provided in NIEIR s annual data sheets. We were informed that AEMO used a spline interpolation method to obtain quarterly prices. This is a reasonable approach to obtaining quarterly price data, but it could produce quarterly price series that are smoother over time than the actual prices. The electricity consumption data used in the econometric models are presented in Figure 4 to Figure 8 on a per capita basis by season. There are no obvious outliers in these data that could affect the modelling outcomes. Figure 4: Historical QLD non-large industrial energy consumption, by season Q1 Q2 Q3 Q4 Note: Non-large industrial energy consumption is the load on the network from residential, commercial and small industrial customers plus estimated rooftop PV generation. Review of AEMO s electricity consumption forecasting process

35 Non-large industrial energy consumption per capita (in kwh) Non-large industrial energy consumption per capita (in kwh) November 2013 Frontier Economics 17 Figure 5: Historical NSW non-large industrial energy consumption, by season Q1 Q2 Q3 Q4 Note: Non-large industrial energy consumption is the load on the network from residential, commercial and small industrial customers plus estimated rooftop PV generation. Figure 6: Historical VIC non-large industrial energy consumption, by season Q1 Q2 Q3 Q4 Note: Non-large industrial energy consumption is the load on the network from residential, commercial and small industrial customers plus estimated rooftop PV generation. Review of AEMO s electricity consumption forecasting process

36 Non-large industrial energy consumption per capita (in kwh) Non-large industrial energy consumption per capita (in kwh) 18 Frontier Economics November 2013 Figure 7: Historical SA non-large industrial energy consumption, by season Q1 Q2 Q3 Q4 Note: Non-large industrial energy consumption is the load on the network from residential, commercial and small industrial customers plus estimated rooftop PV generation. Figure 8: Historical TAS non-large industrial energy consumption, by season Q1 Q2 Q3 Q4 Note: Non-large industrial energy consumption is the load on the network from residential, commercial and small industrial customers plus estimated rooftop PV generation. Figure 9 and Figure 10 show the economic explanatory variables used in the models; income per capita and electricity price. In the QLD and VIC models, the income variable is real Gross State Product (GSP) per capita; while in NSW, SA Review of AEMO s electricity consumption forecasting process

37 2000Q1 2000Q4 2001Q3 2002Q2 2003Q1 2003Q4 2004Q3 2005Q2 2006Q1 2006Q4 2007Q3 2008Q2 2009Q1 2009Q4 2010Q3 2011Q2 2012Q1 2012Q4 Income per capita November 2013 Frontier Economics 19 and TAS model, the income variable is real State Final Demand (SFD) per capita. In all regional models, except SA, the price variable is the real average retail electricity price for residential and business customers. In the SA model, the price variable is the real average retail residential electricity price. In the 2013 Forecasting Methodology Information Paper, AEMO explains that its choice of explanatory variables was based on the statistical significance of candidate variables, the intuitiveness of the estimated coefficients, and the results of diagnostic tests. Both the income and price series contain strong positive trends which are suggestive of non-stationarity. The price series are much smoother than the income series, most likely due to the interpolation used to obtain quarterly prices from NIEIR s annual data. Figure 9: Historical real income per capita QLD NSW VIC SA TAS Note: income = GSP/capita in QLD and VIC models; income = SFD/capita in NSW, SA and TAS models. Review of AEMO s electricity consumption forecasting process

38 Residential electricity price index (2009/10=1) 20 Frontier Economics November 2013 Figure 10: Historical residential electricity price index QLD NSW VIC SA TAS Note: We graph residential electricity price index to provide some indication of price movements over the modelling period Model structure All the regional models have the same basic structure, which is a modified version of Engle and Granger s two-step approach to modelling non-stationary data. In the first step a long-run cointegrating equilibrium equation is estimated using the dynamic ordinary least squares (DOLS) approach. The DOLS approach has been shown to have better statistical performance in small samples than Engle and Granger s original approach. In the second step, the lagged residuals from the DOLS equation (lagged 4 quarters) are included in a so-called seasonal error correction model (SECM) that models how short-term deviations from the long-term equilibrium path return back to equilibrium Specification of the long-run DOLS equations In Table 4 we summarise the specification of the long-run DOLS equilibrium equations. 24 As previously explained, AEMO refers to its models as Integrated Dynamic Models (IDMs). Review of AEMO s electricity consumption forecasting process

39 November 2013 Frontier Economics 21 Table 4: Long-run model specification Dependent variable Independent variables Real income per capita 1 Electricity consumption in kwh per capita 1 state final demand per capita (NSW, SA, TAS) gross state product per capita (QLD, VIC) Real average retail electricity price (in c/kwh) 1 electricity price for residential and business customers (QLD, NSW, VIC, TAS) electricity price for residential customers (SA) Seasonal dummy variables (base = first quarter) Heating degree days (NSW, VIC, SA, TAS) Cooling degree days (QLD,NSW,VIC,SA) 1) Variables enter the model in logarithms. Methodology The DOLS methodology augments the independent variables in the long-run equation with the differences of the income and price variables, as well as lags and leads of the differences. In the NSW, VIC and TAS models only the current differences of the income and price variables and the one-period lagged differences are included. In the QLD and SA models, the one-period leads of the differences are also included. It is common practice to include leads as well as lags in the DOLS estimation of long-run relationships; in three of its regional models AEMO includes only lags in the DOLS equation. Including only lags is justified in cases where the explanatory variables Granger-cause the dependent variable (Hayakawa & Kurozumi, 2006). 25 AEMO states that the number of lags and leads in each regional DOLS model was selected by investigating the stability of the estimated coefficients. Starting with one lag, first a lead and then extra lags and leads are added, and the impact on the long-term elasticities is investigated. If an extra lead or lag does not materially change the elasticities, then the simpler model is preferred. 25 Hayakawa, K. & E. Kurozumi(2008), The role of leads in the dynamic OLS estimation of cointegrating regression models, Mathematics and Computers in Simulation, 79(3), pp Review of AEMO s electricity consumption forecasting process

40 22 Frontier Economics November 2013 This approach makes intuitive sense; however, in econometrics a more formal approach to variable selection is usually based on a criterion such as the Schwarz Bayesian (BIC) and/or Akaike (AIC) information criteria. Using this approach could lead to different specifications. To illustrate this point, we estimated four alternative versions of the QLD model by varying the number of lags and leads of the differences in income and price variables. The results in Table 5 show that the estimated elasticities differ considerably across the five variants. The table confirms that the elasticities do not change markedly between the model with (1 lag, 0 leads) and the model with (1 lag, 1 lead). However, if we continue with AEMO s approach, then the next step, either the model with (2 lags, 0 leads) or the model with (2 lags, 1 lead), results in greater changes to the elasticities. Based on the adjusted-r 2 or the BIC and AIC information criteria, one would not have selected the (1 lag, 1 lead) variant as the preferred model for QLD. We appreciate the difficulty in developing models with stable coefficients given the fairly short history of data available for analysis, and we acknowledge that the information criteria are large sample criteria. However, it should also be recognised how fragile some of the estimated elasticities are to slight changes in model specification. We recommend that AEMO give detailed consideration to this issue in future NEFRs. Table 5: Results for QLD DOLS models with different numbers of lags and leads Structure of DOLS Income coefficient Price coefficient Adjusted-R 2 BIC AIC 1 Lag & 0 Leads 0.25 (0.0000) (0.0000) Lag & 1 Lead * 0.23 (0.0056) (0.0000) Lag & 0 Leads 0.11 (0.0830) (0.0000) Lags & 1 Lead 0.17 (0.0015) (0.0000) Lags & 2 Leads 0.19 (0.0019) (0.0000) Notes: (1) Prob-values in parentheses. (2) Sample period 2000Q1 to 2012Q as selected by AEMO. (3) AEMO s preferred model is the model with one lag and one lead, indicated by an asterisk (*). Model adequacy The statistical fit of the models is very good with adjusted-r 2 values of 0.97 for VIC, 0.95 for TAS, 0.92 for NSW and SA, and 0.87 for QLD. However, we note Review of AEMO s electricity consumption forecasting process

41 November 2013 Frontier Economics 23 that for a cointegrating regression, the usual statistical tests and goodness-of-fit statistics cannot be interpreted in the usual way very high R 2 values are not uncommon. We have conducted a number of diagnostic tests on the residuals of the models, and the models pass most of the standard tests, with only the odd borderline diagnostic result. The exception is the QLD model which fails the normality and serial correlation tests, possibly due to an outlying observation in 2001Q2 when the level of electricity consumption was unusually high. We recommend that for the 2014 NEFR this data point, and its impact on the estimated model, be investigated further. There are two more general issues that raise concern: a) the presence of a structural break in all models except the VIC model, and b) the presence of multicollinearity between income and price, particularly in the SA and TAS models. We expand on these issues further in section after reviewing the short-run model and the modelling results Specification of the short-run EC equations In Table 6, we summarise the specification of the short-run error correction (EC) equations. Table 6: Short-run model specification Dependent variable Independent variables Year on year difference in per capita real income 1 state final demand (NSW, SA, TAS) gross state product (QLD, VIC) Year on year difference in per capita electricity consumption 1 Year on year difference in real average electricity price 1 electricity price for residential and business customers (QLD, NSW, VIC, TAS) electricity price for residential customers (SA) Heating degree days (NSW, VIC, SA, TAS) Cooling degree days (QLD,NSW,VIC,SA) Residual from the long-run DOLS model lagged 4 quarters (error correction term) 1) Variables enter the model in logarithms. Review of AEMO s electricity consumption forecasting process

42 24 Frontier Economics November 2013 Model adequacy The statistical fit of the short-run models is very good with adjusted-r 2 values of 0.92 for VIC and SA, 0.79 for NSW, 0.75 for QLD, and 0.64 for TAS. For differenced data, the model fit to the data is usually worse than for levels data; hence these fits are quite impressive. We have conducted a number of diagnostic tests on the residuals of the models, and the models pass most of the standard tests, with only the odd borderline diagnostic result. As with the long-run model, the exception is the QLD model which fails the normality and serial correlation tests, possibly due to the outlying data point at 2001Q Results Estimated coefficients The long-run price and income elasticities, presented in Table 7, have the correct sign and are statistically highly significant (prob-values 0.006). In terms of the magnitude of the estimated elasticities, the results appear reasonable for all states except TAS. It is not clear why the magnitude of the estimated long-run price and income elasticities for TAS are two (or more) times the size of the estimated elasticities in other states. AEMO will investigate whether the larger TAS longrun elasticities compared to other states are due to issues relating to the model or due to energy consumption in Tasmania being more sensitive to changes in income and price. Table 7: Results summary Model Income elasticity Price elasticity Short-run Long-run Short-run Long-run QLD 0.15 (0.1401) 0.23 (0.0056) (0.0498) (0.0000) NSW 0.07 (0.4998) 0.37 (0.0000) (0.0000) (0.0000) VIC 0.21 (0.0619) 0.31 (0.0000) (0.0030) (0.0000) SA 0.39 (0.0001) 0.30 (0.0000) (0.1625) (0.0000) TAS 0.44 (0.0004) 0.70 (0.0000) (0.0055) (0.0004) Note: Prob-values in parentheses. Review of AEMO s electricity consumption forecasting process

43 Cumulative impact response November 2013 Frontier Economics 25 The short-run income elasticities for all states have the expected sign, but are statistically highly significant only for SA and TAS models (prob-values ) and borderline for VIC (prob-value of ). The short-run price elasticities for all states have the expected sign. They are statistically significant for all states (prob-values ) except for SA. In terms of magnitude, the short-run price and income elasticities for QLD, NSW and VIC appear reasonable (they are lower than the long-run elasticities as one would expect them to be). The short-run price and income elasticities for TAS are high compared to other states, but in line with the long-run elasticities for TAS. The magnitude of the short-run income and price elasticities for SA are somewhat different to the results for other states. The short-run price elasticity is very small, indicating minimal response to price changes in the short-run; while the short-run income elasticity is higher than the long-run income elasticity. The impulse response functions in Figure 11 and Figure 12 provide some assurance that the estimated short-run effects are not too extreme. After a shock, electricity consumption reverts to a long run state within a reasonably short time period of up to about 12 quarters. Figure 11: Impulse response function for a shock to per capita income Time (shock at t=0) QLD NSW VIC SA TAS Review of AEMO s electricity consumption forecasting process

44 Cumulative impact response 26 Frontier Economics November 2013 Figure 12: Impulse response function for a shock to the electricity price Time (shock at t=0) QLD NSW VIC SA TAS The EC terms in all models are negative and statistically highly significant (probvalues of ). The coefficients lie between (QLD) and (SA), indicating quite rapid adjustment back to long-run equilibrium. Econometric forecasts of electricity consumption In Figure 13 to Figure 17, we present the historical quarterly per capita NLI electricity consumption and the quarterly electricity consumption forecasts produced by AEMO s regional econometric models for the next 10 years. In each figure, we also include historical and projected per capita income and price (expressed as indices). It can be seen that the electricity consumption forecasts for NLI, derived from the econometric models, follow the projected growth in per capita income more closely than the projected price changes. 26 Projected prices are fairly flat, and decreasing after Hence, unlike in recent years, prices are not expected to dampen the impact of the projected growth in per capita income. 26 The discussion of the relative importance of income and price in driving electricity consumption refers to per capita consumption. To obtain aggregate electricity consumption forecasts, the per capita forecasts need to be multiplied by future population projections. For the 2013 NEFR, future population growth is the strongest driver for aggregate electricity consumption for all the NEM states. Review of AEMO s electricity consumption forecasting process

45 2000Q1 2001Q1 2002Q1 2003Q1 2004Q1 2005Q1 2006Q1 2007Q1 2008Q1 2009Q1 2010Q1 2011Q1 2012Q1 2013Q1 2014Q1 2015Q1 2016Q1 2017Q1 2018Q1 2019Q1 2020Q1 2021Q1 2022Q1 2023Q1 NLI energy consumption (kwh per capita) Income and price indices (2000Q1=100) 2000Q1 2001Q1 2002Q1 2003Q1 2004Q1 2005Q1 2006Q1 2007Q1 2008Q1 2009Q1 2010Q1 2011Q1 2012Q1 2013Q1 2014Q1 2015Q1 2016Q1 2017Q1 2018Q1 2019Q1 2020Q1 2021Q1 2022Q1 2023Q1 NLI energy consumption (kwh per capita) Income and price indices (2000Q1=100) November 2013 Frontier Economics 27 Figure 13: Per capita NLI energy consumption (actual and forecast) QLD Energy consumption Income (per capita) index Electricity price index Note: NLI energy consumption is the load on the network from residential, commercial and small industrial customers plus estimated rooftop PV generation. Figure 14: Per capita NLI energy consumption (actual and forecast) NSW Energy consumption Income (per capita) index Electricity price index Note: NLI energy consumption is the load on the network from residential, commercial and small industrial customers plus estimated rooftop PV generation. Review of AEMO s electricity consumption forecasting process

46 2000Q1 2001Q1 2002Q1 2003Q1 2004Q1 2005Q1 2006Q1 2007Q1 2008Q1 2009Q1 2010Q1 2011Q1 2012Q1 2013Q1 2014Q1 2015Q1 2016Q1 2017Q1 2018Q1 2019Q1 2020Q1 2021Q1 2022Q1 2023Q1 NLI energy consumption (kwh per capita) Income and price indices (2000Q1=100) 2000Q1 2001Q1 2002Q1 2003Q1 2004Q1 2005Q1 2006Q1 2007Q1 2008Q1 2009Q1 2010Q1 2011Q1 2012Q1 2013Q1 2014Q1 2015Q1 2016Q1 2017Q1 2018Q1 2019Q1 2020Q1 2021Q1 2022Q1 2023Q1 NLI energy consumption (kwh per capita) Income and price indices (2000Q1=100) 28 Frontier Economics November 2013 Figure 15: Per capita NLI energy consumption (actual and forecast) VIC Energy consumption Income (per capita) index Electricity price index Note: NLI energy consumption is the load on the network from residential, commercial and small industrial customers plus estimated rooftop PV generation. Figure 16: Per capita NLI energy consumption (actual and forecast) SA Energy consumption Income (per capita) index Electricity price index Note: NLI energy consumption is the load on the network from residential, commercial and small industrial customers plus estimated rooftop PV generation. Review of AEMO s electricity consumption forecasting process

47 2002Q1 2003Q1 2004Q1 2005Q1 2006Q1 2007Q1 2008Q1 2009Q1 2010Q1 2011Q1 2012Q1 2013Q1 2014Q1 2015Q1 2016Q1 2017Q1 2018Q1 2019Q1 2020Q1 2021Q1 2022Q1 2023Q1 NLI energy consumption (kwh per capita) Income and price indices (2002Q1=100) November 2013 Frontier Economics 29 Figure 17: Per capita NLI energy consumption (actual and forecast) TAS Energy consumption Income (per capita) index Electricity price index Note: NLI energy consumption is the load on the network from residential, commercial and small industrial customers plus estimated rooftop PV generation Stability and multicollinearity in long-run models Stability To investigate the models stability through time, we performed a Quandt- Andrews (Q-A) structural breaks test on each of the models. This test splits the data into two sub-periods, and tests whether the estimated coefficients in the model differ between the two sub-periods. It does this for different dividing points between the two sub-periods to try to identify the point that provides the sharpest separation between the two sub-periods. The test results, shown in Table 8, indicate that all models, apart from VIC, have a significant break point. The break points are at: 2006Q1 for QLD, 2007Q2 for NSW, 2006Q2 for SA and 2007Q4 for TAS. The results for these four states are statistically highly significant. For VIC, the test indicates a break point at 2003Q4, but one could argue that the test result is marginal. Review of AEMO s electricity consumption forecasting process

48 30 Frontier Economics November 2013 Table 8: Structural break test for long-run models State Statistic Value Prob QLD Maximum LR F-statistic (2006Q1) NSW Maximum LR F-statistic (2007Q2) VIC Maximum LR F-statistic (2003Q4) SA Maximum LR F-statistic (2006Q2) TAS Maximum LR F-statistic (2007Q4) Notes: (1) QLD and SA models: (a) Null Hypothesis: No breakpoints within 30% trimmed data. (b) Equation Sample: 2000Q1 to 2012Q3. Test Sample: 2004Q1 to 2008Q4. (c) Number of breaks compared: 20. (2) NSW and VIC models: (a) Null Hypothesis: No breakpoints within 25% trimmed data. (b) Equation Sample: 2000Q1 to 2012Q4. Test Sample: 2003Q2 to 2009Q4. (c) Number of breaks compared: 27. (3) TAS model: (a) Null Hypothesis: No breakpoints within 30% trimmed data. (b) Equation Sample: 2002Q1 to 2012Q4. Test Sample: 2005Q3 to 2009Q3. (c) Number of breaks compared: 17. (4) Probabilities calculated using Hansen's (1997) method. We recommend that, for the 2014 NEFR, AEMO investigate which of the parameters has a structural break in it, and whether and how the models should be modified to account for the break. Further, we recommend that AEMO undertake routine testing for structural breaks for future NEFRs. Correlation between income and price Figure 9 and Figure 10 show a similar rising trend in the income and price data. These variables are included in the long-run equations in levels, and the correlation between these variables could lead to imprecisely estimated elasticities in the long-term models. Imprecisely estimated elasticities will not have much impact if the correlation between income and price observed in the past continues into the future. However, as income and price move along different trends in the future (see Figure 13 to Figure 17), imprecisely estimated coefficients may lead to biased forecasts. One way of investigating whether the correlation between income and price is an issue in the long-run models is to calculate so-called variance inflation factors (VIF). The results of these calculations are shown in the tables below. Table 9: Variance inflation check for QLD long-run model Variable Coefficient variance Centered VIF LOG (G) LOG (P) Note: Sample 2000Q1 to 2012Q3. Review of AEMO s electricity consumption forecasting process

49 November 2013 Frontier Economics 31 Table 10: Variance inflation check for NSW long-run model Variable Coefficient variance Centered VIF LOG (I) LOG (P) Note: Sample 2000Q1 to 2012Q4. Table 11: Variance inflation check for VIC long-run model Variable Coefficient variance Centered VIF LOG (G) LOG (P) Note: Sample 2000Q1 to 2012Q4. Table 12: Variance inflation check for SA long-run model Variable Coefficient variance Centered VIF LOG (I) LOG (P) Note: Sample 2000Q1 to 2012Q3. Table 13: Variance inflation check for TAS long-run model Variable Coefficient variance Centered VIF LOG (I) LOG (P) Note: Sample 2002Q1 to 2012Q4. The most common recommendation for a critical value for the VIF is that it should be no larger than 10. Other authors recommend lower values, e.g. 5. We can see that for the SA model both income and price have VIFs larger than 10 with values of 18.7 and 25.9 respectively. In the TAS model, income and price have VIFs around 10 (8.3 and 11.7 respectively). This suggests that the long-run elasticities may not be estimated precisely. Income and price might proxy each other to some extent so that part of the income elasticity is in essence a price response, and vice versa. It could be, for example, that the true income and price Review of AEMO s electricity consumption forecasting process

50 32 Frontier Economics November 2013 elasticities are closer to zero, but that their combined effect over the historical data is similar to inflating both elasticities. As we indicated before, this is not of concern in a forecasting context if the past correlation between income and price continues into the future. In Table 14 we list the correlations between income and price for all regions over several time periods. The table shows that for SA and TAS the correlation between income and price over the next five, ten and twenty years is quite different to the historical period. Under the postulated scenario, the high VIFs for SA and TAS are of considerable concern for long-run forecasting. We recommend that AEMO give close consideration to this issue for future NEFRs. Table 14: Correlation between income and price Model Actual income and price Projected income and price QLD NSW VIC SA TAS Note: The correlations were calculated for the logarithms of the variables, since the variables are included in the long-run models in logarithms. 3.4 High level review of additional modelling steps Post-modelling adjustments AEMO s current forecasting approach is to use an econometric model to forecast NLI energy consumption, and then to adjust the econometric forecasts for factors which have not been captured by the econometric model. Implicit in this approach is the assumption that the historical trends in energy efficiency excluded from the econometric model are adequately proxied by the key drivers included in the econometric model (i.e. GSP and/or price), and that only the future changes in these factors over and beyond the trends captured by the economic drivers should be accounted for through post-modelling adjustments. If this assumption is not correct, then: (1) estimated elasticities in the econometric model might be biased; and (2) forecasted electricity consumption might also be biased. However, without further investigation, it is not possible to determine the extent or the direction of the bias. AEMO should investigate the Review of AEMO s electricity consumption forecasting process

51 November 2013 Frontier Economics 33 materiality of this issue for the elasticity estimates and forecasts for future NEFRs Methodology for forecasting large industrial loads AEMO s current methodology is based on discussions with large industrial users. One risk of this approach is that these projections may not be best estimates or reflect a reasonable range as: (a) they might otherwise reflect or reveal commercially sensitive information, and (b) the users may err on the side of overestimating future requirements. For example, an aluminium smelter may not reveal that they may reduce load or close. The net effect may result in conservative load projections (i.e. they err on the high side). This relates to the objective of the forecasts in each scenario: whether they are intended to reflect a best estimate, a conservative (higher) estimate, or a reasonable range of all possible alternatives. An alternative that may be considered in future is to rely on the large user estimates for a base case scenario but to develop a range of alternatives based on independent views Methodology for forecasting small non-scheduled generation To forecast small non-scheduled generation, AEMO uses a database of existing and possible future small non-scheduled generators. AEMO develops historical and future capacity profiles for each project based on estimated start-up dates and installed capacities. Frontier s review of AEMO s methodology is based solely on the information provided in the draft 2013 Forecasting Methodology Information Paper (FMIP). Based on that information, our view is that AEMO s approach is reasonable Methodology for forecasting rooftop PV generation In general, AEMO s approach (as we understand it) appears sound, however the methodology could be explained more clearly. With regard to: Saturation: we agree with the described approach, which reduces growth rates as installed capacity approaches threshold. Assumed system size (3.5kW): this reasonably reflects the average system size installed in the past 3-4 months in the NEM (so is a reasonable assumption on this basis). The incremental average over the past 12 months in the NEM is closer to 3kW, but given the growth in size over time (and uncertainty regarding numerous changes in solar PV policy support and the incentives these create), the 3.5kW assumption does not appear unreasonable. Output estimates: the approach adopted is reasonable. Review of AEMO s electricity consumption forecasting process

52

53 November 2013 Frontier Economics 35 4 Findings and recommendations for AEMO s electricity consumption models In this section we summarise our findings and recommendations for improving AEMO s econometric models. We categorise our recommendations into main recommendations (summarised in section 4.2) and additional recommendations (summarised in section 4.3). We advise that AEMO give high priority to addressing all our main recommendations summarised in section 4.2. We recognise, however, that addressing some of recommendations may require significant effort in terms of analyses and investigations that would need to be undertaken, and may have to be implemented over a longer period. 4.1 Overall assessment The AEMO electricity consumption model possesses many of the desirable elements of an electricity consumption model. Of particular note is AEMO s commitment to transparency. To our knowledge, no other electricity organisation internationally, with similar responsibilities, provides as much data and detail on forecasting methodology as AEMO. There are, of course, different types of data sources available in different jurisdictions and different levels of resources and modelling experience that affect the modelling done in a particular jurisdiction. But the econometric modelling undertaken by AEMO puts it within the ranks of the more sophisticated similar organisations internationally. That does not mean that AEMO s models are the best possible and cannot be improved. There are always lessons to be learnt and opportunities to improve. Our comments and recommendations should be seen against this background. They are offered to assist AEMO s ongoing commitment to enhancing its forecasting methodology and transparency. 4.2 Main findings and recommendations Accounting for energy efficiency AEMO s current forecasting approach is to use an econometric model to forecast NLI electricity consumption, and then adjust the econometric forecasts for incremental energy efficiency savings which have not been captured by the econometric model. As energy efficiency savings continue to grow, it will become increasingly difficult to determine how much of the energy efficiency saving is captured by the econometric models, and how much post-modelling adjustment needs to be made. This forecasting challenge is not faced only by AEMO; it is Findings and recommendations for AEMO s electricity consumption models

54 36 Frontier Economics November 2013 faced by operators, regulators and utilities around the world, and there is no obvious solution to the overcoming this difficulty. One alternative approach that could be investigated is to model econometrically the demand for energy services rather than energy. This would involve adding estimated energy efficiency savings over the estimation period to native NLI electricity consumption prior to undertaking the econometric estimation. This should produce elasticity estimates more closely aligned with economic reasoning than the present approach, since the economic drivers in the model drive the demand for energy services rather than electricity per se. We acknowledge, however, that whether or not this approach will produce more satisfactory econometric models and forecasts remains an empirical question. The forecasts made using the estimated econometric model would be forecasts of energy services. Suitable post-modelling adjustments would have to be made to obtain forecasts of NLI energy consumption at the sent-out level. The suggested approach involves making estimates of the amount of energy efficiency savings over the historical period and the forecasting horizon. But such estimates are already part of AEMO s forecasting methodology. In a more exploratory vein, AEMO could consider the so-called stochastic trend modelling framework to obtain indirect estimates of the impact on electricity consumption of factors such as energy efficiency savings. This approach was used recently to model energy consumption in Britain. 27 Another approach, used by a number of utilities in the U.S., is the so-called Statistically Adjusted End-Use (SAE) forecasting framework developed by Itron Inc. The SAE methodology involves constructing end-use variables based on regional estimates of end-use penetration levels, average efficiency levels, thermal efficiency levels, and end-use energy usage estimates published by the U.S. Department of Energy. The end-use energy usage estimates are then adjusted to match use-per-customer data and the aggregate weather sensitivity of a relevant local utility. Once the end-use variables are constructed, an econometric equation calibrates the end-use inputs to the historical energy consumption/sales data and generates forecasts. 28 However, we acknowledge that, at present, AEMO does not possess the detailed end-use data required to implement this approach. It was agreed that in 2014 AEMO would investigate alternative approaches to account for energy efficiency savings. Frontier recognises that it will take significant effort to investigate alternative approaches and that the outcome is 27 Agnolucci, P. (2010), Stochastic trends and technical change: The case of energy consumption in the British industrial and domestic sectors, Energy Journal, 31, pp Itron (2010), Incorporating DSM into the Load Forecast, Itron White Paper, available at 20Forecast.pdf. Findings and recommendations for AEMO s electricity consumption models

55 November 2013 Frontier Economics 37 uncertain. Hence we understand that it will not be possible to implement this recommendation in the 2014 NEFR, and that its implementation in future NEFR reports will depend on the results of investigations to be undertaken in Economic activity driver AEMO s econometric models are used to forecast energy consumption by NLI customers, which include residential, commercial, and small industrial customers. AEMO postulates that economic activity is one of the main drivers of NLI energy consumption, and defines economic activity in terms of real Gross State Product (GSP) per capita or real State Final Demand (SFD) per capita. Both variables were tested in each regional model, and the one with the best explanatory power was selected (GSP for the QLD and VIC models; and SFD for the NSW, SA and TAS models). Both GSP and SFD are measures of economic activity, but they do not have the same coverage. According to the Australian Bureau of Statistics, SFD approaches the measurement of economic activity from an expenditure view of the economy; while GSP approaches it from a production view of the economy. 29 Based on these definitions, it could be argued that SFD might be a good economic driver for residential energy consumption, while GSP is probably more appropriate for commercial and industrial consumption. Since the energy consumption being modelled econometrically by AEMO is a combination of residential, commercial and small industrial users, and considering that the customer mix might have changed over time, AEMO should investigate alternative economic drivers that could potentially perform better in the models. One possibility is to construct a composite economic driver, which could include a weighted combination of SFD and GDP (and possibly additional relevant variables). An example of this approach can be found in the modelling manual of the PJM Interconnection LLC (PJM), a regional transmission utility in the United States. 30 In its energy consumption and peak demand forecasting models for 18 transmission zones, PJM models economic conditions using an economic index variable. This variable is a weighted combination of U.S. GDP, Gross Metropolitan Product, personal income, population, households, and non- 29 See B0012C0AD?opendocument. 30 PJM coordinates the movement of wholesale electricity in all or parts of 13 states and the District of Columbia. Findings and recommendations for AEMO s electricity consumption models

56 38 Frontier Economics November 2013 manufacturing employment, with electricity sales shares by customer class (i.e. residential, commercial, and industrial) used as weights Structural breaks To investigate the models stability, we performed a Quandt-Andrews (Q-A) structural breaks test on each of the regional models (see section 3.3.6). This test splits the data into two sub-periods, and tests whether the estimated coefficients in the model differ between the two sub-periods. It does this for different dividing points between the two sub-periods to try to identify the point that provides the sharpest separation between the two sub-periods. The test results show that all models, except VIC, have a significant break point. The break points are at: 2006Q1 for QLD, 2007Q2 for NSW, 2006Q2 for SA and 2007Q4 for TAS. The results for these four states are statistically highly significant. Further investigation would have to be undertaken to determine what caused these structural breaks. Based on our findings, AEMO agreed to investigate which of the parameters in the modelled equations has a structural break in it, and whether and how the models should be modified for the 2014 NEFR to account for the break. Further, we recommend that AEMO undertake routine testing for structural breaks for future NEFRs Correlations between main drivers To investigate and quantify the severity of multicollinearity in the long-run models, Frontier calculated so-called variance inflation factors (VIF) (see section 3.3.6). The most common recommendation for a critical value for the VIF is that it should be no larger than 10 (although, some authors recommend lower values, such as 5). In the SA model both income and price have VIFs larger than 10, with values of 18.7 and 25.9 respectively. In the TAS model, income and price have VIFs around 10 (8.3 and 11.7 respectively). This suggests that the long-run elasticities in the SA and TAS models may not be estimated precisely. Imprecisely estimated elasticities will not have much impact on the forecasts if the correlation between income and price observed in the past continues into the future. However, if income and price move along different trends in the future, imprecisely estimated coefficients may lead to biased forecasts. Our analysis indicates that for SA and TAS the correlation between income and price over the next five, ten and twenty years is quite different to the historical period. The high VIFs for SA and TAS are therefore of considerable concern for long-run forecasting. 31 PJM (2012), PJM Manual 19: Load Forecasting and Analysis, p. 17. Findings and recommendations for AEMO s electricity consumption models

57 November 2013 Frontier Economics 39 Based on our findings, AEMO has agreed to investigate the materiality of the high correlation between the income and price variables for the forecasts, and whether and how the models could be modified for future NEFRs to account for this issue Selection of weather stations AEMO uses temperature data from a single weather station in each NEM region to calculate the CDD and HDD variables used as explanatory variables in the econometric models. 32 We were informed that the selected weather stations are all located in the capital cities. In our experience, utilities often select a number of weather stations and use a simple or weighted average to obtain temperature variables. We recommend that for future NEFRs AEMO investigate how sensitive the forecasts are to the selection of weather stations and consider using a combination of weather stations for each region Model evaluation AEMO s 2013 Forecast Accuracy Report (FAR) provides an assessment of the forecasts provided in the 2012 NEFR and highlights key improvements to the forecasting process for the 2013 NEFR. We commend AEMO for this effort. However, we recommend that in future FARs AEMO: adopt the methodologies used in the 2011 FAR for assessing the quality of forecasting models. These methodologies are more comprehensive than those used in the 2013 FAR decompose the forecasting errors into errors in the driver projections and errors due to the forecasting models. For example, if weather were unusual in a given year, this decomposition would make it clear that the main contributor to the forecasting error in that year was the weather and not the econometric model. This decomposition would include comparisons of: economic projections (i.e. GSP and price) compared to actual outcomes temperatures used in the forecasts compared to actual outcomes residual errors in the forecasts due to modelling errors compared to errors due to errors in the drivers. 32 For forecasting maximum demand, AEMO uses two weather stations in each NEM region, except Queensland where it uses three stations. Findings and recommendations for AEMO s electricity consumption models

58 40 Frontier Economics November Additional recommendations Other drivers In addition to income and price variables, AEMO tested several other variables for inclusion in its econometric models, e.g. the prices of substitutes for electricity natural gas and other household fuels and the standard variable mortgage rate. 33 AEMO explains that these drivers were considered but were found to be statistically insignificant in explaining energy consumption or that the estimated long run coefficients for these variables were unrealistic once they were entered into the long run equation. 34 We recommend that AEMO provide more information on these tests, either by providing regression results, or by summarising the key results in tabular format. The key information would include: an explanation as to why a driver was considered and included or excluded (e.g. based on economic theory, legacy models); the magnitude and sign of the estimated coefficients; associated t- statistics or prob-values; and the effect on the price and income elasticities, the overall fit of the model, and the forecasts. Providing this information would assist in gaining a better understanding of AEMO s modelling approach and in assessing the models robustness Long-run model Consider the use of annual data instead of quarterly data AEMO estimates its electricity consumption models using quarterly data. AEMO may wish to investigate whether more precise estimates of long-run elasticities can be obtained by estimating the DOLS models using annual data. AEMO sources historical annual electricity prices from NIEIR, and then uses a spline interpolation method to derive quarterly prices from annual prices. The interpolation method (by construction) introduces some autocorrelation in the models, which affects the precision of the estimated parameters. This issue could be addressed in the long-run models by using annual data. While using annual data will reduce the number of observations in the estimation sample, more data is better only if the extra observations provide additional 33 In the 2012 NEFR, AEMO included in the NSW model an air-conditioning ownership variable, which entered the econometric model as an interaction term with the CDD variable; see AEMO (2012), Forecasting Methodology Information Paper, p We were informed that this variable was not tested for inclusion in the 2013 NEFR models as AEMO considers the underlying data to be obsolete. We were also informed that AEMO is investigating alternative sources for air-conditioning ownership data. 34 AEMO communication (22 May, 2013) Draft NEFR 2013 Forecasting Methodology. Findings and recommendations for AEMO s electricity consumption models

59 November 2013 Frontier Economics 41 information of interest. The extra information one gains by using quarterly (instead of annual) data in the DOLS models mainly throws light on how energy consumption varies across seasons, and may not add much relevant information for estimating long-run elasticities. For an example of combining long-run forecasting model estimated using annual data with a short-run forecasting model estimated using monthly data see Engle et al. (1989). 35 Dynamic specification for long-run model The current version of AEMO s long-run DOLS model includes as variables the changes in the economic drivers, and leads and lags in these changes. The changes are measured as the change in the value of the variable in one quarter compared to the previous quarter. By contrast, the short-run EC model includes changes in the variables from one quarter to the same quarter one year previously, i.e. year-on-year changes. Consideration should be given to adopting the same approach in the long-run model. While this cannot be implemented in EViews using the DOLS command, it is quite easy to formulate the DOLS specification within the standard OLS command, which enables this variant of the DOLS model to be estimated Estimating the models as a system AEMO currently estimates its econometric models separately for each NEM region. AEMO could consider estimating all the regional models together as a system of equations using the so-called seemingly unrelated regression (SUR) method. 36 If there are shocks to energy consumption across the NEM regions that are not captured by the independent variables, then the residuals in the models are likely to be correlated across regions. In that situation SUR estimation would produce more precise estimates of the elasticities Functional form The most widely used functional form used to link energy consumption to the explanatory variables (i.e. drivers), and one used by AEMO, is the log-log functional form, where both the dependent and the economic explanatory 35 Engle, R., C. Granger and J. Hallman (1989), Merging short- and long-run forecasts: An application of seasonal cointegration to monthly electricity sales forecasting, Journal of Econometrics 40, pp Zellner, A. (1962), "An efficient method of estimating seemingly unrelated regression equations and tests for aggregation bias", Journal of the American Statistical Association 57, pp Findings and recommendations for AEMO s electricity consumption models

60 42 Frontier Economics November 2013 variables are included in the model as logarithms. 37 The reason for this is twofold: (1) estimation of log-log models is relatively straightforward; and (2) the estimated coefficients can be interpreted as elasticities. The drawback, however, is that the estimated elasticities are assumed to be constant (i.e. consumers reaction to a change in price or income is assumed to be the same regardless of the price or income level). Recently there has been discussion in the energy economics literature regarding the importance of specifying the appropriate functional form, and the consequences of misspecifying the functional form. For example, Karimu and Brännlund (2013) use a nonparametric approach to investigate whether the loglog functional form is the appropriate specification for aggregate electricity consumption models. 38 Their analysis is done using a panel data on aggregate energy consumption in 17 OECD countries. Their results indicate that the loglog specification appears to be inadequate as a functional form for aggregate electricity consumption modelling. Whether the log-log functional form is appropriate for modelling regional NLI energy consumption is an empirical question. It is our understanding that AEMO has not investigated any alternative functional forms. We therefore recommend that AEMO undertake such an investigation in order to assess whether and how the forecasts are affected by alternative functional forms Projections for economic and demographic variables AEMO currently sources all historical and projected economic and demographic data from a single source, NIEIR, and makes the projections publicly available. In the 2013 NEFR, AEMO states that it compared NIEIR price projections to, and found them to be broadly in line with, the price projections published by regulatory agencies in each state. It is not clear whether the assessment was carried out for the entire forecasting horizon since price projections by different agencies/organisations used by AEMO as benchmarks are available for different time periods within the forecasting horizon. We suggest that AEMO clarify this in the future Economic Outlook Information Papers. In the 2013 Economic Outlook Information Paper, AEMO assessed NIEIR s forecasts of GSP/SFD for reasonableness by comparing them to forecasts from 37 The log-log model is also referred to as the double-log model and the log-linear model. The appeal of log-log model is not unique to modelling electricity consumption; it is the default functional form for modelling demand across most industries. 38 Karimu, A. and R. Brännlund (2013), Functional form and aggregate energy demand elasticities: A nonparametric panel approach for 17 OECD countries, Energy Economics 36, pp Monash uses a semi-log functional form in its study of price elasticities rather than the log-log, but notes that its results are similar to AEMO s. This comparison should be done in a more formal way. Findings and recommendations for AEMO s electricity consumption models

61 November 2013 Frontier Economics 43 state government agencies and Deloitte Access Economics. We suggest that AEMO clarify that the comparison against Deloitte Access Economics was done for the period up to (as the spreadsheets provided to Frontier indicate) and comment on similarities and differences between the projections over the forecasting horizon Large industrial load forecasts AEMO s methodology for forecasting electricity consumption for large industrial users is based on information provided by the individual users. This approach has merit since individual users are likely to be better informed than AEMO about their current position in the market and their future prospects. However, a large industrial user may not have an incentive to reveal all the relevant information to AEMO; for example, it may not wish to disclose commercially sensitive information such as a plan to downsize or expand capacity in the near future. An alternative approach would be for AEMO to ask large industrial users to provide forecasts under a base case scenario (i.e. business as usual), and then for AEMO to develop a range of alternative scenarios based on independent assessments. Findings and recommendations for AEMO s electricity consumption models

62

63 November 2013 Frontier Economics 45 5 Maximum demand forecasting In this section we give a brief overview of different approaches to forecasting maximum demand and the desirable features of a maximum demand forecasting model. 5.1 Approaches to forecasting maximum demand There is vast research literature on modelling electricity load and peak demand for long-term forecasting. This literature covers a wide variety of methods that vary in their analytical sophistication and data requirements. Literature surveys can be found in Lotufo & Minussi (1999), Alfares & Nazeeruddin (2002) and Singh et al. (2013). 40 The different approaches fall into several broad categories, which we summarise in Table 15. For each of the more sophisticated approaches we provide a few references to give the flavour of the wide range of methods that are being developed and explored by researchers in the area. Until recently, most utilities used the technically less sophisticated methods that belong to the first three categories in the table. However, in the last decade utilities have started to adopt more sophisticated methods, in particular, semiparametric regression methods such as splines, and simulation techniques to estimate probabilities of exceedance (POEs). A general issue for maximum demand modelling is that historical peak demands occur under different weather conditions and at different times, and hence are not directly comparable across years. To overcome this difficulty, many utilities estimate the standard weather peak demand for each year. One way of doing this is to estimate the relationship between daily peaks and weather variables for a particular season, and then use this relationship to adjust the actual peak demand for the season to obtain the standard weather peak demand. The historical time series of standard weather peak demands is then forecast into the future. This approach can be used with any of the simpler approaches in Table 15. The more sophisticated approaches deal with the weather and time dependence of peak demand by modelling demand for all half hours, not just the peaks, as a function of weather variables and calendar effects. Peak demand forecasts can then be determined after forecasts have been made for all future half hours under different weather scenarios. 40 (1) Lotufo, A. and Minussi, C. (1999), Electric power systems load forecasting: A survey, IEEE Power Tech 99 Conference; (2) Alfares, H. and Md. Nazeeruddin (2002), Electric load forecasting: Literature survey and classification of methods, International Journal of Systems Science, 33(1), pp ; (3) Singh, A., I. Khatoon, Md. Muazzam& D. Chaturvedi (2013), An overview of electricity demand forecasting techniques, Network and Complex Systems, 3(3), pp Maximum demand forecasting

64 46 Frontier Economics November 2013 Table 15: Approaches to forecasting maximum demand Approaches Apply constant load factor Trend fitting techniques Regression methods Advanced regression methods Advanced statistical methods Frequency distribution approach Description Assume the ratio between consumption and maximum demand (load factor) is constant, and apply this factor to an electricity consumption forecast to obtain a peak load forecast Fit a trend function to historical load factor data, project the load factor into the future, and apply this factor to an electricity consumption forecast to obtain a peak load forecast Estimate a regression model linking peak demand to economic, socio-demographic and weather variables, as well as calendar effects (e.g. time of day, day of week, and/or month of year effects). Probability of Exceedance (POE) values for demand can be obtained by substituting temperature values in the model that correspond to particular POE values for temperatures Estimate more advanced models such as regression splines, quantile regressions and extreme value theory, and use these to forecast peak demand 41 Use statistical models developed by engineers for short-term forecasting that have been adapted to medium and long-term load forecasting; e.g. neural networks, support vector machines 42 Model half-hourly demand using regression methods or statistical methods, then simulate many alternative future weather scenarios, and possibly other stochastic input variables, to obtain forecasts of frequency distributions for maximum demand. Probability of Exceedance (POE) values can be determined from the frequency distributions See, for example: (1) Harvey, A. and S.J. Koopman (1993), Forecasting hourly electricity demand using time-varying splines, Journal of the American Statistical Association, 88, pp ; (2) Hendricks, W. and R. Koenker (1992), Hierarchical quantiles and the demand for electricity, Journal of the American Statistical Association, 87, pp ; (3) Veall, M. (1983), Industrial electricity demand and the Hopkinson rate: An application of the extreme value distribution, The Bell Journal, 14, pp See, for example: (1) Carpinteiro, O., et al (2007), Long-term load forecasting via a hierarchical neural model with time integrators, Electric Power Systems Research, 77, pp ; (2) Chen, T. and Wang, Y-C (2012), Long-term load forecasting by an elaborative fuzzy-neural approach, Electrical Power and Energy Systems, 43, pp ; (3) Du, Z-G, L. Niu and J-G. Zhao (2007), Long-term electricity demand forecasting using relevance vector learning mechanism, Lecture Notes in Computer Science, 4491, pp See, for example: (1) Veall, M. (1987) Bootstrapping the probability distribution of peak electricity demand, International Economic Review, 28(1), pp ; (2) McSharry, P, S. Bouwman and G. Bloemhof (2005), Probabilistic forecasts of the magnitude and timing of peak electricity demand, IEEE Transactions on Power Systems, 20, pp ; (3) Hyndman, R. and S. Fan (2010), Density forecasting for long-term peak electricity demand, IEEE Transactions on Power Systems, 25(2), pp Maximum demand forecasting

65 November 2013 Frontier Economics Desirable elements of a maximum demand forecasting model Maximum demand in an electricity system is determined by both long-term, slowly evolving influences, like economic activity, that affect the average level of demand, and short-term factors such as weather and calendar effects, that affect short-term fluctuations around the average level of demand. The desirable elements of the model component concerned with the average level of demand are the same as those for the electricity consumption model and we refer the reader to section 2 for the relevant discussion. In this section we restrict ourselves to additional elements applying specifically to modelling maximum demand. Dependent variable One of the main differences between an energy consumption model and a peak demand model is that the dependent variable is different. The variable of concern is the maximum demand among all the half hours in the year, in the summer or winter season, or in a shorter period. One could consider only the peak demand in each year, or in each season, or month, or day, and estimate a model to determine the drivers of the observed peak demands. An early example of this is Veall (1983), who modelled monthly peak demands of industrial customers as a function of the demand charge and kwh usage in the month, as well as other drivers of peak demand, and used an extreme value distribution for the peak demands. 44 However, it is now quite common to model the demand for all half hours in the period, or the demand for all half hours that have the potential to be peaks. Economic drivers The same economic drivers that one expects to find in a consumption model also affect peak demand an economic activity variable, an own price variable and a substitute price. However, the granularity of the data is different. One would wish to tie the economic variables as closely as possible to the half-hourly demands being modelled. For variables that are only available at, say, an annual frequency or a quarterly frequency, it might be inappropriate to keep those variables constant year to year or quarter to quarter. One could use interpolation to obtain a closer match of the economic variables to the granularity of the dependent variable, to say monthly values. In the case of time-differentiated prices, such as time-of-use, dynamic peak, or real time pricing, the price naturally varies for different half hours and hence has 44 Veall, M. (1983), footnote 41 (3). Maximum demand forecasting

66 48 Frontier Economics November 2013 the same granularity as the dependent variables. Such tariffs are generally designed to impact on the load shape, and hence are expected to have a different impact on peak demand than on average electricity consumption. Weather variables For annual energy consumption models it is common to use summary weather variables, such as cooling degree days (CDD) and heating degree days (HDD) to account for different weather conditions between years. With half-hourly data it is desirable to have weather variables of the same granularity as the demand variables. Weather affects demand differently through the day and through the year, and half-hourly demand models need to be able to capture these timedifferentiated impacts. Moreover, lags of weather variables should be included to account for the persistence in weather patterns. A rule of thumb sometimes used in the industry is that the winter peak demand occurs, not on the coldest day of the year, but on the third day of a cold spell. A peak demand model should be able to capture such a momentum effect. And since the impact of temperature on demand is known to be non-linear, the model should also be able to capture such non-linearity. Some modellers 45 have had success using modified weather indexes to capture perceived temperatures rather than actual temperatures. Perceived, or apparent, heat or cold can be captured by variables such as the Heat Index and the Wind Chill Factor 46. Such indexes take into account humidity and/or wind speed, as well as temperature, since it is thought that it is the combination of such factors that makes consumers turn on air conditioners or heaters. Many modellers include measures of cooling and/or heating capacity in their models, for example, wattage of installed air conditioning. Such capacity measures often grow at a different rate to electricity sales, and capacity can have a major impact on consumers ability to respond to weather conditions. 47 This suggests that capacity variables should not be included in the model like other independent variables, but rather as modifiers of the coefficients of the weather variables. 45 See, for example, PJM (2013), Load Forecasting and Analysis: Manual, v22, p See, for example, for comments on this point by the Bureau of Meteorology. 47 The installation of cooling or heating capacity depends on prevailing weather conditions, which vary considerably between jurisdictions. While some Australian distributors face relatively mild weather conditions throughout the year, the weather in the southern states of South Australia and Victoria is generally mild with occasional extremes. Thus, in the southern states it is more likely that average demand and maximum demand have different growth rates. Maximum demand forecasting

67 November 2013 Frontier Economics 49 Calendar effects When modelling daily data or intra-daily data, calendar effects become important. For example, consumers demands tend to be different on weekends compared to weekdays. Factories are closed, workers are at home so both business and residential customers are likely to be affected. Public holidays and school holidays also have a noticeable impact on demand, both on average half-hourly demand, and on the profile through the day. Such factors should be incorporated in any maximum demand model. Energy efficiency initiatives and post-modelling adjustments In addition to dynamic tariffs, in Australia there are a host of other initiatives at various stages of implementation or consideration. Examples are, feed-in tariffs for small PV generation units, new standards for lighting, and so on. A maximum demand model should be able to incorporate the impact of such initiatives. This is often done by simulating the impact on consumption and then applying the same proportional impact to peak demand. However, many of these initiatives have different impacts at different times, either by design, or because customers respond differently to the initiative at different times. This indicates that a peak demand model should be able to accommodate post-modelling adjustments that change the load factor and the shape of the load curve. Maximum demand forecasting

68

69 November 2013 Frontier Economics 51 6 Review of maximum demand forecasting methodology 6.1 Background To assist in producing the peak demand forecasts for each NEM region, AEMO engaged the Business and Economics Forecasting Unit at Monash University. We refer to the Monash modelling team as Monash, and to their methodology as the Monash model. Our review is based on the reports prepared by Monash for AEMO, as well as the description of Monash s modelling methodology in a number of published and working papers. It was outside the scope of our engagement to review computer code and data files used by Monash to produce the forecasts. As a result, our review of the peak demand models is more limited than our review of AEMO s energy consumption models. There are quite a number of Monash reports released over the years that describe and apply Monash s methodology. The reports we have relied on most for details of the statistical methodology underpinning the Monash model are Hyndman (2007) and Hyndman & Fan (2010). 48 We have also reviewed the forecasting reports for each of the five NEM regions produced by Monash for AEMO for the 2013 NEFR. 49 In addition, we reviewed two research reports that Monash produced in preparation for the 2013 forecasting exercise. 50 Monash s maximum demand model is a model for half-hourly demand that places special emphasis on the impact of temperature variables and calendar effects on demand. For each NEM jurisdiction, separate models are developed for summer and for winter. 51 Monash s approach to forecasting long-term half- 48 (1) Hyndman, R. (2007), Extended Models for Long-Term Peak Half-Hourly Electricity Demand for South Australia A report for ESPIC, Business & Economic Forecasting Unit, Monash University; (2) Hyndman, R and S. Fan (2010), Density forecasting for long-term peak electricity demand, IEEE Transactions on Power Systems, 25 (2), pp The report for NSW is Fan, S. and R. Hyndman (3 June 2013), Forecasting Long-Term Peak Half-Hourly Electricity Demand for New South Wales A Report for AEMO, Business & Economic Forecasting Unit, Monash University. The reports for the other regions have analogous titles. We refer to these as the Monash regional reports. 50 (1) Fan, S. and R. Hyndman (30 Jan 2013), The Price Elasticity of Electricity Demand in the National Electricity Market - A Report for AEMO. Business & Economic Forecasting Unit, Monash University [Henceforth referred to as the Monash (2013) elasticity report]. (2) Fan, S. and R. Hyndman (19 Feb 2013), Load Factor Analysis in the National Electricity Market and the Implications for the Peak Demand Forecast A Report for AEMO. Business & Economic Forecasting Unit, Monash University [Henceforth referred to as the Monash (2013) load factor report]. 51 Summer is defined as the months from October to March, and winter as the months April to September. For the 2013 NEFR forecasts, Monash has restricted the definition of summer for Tasmania to include only the months December to February. Review of maximum demand forecasting methodology

70 52 Frontier Economics November 2013 hourly electricity consumption and estimating probabilities of exceedance (POE) was first developed around 2007 and several refinements to the original approach have been implemented in subsequent years. The Monash modelling approach belongs to the category of advanced regression methods which is combined with simulation to obtain frequency distributions for future half-hourly demands and POEs. An early contribution towards obtaining a probabilistic forecast for peak demand, with a full set of probabilities attached to different potential outcomes, is Veall (1987). 52 A model with a similar approach to the Monash model with respect to bootstrapping future temperature scenarios can be found in McSharry et al. (2005). 53 However, Monash has incorporated a number of modifications to this approach. The remainder of this section is organised as follows. To understand the Monash methodology, it is helpful to consider the model used for estimation and the model used for forecasting separately. Hence, in section 6.2 we describe Monash s estimation model and in section 6.3 Monash s forecasting model. Our high level review of post-modelling adjustments to the maximum demand forecasts and of the treatment of PV in terms of maximum demand is provided in section Estimation model Model specification The general algebraic form of Monash s estimation model is shown in Figure 18. The dependent variable in the model is half-hourly NLI demand defined as general mass market sales of electricity (including network losses) plus estimated rooftop PV generation and auxiliary loads. The demand for a particular half hour is modelled in terms of a large number of weather variables, day of the week dummy variables, holiday effects, and a day of season effect. For each region, there are 48 such models for summer (one for each half hour of the day), and 48 such models for winter. As is common with demand models, the demand variable (the dependent variable in the equation) is included in the model in logarithmic form. The nonweather variables are referred to collectively as calendar effects. Typically, there is a large number of temperature variables in the Monash model. For example, Model 15 in Table 1 on page 17 of the Monash report for NSW, 52 Veall, M. (1987) Bootstrapping the probability distribution of peak electricity demand, International Economic Review, 28(1), pp McSharry, P, Bouwman S. and Bloemhof, G. (2005), Probabilistic forecasts of the magnitude and timing of peak electricity demand, IEEE Transactions on Power Systems, 20, pp Review of maximum demand forecasting methodology

71 November 2013 Frontier Economics 53 referred to as the best temperature/calendar model, contains more than 25 temperature variables as well as three calendar effects. Each of the temperature variables is modelled as a cubic regression spline with two knot points, with the splines being constrained to become linear at the ends of the curves. The splines are then combined additively in the model. Model 15 is the model for a particular half hour for summer days. It is one of the set of 48 half-hourly summer models for NSW, one for each half hour of the day. A model of this type is called semi-parametric, because the use of cubic splines allows each weather variable to have a flexible nonlinear impact on demand. 54 On the other hand, there is also a lot of data with at least 10 years of data, there are more than 1,800 observations available for estimating each half hourly model. Figure 18: General algebraic form of the Monash estimation model Estimation model where is half-hourly native NLI demand at half hour inclusive of PV is the average half hourly demand in the season that half hour belongs to represents a set of calendar variables represents temperature variables represents the impact of calendar effects on demand at half hour represents the impact of temperature variables on demand at half hour and, are parameters to be estimated (e.g. temperature sensitivity and price elasticity), and is a random error term Source: Adapted by Frontier Economics from Monash reports Note the term in the equation in Figure 18; this is the logarithm of the average half-hourly demand for the particular summer (or winter) season that half hour falls into, and it sets the overall level for the half-hourly demands for that season. For estimation purposes, Monash defines a new measure of half-hourly demand which it terms adjusted demand. This is equal to the ratio of the half-hourly demand divided by the average half-hourly demand for the season. Hence it shows how large or small the demand in any half hour is compared to the 54 Despite their name, such models are, in fact, highly parameterised in the sense that they contain many parameters that need to be estimated. However, they do not constrain the relationship between demand and temperatures to follow a simple functional form. Review of maximum demand forecasting methodology

72 54 Frontier Economics November 2013 season s average half-hourly demand. Figure 19 presents the algebraic form of the estimating equation, with adjusted demand as the dependent variable. Figure 19: Adjusted demand form of the Monash estimation model Estimation model in adjusted demand form where is the adjusted or standardised half-hourly demand at half hour. It shows how much larger or smaller the demand at half hour is compared to the average halfhourly demand for the season 55 Other terms are as defined in Figure 18 Source: Adapted by Frontier Economics from Monash reports Model selection and goodness of fit When estimating the model shown in Figure 19, a model selection procedure is used to determine the final set of explanatory variables. There are potentially 29 temperature variables and 3 calendar effect variables. To determine which of temperature/calendar variables could be deleted from the model, a crossvalidation process is used in an ordered general-to-specific approach that could be included in the model. For a given specification the model is fitted twice to the data for every half hour; first excluding the second last year of data, and then last year of data. The average out-of-sample mean square error (MSE) is then calculated for the models fit to the omitted data for the time periods 12noon 8.30pm. The specification with the smallest MSE is chosen as the preferred model. The fit of the preferred models to the historical data, as measured by the R 2 criterion, is typically very good for all half-hourly demand models Dynamic bias adjustment As a further check on the adequacy of the models, Monash plots the residuals of the half-hourly estimated models against the predicted adjusted demand values. 55 Although the Monash group refers to this as adjusted demand, it is more accurately described as relative demand, since it is a ratio the ratio of actual demand relative to average demand. Note that the vertical axes in the bottom panels of Figures 10 and 34 of the Monash regional reports wrongly label the adjusted demands as being in GW when they are, in fact, dimensionless. To avoid confusion, we will continue to refer to the ratio as adjusted demand. Review of maximum demand forecasting methodology

73 November 2013 Frontier Economics 55 In previous years, Monash has found that these plots indicate that there is a bias in the models often a model s predicted demands deviate from the actual demands for extremely high and/or extremely low values of predicted demand. To compensate for this bias, Monash makes a bias adjustment to the models when necessary. We understand that this bias adjustment consists of adding a linear term to the tails of the model. These linear terms are determined subjectively. Recent investigations by Monash have revealed that the required bias adjustment terms may change over time. Hence, for the 2013 NEFR, Monash has opted to not use the full period of fitted data to determine the bias adjustment terms, but only the data for the last few years, from 2006 to Monash refers to this as a dynamic bias adjustment Model improvements For the 2013 NEFR, Monash implemented several improvements to the maximum demand forecasting methodology. Two of these were implemented into the estimation model, namely: (1) adding PV load to NLI load prior to undertaking the modelling, and (2) making the bias adjustment dynamic. The first change accounts for the fact that PV generation is becoming an increasingly important element in providing electricity to end users. Including PV as part of the dependent variable leads to superior model fits. The second change addresses the fact that the bias in the estimation model does not seem to be constant over time. 6.3 Forecasting model Model specification The algebraic form for the forecasting model used by Monash is shown in Figure 20. The new term in this model is. This term takes the place of in the estimation model in Figure 18, the average half-hourly demand for the season. The reason is that the average half-hourly demand for the season is not known for future years but needs to be forecast. These forecasts are derived from AEMO s electricity consumption models. The model in Figure 20 represents the estimated version of the forecasting model and can be used to obtain forecasts. The functions and the parameters in the expressions, and are all known in the estimated model. The only terms whose future values are not known are the variables in the model. Hence, in order to calculate a forecast, all that is needed is to substitute the projected future values for the variables into the equations; the calendar Review of maximum demand forecasting methodology

74 56 Frontier Economics November 2013 variables, the temperature variables, and the variables in the electricity consumption model (namely, income, price, population, CDD and HDD). Figure 20: General algebraic form of the Monash forecasting model Forecasting model where is the forecast for half-hourly NLI demand at a future half hour represents a set of calendar variables at half hour represents the simulated temperature variables assumed to affect demand at half hour represents the projections for half hour of the variables appearing in the AEMO electricity consumption model is the forecast for the average half-hourly electricity consumption derived from the AEMO electricity consumption model for the season that half hour belongs to,, and are estimated parameters (e.g. temperature sensitivity and price elasticity) Other terms are as defined in Figure 18 Source: Adapted by Frontier Economics from Monash reports Forecasting probability distributions and POEs The previous section describes how the forecasting model in Figure 20 can be used to obtain point forecasts for half-hourly demand. The Monash demand forecasting approach is more sophisticated than that. Instead of providing a point forecast for demand for each half hour in the forecast horizon, it aims to provide the full density function of future peak demands. Bootstrap simulation techniques are used to produce 1,000 different years of temperature projections, so that, for each future half-hour, 1,000 possible forecasts for demand are generated. Monash achieves this by splitting past years up into sub-periods and then creating synthetic temperature profiles by combining sub-periods from different years (e.g. June s temperatures might come from 2003 and July s from 2007). Each of the sub-periods is also allowed to have different lengths. This approach is called a Review of maximum demand forecasting methodology

75 November 2013 Frontier Economics 57 double seasonal block bootstrap with variable blocks. All time periods within each block also receive a small random shock to their temperatures. 56 In addition to the simulated weather scenarios, Monash also makes an allowance for the impact of climate change on temperatures, and incorporates different PV scenarios. The results of these simulations can be summarised as a probability distribution for the forecast demand for each half hour. This is analogous to a histogram for the possible future outcomes for the demand in a given half hour. The simulation process also produces probability distributions for maximum seasonal demand and maximum weekly demand. By summarising the forecast loads across all half hours in the season, it is possible to produce a probability distribution for half-hourly loads from which one can readily determine the POE value for any given level of probability Model improvements For the 2013 NEFR, Monash implemented several improvements to the maximum demand forecasting methodology. One of the limitations of Monash s methodology in the past was that it implicitly assumed that load factors remain constant over time. This is a consequence of the model s specification. The adjusted demands do not depend on time, or on the size of the average load. This is equivalent to assuming that the relationship between maximum demand and average demand does not change over time, apart from the short-term effects of temperatures and calendar effects (see Figure 19). For the 2013 NEFR Monash undertook an investigation into the historical movements in load factors. 57 The analysis shows that load factors have varied over the years, in some regions by more than 5% in the case of summer loads. Monash has introduced a number of measures to help address the variations in the load factors over time. These measures primarily affect the forecasting model, not the estimation model. The main measures are: adjusting the price elasticities in the electricity consumption model for high levels of demand 56 The Monash approach builds on the work of McSharry, P, Bouwman S. and Bloemhof, G. (2005), Probabilistic forecasts of the magnitude and timing of peak electricity demand, IEEE Transactions on Power Systems, 20, pp Fan, S. and R.J. Hyndman (2013), Load factor analysis in the National Electricity Market and the implications for the peak demand forecast, Monash University Business and Economic Forecasting Unit [Monash load factor report]. Review of maximum demand forecasting methodology

76 58 Frontier Economics November 2013 using the simulated temperature profiles to obtain corresponding values for the CDD, HDD variables in the electricity consumption component of the model. The second of these changes is straightforward. But the process for adjusting the price elasticities is more involved. We review that process in the next section Adjusted price elasticities Monash is primarily interested in modelling and forecasting demand when demand levels are high. For the 2013 NEFR, Monash undertook a separate study to investigate how price elasticities vary at different levels of demand and time periods. 58 This study showed that price elasticities do indeed vary by the level of demand. To allow for the fact that the price elasticity might be different at high levels of demand compared to low levels of demand, Monash adjusts the price elasticities in the average electricity consumption component of the forecasting model. These price elasticities, which are a component of the parameter vector in the average demand model, were estimated as part of developing the electricity consumption models. The scaling factors used by Monash to adjust the price elasticities for high levels of demand are obtained from Monash s elasticity study by taking the median of the estimated price elasticities across different time periods at the 95 th demand percentile, divided by the overall price elasticity for seasonal average demand. These scaling factors are applied to the estimated price elasticities in AEMO s electricity consumption models to obtain the adjusted elasticities. We summarise the scaling factors mentioned in the five regional Monash reports in Table Fan, S. and R.J. Hyndman (2013), The price elasticity of electricity demand in the National Electricity Market, Monash University Business and Economic Forecasting Unit [Monash elasticity report]. Review of maximum demand forecasting methodology

77 November 2013 Frontier Economics 59 Table 16: Scaling factors applied by Monash to obtain adjusted price elasticities Jurisdiction Elasticity adjustment Comment NSW 0.2 / 0.17 VIC No adjustment QLD 0.09 / 0.04 SA 0.08 / 0.12 TAS 0.2 / 0.17 In Table 10 of the Monash elasticity report, the median elasticity at the 95% point is 0.44 rather than 0.2 Note: The elasticity adjustments are reported in Monash s forecasting reports for the individual regions. The numerators for the scaling factors generally correspond to the median elasticities at the 95% point as reported in the summary tables in Monash elasticity report. Monash does not explicitly report the information required to calculate the denominators in the scaling factors. 6.4 High level review of additional modelling steps Post-modelling adjustments AEMO estimates the maximum demand reductions from energy efficiency based on the estimated annual energy savings divided by the average load factor for a given region/season. For example, if energy savings due to energy efficiency policies are 1,000GWh and the average load factor for the region/season is 66% then the estimated reduction in maximum demand is calculated as: 1,000GWh /8,760 hours /66% load factor = 173MW reduction in maximum demand This implies that all reduction in load due to energy efficiency is assumed to be in proportion to the average load factor for all energy use in the market as a whole. This is a simple and transparent approach, though it does not consider the potential source of energy savings from energy efficiency. For example, policies targeted at air-conditioning would likely have a greater impact on reducing peak demand use than average demand use. This might have the effect of reducing the peakiness of the market (increasing average load factors) but the AEMO assumption means that average load factor will be unchanged as a result of the energy efficiency policies. However, more detailed analysis based on the actual energy efficiency policies may be unreasonably complicated. As AEMO correctly points out, the simplified approach that has been adopted will likely result in a conservative estimate of savings from energy efficiency policies (where conservative is defined as erring Review of maximum demand forecasting methodology

78 60 Frontier Economics November 2013 on the low side of demand savings or the high side of maximum demand forecasts) Treatment of rooftop PV generation impact on maximum demand The half-hourly load traces of PV used by Monash in the regional maximum demand models are provided by AEMO. Based on the documentation provided to Frontier, AEMO s approach generally appears reasonable. In the 2013 FMIP (pp. 4-40), AEMO states that: Rooftop PV generation at the time of regional maximum demand was forecast for each region by multiplying the region s forecast installed capacity by a factor reflecting the ratio of rooftop PV output on high demand days to installed capacity. The data sources used in calculating the contribution to maximum demand forecasts per NEM region included the half-hourly rooftop PV historical contribution factor and the half-hourly native demand. The following steps were taken to derive the rooftop PV contribution to maximum demand forecasts: 1) Calculate average time of maximum demand based on historical maximum native demand times. 2) Gather rooftop PV performance at the average maximum demand time +/-1 hour for each historical maximum demand day. 3) Calculate the average rooftop PV contribution from the sample data and multiply this with the installed capacity forecast to derive the contribution to maximum demand forecasts. The use of average output during periods of maximum demand (and 1 hour either side) is inconsistent with AEMO s use of an 85% confidence level when estimating the contribution of wind output to peak demand. Using average output will overstate how much reliable solar output can be expected during maximum demand periods compared with an 85% confidence level. We agree with AEMO s caveat that maximum demand may shift in the future as rooftop PV capacity increases (reducing the rooftop PV contribution at peak), and that this should be investigated in future NEFRs We were not able to review the report by George Wilkenfeld and Associates which AEMO cited for its analysis as it is not publicly available. The report is: George Wilkenfeld and Associates, (2012), Review of Impact Modelling for E3 Work Program, unpublished report to the Department of Climate Change and Energy Efficiency (DCCEE). 60 We have been informed that AEMO tested the implementation of the methodology used for the wind study to determine the maximum demand contribution of rooftop PV but that this produced results that were lower than historical actuals. AEMO will revisit the rooftop PV forecasting methodology for the 2014 NEFR. Review of maximum demand forecasting methodology

79 November 2013 Frontier Economics 61 7 Findings and recommendations for AEMO s maximum demand models In this section we discuss the findings from our review of Monash s maximum demand forecasting methodology and present our recommendations for future refinement of the methodology. Section 7.1 presents an overview of our findings. In sections 7.2 and 7.3 we present and discuss our main recommendation for Monash s estimation model and forecasting model respectively. Some additional recommendations are presented in section 7.4. In section 7.5 we comment on the computational implications of our recommendations. We advise that Monash give high priority to addressing our main recommendations (those summarised in sections 7.2 and 7.3) and attempt to incorporate them in the 2014 NEFR. We recognise, however, that some the recommendations may involve significant effort in terms of analysis and investigations, and may have to be implemented over a longer period. 7.1 Overall assessment The Monash peak demand model possesses many of the desirable elements of a peak demand model. The modelling of the weather variables is very detailed and sophisticated. The modelling of the calendar effects also seems appropriate. Further, we believe that the Monash team s approach to calculating the distributions of maximum demand and probability of exceedance (POE) levels is statistically sound. Electricity distribution and retail businesses have access to much more data from billing records and customer surveys than is available to AEMO. This would enable them to develop a better understanding of customer consumption patterns to inform their forecasting models. However, we are not aware of any other energy organisation in Australia that has forecasting models that are technically as sophisticated as AEMO s maximum demand models. The recommendations made in the following sections should therefore be seen as part of the continuing model improvement process that Monash is engaged in. 7.2 Main recommendations for the estimation model Functional form for weather variable impacts The fact that Monash undertakes a bias adjustment to its estimated semiparametric models suggests that there is a functional misspecification in the model. Monash s approach to dealing with this bias is to make a post-modelling adjustment to remove the bias. Findings and recommendations for AEMO s maximum demand models

80 62 Frontier Economics November 2013 An alternative approach would be to re-visit the way Monash specifies the cubic splines in its semi-parametric models. The location and number of knots could be varied, and more weight could be given to the fit of the splines at extreme temperatures. We recommend that Monash investigate whether this approach can reduce or eliminate the need for the post-modelling bias adjustment Testing for structural breaks Our review of AEMO s electricity consumption models provided strong evidence of structural breaks in four of the NEM regions around 2006 and The graphs in Monash s load factor report similarly show evidence of a change in the load factors at around the same time. Further, in implementing its dynamic bias correction method, Monash has focused its analysis on the period 2006 to These observations lead us to have some concerns about possible structural breaks in Monash s estimation models. We recommend that for the 2014 NEFR Monash undertake formal statistical tests to investigate whether these concerns are justified Temperature correcting average demand At present Monash calculates adjusted demands taking the ratio of actual demands divided by the average half-hourly demand for the season that the actual demands fall in. This implies that customers respond not to temperatures per se but rather to temperatures relative to an average temperature for the season. We have some doubt that this characterises how temperatures affect demand. We think it is likely that consumers respond to the amount by which temperatures deviate from long-term average temperatures for the season. We recommend that, as an alternative approach, Monash investigate normalising the average demands to standard temperatures (or standard CDDs and HDDs) before calculating standardised demands. It is an empirical question whether this would also improve the fit of the estimation models. In a separate note we have indicated how such a temperature normalisation could be implemented. 7.3 Main recommendations for the forecasting model Adjusted price elasticities To allow for the fact that price elasticities might vary by the level of demand, Monash applies a scaling factor to the price elasticities in electricity consumption models to obtain adjusted high demand price elasticities. While we commend Monash s efforts to incorporate different price elasticities in its forecasts for peak demand and average demand, the current approach has some undesirable consequences resulting from the fact that the scaling factor is applied to the Findings and recommendations for AEMO s maximum demand models

81 November 2013 Frontier Economics 63 average seasonal demand component of the model, rather than to high demand levels per se. In other words, Monash applies the high demand price elasticities to all levels of demand, not just high demands. Hence, results for frequency distributions for half-hourly demands in the regional Monash reports (rather than maximum demands) need to be interpreted with caution. It is difficult to know how to interpret these results. This applies to a number of figures and tables in Monash s regional reports. 61 One implication of Monash s conclusion that price elasticities differ according to the level of demand is that the convenient decomposition of half-hourly demand into an average demand component driven by economic variables, and a shortterm component driven by temperature and calendar variables is no longer tenable. If the price elasticity depends on the half-hourly levels of demand, then these two components will have to be integrated. Another implication of Monash s conclusion that the price elasticity depends on the level of demand is that this makes the price elasticity endogenous. This would lead to quite complex specifications and estimation issues. A preferable approach to incorporating a flexible price elasticity is to adopt a functional coefficient modelling approach allowing the price elasticity to be influenced by the exogenous variables that drive demand, rather than demand itself. However, this would again necessitate a blending of the two components of the model, since temperature would be one of those exogenous drivers. We recommend that in order to guide the further development of the elasticity adjustment, Monash develop an explicit statistical/economic specification to provide a coherent framework for how price elasticities vary across different time periods and different levels of demand Monash s elasticity study We also have some comments on Monash s elasticity study that underpins the elasticity adjustments. The models used in this study have similar drivers to AEMO s long-run energy consumption models, i.e. price and income variables, as well as CDD and HDD variables. However, Monash s models have a different specification to AEMO s electricity consumption models, and the estimation procedure seems to pay no attention to possible non-stationarity in the data. The models also do not seem to have 61 In the final round of forecasts, these comments do not apply to Victoria, since the adjustment methodology results in the high demand price elasticity being the same as the average demand level price elasticity. 62 We note that the description of Monash s approach to estimating elasticities at different levels of demand is not detailed enough to fully understand the approach. Findings and recommendations for AEMO s maximum demand models

82 64 Frontier Economics November 2013 undergone the same degree of model development and diagnostic testing as AEMO s electricity consumption models. As a result, the electricity consumption models used by Monash in its elasticity study are likely to have less satisfactory statistical properties than AEMO s electricity consumption models. In our view, the models used to underpin the adjustments made to the price elasticities for high demand levels should be regarded as in an early stage of development. We therefore recommend that for the 2014 NEFR a new elasticity study be undertaken that uses models for electricity consumption that are compatible with AEMO s models for electricity consumption. 7.4 Additional recommendations Model assessment There are a number of ways to evaluate a forecasting model s performance. One powerful tool for improving a model is to decompose the forecasting/prediction errors of past versions of the model into the contributions to the forecasting errors due to each of the drivers versus the contributions that are due to modelling weaknesses. Such a decomposition helps to identify where future development effort should be directed. We have been advised that Monash undertakes such evaluations; we recommend that the results of this evaluation be included in Monash s reports for future NEFRs Interpretation of estimated coefficients An important part of assessing a forecasting model is to examine the reasonableness of the estimated parameters in the model; for example, the size and sign of a price elasticity. We were not able to perform this task for the Monash models because we were not provided with sufficient detail on the estimated models. We would have liked to assess, for example, whether the coefficients on the temperature variables are intuitively reasonable; or the impact of different days of the week on demand. We recommend that for future reviews, Monash be asked to provide sufficient detail on the estimated models to enable the reviewers to undertake such checks Treatment of uncertainty The KEMA review 63 indicated that it is world best practice to interpret POE as reflecting the uncertainties in all the components of the forecasts, not just weather. KEMA suggested that planners consider developing a probabilityweighted average of the low, medium and high economic scenario forecasts for 63 KEMA (2005), Review of the Process for Preparing the SOO Load Forecasts, p. 47. Findings and recommendations for AEMO s maximum demand models

83 November 2013 Frontier Economics 65 each POE level in order to provide forecasts with 10%, 50% and 90% long-run POEs that take account of the uncertainty in the economic projections. The bootstrapping approach implemented in the Monash model deals with uncertainty in regard to future temperature profiles. Monash s bootstrapping approach could be extended to deal with uncertainty in the economic variables (i.e. price and economic activity). To implement this idea, probabilities, similar to POEs, would have to be assigned to the high, medium and low economic scenarios for the aggregate energy consumption model, and a joint distribution would need to be developed for the economic drivers in that model. Since the weather variables in the peak demand model are independent of the future values for the economic drivers, future values for the economic drivers could be drawn independently of the bootstrapped temperatures. Another source of uncertainty derives from the fact that the parameters in the aggregate energy consumption and peak demand models have been estimated and are not known with certainty. The standard errors of the estimated parameters give an indication of the degree of uncertainty in these estimates. We recommend that Monash investigate how its simulation methodology can be extended to incorporate these additional sources of uncertainty in the forecasts. We accept that this would be a long-term objective Whole-of-year POEs Monash currently estimates POEs separately for summer and for winter. While these POEs are referred as annual POEs, for a region where the annual peak can occur in either summer or winter, the seasonal POEs do not provide a true indication of the probability of exceedance for the whole year. We recommend that Monash supplement its current set of POEs to conclude whole-of-year POEs for regions where there is some probability of peak demand switching between seasons. 7.5 Computational implications The Monash model is much more computer intensive to develop and operate than AEMO s electricity consumption model or the models typically used by Australian energy consumption businesses. We recognise that some of our suggestions would add to the computational effort, and it could be argued that it is not feasible to consider our recommendations due to computational limitations. In regard to this point we make the following observations: Most of our recommendations relate to the estimation stage of the modelling process. This is far less computer intensive than the forecasting stage, since it does not involve computer-hungry bootstrapping, and the estimation models Findings and recommendations for AEMO s maximum demand models

84 66 Frontier Economics November 2013 are essentially linear-in-parameter regression models (though somewhat complex ones). Since Monash first developed its general modelling approach around 2007, there has been a considerable increase in the computational speed of individual computers; moreover, it is now quite common for organisations to link a number of personal computers to create a distributed computing network with supercomputer-like capabilities. In view of these observations, we do not think that computational limitations are a significant barrier to the implementation of most of our recommendations. The one recommendation where we think computation might be an issue is our suggestion to account for additional sources of uncertainty in the forecasts; for example, by simulating many future economic scenarios and/or treating the estimated coefficients in the forecasting equations as being stochastic and simulating their values from the relevant sampling distributions. We recognise that this is an aspirational objective for the longer-term development of the modelling approach. Findings and recommendations for AEMO s maximum demand models

85 November 2013 Frontier Economics 67 8 AEMO s forecasting methodology compared to international best practice Frontier was asked to compare AEMO s forecasting methodology to forecasting methodologies used by organisations (globally) with similar responsibilities as AEMO. Due to a lack of information made publicly available by such organisations, it was not possible to undertake a thorough comparison of forecasting methodologies (including inputs, assumptions, and modelling techniques). Our review is therefore limited to a high level comparison of specific analytical steps. We have also extended the scope of our review to include organisations that develop electricity consumption and/or maximum demand forecasts but do not have the same responsibilities as AEMO. The organisations included in our review are: AEMO-equivalent organisations PJM Interconnection (PJM), United States Alberta Electric System Operator (AESO), Canada Transpower, New Zealand National Grid Electricity Transmission (NGET), United Kingdom Other organisations (all in the United States) U.S. Department of Energy Southern California Edison California Energy Commission. We first present insights from our review for all seven organisations. We then provide a more detailed overview of forecasting methodologies used by the four AEMO-equivalent organisations. Our review is based on publicly available information contained in documents such as: annual electricity consumption and maximum demand forecasting reports, model documentation (e.g. forecasting manuals), and third-party reviews of forecasting models. 8.1 Summary of findings Transparency Our search for publicly available information on forecasting methodologies used by AEMO-equivalent organisations revealed that AEMO, in our opinion, is leading the way in terms of transparency of its forecasting process. In addition to publishing electricity consumption and maximum demand forecasts, AEMO also publishes annually an accompanying report titled Forecasting Methodology AEMO s forecasting methodology compared to international best practice

86 68 Frontier Economics November 2013 Information Paper, which contains a detailed description of AEMO s forecasting procedures. The amount and type of information made publicly available by other organisations varies considerably. The most detailed information on forecasting methodologies seems to be available in peer review reports. However, in most overseas jurisdictions, peer review reports are not produced on a regular basis, and it is not always clear which of the recommendations in these reports have been implemented. Forecasting horizon AEMO produces electricity consumption and maximum demand forecasts for a 20-year horizon. Across the reviewed organisations, the forecasting horizon spans from 10 years (NGET) to 20 years (AESO). Forecasting methodology for electricity consumption The dominant methodology for forecasting electricity consumption seems to be the top-down econometric (i.e. regression) approach with economic variables as the main drivers. This is the approach used by AEMO. The sophistication of estimated regression models varies widely across the organisations, and to some extent is driven by data availability. Of the reviewed AEMO-equivalent organisations, Transpower appears to be the only one that does not include any weather variables in its econometric electricity consumption models (although it plans to do so as part of future model development). Similarly to AEMO s approach, PJM, Transpower, and NGET all use aggregate data in their electricity consumption estimations. Only AESO models separately electricity consumption by different customer segments. Forecasting methodology for maximum demand AEMO and PJM both model system peak demand or system half-hourly demand directly using an econometric or a statistical model. This approach is preferred to an approach that derives peak demand forecast from an electricity consumption forecast by applying a load factor. In terms of sophistication, both use sophisticated models which fall into the category of frequency distribution techniques, with future synthetic weather scenarios created through Monte Carlo simulations of past weather conditions. NGET derives system maximum demand forecasts by applying a load factor to forecast electricity consumption. 64 Transpower uses a similar load factor approach, but allows the load factor to vary randomly from year to year. This variation in the load factor was included in the models to address criticism from 64 It is not clear from the publicly available information whether NGET uses a constant load factor. AEMO s forecasting methodology compared to international best practice

87 November 2013 Frontier Economics 69 two peer reviewers who suggested that a more appropriate approach to modelling maximum demand would be to model it directly. Transpower s approach to modelling maximum demand is unique as it uses a socalled ensemble approach which involves combining forecasts from several different forecasting models. The models used to prepare the 2013 Annual Planning Report include: an econometric model of electricity consumption (estimated by Transpower) to which a load factor is applied; a load-factor model based on electricity consumption projections from the New Zealand Ministry of Economic Development; and a trend line extrapolation model. In his review of Transpower s forecasting methodology, Professor Rob Hyndman, of Monash University, states that there is a considerable academic literature on the value of combining forecasts, especially when forecasts are generated using different types of models. 65 Economic activity driver In its regional electricity consumption models, AEMO defines economic activity in terms of real GSP per capita or real SFD per capita. Since the energy consumption being modelled econometrically by AEMO is a combination of residential, commercial and small industrial users, the GSP/SFD variable may not be capturing adequately the changes in the customer mix (and hence the energy intensity of the economy). Similarly, Transpower uses GDP as the main economic driver in its electricity consumption and maximum demand models. However, in its 2013 Annual Planning Report it echoes our concern that structural changes in the economy affecting energy consumption are not well captured by a GDP variable: More recently, real GDP appears to be growing at higher rates than demand. Partly, this appears to be a result of different sectors of the economy growing at different rates, and in particular, a result of a decline in demand in the energy-intensive industrial sector. We are currently investigating whether these sectoral differences in growth are adequately being accounted for in our modelling. Of the reviewed AEMO-equivalent organisation, PJM appears to use the most elaborate economic variable, expressed as an index. This economic index variable is a weighted combination of U.S. GDP, Gross Metropolitan Product (GMP), personal income, population, households, and non-manufacturing employment, with the shares of electricity sales by customer class (i.e. residential, commercial, and industrial) used as weights. 65 Professor Hyndman s report is available at AEMO s forecasting methodology compared to international best practice

88 70 Frontier Economics November 2013 Handling of uncertainty AEMO handles uncertainty in its electricity consumption forecasts by considering three different economic scenarios. Uncertainty in future maximum demand is handled by combining the economic scenario analysis with Monte Carlo simulations of future weather scenarios. One or both of these approaches seem to be used by the reviewed organisations. As explained previously, Transpower is somewhat unique in its approach. It does not rely on a single model, but rather combines forecasts from several different models. Model validation and testing In the 2013 Forecast Accuracy Report (FAR), AEMO provides an assessment of the forecasts published in the 2012 NEFR. From the reviewed documents, AESO and PJM also test their models for forecasting accuracy. In general, not enough information/explanation is provided by any of these organisations on the actual methodology used to assess their models forecasting accuracy. Our review indicates that it is a standard practice to have one s models reviewed by an independent external party, as is currently done by AEMO. AESO s, PJM s and Transpower s models have been peer-reviewed; with PJM and Transpower making the peer-review reports publicly available. Projections for economic and demographic drivers AEMO currently sources all historical and projected economic and demographic data from a single source, NIEIR. In the 2013 NEFR AEMO states that it compared NIEIR price projections to, and found them to be broadly in line with, the price projections published by regulatory agencies in each state. In the 2013 Economic Outlook Information Paper, AEMO assessed NIEIR s forecasts of GSP/SFD for reasonableness by comparing them to forecasts from state government agencies and Deloitte Access Economics. Our literature review of international practices indicates that many organisations validate the projections of economic drivers obtained from one source against data from other publicly available or commercial sources; while other organisations source such data from multiple data providers. For example, AESO sources projections for economic variables from the Conference Board of Canada s annual long-term provincial economic forecast. AESO then validates the reasonableness of these economic projections against other recognised thirdparty economic forecasts, including IHS Global Insight, major Canadian banks, and the Government of Alberta AESO (2012), 2012 Long-term Outlook, p.8, available at AEMO s forecasting methodology compared to international best practice

89 November 2013 Frontier Economics 71 The U.S. Department of Energy s Energy Information Agency (EIA) publishes its energy outlook annually, which includes electricity consumption forecasts over a 30-year horizon. In its 2013 publication, EIA assessed its assumption on the average annual economic growth projections over different time periods against projections from 10 different sources, including other government agencies, commercial data providers, private companies publications, and a survey of expert opinion. 67 To produce electricity consumption forecasts, Southern California Edison (SCE) sources data from two established third-party forecasting sources, IHS Global Insight and Moody s. SCE has found that one of these two sources tends to be too optimistic about the Californian economy (and in particular manufacturing output projections), while the other one tends to be too pessimistic. SCE states that these differences can result in a large discrepancy between the two sources projections. When this occurs, SCE averages the two sets of data to arrive at a single projection. 68 The California Energy Commission (CEC) collects economic and demographic projections from a number of commercial, academic, and government sources (such as IHS Global Insight, Moody s, Oxford Economics, University of California at Los Angeles, and the California Department of Finance). 69 CEC then selects the base, high, and low case scenarios from the range of projections provided by these different forecasting sources U.S. Energy Information Administration (2013), Annual Energy Outlook 2013 with Projections to 2040, available at 68 Southern California Edison (2013), California Energy Commission Docket No. 13-IEP-1C/1L Comments on Workshop on Economic, Demographic, and Energy Price Inputs for Electricity, Natural Gas and Transportation Fuel Demand Forecasts, available at Comments_on_Workshop_on_Econ omic_demographic_an_workshop_ _tn pdf. 69 The California Energy Commission is the state's primary energy policy and planning agency. One of its main responsibilities is forecasting the state s future energy needs. CEC produces total energy forecasts (by sector) and peak demand forecasts. 70 California Energy Commission (2013), 2013 IEPR Preliminary Electricity and Natural Gas Demand Forecast General Approach and Assumptions, available at 19_workshop/presentations/03_Kavalec_Chris_Demand_Forecast_Assumptions.pdf. AEMO s forecasting methodology compared to international best practice

90 72 Frontier Economics November Review of forecasting procedures PJM Interconnection (United States) PJM Interconnection, L.L.C. (PJM) is a regional transmission organization (RTO) in the United States that coordinates the movement of wholesale electricity in all or parts of 13 states and the District of Columbia. Each year, PJM produces electricity consumption and maximum demand forecasts for a 15-year time horizon. Forecasts are produced for each of the 18 transmission zones in the RTO, and for the RTO as a whole. The maximum demand forecasting models currently in use were developed in 2005 to provide inputs to PJM s system planning functions. The most comprehensive description of PJM s maximum demand modelling process (including selection of the modelling techniques and explanatory variables) is available in its 2007 White Paper titled PJM Load/Energy Forecasting Model (hereafter the 2007 White Paper). 71 The maximum demand models were reviewed by the Brattle Group, a consultancy, in 2006, and their critique and recommendations presented in a report titled An Evaluation of PJM s Peak Demand Forecasting Process (hereafter the 2006 Brattle Review). The most recent document publicly available on the PJM s forecasting process is PJM Manual 19: Load Forecasting and Analysis, published in While this document offers some insight into how the models have evolved over time (namely with respect to explanatory variables included in the regressions), its focus is mainly procedural. The latest long-term electricity consumption and maximum demand forecasts for the period 2013 to 2028 were published in January 2013 in PJM Load Forecast Report. 73 This report does not contain any information on the forecasting methodology. At the onset of its long-term maximum demand model development, PJM considered two types of modelling techniques. The two techniques, both using an economic variable as a main demand driver, were: (1) a multiple regression (i.e. econometric) technique, and (2) a neural network technique. Although PJM acknowledged that a neural network modelling technique may be better in estimating the nonlinear relationships inherent in system demand, it chose to develop its maximum demand forecasting models using a top-down econometric technique. The reasons given were that: (1) the econometric models results (i.e. coefficient estimates) are easier to understand and interpret; and (2) econometric 71 Available at 72 Available at 73 Available at AEMO s forecasting methodology compared to international best practice

91 November 2013 Frontier Economics 73 models are more compatible with use in scenario analysis (the 2007 White Paper, p. 4). Electricity consumption In 2006, to satisfy PJM corporate needs, maximum demand models were modified to produce electricity consumption forecasts. This modification, as explained in the 2007 White Paper, consisted of changing the models dependent variable from daily maximum demand to daily energy consumption. Very little additional information is publicly available on PJM s electricity consumption forecasting process. Maximum demand PJM s forecasting methodology consists of the following analytical steps: 1. Model parameter estimation: estimate an econometric forecasting equation that specifies the relationship between each PJM zone non-coincidence daily peak (NCP) load and load drivers (which include economic variables, weather variables and calendar effects). 2. Forecast drivers: obtain projections for the driver variables from a third-party expert (economic variables) and through Monte Carlo simulations (weather variables). 3. Derive Forecasts: substitute the projected values for the drivers into the NCP load forecasting equation and calculate daily NCP load forecasts for each zone. Below we discuss these analytical steps in more detail. Model parameter estimation The PJM zonal NCP model specification consists of over 50 explanatory variables, including calendar effects, weather variables, and economic variables. Because models are estimated with daily data, most of the explanatory variables capture intra-week and intra-year variations in consumption patterns (i.e. calendar effect variables include: day-of-the-week, month-of-the-year, holidays, minutes of daylight). Weather variables used are expressed in terms of HDD and CDD, 74 and are lagged over three days to capture the impact of heat/cold build-up on demand. The impact of economic conditions is currently modelled using an economic index variable (see Figure 21). This index variable is a weighted combination of U.S. GDP, GMP, personal income, population, households, and nonmanufacturing employment, with shares of electricity sales by customer class (i.e. residential, commercial, and industrial) used as weights. This variable captures 74 The HDD variable is calculated using wind-speed adjusted dry bulb temperature, while CDD is calculated using a temperature humidity index. AEMO s forecasting methodology compared to international best practice

92 74 Frontier Economics November 2013 changes in the value of goods produced and also changes in the composition of the economy. 75 In cases where a transmission zone experienced a large and unexpected change in load, a dummy variable is added to the model as a way of capturing the load adjustment. Models are estimated using historical data, with data beginning in January The preferred forecasting models are selected based on goodness-of-fit and on in-sample and out-of sample performance. Figure 21: Economic Index Source: PJM (2013), PJM Manual 19: Load Forecasting and Analysis, p. 17. Forecast drivers Inputs for the economic index projection are sources from Moody s Analytics. The weather projections are obtained using a Monte Carlo simulation of historical hourly weather data (with weather data going back to early 1970s). 75 The economic index variable is one illustration of how the PJM s forecasting models have evolved over time. The original models, developed in the mid 2000s, only included GDP and/or GMP (see the 2007 White Paper). AEMO s forecasting methodology compared to international best practice