An Examination of Temporal Trends in Electricity Reliability Based on Reports from U.S. Electric Utilities

Transcription

1 LBNL-XXXX ERNEST ORLANDO LAWRENCE BERKELEY NATIONAL LABORATORY An Examination of Temporal Trends in Electricity Reliability Based on Reports from U.S. Electric Utilities Joseph H. Eto, Kristina Hamachi LaCommare, Peter Larsen, Annika Todd, and Emily Fisher The work described in this report was funded by the Office of Electricity Delivery and Energy Reliability of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231.

2 Disclaimer This document was prepared as an account of work sponsored by the United States Government. While this document is believed to contain correct information, neither the United States Government nor any agency thereof, nor The Regents of the University of California, nor any of their employees, makes any warranty, express or implied, or assumes any legal responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by its trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof, or The Regents of the University of California. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof, or The Regents of the University of California. Ernest Orlando Lawrence Berkeley National Laboratory is an equal opportunity employer.

3 An Examination of Temporal Trends in Electricity Reliability Based on Reports from U.S. Electric Utilities Joseph H. Eto, Kristina Hamachi LaCommare, Peter Larsen, Annika Todd, and Emily Fisher Ernest Orlando Lawrence Berkeley National Laboratory 1 Cyclotron Road, MS Berkeley CA The work described in this report was funded by the Office of Electricity Delivery and Energy Reliability of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231.

4

5 Abstract Since the 1960 s, the U.S. electric power system has experienced a large-scale electricity blackout roughly once every ten years. Each has been a vivid reminder of the importance society places on the continuous availability of electricity at any time of day or night. Each brings forth calls for changes in the ways the industry should be regulated, and on how investment and operating decisions should be made to maintain or enhance reliability. At the root of these calls are implicit and explicit judgments on what reliability is worth and how much society should be prepared to pay for reliability. In principle, information on the actual reliability of the electric power system and on how proposed changes would affect that reliability ought to play a role in making these judgments. This paper seeks to inform discussions on the reliability of the U.S. electric power system by assessing trends in U.S. electricity consumers total reliability experience. It augments prior investigations, which have focused only on power interruptions originating on the bulk power system, by considering interruptions originating on both the bulk power system and the local distribution system. It accounts for differences among utility reporting practices for power interruptions by employing statistical techniques that remove the influence of these inconsistencies on the trends that are identified. The paper presents an analysis of up to 11 years of electricity reliability information collected from 155 U.S. electric utilities. The questions explored include: 1. Are there are utility-specific differences in reported electricity reliability? 2. Are there trends in reported electricity reliability over time? 3. How are trends in reported electricity reliability affected by the installation or upgrade of an automated outage management system? 4. How are trends in reported electricity reliability affected by the use of IEEE Standard ? i

6

7 Acknowledgments The work described in this report was funded by the Office of Electricity Delivery and Energy Reliability (OE) of the U.S. Department of Energy (DOE) under Contract No. DE-AC02-05CH iii

8

9 Table of Contents Abstract... i Acknowledgments... iii Table of Contents...v List of Figures and Tables... vii Acronyms and Abbreviations... ix 1. Introduction Data Collection and Review Sources of Data Utility reliability metrics Weather Data Retail Electricity Sales Review of Utility-Reported Reliability Data Geographic Representation and Coverage Reliability Data Completeness over Time and by Utility Average SAIDI and SAIFI over Time Outage Management System Major Events Findings from the Statistical Analysis of Reliability Data Reported by Electric Utilities Introduction to the Statistical Methods Used in the Analysis Application of the Statistical Models Findings Are there are utility-specific differences in reported electricity reliability? Are there trends in reported electricity reliability over time? How are trends in reported electricity reliability affected by the installation or upgrade of automated outage management systems? How are trends in reported electricity reliability affected by the use of IEEE Standard ? Summary and Interpretation of Findings...25 References...27 Appendix A...29 v

10

11 List of Figures and Tables Figure 1. Map of U.S. by NERC Region... 7 Figure 2. Number of utilities with SAIDI and SAIFI data... 8 Figure 3. Completeness of time series... 8 Figure 4. Average SAIDI by year with and without inclusion major events... 9 Figure 5. Average SAIFI by year with and without inclusion of major events... 9 Figure 6. Number of utilities by year that installed or upgraded the OMS by NERC region Figure 7. SAIDI Percentage Difference between Using and Not Using IEEE Standard Figure 8. SAIFI Percentage Difference between Using and Not Using IEEE Standard Table Sales of Utilities for Which Data Were Collected, by NERC Region... 6 Table 2. Summary of Utilities Using IEEE Table 3. F-test of the Hypothesis that there are No Fixed Effects Table 4. One-way Random Effects Regression (Model 1): The effect of sales, HDD, CDD, time, and OMS on frequency and duration of interruptions (with major events) Table 5. One-way Random Effects Regression (Model 1): The effect of sales, HDD, CDD, time, and OMS on frequency and duration of interruptions (without major events) Table 6. No Utility Fixed Effects Regression (Model 2): Effect of IEEE, sales, HDD, CDD, time, and OMS on frequency and duration of interruptions (without inclusion of major events) Table A- 1. One-way Fixed Effects Regression for the Effect of Sales, HDD, CDD, time, and OMS on Frequency and Duration of Grid Disruptions Table A- 2. Two-way Fixed Effects Regression for the Effect of Sales, HDD, CDD, and OMS on Frequency and Duration of Grid Disruptions vii

12

13 Acronyms and Abbreviations ASCC CAIDI CDD DAWG DC DOE EIA FRCC HDD HICC IEEE IOU LBNL MED MRO MW NCDC NERC NOAA NPCC OE OMS PUC RFC SAIDI SAIFI SERC SPP TRE WECC Alaska Systems Coordinating Council Customer Average Interruption Duration Index cooling degree-day Disturbance Analysis Working Group District of Columbia U.S. Department of Energy Energy Information Administration Florida Reliability Coordinating Council heating degree-day Hawaiian Islands Coordinating Council Institute of Electrical and Electronics Engineers investor-owned utility Lawrence Berkeley National Laboratory major event day Midwest Reliability Organization megawatt (10 6 watts) National Climatic Data Center North American Electric Reliability Corporation National Oceanic Atmospheric Administration Northeast Power Coordinating Council DOE Office of Electricity Delivery and Energy Reliability outage management system Public Utility Commission Reliability First Corporation System Average Interruption Duration Index System Average Interruption Frequency Index Southeast Electric Reliability Council Southwest Power Pool Texas Regional Entity Western Electricity Coordinating Council ix

14

15 1. Introduction Since the 1960 s, the U.S. electric power system has experienced a large-scale electricity blackout roughly once every ten years. Each has been a vivid reminder of the importance society places on the continuous availability of electricity at any time of day or night. Each brings forth calls for changes in the ways the industry should be regulated, and on how investment and operating decisions should be made to maintain or enhance reliability. At the root of these calls are implicit and explicit judgments on what reliability is worth and how much society should be prepared to pay for reliability. In principle, information on the actual reliability of the electric power system and on how proposed changes would affect that reliability ought to play a role in making these judgments. And, in fact, such practices are common at the local level, for example between an investorowned utility and its state public utilities commission. Yet, to date, application of these principles at the national level remains under-developed because the national-level information represents only a small portion of electricity consumers total reliability experience and the more comprehensive data that are available from individual electric utilities are sometimes inconsistent with one another due to differences in reporting practices, among other things. This paper seeks to inform discussions on the reliability of the U.S. electric power system by assessing trends in U.S. electricity consumers total reliability experience. It augments prior investigations, which have focused only on power interruptions originating on the bulk power system, by considering interruptions originating on both the bulk power system and the local distribution system. It accounts for differences among utility reporting practices for power interruptions by employing statistical techniques that remove the influence of these inconsistencies on the trends that are identified. The principle focus of prior published investigations on the reliability of the U.S. electric power system has been on the reliability of the bulk power system. For example, Amin (2008) suggests that the reliability of the bulk power system has been declining over time based on a simple review of the frequency and size of reported events. The response by Hines et al. (2009) rejects that hypothesis based on a rigorous statistical examination of the same data. Yet, power interruptions originating on the bulk power system do not represent the only source of interruptions experienced by electricity consumers. Hines et al. (2009) find evidence, also corroborated by Eto and LaCommare (2008), that interruptions originating on the bulk power system represent only a small fraction of the power interruptions experienced by electricity consumers. This means that the vast majority of interruptions experienced by electricity consumers are caused by events affecting primarily the electric distribution system. Thus, analysis of power interruptions originating on the bulk power system alone will address only a portion of electricity consumers total reliability experience. Information on electricity consumers total reliability experience is collected routinely by electric utilities. The information collected almost always includes all power interruptions experienced by electricity consumers, including both those originating on the bulk power system, as well as 1

16 those originating on the electricity distribution system. The principle metrics used by utilities to report this information focus separately on the frequency and duration of power interruptions. Previous work examining electric utility practices for reporting reliability information revealed significant variability (Eto and LaCommare 2008). Despite the existence of standards - albeit voluntary ones - promulgated by the industry s professional society, the Institute for Electrical and Electronics Engineers (IEEE), differences in the definition and classification of power interruption events by utilities render direct comparisons problematic and potentially misleading. In this paper, we present an analysis of up to 11 years of electricity reliability information reported by 155 U.S. electric utilities. We first quantify trends in electricity reliability and seek to relate these trends to corollary information on the characteristics of the utilities, the climates in which their customers are served, and on the adoption of advanced technologies for recording power interruptions. Our analysis is conducted using statistical techniques that take into account differences in reliability reporting practices among electricity utilities. We then explore the effect of differences in reporting practices. The questions we examined and the motivation for examining them are as follows: 1. Are there utility-specific differences in reported electricity reliability? Eto and LaCommare (2008) observe wide variability in reported reliability among electric utilities. They also observe wide variation in reliability metric reporting practices. They conclude that variations in reporting practices complicate meaningful analysis to understand underlying sources of variability in reported reliability. This analysis seeks to determine whether these utility-specific effects can be identified consistently. If so, can they also be treated explicitly to separate their potential influence on other correlations we examine? 2. Are there trends in reported electricity reliability over time? As noted above, Hines et al. (2009) conclude that there are no statistically significant trends over time based on a rigorous statistical examination of data on the reliability of the bulk power system. Taking explicit account of utility specific differences and comprehensive information on the totality of consumers reliability experience, our analysis seeks to determine, whether temporal trends can be identified that are statistically significant. 3. How are trends in reported electricity reliability affected by the installation or upgrade of an automated outage management system? McGrananghan (2006) presents anecdotal information on adoption of automated outage management systems for one utility and its apparent influence on reported reliability. It speculates that, in this instance, adoption of the automated outage manage system led to lower reported reliability because of under-reporting of customer power interruptions prior to adoption. Our analysis seeks to explore the effect of installing or upgrading an OMS and how any such advanced reporting system changes the reported reliability over time. 2

17 4. How are trends in reported electricity reliability affected by the use of IEEE Standard ? Eto and LaCommare (2008) compare reliability metrics reported by 11 electric utilities using both historic company practices and the IEEE Standard Based on this small sample, they find no evidence of systematic bias resulting from use of the IEEE Standard. This analysis seeks to update our findings based on more recent data. The paper is organized as follows. In Section 2, we describe and review aspects of the information we collected to support the analysis, including electricity reliability metrics, the size of the electric utilities reporting the metrics, the climate in which their customers are located, the adoption by the utility of automated technologies for recording power interruptions, and the utility practices for reporting power interruptions. Section 3 describes the statistical methods used to analyze the information and then presents the findings from application of these methods on reliability trends and their correlation with various factors, including size of utility, climate, installation of adoption of an automated outage management system, and utility reporting practices vis-à-vis IEEE Standard Section 4 provides a summary of our findings and a discussion of next steps in the analysis. A technical appendix provides information on the results from an alternative specification of the statistical model that is the basis for findings presented in Section 3. 3

18

19 2. Data Collection and Review The data we collected for this study includes: Reliability metrics, focusing on the System Average Interruption Frequency Index (SAIFI) and the System Average Interruption Duration Index (SAIDI), Installation or upgrades to automated outage management systems (OMS), Adoption of IEEE Standard for reporting reliability metrics, Retail electricity sales, and Weather. This section describes the sources for these data and reviews selected aspects of the data we collected on reliability metrics. 2.1 Sources of Data Utility reliability metrics The principle source for the utility reliability metrics we collected was state utility regulatory commissions because these data are routinely filed by utilities and often made publicly available (Eto and LaCommare 2008). In addition, a small portion was also collected directly from individual utilities. Two reliability metrics, SAIDI and SAIFI, were collected for the years We requested SAIDI and SAIFI both with and without the inclusion of major events. 1 See section for a discussion of major events and the reason why utilities sometimes report reliability metrics both including and not including them. We also collected information on whether and in what year a utility installed or upgraded its automated outage management system (OMS). An OMS provides an automatic and consistent means for collecting information on the frequency, extent, and duration of electric service interruptions. This automation technology generally replaces manual record keeping, which is widely recognized as a less reliable means for collecting service interruption information. Finally, we also collected information on whether the utility relied on IEEE Standard in reporting its reliability metrics. Among other things, the IEEE standard features a systematic, statistically based method for reporting SAIDI and SAIFI without the inclusion of major events Weather Data We collected information on climate in the form of annual heating and cooling degree-days (HDD and CDD) for the 48 contiguous states for from the National Oceanic & 1 In some instances when SAIDI was not reported, the Customer Average Interruption Duration Index (CAIDI) was collected to derive SAIDI using the simple mathematical expression CAIDI = SAIDI/SAIFI. 5

20 Atmospheric Administration s (NOAA) National Climatic Data Center (NCDC) (NCDC 2011). 2 HDD and CDD are measures of the need for heating or cooling in a building. Thus a HDD is positive if ambient air temperature is cool and a building needs to be heated; a CDD is positive if ambient air temperature is warm and a building needs to be cooled. A HDD is defined as 65 minus the average of the daily high and low temperature where HDD is set to 0 if the average daily temperature is more than 65 F. A CDD is defined as the average of the daily high and low temperature minus 65 where the CDD is set to 0 if the average daily temperature observed is less than 65 F. We assigned state-level HDD and CDD s to each utility based on its location Retail Electricity Sales We collected retail electricity sales for each utility for years from information that is published annually by the U.S. Energy Information Administration (EIA 2010). 2.2 Review of Utility-Reported Reliability Data Geographic Representation and Coverage We collected reliability data from 155 different U.S. utilities. Of these, 139 are investor-owned utilities (IOUs) and 16 are either municipal utilities or electricity cooperatives. Figure 1 shows the geographic distribution of these utilities by NERC region, including the number of utilities and the percentage of total retail electricity sales within the region that are accounted for by these utilities based on sales in Table Sales of Utilities for Which Data Were Collected, by NERC Region NERC Region Total Electricity Sold in 2009 (MWh) Total Electricity Sold by Utilities for which Data were Collected (MWh) Percentage of Region Percentage of U.S. Total WECC 658,714, ,434,528 63% 12% MRO 205,459,370 77,647,308 38% 2% SPP 186,070,325 67,928,200 37% 2% NPCC 222,727, ,152,714 98% 6% RFC 919,727, ,188,248 63% 16% SERC 876,326, ,166,898 26% 6% FRCC 217,936, ,961,806 78% 5% TRE 271,354,767 28,414,156 10% 1% HICC 10,125,934 9,689,661 96% 0% ASCC 6,269,927-0% 0% TOTAL 3,574,712,272 1,799,583,519 50% 50% 2 Temperature records came from observation stations located in climatologically homogenous regions within a state. The station s observations are weighted by the area of its climate region as a proportion of the state s area. This produces a weighted average for temperature in the state. For further details on the weighting procedures, see NOAA National Climatic Data Center (2011). 6

21 Source: EPA EGrid 2010 Map Figure 1. Map of U.S. by NERC Region Table 1 shows by NERC region the same information presented on Figure 1 and the percentages of total U.S. sales in 2009 represented by the utilities for which we collected reliability data. The reliability data we collected for this study represent the reliability reported by electric utilities accounting for half of total U.S. electricity sales in The percentages of sales represented vary by region, from a low of 0% (ASCC) to a high of 98% (NPCC). Reliability data from utilities representing more than 50% of total sales were collected for HICC, FRCC, NPCC, RFC, and WECC Reliability Data Completeness over Time and by Utility Figure 2 shows, by year from 2000 to 2010, the number of utilities whose reliability data we collected. The figure shows a general increase in the number of utilities from year 2000 to 2006 for both SAIFI and SAIDI both with and without major events included. The trend declines after 2006 for SAIFI and SAIDI without inclusion of major events and after 2009 for SAIFI and SAIDI with inclusion of major event days due to increased difficulty in getting the most recent data. Figure 3 shows the number of years of reliability data we collected from each of the 155 utilities. We were able to obtain a complete time series of 11 years of SAIFI and SAIDI without inclusion 7

22 of major events for over a third of the utilities (or 56 utilities). We collected six or more years of reliability data for over 90% of the utilities (or more than 140 utilities). 180 Reliability Data from Utiilities # of utilities SAIDI w/ MEDs SAIFI w/ MEDs SAIDI w/o MEDs SAIFI w/o MEDs year Figure 2. Number of utilities with SAIDI and SAIFI data 60 Completeness of Time Series # of utilities SAIDI w/ MEDs SAIFI w/ MEDs SAIDI w/o MEDs SAIFI w/o MEDs years of data in each time series Figure 3. Completeness of time series 8

23 2.2.3 Average SAIDI and SAIFI over Time Figures 4 and 5 summarize by year the simple statistical averages of the reported SAIDI and SAIFI values. As expected, the statistical average of SAIDI or SAIFI is greater with the major events included With Major Events Without Major Events Average SAIDI minutes year Figure 4. Average SAIDI by year with and without inclusion major events With Major Events Without Major Events Average SAIFI number of interruptions year Figure 5. Average SAIFI by year with and without inclusion of major events 9

24 2.2.4 Outage Management System Table 2 summarizes, by NERC region, the number of utilities that installed or upgraded their OMS by We found that 118 utilities or 76% of 155 utilities for which we collected reliability data had installed or upgraded OMS by Table 2. Summary of Utilities with OMS NERC Region # Utilities We Obtained From # Utilities that Had Installed or Upgraded their OMS by 2010 WECC MRO SPP 11 6 NPCC RFC SERC FRCC 5 5 TRE 8 3 HICC 4 1 ASCC 0 0 TOTAL Figure 6 presents, by NERC region, the number of utilities that installed or upgraded their OMS in a given year, as well as the cumulative number over time. The figure indicates that a significant number had installed OMS prior to the first of our data collection (i.e., prior to 2000). The figure also indicates that there are a number of utilities that have installed or upgraded their OMS, but for which we could not determine the year of installation or upgrade. 10

25 # of utilities OMS Installations or Upgrades ASCC HICC TRE FRCC SERC RFC NPCC SPP MRO WECC Cumulative or earlier year unknown 0 year Figure 6. Number of utilities by year that installed or upgraded the OMS by NERC region Major Events Information on reliability is sometimes segmented using the concept of major events. Major events are defined by a variety of criteria to differentiate between routine power interruptions and non-routine or extraordinary power interruptions. There are scores of different definitions for how a major event is defined as discussed in Eto and LaCommare (2008). IEEE Standard is a voluntary industry standard that articulates a consistent set of definitions and procedures for measuring and reporting distribution reliability information, including a statistically-based definition of major events. Adoption of IEEE is still in its early stages. In 2006, we found 14 utilities (of the 120 utilities whose data we obtained) reported reliability information using this standard (Eto and LaCommare 2008). For this study, we collected SAIDI and SAIFI for 38 utilities (of the 155 utilities whose data we obtained) that used the IEEE standard. See Table 3. In general, the reliability data we collected were either prepared using IEEE Standard or they were not prepared using the standard. 3 For this study, we obtained reliability metrics reported both using IEEE Standard as well as some other non-ieee Standard for eight utilities. In preparation for the more rigorous analysis of the relationship between reported reliability and reporting practices presented in the next section, we look specifically at 3 In every instance in which a utility relied on IEEE Standard to report its reliability metrics, the standard was used to prepare the metrics for each year for which data were obtained. In many instances, this meant that the utility had recalculated its reliability metrics using the standards for the years prior to the utility s decision to use the standard. 11

26 the differences in reported SAIDI and SAIFI without inclusion of major events, as reported by these eight utilities. Table 2. Summary of Utilities Using IEEE NERC Region # Utilities For Which We Obtained Reliability Data # Utilities that Used IEEE Std to Report Reliability Data WECC MRO 14 1 SPP 11 1 NPCC 21 4 RFC SERC 30 6 FRCC 5 0 TRE 8 0 HICC 4 0 ASCC 0 0 TOTAL Figure 7 and 8 present percentage differences in SAIDI and SAIFI without inclusion of major events, respectively, between use of some other method and use of the standard. Visual inspective of Figure 7 shows that SAIDI reported using the standard is lower, on average, than SAIDI reported using some other method. However, for a given utility, there is also significant variation in these differences from year to year and these variations appear to be as large or even larger than the average of the percentage differences over the years. Figure 8 indicates that the percentage differences for reported SAIFI using the two methods are even less discernable (i.e., the percentage differences are close to zero). In addition, the variations from year to year for a given utility are also with a few notable exceptions - smaller on a percentage basis. It is difficult to draw definitive conclusions from such a small sample of utilities. In Section 3, we will apply more sophisticated statistical methods to re-examine this topic using a much larger small. 12

27 100% % Difference in SAIDI Using IEEE Compared to Not Using IEEE % 60% 40% 20% 0% 20% 40% 60% 80% 100% year Figure 7. SAIDI Percentage Difference between Using and Not Using IEEE Standard % % Difference in SAIFI Using IEEE Compared to Not Using IEEE % 60% 40% 20% 0% 20% 40% 60% 80% 100% year Figure 8. SAIFI Percentage Difference between Using and Not Using IEEE Standard

28

29 3. Findings from the Statistical Analysis of Reliability Data Reported by Electric Utilities This section describes the statistical methods we used to analyze the electric utility reliability and other data we collected and then presents our findings from application of these methods to identify reliability trends and their correlation with the various factors we considered. The questions we explore in this analysis include: 1. Are there are utility-specific differences in reported electricity reliability? 2. Are there trends in reported electricity reliability over time? 3. How are trends in reported electricity reliability affected by the installation or upgrade of an automated outage management system? 4. How are trends in reported electricity reliability affected by use of IEEE Standard ? 3.1 Introduction to the Statistical Methods Used in the Analysis As described in Section 2, the reliability data we analyzed consist of up to 11 years of SAIDI and SAIFI reliability metrics (both with and without the inclusion of major events) collected from 155 electricity distribution companies (utilities). Statistical analysis of this type of data is commonly referred to as an analysis of panel data because the methods and results involve data that have both a cross-sectional (i.e., multiple utilities) and a time-series (i.e., multiple years) element. The structure and completeness of the panel data influenced both the choice of tests used to specify the statistical models methods used to estimate the model parameters and standard errors. Cameron and Trivendi (2009) refer to the specific type of panel data we analyzed as short, because the data structure has many entities (i.e., utilities), but only a few time periods (compared to the number of entities). In addition, our panel data are unbalanced because they do not contain reliability metrics for every year from all utilities (Wooldridge 2002). So, to summarize, the statistical analysis we conduct one that is based on a short, unbalanced panel data set. The conventional statistical method used to analyze short, unbalanced panel data is multivariate regression. Multivariate regression models provide quantitative estimates of the strength of the correlation between an outcome variable (i.e., the reliability metric SAIDI or SAIFI) and a set of explanatory variables. The specific forms of the multivariate regression models we estimated are called either fixed effects or random effects models. Fixed and random effects models are particularly useful for this type of analysis because they enable the regressions to take explicit account of differences in the outcomes (i.e., SAIDI and SAIFI) that are correlated with differences in the sources of the data for these outcomes (i.e., the utilities). As noted earlier, utilities follow different practices in reporting reliability, such as whether or not they use IEEE Standard Fixed and random effects models can take explicit account of these correlations and thereby remove the 15

30 influence of these differences from the other correlations under consideration, namely, correlations with the other explanatory variables. 3.2 Application of the Statistical Models The application of the statistical methods involved three sequential steps. First, we transformed the reliability metrics by re-expressing them as natural logarithms. Second, we conducted F-tests on the transformed reliability metrics to confirm the appropriateness of using fixed and random effects models. Third, we estimated two sets of models. We now briefly describe each of these steps. We decided to transform the reliability metrics by re-expressing them as natural logarithms for two reasons. First, it is well known that the metrics themselves tend to follow a log-normal distribution, so transforming them results in a distribution that follows a normal distribution. Second and perhaps more importantly, re-expression allows for a natural interpretation of the estimated coefficients from the regression equations as percentages. For example, if an estimated coefficient for an explanatory variable has a value of 0.02, the natural interpretation is that a step change in that variable is correlated to a 2% increase in the reliability metric. The F-test is a standard statistical test that is used to determine the appropriateness of estimating fixed and random effects models. The F-test is a test of the null hypothesis that there are no fixed or random effects. If these null hypotheses can be rejected with some degree of statistical confidence, it means there may be fixed or random effects and therefore the use of fixed and random effects models to estimate these effects is warranted. We estimated two sets of log-linear models. First, we estimated a set of models that include utility-specific effects, and called this Model 1. These models seek to control for systematic differences across utilities such as time, region (climate), system size, as well as the installation or upgrade of an automated outage management system (OMS). The set consists of separate models for SAIFI and SAIDI, both with and without inclusion of major events. Both fixed and random effects versions of Model 1 were estimated. Second, we estimated a set of models that did not include utility-specific effects, called Model 2. These model seeks to examine how reporting differences, specifically utilization of IEEE Standard , is correlated with reported reliability. The specification of first model is as follows: Where: y it = α + γ i + β 1 Sales it + β 2 HDD it + β 3 CDD it + β 4 YR t + β 5 OMS it + β 6 POST OMS it + ε it (1) y it is the natural log of the reliability metric (SAIDI or SAIFI) for utility i=1,2,,n in year t=1,2,,t; γ i is a company-specific effect; Sales it is annual electricity sales in Millions of MWh; HDD it is heating degree-days; 16

31 CDD it is cooling degree-days; YR t is a time trend in years, OMS it is an indicator variable that takes the value 1 if utility i has an OMS installed in year t, and 0 otherwise. POST OMS it takes the value 1 for the first year after utility i installs an OMS, 2 for the second year after an OMS is installed, etc, and 0 for earlier years prior to the installation of an OMS. We applied the Hausman (1978) specification test to the Model 1 to determine whether a fixed or random effects version of this model was more appropriate. 4 The test did not reject the null hypothesis. We therefore concluded that the random effects model was consistent and more efficient than fixed effects in this case. 5 In the remainder of this subsection, we present results for the random effects model only. 6 The second model specification, which does not include utility-specific effects, γ i, is as follows: 7 y it = α + β 1 IEEE i + β 2 Sales it + β 3 HDD it + β 4 CDD it + β 5 YR t + β 6 OMS it + β 7 POST OMS it + ϵ it (2) Where, in addition to the variables defined above: IEEE i takes the value 1 if utility i reports interruptions using IEEE Standard and 0 otherwise. Note that for both Model 1 and Model 2 the time trend, YR t, enters as a linear time variable rather than as a year-specific effect. This additional assumption was justified because the cost of including it as a year-specific effect is high in terms of degrees of freedom for such a short dataset. See the appendix for a model that includes the time trend as a year-specific effect; the time trend is similar but each individual year is not statistically significant. We estimated standard errors for the one-way unbalanced data model using a specialization (Baltagi and Chang 1994) of the approach proposed by Wansbeek and Kapteyn (1989) for unbalanced two-way models. The Wansbeek and Kapteyn method for estimating variance 4 The Hausman test examines whether under the null hypothesis that the individual utility effects are uncorrelated with the other regressors in the model. If the null hypothesis is not rejected, both the random effects and the fixed effects models are consistent but only the random effects model is efficient. This means that fixed and random effects models will give the same coefficient estimates, but the random effects model will have much smaller standard errors. Using a fixed effects model when the random effects model is consistent may lead to an erroneous interpretation of the statistical significance of coefficients. See Greene (2000) for a more detailed discussion of the difference between fixed versus random effects. 5 Note that because this is an unbalanced dataset, the Breusch-Pagan test for random effects is not appropriate here (SAS 2011a). 6 See the appendix for fixed effect results; as expected, the coefficient estimates are similar but less efficient (that is, they are not statistically significant). 7 In order to understand the effect of utilization of IEEE Standard on reported reliability, utility effects cannot be included in the model because utility effects take account for all systematic differences among utilities, including whether or not the utility used IEEE Standard

32 components is the default approach used by SAS in the one-way random effects estimation of unbalanced panel data (SAS 2011b). 3.3 Findings Are there are utility-specific differences in reported electricity reliability? Table 4 presents the results from the application of the F-test to the reliability metrics. The table indicates that both one-way (utility only) and two-way (utility and year) fixed effects are statistically significant (at the 0.01%) level for all four reliability metrics SAIDI and SAIFI both with and without inclusion of major events. That is, there are very strong correlations both between the utility and the values of the reliability metrics and between the utility and year, and the values of the reliability metrics. The strong correlation between the utility alone, and the reliability metrics indicates that it is important to take this correlation into account when examining correlations between the reliability metrics and other correlated (or explanatory) variables. In other words, there are strong utility-specific effects that are systematically correlated with the reliability metrics. At this point, we cannot determine the exact or complete set of sources or causes of these effects, but they are consistent with the existence of utility-specific differences in reporting practices. Hence, taking this correlation into account appropriately means that subsequent correlations with other variables will not be contaminated by these differences in reporting practices (nor any other utility-specific effects). The correlation between both utility and year, and the reliability metrics means that year-to-year correlations are also important to take into account when examining correlations with other variables. This finding supports examining the reliability metrics, by utility, as a time series, rather than as a handful of observations randomly drawn from different years. Taken together, these findings motivate and provide a basis for the estimation of Model 1 (the random effects model) and Model 2 (without fixed utility effects), which are discussed next. Table 3. F-test of the Hypothesis that there are No Fixed Effects One-way Fixed Effect (Utility) Two-way Fixed Effects (Utility and Year) Reliability Metric F Value Degrees of Freedom (among/within) Prob. > F F Value Degrees of Freedom (among/within) Prob. > F ln SAIDI (w/o MEs) /1037 < /1029 < ln SAIFI (w/o MEs) /1034 < /1026 < ln SAIDI (w/mes) /595 < /587 < ln SAIFI (w/mes) /595 < /587 <

33 3.3.2 Are there trends in reported electricity reliability over time? We estimated each model separately for each of the four different reliability metrics: SAIFI and SAIDI, both with and without major events. We estimated Model 1 was with two specifications for the treatment of installation or upgrade of an OMS. The first version of this model considers only the differences in reported reliability before and after installation or upgrade. The second version of this model considers a learning effect in which the model seeks to estimate the correlation with installation or upgrade in subsequent years. Tables 5 and 6 present the results for the Model 1. Table 5 presents the results for SAIFI and SAIDI with major events for both versions of Model 1. Table 6 presents the results for SAIFI and SAIDI without major events for both versions of Model 1. Evidence that there is a secular (long-term, non-periodic) trend of increasing frequency and duration of interruptions over time is found in both Tables. 8 In Table 5 (SAIFI and SAIDI with major events), columns I and II, the coefficients for YR are and for SAIFI and SAIDI, respectively. Both estimated coefficients are statistically significant at the 1% level. The natural interpretation of these coefficients is that SAIFI and SAIDI are increasing annually by about 2% and slightly more than 2.5%. These trends are also confirmed when major events are not included in SAIFI and SAIDI. In Table 6, columns I and II, the coefficients for YR are and for SAIFI and SAIDI, respectively. Again both are statistically significant at the 1% level. The natural interpretation of these coefficients is that SAIFI and SAIDI are both increasing annually at about 2%. It is important to observe that, in making these estimates, Model 1 also takes into account the potential for correlations with electricity sales, as well as climate (in addition to taking into account the effect of utility-specific differences, as discussed in Section 3.3.1). In fact, the model finds that these external factors are generally not at all correlated with the increasing secular trends observed for both SAIFI and SAIDI. One exception is that correlation between SAIFI without major events and cooling degree days is statistically significant at the 5% level. The natural interpretation of this correlation is that SAIFI is very slightly higher when there are more cooling degree days. It is premature to speculate or draw conclusions on the causes underlying these trends without more explicit treatment of potential sources of bias in reported reliability. In the next subsection, we focus on measurement error as one potential source of bias. 8 Model (1) restricts the time trend to be linear. In the appendix, we present results from a model that includes year fixed effects rather than a linear time trend. The estimates of the time fixed effects increase in a relatively linear fashion, but the loss in degrees of freedom results in estimates that are not statistically significant. We feel that the assumption of a linear time trend is therefore appropriate in this case. 19

34 Table 4. One-way Random Effects Regression (Model 1): The effect of sales, HDD, CDD, time, and OMS on frequency and duration of interruptions (with major events) Table 5. One-way Random Effects Regression (Model 1): The effect of sales, HDD, CDD, time, and OMS on frequency and duration of interruptions (without major events). Notes: Standard errors in parentheses. * p<0.10, ** p<0.05, *** p<

35 3.3.3 How are trends in reported electricity reliability affected by the installation or upgrade of automated outage management systems? The estimates for the OMS coefficients in Table 5 and Table 6, columns I and II, describe the strength of the correlation between installation or upgrade of an automated outage management system and SAIFI and SAIDI. The results differ for SAIFI and SAIDI, both in terms of the direction of the correlation and in terms of the strength of the correlation. The correlation of SAIFI (both with and without inclusion of major events) with the installation or upgrade of an automated outage management system is mixed and not statistically significant, even at a 10% level. The correlation with SAIDI, however, is always positive and is statistically significant at the 1% level. The natural interpretation of this correlation is that adoption of automated outage management system is accompanied by an increase in reported SAIDI by nearly 29% when major events are included and by nearly 14% when major events are not included. Tables 5 and 6, columns III and IV, further explore the relationship between installation or upgrade of an automated outage management systems and reported reliability by introducing a time-element, POST OMS, which tracks how the correlation with SAIFI and SAIDI changes in the years following adoption. The results are suggestive, but not conclusive. The results suggestive because there is evidence that installation or upgrade of an automated management system is correlated with an increase in SAIFI or SAIDI, initially, but that SAIFI and SAIDI decrease in the years following installation or upgrade. The results are not conclusive because the estimated coefficients are not consistently statistically significant. The only SAIFI coefficient that is statistically significant is POST OMS with major events included at a 5% level. The natural interpretation is that there is an annual reduction in SAIFI of 2.4% following installation or upgrade of an automated outage management system. The SAIDI coefficients that are statistically significant include OMS (1% level) and POST OMS (10% level) with major events included, and OMS (1% level) when major events are not included. The natural interpretations are, when major events are included, there is a one-time increase in SAIDI of 38% followed by an annual decrease of 4%. When major events are not included, there is a one-time increase in SAIDI of 16%. The effects of installation or upgrade of an automated outage management system in the years following installation or upgrade are not consistently statistically significant. However, the coefficients on the magnitude of the year-to-year changes in the reliability metrics remain highly statistically significant at the 1% level. In fact, SAIFI and SAIDI with major events increase annually at faster rates (of about 3% and 6.5%, respectively) than the estimates that do not consider this effect (columns I and II). SAIFI and SAIDI without major events increase annually at rates of about 2% and 2.5%, respectively, which is roughly consistent with the estimates that do not consider this effect (columns I and II). 21

36 3.3.4 How are trends in reported electricity reliability affected by the use of IEEE Standard ? Tables 7 presents the results for Model 2, which removes γ i the company-specific effect and replaces it with IEEE i, which indicates whether the company relied on IEEE Standard in reporting its reliability metrics. Table 7 was developed for only SAIFI and SAIDI without major events because reliance on IEEE Standard involves implementing a specific method for not including these events. 9 Table 7 reports that coefficient for reliance on IEEE Standard is not statistically significant for SAIFI and is statistically significant at the 5% level for SAIDI. The natural interpretation of the latter result is that reliance on IEEE Standard is correlated with a lower reported SAIDI of about 11%. The removal of utility-specific effects also affects the values and statistical significance of other correlates in the model, compared to the values estimated for them in Model 1. For SAIFI, the coefficients on Sales and CDD are also statistically significant in Model 2. For SAIDI, the coefficient on CDD is statistically significant in Model 2. The statistical significance of these correlates is likely reflective of a utility-specific effect, since these correlates were not at all or less statistically significant when utility-specific effects were taken into account (in Model 1). Table 6. No Utility Fixed Effects Regression (Model 2): Effect of IEEE, sales, HDD, CDD, time, and OMS on frequency and duration of interruptions (without inclusion of major events). ln SAIFI ln SAIDI Intercept ** *** ( ) ( ) IEEE ** (0.0425) (0.0470) Sales ** (0.0010) (0.0011) HDD (0.0000) (0.0000) CDD *** *** (0.0000) (0.0000) YR ** *** (0.0068) (0.0075) OMS (0.0602) (0.0663) POST OMS (0.0114) (0.0126) Utility Fixed Effects No No Notes: Standard errors in parentheses. * p<0.10, ** p<0.05, *** p< SAIFI and SAIDI reported with inclusion of major events would be identical for utilities regardless of whether they implement this aspect of standard or use some other means to identify major events. Moreover, Statistically significant coefficients that might result from estimating Model 2 for SAIFI and SAIDI with major events included would pick up both this effect as well as other correlated factors that have now been allowed to enter the estimation due to the elimination of a utility-specific effect. 22