N ORTH A MERICAN ELECTRIC RELIABILITY COUNCIL

Transcription

1 N ORTH A MERICAN ELECTRIC RELIABILITY COUNCIL Princeton Forrestal Village, Village Boulevard, Princeton, New Jersey JOINT MEETING OF THE MC, PC, OC, AND CIPAG January 13, 2004, 8:00 a.m. 12:00 p.m. The Caribe Royale Suites Resort Orlando, Florida Tel: AGENDA 1. Introduction 8:00 a. Chairman s Welcome and Opening Remarks M. Fidrych b. NERC Antitrust Compliance Guidelines G. Cauley 2. Overview of August 14, 2003 Blackout Investigation 8:15 a. Perspective of the United States Canada Power System Outage TF A. Silverstein b. Blackout Investigation: Process, Organization and Data Management D. Hilt (Attachment 1) c. Blackout Sequence and Root Causes J. Robinson d. Draft NERC Recommendations (Attachments 2 and 3) G. Cauley Questions and Comments 3. Additional Lessons Learned 10:00 a. System Operation and Reliability Coordination: Emergency Response D. Hilt (Attachment 4) b. System Operating Tools: Information, Applications, Human Factors T. Kucey c. System Planning, Design, Ratings, and Operating/Planning Criteria F. Macedo (Attachment 5) d. System Data, Modeling, and Analysis (Attachment 6) e. Protection and Controls Generation (Attachment 7) R. Cummings G. Bullock f. Protection and Controls Transmission (Attachment 8) T. Weidman g. System Restoration TBD h. Vegetation Management (Attachment 9) TBD i. Data Requirements (Attachment 10) TBD Questions and Comments 4. Requested Standing Committee Activities G. Cauley 11:45 a. Comment on Investigation Results and Draft NERC Recommendations b. Develop Committee Work Plans and Assignments Phone Fax URL

2 Attachment 1 NERC Report on August 14, 2003 Blackout Lessons Learned and Recommendations Storyboard Outline 12/03/03 I. Recommendations A. What recommendations are there to address the root causes of the outage, to minimize the spread and impact of potential future cascading outages, and to address other lessons learned from August 14? B. What will happen after the final report? What is the process for making sure the recommendations are implemented and that future blackouts do not occur? What is the action plan for implementing these recommendations and who is responsible for implementation of the action plan? What additional research is recommended and how will that get done? What areas of further investigation are needed and who should do that? What is the institutional framework to carry the ball from here? II. Introduction A. Who developed the report and how was the report developed? What has happened since the U.S. Canada TF interim report was released: continuing investigation, public hearings, etc.? B. What is the purpose of the report? How does this final report fit with the interim report released on November 19? How is this final report organized and how is the interim report referenced by (or incorporated into) this report? C. What inputs and recommendations were received through the public forums and written comments in response to the interim report? Provide a synopsis of comments received, with a link to the hearing transcripts and written comments filed. III. What caused the cascade? What can be done to prevent the start of future cascades? A. Review the triggering events that started the cascade. What refinements are there to the sequence and root cause analysis of trigger events since the interim report? What specifically went wrong regarding the roles, responsibilities, tools, training, and authorities, etc. of system operators and reliability coordinators involved in the August 14 power outage? What is considered good performance in these areas? What recommendations are there to avoid a repeat of the types of human errors and computer failures that occurred leading to the cascade on August 14? B. What is considered good performance with respect to vegetation management and right of way maintenance for high voltage transmission lines? What recommendations are there to address vegetation management and right of way maintenance performance issues to ensure problems similar to those experienced on August 14 do not occur in the future? C. Besides the immediate causes triggering the cascade, were there broader underlying (e.g. organizational or institutional) deficiencies that allowed these trigger events to happen? Chapter 2 of the interim report outlines the reliability safeguards that are supposed to prevent the start of a cascade? Why did these safeguards break down on August 14? What happened or did not happen to allow them to break down? Looking at August 14 and prior large blackouts, do we see isolated, random failures or are there common underlying deficiencies enabling these large blackouts to occur? What recommendations are there to ensure reliability safeguards are effective in preventing future cascades? IV. The cascade why did the outage spread as far as it did and affect so many customers? Why wasn t it contained? What can be done to contain the impacts of any future cascades? Gerry Cauley Page 1 11/25/2003

3 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION A. Refine and expand the explanation of how the cascade propagated and why it stopped as it did. Provide a more detailed description of loads that were unserved as a result of the blackout and load areas that remained energized. Why did the disturbance progress to impact such a large geographic area? Why did the disturbance stop where it did to contain the blackout impact area within the rip fence boundary formed naturally by the rapid sequence of line trips? Did line and generator protection work as designed? Did line and generator protection help to propagate or help to stop the spread of the cascade, or both? What would have reduced the amount of load and generation lost and/or contained the blackout to a smaller geographic area? What could have been done to minimize the extent and duration of unserved demand? B. Summarize the results of voltage stability and dynamic/transient simulations and sensitivity analyses in the times surrounding key events to determine the susceptibility of the system on August 14 to power angle instability and voltage transients. To the extent data are available for verification, simulate the modes of breakup and collapse of various electrical islands that formed during the blackout and provide a dynamic graphical visualization of the cascade. C. What additional analyses should be done to evaluate alternatives to slow the onset of a cascade to allow corrective action, to minimize the geographic spread of a cascade, and to minimize the overall impacts to electricity customers? V. Reliability standards and compliance enforcement A. Identify all violations of NERC and regional standards, policies, procedures, and criteria that that contributed to the start of the cascade on August 14. Identify all violations of local operating procedures and criteria that contributed to the start of the cascade. B. What reliability compliance reviews were done in the two years priors to August 14 at FE and MISO? What were the results of those reviews and what deficiencies were identified with respect to meeting NERC and regional standards, policies, procedures, and criteria? Was the compliance review program, as designed and as conducted, adequate to identify the deficiencies that led to the August 14 outage? C. Are NERC and regional standards adequate to address the root causes leading to the cascade, and these standards were simply violated? Or are there areas in which additional standards are necessary? Are the current reliability standards sufficiently detailed and specific to provide reasonable assurance that future cascades will be prevented? Should there be standards requiring state estimators, real-time contingency analysis, voltage monitoring, control systems and alarms, and backup control systems? Should there be standards for facilities maintenance, including transmission line right of way maintenance and vegetation management? D. What changes should be made to the reliability standards development and compliance enforcement processes? VI. Technical issues and recommendations A. Operations and operating tools 1. Operations planning and resource management (1) Given the initial outages of East Lake 5 and three transmission lines, and assuming the FE operators knew what was happening on their system, what could they have done to prevent the cascade? Did they have sufficient resources available to them to stop the evolving emergency? Were there sufficient resources for other credible emergencies on the FE system on August 14? At what point in the sequence were there insufficient resources to prevent the cascade? Were FE s operations planning and emergency resources adequate for credible system emergencies? (2) Are the industry-accepted methods and practices of operations planning for response to abnormal and emergency conditions

4 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION adequate, considering the speed at which conditions can degrade on the power system? Is 30 minutes to restore the system back to within normal limits following a contingency that causes an operating security limit violation an appropriate amount of time? 2. Schedules and transfers (1) Were FE and other parties involved in the blackout in conformance with their transmission tariffs on August 14? Was posted TTC and ATC accurate? Was FE operating within its posted TTC for imports? (2) Were control areas in the Eastern Interconnection conforming to interchange schedules on August 14 and properly tagging those schedules for input to the IDC? What is the process for assessing and approving transaction schedules/tags? Analyze tagged transactions and actual schedules for August 14 to determine their conformance to system operating limits, including reactive constraints. Determine the change in system transfers caused by schedule changes in the period noon to 1600 hours EDT on August 14. How did the schedule/tags over this period compare with previous days and seasons? (3) Is TLR effective as a transmission loading relief tool if there are mismatches between schedules and tags of the magnitude seen on August 14? What can be done to make it more effective? What alternatives should be considered in the future? (4) Although the interim report noted the system was within defined non-simultaneous transfer limits, what simultaneous transfer limits are appropriate and would the system have been within those limits on August 14? What simultaneous transfer limits should be developed and by what process in the future? 3. Reliability coordination (1) What failures to effectively coordinate reliability occurred among reliability coordinators on August 14? How can the coordination across the complex interface between PJM and MISO be improved? What recommendations are there to make this reliability coordination interface effective in providing reliability safeguards? (2) What other lessons were there regarding reliability coordination and what changes are recommended to improve reliability coordination? Has the wide-area overview of the power system concept been implemented effectively? If not, why not? What can be done to assure redundant sets of eyes are watching and coordinating the reliability of the system? (3) What can be done to improve reliability while RTOs are in development and reliability boundaries are in a state of transition? What are the key performance criteria and factors for good reliability coordination? Examples include authority; modeling, tools and observability; wide-area overview; communications and coordination; and integration of reliability functions. What recommendations are there to achieve effective reliability coordination? (4) How should reliability functions be shared between reliability coordinators and their member control areas/transmission systems to ensure reliability safeguards are effective? To what extent are these responsibilities and authorities distinct and to what extent are they shared (overlapping)? Are the size and number of reliability coordinators and control areas factors in determining effectiveness, and in what way is that so? 4. Operating/reliability coordinator personnel qualifications and training. (1) What recommendations should be made to improve the ability of operators to identify, declare, and respond to emergencies? What verification and certification should be required? (2) What recommendations are there to improve communications among operators and reliability coordinators?

5 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION (3) Have system operations and operators jobs become too complex or distracted from reliability? If so, what should be done to correct the situation? 5. Operating tools what constitutes good performance in the following areas: (1) computer tools, alarms, displays, etc.; (2) system observability; (3) real-time contingency analysis; and (4) backup capabilities. 6. Frequency anomalies what issues related to reliability were identified regarding frequency anomalies on August 14? What recommendations are appropriate? B. Planning, design and system studies 1. Voltage and reactive management (1) Summarize the results of analysis of the voltage conditions on August 14 showing PV-PQ curves for FE and the surrounding region. Did the FE system have adequate capacity and reserves; static and dynamic reactive reserves; and delivery capability? Was the FE system near voltage collapse prior to the outage (pre- and post East Lake 5; Chamberlin Harding; Hanna-Juniper; and Star-S. Canton)? (Monitor redundancy with Chapter IV. B) (2) Review the results of reactive power planning studies for the summer of 2003 to determine the adequacy of reactive supply and reserves including the balance between static and dynamic resources. Assess the sufficiency of sensitivity studies carried out to allow for the uncertainty in reactive power demand and the unavailability of static and dynamic reactive supply resources. Identify the status of plans to provide additional reactive power support. Review the FE engineering justifications for two reactive projects recently completed the transformer at East Lake and the Avon capacitors to determine FE s assumptions regarding system reactive power needs. (3) Review the reactive power management strategies and operations measures in place to maintain an acceptable voltage profile. Determine if any of these were deployed on August 14. Are the FE criteria for voltage limits adequate? Were voltage and reactive effectively coordinated among systems prior to and on August 14? Was FE reactive reserve planning adequate? Assess the appropriateness of the voltage schedules and profiles in the affected systems and the deployment of reactive resources including the balance of static and dynamic reserves for the load levels and transactions over the period noon to 1600 hours EDT. Compare voltage schedules/profiles of August 14 with previous days and if atypical, determine conditions that made it so. Is reactive reserve planning adequate in general in the region and industry? What recommendations are needed? (4) Is the lack of standards on voltage and power factor (for generators and loads) a problem? Was it a contributing factor to the August 14 outage? What recommendations are appropriate for voltage/reactive coordination and standards on voltage and/or power factor? (5) Review the validity of reactive power demand and reactive capability of generating units used to assess system performance. Determine how these data are validated. Review the assumptions regarding load power factor for system studies to determine if these assumptions are adequate. 2. Planning studies were planning studies in effect for August 14 accurate and effective for the conditions seen on August 14? Were generator and load models adequate? Were regional studies sufficiently coordinated among regions? What improvements should be made in the areas of system studies and coordination of studies on a regional basis? (1) Review the criteria used for identifying deficiencies in the planning timeframe and in setting system operating limits and

6 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION compare with NERC and regional criteria. Examine where exemptions have been applied by the entities and approved by NERC or Regional Councils. (2) Assess the methods and assumptions used in carrying out power system studies in all time frames and compare with good utility practice. In particular evaluate the approaches used in determining the more restrictive of thermal, voltage stability and transient stability limits. Also evaluate the extent to which studies are coordinated with adjacent and neighboring systems (3) Review system studies for the summer of 2003 to assess the sufficiency and effectiveness of the analyses of system performance. In particular assess whether the studies failed to expose system deficiencies, which if addressed could have materially affected the event. Where deficiencies were identified what was the status of remedial action plans? (4) Compare system conditions and assumptions used to study system performance for August 2003 with conditions that actually existed on August 14. How typical was August 14 compared to earlier in the week or a typical summer day? (5) Review the extent to which the resiliency or robustness of bulk power systems were assessed by examining system performance for contingencies beyond normal criteria. Review what measures were taken to prevent cascading outages either through minimizing the probability of the event or its impact. 3. Design criteria what design criteria were in effect to prevent a cascade starting in the northern Ohio area and why were these criteria ineffective? Can improvements be made in system design criteria to reduce the risk of future cascading outages? 4. Equipment ratings and system limits should there be standards on setting of equipment ratings and system limits? How can ratings and limits be better coordinated? (1) Review the responsibilities and procedures used for setting system operating limits and communicating them to the control room operators. Compare these with good utility practice for prevailing system conditions and changing system conditions. Review the process for ensuring that the system operating limits are updated as system conditions change and events unfold in real time. (2) Determine the appropriateness and adequacy of the system operating limits in place prior to and following each generator and/or transmission element outage that occurred on August 14 prior to the start of the cascade. (3) Determine when on August 14 the system transitioned from a studied and known state to an unstudied and unknown state. Identify limit violations and review what actions were taken. C. System protection and controls 1. Are the line protection schemes commonly used today sufficient to arrest and minimize the effects of system cascades? What improvements are recommended to transmission system protection approaches? What is the feasibility of islanding schemes or other special protection schemes to reduce the risk of uncontrolled cascade? 2. Are the generator protection and control schemes commonly used today sufficient to arrest and minimize the effects of system cascades? What improvements are recommended to generation protection and controls based on the performance of generators on August 14 during the cascade and subsequent restoration? 3. Is the under-frequency load shedding philosophy commonly used today sufficient to arrest and minimize the effects of system cascades? Was automatic under-frequency load-shedding effective and can it be improved based on performance on August 14? Should under-voltage load shedding be given more consideration for certain applications?

7 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION 4. What could be done to contain and minimize the effects of potential future cascades? What changes are needed in the electricity infrastructure to make the electricity supply reliable and avoid future cascading outages? What changes are needed to system design criteria to reduce the risk of uncontrolled cascading? What existing or new technologies could be cost-effectively adopted, such as electronic tools for automated monitoring (e.g. phasor measurements); special protection schemes; and controls? Are there technologies that can be deployed to improve reliability performance of the grid, such as measuring line sag, conductor temperature monitors, wind speed, etc.? 5. What improvements can be made to system measurements and diagnostics for event analysis? What were the lessons learned regarding measurements and diagnostic data from the analysis of the August 14 blackout? Can more robust measurements be used to validate system analysis models? What is the value of adding time synchronized equipment?

8 Pre-Decisional Draft for Discussion August 14, 2003 Blackout Attachment 2 NERC Recommendations to Prevent and Mitigate the Impacts of Future Cascading Blackouts Proposed Immediate Actions to Be Initiated by June 30, Reinforce the existing NERC Compliance Enforcement Program to verify and report compliance with existing reliability standards and policies. NERC and the Regional Reliability Councils will jointly modify the existing Compliance Enforcement Program to include periodic audits of Reliability Coordinators and Control Areas for the purpose of verifying compliance with existing NERC and Regional reliability standards, policies, and procedures. These compliance audits will be conducted by experts who are independent of the entities being audited. The audits will be based on published criteria that are objective and measurable. As such, NERC will establish a set of Board-approved compliance templates 1. These compliance templates will build on, and as necessary expand upon, existing approved planning standards templates and draft operating policy templates. NERC will require Regions to confidentially report quarterly to NERC details regarding the nature and potential reliability impacts of violations and the identity of parties which are found to be non-compliant with NERC and Regional Reliability Council standards. NERC and the Regions will prioritize these compliance audits so as to a) provide the greatest potential benefit in preventing future large-scale cascading outages and b) address the root causes of the August 14, 2003 blackout and other historical large-scale system disturbances. Assign to: NERC Compliance and Certification Committee Justification: Violations of NERC operating policies and planning standards contributed to the August 14 blackout. 2. Establish a program of Control Area and Reliability Coordinator reliability readiness audits. In addition to the compliance audits described above, NERC and the Regions will establish a program to conduct periodic reliability readiness audits 2 for each Reliability Coordinator and Control Area. The program will audit each Control Area s readiness to meet the current requirements of the NERC Control Area Certification Procedure. For Reliability Coordinators, the audits will verify readiness to meet established reliability coordinator criteria by assessing its plans, procedures, processes, tools, personnel qualifications, and training. In addition to reviewing written documents, the audits will examine actual practices of Control Areas and Reliability Coordinators. All Control Areas and Reliability Coordinators will be audited within 3 years of the program s initiation and every 3 years thereafter. 1 The performance audits will be based upon the compliance templates NERC and the regions are currently using to evaluate compliance with existing NERC operating policies and planning standards until such time as they are replaced by new reliability standards. 2 Reliability assessment audits are similar in scope and purpose to certification audits and are proposed as an interim measure until Organization Certification criteria are adopted and implemented. Draft

9 Pre-Decisional Draft for Discussion Assign to: Compliance and Certification Committee Justification: The November 19, 2003 U.S.-Canada report identified deficiencies in Control Area and Reliability Coordinator capabilities to perform assigned functions as contributing to the August 14 blackout. 3. Establish a program to report bulk transmission line outages caused by vegetation. NERC and the Regional Reliability Councils will jointly initiate a program for reporting of all bulk electric system 3 transmission line forced outages resulting from vegetation contact 4. The program will use the Western Electricity Coordinating Council (WECC) vegetation monitoring program as a model. Each Transmission Operator will submit an annual report of all such vegetation-related forced outages to its respective Region. Regions will provide detailed annual reports of vegetation-related outages to NERC. Based on the results of these reports, NERC will consider the need for standards related to vegetation management. Assign to: Compliance and Certification Committee Justification: Inadequate vegetation management is identified as a root cause of the August 14 blackout. 4. Establish a program to track implementation of recommendations. NERC and the Regional Reliability Councils will establish a program for tracking the implementation of recommendations coming from NERC and regional reports on reliability violations, compliance audits, and lessons learned from system disturbance reports, including the August 14 blackout. The Western Electricity Coordinating Council (WECC) and Northeast Power Coordinating Council (NPCC) procedures for tracking and follow up on recommendations will serve as models for these procedures. Regions will report regularly to NERC on the status of their follow-up to these recommendations. NERC staff will provide summary reports to its board on both NERC and regional tracking activities. Assigned to: TBD Justification: A number of the root causes of the August 14, 2003 blackout had been identified as unfulfilled recommendations from prior blackouts. 5. Review the remediation plans of FirstEnergy, the Midwest Independent System Operator, and PJM, and monitor implementation of those plans. NERC will review the plans of FirstEnergy, MISO, and PJM to remediate the root causes of the August 14, 2003 blackout, as identified in the U.S. Canada Task Force report, and will monitor the implementation of those plans. NERC will review progress by FirstEnergy, MISO, and PJM in executing their respective remediation plans until such time that all identified actions have been completed. NERC will advise FERC, the Ohio Public Utility Commission, and other 3 All transmission lines operating at 100 kv and higher voltage shall be included in the program. 4 Contact with vegetation includes both physical contact and arcing due to insufficient clearance. Draft

10 Pre-Decisional Draft for Discussion jurisdictional agencies as appropriate, regarding the efficacy of the remediation plans and results, as they relate to reliability of the bulk electric system. Assign to: Technical Steering Committee assisted by Compliance and Certification Committee, Operating Committee, and Planning Committee. Justification: These three entities were citied in the November 19, 2003 U.S.-Canada Task Force report as having one or more deficiencies leading to the August 14 blackout. 6. Clarify NERC operating policies and procedures defining Reliability Coordinator and Control Area functions, responsibilities, and authorities. NERC will review and revise its operating policies and procedures to ensure Reliability Coordinator and Control Area functions, responsibilities, and authorities are completely and unambiguously defined. Such policies will also include Reliability Coordinator and Control Area characteristics and requirements to enable prompt recognition and effective response to system emergencies. This review will incorporate the causes of the August 14, 2003 blackout and other issues necessary to prevent future large-scale outages. Assign to: Operating Committee (ORS, RCWG, TS) Justification: Ambiguities in the operating policies may have allowed entities involved in the August 14 blackout to make different interpretations regarding the functional responsibilities and authorities of Reliability Coordinators and Control Areas. 7. Improve mechanisms for the timely exchange of outage information among Control Areas and Reliability Coordinators. NERC should review options for sharing of outage information in the operating time horizon (e.g. 15 minutes or less), so as to ensure the accurate and timely sharing of outage data necessary to support real-time operating tools such as state estimators, real-time contingency analysis, and other system monitoring tools. A selected data sharing capability should be implemented by affected parties, such as Reliability Coordinators, Control Areas, and Transmission Operators. Data sharing criteria should be established, including addressing confidentiality issues. Assign to: Operating Committee (ORS working with TWG and IDCWG) Justification: Lack of timely and accurate outage information resulted in degraded performance of state estimator and reliability assessment functions on August 14. Draft

11 Pre-Decisional Draft for Discussion Proposed Actions Requiring Analysis and Development beyond June 30, Address Issues Remaining in the Functional Model. NERC will revise the Functional Model to address more specifically the relationship between Reliability Coordinators and Reliability Authorities, and other outstanding issues. Assigned to: Operating Committee Justification: Reliability Coordinators provide a critical reliability function, yet the relationship with Reliability Authorities is not clearly defined in the Functional Model. 9. Reassess and accelerate the development of new reliability standards. NERC will review proposed new reliability standards to assess their sufficiency in addressing all issues identified in the August 14 blackout and prior large-scale blackouts. NERC will modify emerging standards or initiate new standards as necessary to address all reliability lessons learned. Reliability standards will be stated with sufficient detail to ensure that, if followed, the standards would be effective in preventing future cascading outages. New reliability standards will be available for adoption as soon as possible, with a target of completion by December 2006, and will include a plan for the smooth transition from existing operating policies and planning standards. NERC will seek opportunities to accelerate the adoption of reliability standards. Assign to: Standards Authorization Committee assisted by other committees. Justification: Objective and measurable reliability standards are a cornerstone of compliance enforcement. 10. Complete development of the Reliability Organization certification program. NERC will complete the development and adoption of certification criteria for reliability functions identified in the Functional Model, including Reliability Authorities, Balancing Authorities, Interchange Authorities, and Transmission Operators. NERC will register all entities providing these reliability functions by December 2004 and initially certify all registered reliability entities by December Assign to: Compliance and Certification Committee Justification: Reliability organization certification will assure essential capabilities are present at all entities operating the bulk electric system. 11. Modify personnel certification criteria to include emergency response training requirements and other qualifications necessary to assure reliable operations. NERC will modify the current operating personnel certification criteria, currently a test of knowledge of the NERC operating policies, to include additional qualifications that would provide affirmative indicators of competency to operate the bulk electric system. Such Draft

12 Pre-Decisional Draft for Discussion additional criteria will at a minimum include requirements for realistic training to recognize and respond to system emergencies. Assign to: Compliance and Certification Committee (Personnel Certification Governance Committee) and Operating Committee (Personnel Subcommittee) Justification: Lack of situational awareness is a root cause of the August 14, 2003 blackout. Enhanced situational awareness requires effective training, tools, and procedures. Current certification requirements include only knowledge of operating policies. 12. Continue to promote enactment of federal legislation enabling mandatory reliability standards. NERC will continue to promote enactment of reliability legislation. NERC will continue preparing for a smooth transition from the current voluntary approach to setting and enforcing mandatory reliability standards. Assign to: NERC staff Justification: Setting and enforcement of mandatory reliability standards are essential to reliability. 13. Establish guidelines for real-time operating tools. NERC will facilitate a review of necessary real-time operating tools and publish a set of guidelines. The guidelines will identify the provision, operation and maintenance of systems and applications critical to the reliable operation of bulk electric systems. The guidelines will also address minimum backup requirements for critical reliability functions and capabilities. NERC will consider the development of standards in these areas as appropriate. Assign to: Operating Committee (Operating Reliability Subcommittee and Reliability Coordinator Working Group) Justification: The August 14 blackout was caused by a lack of situational awareness that was in turn the result of inadequate reliability tools and backup capabilities. 14. Review operations planning and operating criteria. NERC will conduct a review of operations planning practices and criteria to ensure expected practices are sufficient and understood. The review will reexamine fundamental operating criteria, such as n-1 and the 30-minute limit in preparing the system for a next contingency, and Table I Category C.3 of the NERC Planning Standards. Operations planning and operating criteria will be identified that are sufficient to ensure the system is in a known and reliable condition at all times, and that positive controls, whether manual or automatic, are available and appropriately located at all times to return the Interconnection to a secure condition. Daily operations planning, and subsequent real time operations planning will identify available system reserves to meet operating criteria. Assign to: Operating Committee (Operating Reliability Subcommittee, Reliability Coordinator Working Group, Operating Limits Definitions Task Force) Draft

13 Pre-Decisional Draft for Discussion Justification: The investigation report noted at least one entity that had insufficient resources to remain within operating limits on August Review operator and reliability coordinator communications. NERC will facilitate the development of requirements, guidelines, and tools to aid system operator and reliability coordinator communications. NERC will explore alternatives that would allow verbal communications in emergencies to be more efficient while enhancing the transfer of critical reliability information among system operators and reliability coordinators. Assign to: Operating Committee (Operating Reliability Subcommittee and Reliability Coordinator Working Group) Justification: The U.S-Canada Task Force report of November 19 noted numerous instances of breakdown in communications among Control Areas and Reliability Coordinators as contributing to a lack of situational awareness and effective actions to prevent the cascade. 16. Review transmission facility ratings methods and practices. NERC will facilitate the development of guidelines for rating transmission equipment. NERC Reliability Regions will assemble a regional database that includes the ratings of all bulk electric system (100 kv and higher voltage) transmission lines, transformers and phase angle regulators. NERC and the Reliability Regions will consider standards or other measures to promote use of a consistent set of facilities ratings among and within various entities for both planning and operations. Assign to: Planning Committee Justification: The investigation discovered inconsistent use of ratings existing on August 14, between planning and operating models and among entities. 17. Review reactive power and voltage control practices. NERC will review the effectiveness of existing reactive power and voltage control standards and practices, including static and dynamic reactive power reserves. The review will focus on assuring the bulk electric system does not approach unstable voltage conditions and that systems do not adversely impact voltage profiles or reactive supply of other systems. Assign to: Planning Committee assisted by the Operating Committee Justification: The August 14 blackout identified inconsistent practices with regard to the setting and coordination of voltage limits and possibly insufficient reactive power supply in Northeastern Ohio. 18. Review system design, planning, and study methods and practices. Reliability Regions will review system design, planning, and study methods and practices within their respective regions to ensure such activities are technically adequate and coordinated among the entities within the region. NERC and the Reliability Regions will review the scope, frequency, and Draft

14 Pre-Decisional Draft for Discussion coordination of interregional studies, to include the possible need for additional simultaneous transfer studies based on transactional trends. Study criteria will be reviewed, particularly the maximum credible contingency criteria used for system analysis. Each Control Area will be required to identify the planned emergency import capabilities for each major load area. Assign to: Planning Committee (for planning studies) and Operating Committee (Reliability Coordinator Working Group for operations planning studies) Justification: Prior studies did not adequately define the conditions existing on August 14 and the system in Northeastern Ohio was reported as having insufficient reactive support to serve its loads. 19. Standardize the criteria for system modeling and data exchange methods and practices. Reliability Regions should establish regional power system models that enable the sharing of consistent data among entities in the region. Power flow and transient stability simulations should be periodically validated against actual system events. Establish a viable load and generator testing program (including load power factor) to improve agreement between power flows and dynamic simulations and the actual system performance. Assign to: Planning Committee Justification: The post-mortem models developed to simulate August 14 conditions and events indicate that dynamic modeling assumptions, including generator and load power factors, used in planning and operating models were inaccurate. 20. Facilitate the installation of time-synchronized recording devices. NERC will work with each Reliability Region to define regional criteria for installation of satellite-timesynchronized recording devices in power plants and substations, such as phasor measurement units (PMUs) with continuous archiving. Reliability Regions will facilitate the installation of an appropriate number, type and location of devices within the region to allow accurate recording of future system disturbances and to facilitate validation of simulation studies. Digital Fault Recorder data are also valuable when data capture is triggered, especially for understanding relay actions. Assign to: Planning Committee Justification: Time-synchronized recorders will greatly help in the analysis of future disturbance. Event synchronization was one of the most difficult challenges of the August 14 blackout investigation. 21. Evaluate alternative system protection and automatic remediation schemes. NERC will facilitate a study of technology solutions and methods to limit the spread of potential future cascading outages. Such review will consider actions to slow the propagation of a cascade to allow operator intervention, and fast-acting automatic controls such as under-voltage load shedding and remedial action schemes. NERC Planning Standards III.A.G17 regarding transmission line zone 3 protection will be reviewed and modified as appropriate to slow or Draft

15 Pre-Decisional Draft for Discussion limit the propagation of a cascading outage. While avoiding permanent equipment damage due to faults, the design and application of transmission phase relay protection should not compromise a power system s inherent physical capability to slow down or stop a cascading system event. Assign to: Planning Committee Justification: A key factor in the propagation of the August 14 outage was the large number of zone 3 impedance relay operations for other than fault conditions (e.g. power swings or out-of-step conditions). 22. Establish a reliability performance monitoring function. NERC will establish a program to evaluate and report bulk electric system reliability performance. Such a program would assess large-scale outages and near misses to determine root causes and lessons learned, similar to the blackout investigation. This function incorporates the current Disturbance Analysis WG and expands the work to provide more proactive feedback to the NERC Board regarding reliability performance. This program would also gather and analyze reliability performance statistics to inform the Board of reliability trends. Consideration would be given to incorporating the recommendations tracking activity within this function. It is anticipated that this function would also develop procedures and capabilities to initiate investigations in the event of future large-scale outages or disturbances. Such procedures and capabilities would be shared among NERC and the Reliability Regions for use as needed. Jurisdiction for investigation teams need to be clearly established Assign to: Reliability Performance Function Justification: Feedback on reliability performance and trends could have helped prevent the August 14 blackout, as could effective implementation and tracking of recommendations from prior blackouts. 23. Review lessons learned from August 14, 2003 regarding system restoration and black start. NPCC, ECAR, and PJM will review the black start and system restoration performance within their respective regions following the outage of August 14. The results will be presented to NERC and specific recommendations make based on the analysis. Assign to: Operating Committee (Operating Reliability Subcommittee and Reliability Coordinator Working Group) and Planning Committee (blackstart requirements) Justification: The August 14 outage was a rigorous test of system restoration capabilities in the Northeast U.S. and Ontario. Lessons learned will help improve restoration response in the future. 24. August 14 data availability and management. NERC and the Reliability Regions should develop a data management capability to more effectively and efficiently assemble data following a large-scale blackout or disturbance. NERC should develop a plan for disposition Draft

16 Pre-Decisional Draft for Discussion of data currently held in the August 14, 2003 blackout data repository, considering the value of the data for further analysis and the critical infrastructure sensitivity of the data. Assign to: NERC staff and Regional Managers. Justification: More efficient data gathering mechanisms are necessary in the event of future large-scale blackouts. The August 14 database at NERC is a valuable resource for additional research and analysis. Confidentiality issues, particularly related to infrastructure security, must be resolved prior to making data available for additional research and analysis. Draft

17 Pre-Decisional Draft for Discussion August 14, 2003 Blackout Sample Additional Technical Recommendations to Prevent and Mitigate the Impacts of Future Cascading Blackouts Attachment 3 General Comments Received from Outside Investigation Team 1. NERC policies should include reliability coordinator and control area characteristics, size, and requirements to enable prompt recognition and control of system emergencies. This review should incorporate the causes of the August 14, 2003 blackout and other issues necessary to prevent future large-scale outages. 2. Prompt and effective system emergency operation requires a clear command and control protocol including a single, jointly understood, emergency procedure. Multiple area swiss chess jurisdictions should be minimized to the extent necessary to provide prompt and effective hierarchical control of system emergencies. 3. Whenever full operational capability is compromised on a necessary transmission facility, software or computer hardware function, or operator control function, a control area operations procedure should establish a reasonable period of time to return the facility to full operational capability, or trigger an appropriate more conservative mode of operation. Full operational capability of a necessary facility should not be compromised for an indefinite period of time. 4. One kind of open issue is a discussion around requirements for other entities that could impact control area performance such as generators and/or LSEs. I'm more concerned about the former than the latter, but shouldn't there be some sort of standards around their ability to meet and follow criteria. I would not only concern myself with proper operation of the generator, but even switchyard control. There have been several instances where generators have performed uninstructed operations in their switchyards. At the wrong time, this kind of thing could be devastating. Are there standards around this kind of stuff, or should there be? 5. NERC regions should strive to establish a greater degree of independence from their host utilities. ECAR is tied to AEP, SERC to Southern Company, MAAC to PJM, WECC to WAPA, etc. Compliance loses its meaning if it is a rubber stamp. 6. Review and enhance tools enabling Reliability Coordinators, Control Area operators, and transmission operators understanding of the nature of overloads. Operators need to know whether a loading problem is transactional or native load/dispatch related. NERC should enhance the IDC study capabilities to show the breakdown of impacts on demand by priority bucket for any transmission element. 7. Real-time operating tools need to be developed on platforms that facilitate easier data and model sharing between systems. If NERC made a more open architecture a requirement Draft

18 Pre-Decisional Draft for Discussion for reliability coordinators, control areas, transmission operators, etc.; vendors could be driven to conform to more open information sharing between platforms. Closed database architectures make it very difficult to exchange timely operating information because human intervention is often required to communicate status changes. 8. Review and improve requirements for tagging of dynamic schedules and pseudo ties. NERC should consider tightening requirements on tagging of dynamic schedules and pseudo ties to ensure that all security analysis systems and the IDC have the correct interchange modeled. 9. Given the importance of each Control Area's real time control systems (EMS), the poor Aug 14 e-tag audit and AIE survey results (for every hour surveyed!), and the fact that almost all modeling software is assuming that the IDC info is correct (when it apparently isn't). Review Policy 3 requirements to keep the e-tag system, the IDC, and each Control Area's EMS system (which contains the official scheduled interchange) in agreement with each other. Adjust Policy 3 as necessary. NERC should increase the frequency of Tag Audits and AIE survey, and ensure that future audits compare the info in the IDC to the info in each CA's EMS system. Information regarding these audits should be included in the reliability performance statistics that are reported to NERC and the Board (as outlined in # Review and improve requirements for tagging of dynamic schedules and pseudo ties. NERC should consider tightening requirements on tagging of dynamic schedules and pseudo ties to ensure that all security analysis systems and the IDC have the correct interchange modeled. 11. NERC could establish generator reactive capability reporting requirements that more clearly communicate a generator's true MVAr capability during hot days. Curves need to be based upon operation at high ambient summer conditions. Certain IPPs are filing for an MVAr tariff with the FERC when their actual MVAr output at peak (95 F ambient or higher) is nearly zero. The program should include monitoring to identify generation that is not operating under automatic voltage control. 12. I would add a recommendation to establish frequency response requirements for generators, control areas, reserve sharing groups or some other zonal classification. How do we know if the eastern interconnection has adequate zonal frequency response? Many generators are base loaded or otherwise operated without frequency response to grid disturbances. If frequency response is predominantly in one zone, a capacity disturbance far away could result in high unscheduled flows across long distances. The spread of a black out could potentially be abated with an organized frequency response strategy that recognizes key bottlenecks. I would also encourage NERC to establish under frequency generator and load tripping coordination practices. 13. Implementation of topology processors for this exchange should be investigated. Draft

19 Pre-Decisional Draft for Discussion 14. The NERC must perform surveys for current availability and functional requirements for Control Areas EMS, SCADA and AGC backup systems. NERC must assess survey results and come out with requirements, deployment recommendations, and periodic testing to avoid the system failures observed during the 08/14/03 blackout for this type of real-time control systems. Backups must be available as a hot standby and provide clear indication to dispatchers of its readiness to assume control. Draft

20 Pre-Decisional Draft for Discussion Operations Team Draft Recommendations 1. The separation of real-time generation control and transmission reliability needs to be reviewed in light of reliability concerns. 2. A second set of eyes to effectively see the actions/conditions and direct corrective actions between balancing authorities is needed. 3. Do not delay entering TLR requests into the IDC to see what actions would be necessary for relief, especially if the relief needed is large. This action doesn t initiate the TLR, but gives you information about what relief is available and what would have to be implemented to accomplish the desired relief. 4. If one operator has information of a critical nature that is in the responsibility area of another press to ensure the recipient of the information understands its implications. 5. Ensure all contingent losses that affect facilities in the footprint for which a reliability entity is responsible, are included in the monitoring tools. Note: This is considerably more challenging when the footprints of adjoining entities are not contiguous or are significantly interleaved (such as MISO and PJM). 6. Transmission limits and facility ratings should be set with understanding of and coordination with relay settings so that undesirable tripping does not occur. 7. Develop standards for what capabilities, procedures, and tools are needed to fulfill the roles of a reliability authority (and coordinator) and Balancing Authority (Control area RT) 8. Entities should be verified to be in full compliance with standards before assuming responsibilities or authorities. 9. Assume reliability responsibility should NOT occur until tools and capabilities are fully ready. 10. System operators need a system overview so that they can get a sense of the status of their system at a glance. Dynamic map boards provide a precise, consistent place to access the same system data and status every time. Flashing indication draws the attention of the system operator to abnormal conditions requiring assessment without needing to pull up a console display. 11. A comprehensive set of tools needs to be identified to relieve transmission constraint depending on the system circumstances. If they need relief quickly, access to effective (relevant to the location of the transmission problem) load shedding or effective redispatch options. Schedule reductions from the TLR process are meant for contingency relief proactively in anticipation of a possible contingency do not use them for emergency action or to meet operating within limits NERC criteria. Draft

21 Pre-Decisional Draft for Discussion 12. Operational plans must be developed to utilize these tools appropriately and effectively to manage emergency conditions and meet NERC criteria to operate within limits. 13. Develop standards for operational risk analysis and what generic risks exisit that could lead to an emergency, and how to prepare for them. 14. Each entity with reliability responsibility will apply these standards to develop a set of processes and tools to implement them 15. Reliability operators will be trained for handling emergencies through the use of good fidelity power system simulators for training 16. Suggest survey of practices among ISOs, RTO s, and system operators for best practices in training quantity, quality, and testing requirements. 17. Include in training curriculum A) the need for system operators to be inquisitive about conditions that seem unusual or abnormal find out why B) recognition of developing risk conditions C) recognize when you re in an emergency D) strategically assess and prioritizing control center activity when in an emergency (e.g., delegate routine phone calls, next hour scheduling reconciliation leave at least one senior system operator to develop a plan of action to address the emergency) 18. Authority Recommendations A) The Control Area Operator is the first line of defense for reliability within their boundaries (primarily responsibility). B) The Reliability Coordinator is the first line of defense for managing reliability issues between control areas within their boundaries (primary responsibility) and the 2nd line of defense (secondary responsibility) for reliability within control area boundaries in their footprint. They will order remedial actions should a control area fail to fulfill their responsibilities. C) Reliability Coordinators have a responsibility to each other to manage reliability issues at their seams. D) Footprints of coordinators should be formed to minimize the operational and electrical interfaces that have to be managed. Draft

22 Pre-Decisional Draft for Discussion 19. An effort should be undertaken to identify and implement skills to best fit an unbundled functional environment. 20. Reinforce the need to be curious about unusual conditions and to investigate their implications, given that a full range of information may not be available to them. 21. Encourage active engagement into neighboring systems status that may affect one s own and receptivity for questions about one s own system as well. There has been a culture of hands off systems beyond ones border. We need to change that to one of neighboring entities operating in a reliability partnership. If one operator has information of a critical nature that is in the responsibility area of another press to ensure the recipient of the information understands its implications. 22. Flowgate monitors should not be the primary reliability monitoring tool, since they do not include all known constrained transmission paths. In addition, forced outage conditions may result in transmission constraints that would not have been identified and registered ahead of time. 23. NERC shall establish a standardized minimum methodology for adequately analyzing voltage stability 24. Establish and test procedures between reliability jurisdictions and any other entities that allow execution of all TLR phases including immediate action. 25. Develop common open systems and protocols to communicate instructions for immediate action on generators and loads to relive transmission loading problems anywhere in the interconnected system(s). (Integrate it with other related business systems for inception to settlement transaction management.) 26. Testing of EMS systems must be sufficient to identify failure or poor performance risks of various kinds. A plan shall be implemented so that operating personnel can identify and act appropriately when a system is in one of these conditions. [This should also apply to the alarm issue.] 27. Coordinate the periodicity of data exchange between those responsible for reliability to assure their ability to see and respond to quickly evolving power system condition changes (consider no greater than 5 seconds between updates). b) When discussing or validating power system data, entities shall be clear on what their data source is and assure they are fully aware of the real time actual measured data, using the state estimated data to assure the validity of the data locally as appropriate. 28. Status of electrical devices that are modeled in a state estimator should be obtained wherever possible. Such status should automatically be updated in the state estimator model whenever it is likely that incorrect status would compromise critical operation of the state estimator. Draft

23 Pre-Decisional Draft for Discussion 29. Any entity required to use real time security tools such as state estimator and real time contingency analysis shall obtain sufficient data from adjoining systems to assure accurate results in their area of responsibility 30. Any entity with reliability responsibilities shall obtain sufficient data and develop monitoring, displays, and alarming sufficient to fully exercise their responsibilities under time limited circumstances prior to assuming such responsibility. 31. All those with primary reliability responsibility for substantial (to be defined!) high voltage wholesale electric transmission grid should have fully implemented on-line contingency analysis tools that run automatically at least every 15 minutes with effective notification such that thermal or voltage reliability problems would be readily identified. It shall have a manual study mode, automatically update it model as system status changes (ICCP linked to model or by CIM), alarm if not functioning properly, have dependability (available and accurate) factor of greater than 99% and be verified under stress. Additionally, any outages planned after the day ahead period shall be broadcast ASAP on a common tool such as the SDX. 32. Reliability tools shall be run on a periodic basis (see # 1) 33. The results of reliability assessments that relate to another s system shall be communicated to the other. 34. Assessments encompassing more than seasonal studies of typical conditions more widespread determination of simultaneous transfer limits taking into account generating patterns, reactive requirements and voltage collapse points. 35. Interactive simulation of low probability, but severe system conditions to recognize disturbance characteristics and to practice response techniques for both disturbance response and restoration. 36. Good visualization tools to help with this recognition and to get immediate feedback on the effectiveness of operator actions in real-time. 37. Reinforcement of training curriculum and testing to include simulators using actual disturbance scenarios, recognition of disturbance precursors, mitigation of disturbance severity. Include lessons from other areas/regions. Eliminate the common causes in disturbances because we have effectively transferred learning from one area to others. 38. Status and alarm features minimum fail-safe criteria for EMS systems to include back-up provisions, functionality monitoring and assessment, and transfer of control to other authorities should the primary and back-up systems fail. Consider 24/7 IT monitoring of EMS tools (separate from system operations staff) 39. Thermal, voltage collapse studies either real-time or off-line, but with current system conditions Draft

24 Pre-Decisional Draft for Discussion 40. If cascading occurs for some contingencies more quickly than an operator can reasonably respond must have automatic controls in place to act as a safety net. 41. All the studies in the world and regardless of the care taken in assessing the current status and vulnerabilities of the power system, they mean nothing if there are obstacles in the transmission rights-of-way. Transmission lines that trip out before overload render that assessment invalid. 42. Control Area Operators are the first line of reliability defense should be first to declare an emergency. Should be the entity to take action. 43. Control Areas need to raise the alarm when they re in trouble. 44. However, if they do not, RC s must have sufficient authority and means to carry out actions to preserve reliability. Confirm that authority through regular and rigorous audits. 45. If the RC sees a disturbance in the making authority to declare an emergency and order specific relief. System operators need to comply. If they disagree, comply first, argue later. 46. RC needs enough visibility with enough granularity to assess system problems and determine appropriate response to shortstop a potential cascading problem. 47. RC needs tools sufficient to determine simultaneous limitations on transmission paths (thermal, voltage, transient stability if appropriate). In some areas, SE and Contingency Analysis is necessary but not sufficient. If CA says you have a problem you do if it says you don t have a problem you still might (doesn t provide transient stability limitations). 48. Do operating personnel (and reliability coordinators) have the necessary authority to act in a timely manner during an emergency? Are they empowered to act in an emergency to prevent conditions that could lead to a cascading outage? What can be done to reinforce that authority? 49. Authority to take actions and responsibility to take those actions to prevent a cascading outage. Industry must support operators exercising that authority it will not be obvious if their actions have been successful. 50. Must have and know where collapse points are assessment tools need to include wide area view of current conditions and warning of contingency problems beyond that of non-simultaneous thermal limits of transmission circuits. 51. Need a stack of redispatch options to relieve transmission quickly. 52. Schedule reductions is meant for contingency relief proactively in anticipation of a possible contingency not for emergency action. Draft

25 Pre-Decisional Draft for Discussion 53. Standard protocols common terminology. 54. Emergency drills simulation with both control area or transmission operators and reliability coordinator. 55. See into each other s systems far enough to be effective. Agreement at seams about what facilities will be discussed with relevant CAO and RCs which ones matter 56. Dedicated personnel assessing of status of system possibly leaving the commercial arrangements separate from operation 57. Communicate with each other (within and between control centers) 58. Protocols prioritizing the attention of a control center dispatcher to determine course of action as a disturbance progresses delegate commercial schedules or routine phone calls to others Draft

26 Pre-Decisional Draft for Discussion Transmission Team Draft Recommendations 1. NERC to establish standards for setting Zone 3 relays so that there is adequate margin well above the maximum emergency rating of critical transmission lines 230 kv and above. Zone impedance relays are intended only to trip on faults and not on loads. 2. NERC to establish standards for rating transmission lines based on ambient wind speeds, ambient temperatures and conductor temperatures. Transmission owners have the responsibility for maintaining their rights-of-way and designing their transmission lines so that safe conductor to ground clearances are maintained. 3. NERC and Regional Reliability Councils to review their underfrequency load shedding programs to ensure that it is coordinated with generator underfrequency protection. Model after WSCC underfreqency load shedding program. 4. NERC and Regional Reliability Councils to establish standards for undervoltage load shedding. 5. NERC and Regional Reliability Councils to establish standards for reactive reserves and voltage control. 6. NERC and Regional Reliability Councils to review the use of out-of-step tripping and out-of-step blocking protection to contain disturbances within pre-determined boundaries. 7. NERC and Regional Reliability Councils to review the use of special protection schemes and Remedial Action Schemes (RAS) as safety nets for mitigating and containing system wide disturbances. 8. NERC and Regional Reliability Councils to review high speed reclosing practices to ensure that transmission lines are not reclosed out-of-step with another system/area. 9. NERC and Regional Reliability Councils to review and recommend standard output formats for digital fault recorders and digital relays so that post-disturbance analysis will be facilitated. 10. NERC and Regional Reliability Councils to recommend operating tool standards for certification of all control area operators and RTOs to include robust state estimators, dispatcher power flows and real time contingency analysis as a minimum. 11. NERC and Regional Reliability Councils should review major disturbance reports over the last ten years to identify trends and common modes of failure. Major disturbances such as the recent Italian Blackout and August 10/July disturbances are examples of major disturbances that can be learned from. 12. NERC to establish protection and control compliance audit process for critical bulk power transmission lines and generators. Draft

27 Pre-Decisional Draft for Discussion System Planning Sequence of Events and Root Cause Draft Recommendations Proposed Planning Principle: While avoiding permanent equipment damage due to faults, the design and application of transmission phase relay protection should not compromise a power system s inherent physical capability to slow down or stop a cascading system event. Example Practice: (1.A) The design and application of directional distance phase relays should not compromise a facility s 5-10 minute short time emergency rating under typical winter temperature, and typical wind speeds (40-60% probable wind and ambient temperature conditions). After appropriate automatic load shedding in less than 5 minutes, such a practice would also permit short time emergency protection against permanent facility damage. Phase relay tripping characteristics should be modified or relays replaced to accommodate the above short time emergency current at 90% voltage, and 85% power factor. As necessary blinders should be added or phase protection design changed. Proposed Planning Principle: For each major load area (such as multiple city metropolitan regions) the risk of cascading load loss due to inadequate transmission facilities should be reduced to a reasonably low probability. Based on each major load area and its long lead time planning horizon ; for completing transmission reinforcements, associated firm peak load projection, and associated firm commitments to construct new generation; Each control area planning entity should establish an Emergency Import Objective for each major load area. Example Practice: (2.A) For each major load area the risk of exceeding the transmission Emergency Import Limitation (EIL) should be reduced to a very low loss of load probability. Based on the major load area s specific long lead time horizon such as 6 years; for committing to major transmission reinforcements, associated 6 year firm peak load projection, and associated firm commitment to construct new generation within 6 years to reduce the transmission import requirement, an Draft

28 Pre-Decisional Draft for Discussion Emergency Import Objective (EIO) shall be established for each major load area. The established MW import Objective shall include the impact of an overlapping forced outage of at least one (or two) largest generators in each area. If the calculated long lead time Transmission Import Limit (EIL) is less than the Emergency Import Objective (EIO), then a commitment to start and fund transmission reinforcements shall be made immediately, and the transmission owner fully compensated as work progresses. In the mean time if new firm generation, or load management commitments are made which sufficiently reduce the long lead time EIO import objective, the new transmission reliability project should then be placed on hold as deemed appropriate by the ERO reliability organization. If firm commitments for new generation or load management are not made in a timely fashion, transmission reinforcements must proceed to protect system reliability and firm load. Proposed Planning Principle: Where severe low voltage conditions increase the probability or speed of a cascading event, Safety Nets should be examined, alternatives documented, and the preferred alternative applied. Such schemes should be a backup line of defense against multiple facility outages resulting in a cascade. Such Safety Nets shall not be permanently used as the primary means to avoid long term transmission system or generation reinforcements. Example Practice: (3.A) As a safety net to slow down or avoid a cascade, local under voltage load shedding schemes should be used. Complex centralized schemes should be avoided to the extent practical. At substations with transmission facilities and under-frequency load shedding relays, install under-voltage load shedding relays with multiple step time delay not to exceed 30 seconds. To avoid unintended mis-operations, use two sources of potential, one from the transmission bus and one from the distribution bus. System Operations Proposed Operations Principle: Prompt and effective system emergency operation requires a clear command and control protocol including a single, jointly understood, emergency procedure. Multiple area swiss chess jurisdictions should be minimized to the extent necessary to provide prompt and effective hierarchical control of system emergencies. Example Practice: (A.1) Transmission Control Area jurisdictions should be established to minimize overlapping confusion, and provide prompt and effective system emergency operation. Multiple Generation Control Areas may be included in one Transmission Control Area as deemed appropriate. In very large existing Generation Control Areas more than one Transmission Control Area shall be established if deemed appropriate for reliable contingency analysis Draft

29 Pre-Decisional Draft for Discussion modeling purposes, or more importantly, to maintain prompt and effective system emergency operation. Proposed Operations Principle: Whenever full operational capability is compromised on a necessary transmission facility, software or computer hardware function, or operator control function, an operations procedure should establish a reasonable period of time to return the facility to full operational capability, or trigger an appropriate more conservative mode of operation. Full operational capability of a necessary facility should not be compromised for an indefinite period of time. Example Practices: (B.1) If a control area alarm function, or single contingency analysis is inoperable, the adjacent transmission entities shall be notified immediately. If both entities lose alarm capability for more than 15 minutes, or lose capability for more than 30 minutes to diagnose the system s capability to withstand a single contingency, a system emergency shall be declared. The predefined joint emergency procedure shall be executed, and preparations shall be made to shed load as specified in the emergency procedure. (B.2) During normal weather conditions, if a transmission facility trips and recloses, and repeats this pattern again prior to completion of a field inspection, the transmission facility full load capability shall be declared suspect. The facility ratings shall be lowered to the power flow at time of tripping. Since single contingency analysis will show a contingency overload based on this rating, the compromised facility shall be declared inoperable and assumed to trip in each contingency outage set. This n-2 contingency analysis should continue until a field inspection is completed, and repairs are made as necessary to return the facility to full load capability. Proposed Operations Principle: Daily operations planning, and subsequent real time operations planning should use available system reserves as necessary to maintain a Tv (30 minute) compliance capability to meet the intent of NERC operating Policy #2. Example Practice: (C.1) Daily operations planning, and subsequent real time operations planning should plan on using all available system reserves as necessary to maintain a 30 minute compliance capability. After a forced outage of any facility, sufficient available generation reserve should be started, or load management executed to maintain a 30 minute compliance capability. If and when all such reserves are activated and compliance capability is still compromised, the control area should declare a system emergency. The control area would also declare loss of compliance Draft

30 Pre-Decisional Draft for Discussion capability with NERC operating Policy #2. The predefined system emergency procedure should then be executed, and preparations made to shed load as specified in the emergency procedure. By adopting the above principles A, B, and C and implementing daily operations planning practices as noted in the above examples, control desk operators will have early warnings. If implemented in each control area and RC daily planning, the above will considerably improve operator situational awareness and capability to deal with system emergencies. Draft

31 Pre-Decisional Draft for Discussion System Study System Planning and Design Team Draft Recommendations 1. Planning studies (long term and seasonal) should ensure that analyses are comprehensive, complete and include a full range of sensitivity studies to reflect operating scenarios likely to be encountered. (e.g. N-1 contingency assessments assuming all elements inservice pre-contingency under forecasted peak load conditions and known firm transfers may not be comprehensive enough to ensure an adequate level of reliability.). The sensitivity studies should include multiple generator and transmission equipment outages, load beyond forecast, import levels and transaction bias on the system, load power factors, load representation and conditions on adjacent systems. 2. In carrying out planning studies for Category C contingencies, specific operating measures that are relied upon to achieve system re-adjustment within 30 minutes must be identified, shown to be feasible and communicated to operating personnel with appropriate operator training. 3. Study the effect of special protection systems (or remedial action systems or safety nets) with automatic arming features to cover those Category D contingencies deemed critical by the Planning entity. 4. Carryout a periodic review of the operating limits of interfaces or flowgates to ensure that the most constraining of thermal, transient, dynamic, voltage stability is reassessed as system operating conditions, including transaction bias on the system, changes. 5. Conventional voltage studies assessing normal and abnormal voltage range and percentage voltage decline do not (a) provide a comprehensive indication of a secure operating voltage or (b) ensure adequate dynamic reactive reserves or (c) ensue a proper balance between static and dynamic reactive resources. They must be supplemented by voltage stability analyses, using the well-established P-V and VQ techniques to ascertain a minimum pre-contingency operating level, such that the post-contingency voltage would be above the voltage collapse point with adequate margin expressed in terms of dynamic reactive reserve. 6. Communicate and explain planning/operational planning studies to system operation personnel, neighboring systems and Reliability Authorities. Training should be included. 7. Transmission Owner s Facility Connection Requirements for connection to the Bulk Electric System should require distribution entities and customers connected directly to the transmission system to plan and design their system to operate close to unity power factor under heavy load conditions to minimize the reactive power burden on the transmission systems (Planning Standard 1D Guide G2) 8. As part of regional coordination, develop a formal process for members to submit the results of their seasonal and longer term transmission planning studies for a peer review and approval on a periodic basis. Draft

32 Pre-Decisional Draft for Discussion Coordination of Regional Assessments, Inter-Regional Assessments and Between Control Areas 9. Control Areas must submit data and system conditions representing reasonably stressed internal interfaces based on pre-dominant flow pattern. Control Areas must provide operating limits (or studied limits) associated with key interfaces to their Region to provide a benchmark on the data and system conditions submitted. Data submission should also include an explanation of the basis for deriving the interface limits i.e. transient, voltage or thermal considerations and any pertinent information on maximum import/export levels, external transfers or reliability must run generation requirements. 10. Regional reliability studies must assess region-wide transfers with different transfer biases to identify potential choke points for each Control Area and Reliability Authority to carry out further assessments. Coordinated assessments of individual Control Areas, based on their stated interface limits and appropriate regional transfers, should be carried out by the Region or Control Area as part of their annual assessment process with an aim of covering all of the critical interfaces for all the Control Areas in a given period. 11. Inter-regional reliability studies must continue to assess wider area transfers than regional studies by including several regions in the studies and repeat the process above to come up with the boundaries of inter-regional transfer capabilities. 12. Each Control Area must review the results of Regional and Inter-regional transfer studies and capture these transfers to ensure that the development of internal limits properly captures the effects of external transfers and other conditions on adjacent systems or elsewhere on the Eastern Interconnection. These limitations must be communicated to the transmission operators and Reliability Authorities. Identified restrictions must be submitted to the Regional and Inter-regional study groups so that the impacts will be included in future assessments. 13. In conducting its internal assessment, if one Control Area identifies the potential for adverse impact on its neighbors (e.g. contingencies in one Control Area adversely impacting the neighboring Control Area), then these results should be reviewed as part of a formal Regional process to enable joint development of inter-dependent operating limits and procedures. Modeling and Assumptions 14. Control Areas should conduct an assessment of the availability of distribution capacitors before the summer peak period and use this data as a sensitivity in the seasonal and near term planning studies 15. Reinforce compliance with NERC Planning Standard 111.C related to operating generators with their excitation systems in the automatic voltage control mode and Standards 11.B related to testing and notification of generator Mvar capability. Draft

33 Pre-Decisional Draft for Discussion 16. Detailed modeling of sub-transmission network (below 110 kv) including transformers with the appropriate under-load tap changers, distribution capacitors and customer loads should be used in studies by Control Areas in assessing operating limits that are restricted by voltage performance and voltage stability. If complete detailed modeling on the subtransmission network is not possible, constant MVA load representation should be used. Reactive power Management and Voltage Stability Limits 17. Voltage magnitudes alone are poor indicators of voltage stability or security because the system may be near collapse even if voltages are near normal depending on system characteristics. The system should be planned so that there is sufficient margin between normal operating point and the collapse point to allow for reliable system operation. Real time monitoring of dynamic reactive reserves should be considered to ensure that the minimum requirements are always met. 18. Develop and periodically review a reactive power margin utilizing PV, VQ analysis against which system performance should be evaluated and used to establish maximum transfer levels as well as reliability must run generation requirements. Care should be taken to define appropriate regions o assess reactive margins whose boundaries may not coincide with ownership/commercial boundaries. 19. If systems are designed to depend upon capacitor switching to avoid voltage collapse, then redundancy must be provided 20. The minimum operating voltage (pre-contingency) must be duly reflective of the margin required to maintain a secure post-contingency operating voltage to prevent voltage collapse. 21. Operating instructions on minimum operating voltages must include trigger points such as voltage levels and voltage deviations for system operating personnel to take immediate actions including load shedding via SCADA. 22. Operators should be trained to be vigilant to avoid operating the system under peak load or heavy transfer conditions at the lower end of the acceptable operating range, i.e. within a normal and acceptable range of Operate at the higher end to maintain a higher level of system security when practical, taking into account the need to maintain adequate dynamic reactive reserves. 23. Procedures should be in place and operators trained to mitigate declining voltage profiles 24. Generators should maintain adequate reactive reserves during normal operations to ensure dynamic MVars are available to respond to contingencies. Off-line or near realtime studies can identify the level of adequate reactive reserves required for the secure operation of the transmission system. However, absent these studies, it is good utility practice to strive to operate the transmission system such that generators operate close to unity power factor at the point of interconnection Draft

34 Pre-Decisional Draft for Discussion 25. Processes should be reviewed to ensure both transmission providers and generator entities have proper incentives to maintain adequate reactive power support. As noted in NERC Planning Standard I.D.M2, Generator and transmission providers should work jointly to optimize the use of generator reactive capability. Process to ensure adequate reactive support is in place should include: 26. Minimum design and operating reactive capabilities from generators 27. Identification and implementation of transmission system reactive support through adequate studies by the transmission provider. These studies should ensure that appropriate voltage levels and reactive margins are maintained for all credible contingencies identified in NERC Planning Standards IA Table An ongoing test to ensure both generators and transmission providers have provided adequate reactive support is the ability of generators to be able to hold a voltage schedule as required by the transmission system operator (NERC Planning Standard III.C.S2). 29. Investigate establishment of station/load bus power factor obligations System Emergencies and Extreme or Multiple Contingencies 30. Control Areas and Reliability Coordinators should carryout a periodic review of operating instructions for declaring system emergencies and effective operating measures to deal with them. 31. Operator training must emphasize the need to place the system in a safe-posture so that when the system enters an unknown state or a state not covered by existing instructions or studies, the operator must exercise all possible and effective measures to secure the state without fear of reprisal. 32. Operating strategies must be developed to ensure the return to a (safe) reliable condition following an unplanned event on the system. 33. Emergency operation instructions must be shared and coordinated between adjacent control area and Reliability Authorities to minimize seams issues. 34. Extreme contingency evaluations must be conducted to measure the robustness of the interconnected transmission systems and to maintain a state of preparedness to deal effectively with such events. 35. Extreme contingency assessments must be conducted on a coordinated intra and interregional basis so that all potentially affected entities are aware of the possibility of cascading or system instability events 36. Consider safety nets to prevent cascading outages including use of automatic undervoltage load shedding. Automatic under voltage load shedding should be regarded as a safety net and should not be used as a substitute for adequate system design. Draft

35 Pre-Decisional Draft for Discussion 37. Manual load shedding blocks that operators can implement through SCADA should be in place. 38. Operating guides and/or instructions must be reviewed periodically to ensure procedures and measures exist to restore system to a secure state in the allowable time periods. 39. TLR should not be relied upon to restore the system from an insecure state to a secure state, as these procedures cannot be implemented effectively within the required timeframes in many cases. Transactions, Flowgates and IDC 40. Reliability Authorities should ensure that periodic reviews of the adequacy of existing monitored flowgates and their limits are conducted and modify the flowgates in the IDC as prudent to accommodate changing system conditions. 41. The Reliability Authority must determine the flowgates to be monitored and have mechanisms in place to derive their limits as system conditions and topology changes. Draft

36 Pre-Decisional Draft for Discussion Simulation, Modeling and Analysis Team Draft Recommendations Time Synchronization and time stamping of telemetry ICCP data (volts, flow) Frequency data ACE data All DFRs, DERs, etc. Dynamic Schedules All dynamic schedules must be tagged or dynamically available for input to IDC and other system security analysis packages SDX Near term make timely data submittal mandatory within 10 minutes? Long term use a topology processor to provide data input, either to replace SDX data or to interface with it Line ratings Methodology of calculation Peer review of methods Database of tie lines (in existence for MMWG) Procedure for broadcasting line rating updates to all concerned parties (with sanity checks) Powerflow modeling errors Overly optimistic power factors Incorrect circuit ratings (see above) Need to benchmark to system readings periodically what is the power factor served as observed at the kv system Database of powerflow topology to ensure ratings and topology consistency long discussed by MMWG Interregional Studies Need to take all outages, including generation outages More than N-1 contingencies Need to monitor entire system (not limited to regional interfaces) Need to look at wider variety of simultaneous transactions both magnitude and direction as well as simultaneous transfers Learn from historical transfer trends ongoing analysis needed Postulate other conceivable (economic or weather sensitive) Transfer Sensitivity Study Tool Avoid misdirection of operators, allowing them to know whether a problem is transaction driven, or related to NNL Draft

37 Pre-Decisional Draft for Discussion Near-term include a TLR 5 calculator on demand for any line showing breakdown of impacts by priority bucket Draft

38 Pre-Decisional Draft for Discussion Data for Control and Compliance Frequency Team Draft Recommendations Data quality needs to be improved. At present the FAIT analysis was hampered by missing or inadequate data storage or time stamping. This includes the AIE surveys. A high degree of uncertainty exists on the timing of the generating unit trips and thus the calculated EI frequency responses. It is unclear what other events may have occurred in the EI that could have masked or modified observed SOE events. The FAIT recommends that NERC and/or DOE expedite industry efforts to develop a standard or standards for: Time stamping original field data, including generator events, based on a national standard time standard (GPS, IRIG or whatever) Collecting generator outage data (time, MW, ramp rate and location) and routinely analyzing impacts on interconnection frequency The RS should prepare a data request for PJM, requesting to clarify the time discrepancies and the possible correction between their abnormal ACE and abnormal Frequency Response during the period AGC was not available on 8/14/03. Once causes are identified, RS should assess if recommendations are required, The RS should issue an AIE follow-up letter asking these control areas to explain the difference between their AIE and ACE and specifically ask them if they lost generation or schedules during the hour. Causes for AIE-ACE difference should be assessed and recommendations defined. There was an ACE excursion at the beginning of the HE 14. With the above request the RS should ask the Control Areas to explain the reasons for this sustained the over-generation. Although no potential link between the analyzed extended frequency excursion and the August 14 blackout was identified, the RS should identify the reasons of CPS2 violations in CIN and TVA with these control areas. The RS should request detailed information for the Control Areas to understand the causes for the large frequency excursion HE 13 (double what would be expected from Eastlake 5 outage). The RS should request Frequency Response Survey(s) near the 8/14/03 blackout. Handsome Lake trip is consistent with a 2,500 MW/0.1 Hz. Frequency Response for the Eastern Interconnection, Piney Creek trip is consistent with a 775 MW/0.1 Hz. Frequency Response for the Eastern Interconnection. If this is true, then it would indicate that the Frequency Responsive Resources on the Eastern Interconnection were virtually exhausted by the time of the Piney Creek trip. NERC should require that control areas still not reporting ACE to NERC to submit their reports. Explanation of differences between AIE schedules and IDC schedules and any impact on voltages in Ohio. Approve and implement the proposed Frequency Data Analysis and Storage project. Draft

39 Pre-Decisional Draft for Discussion Identify, report and assess Control Areas experiencing EMS-AGC-data problems on 8/14/03 Publish listing of Control Areas not providing real-time data Obtain real-time net actual and scheduled interchange with adjacent Control Areas. This will help identify unexplained ACE and allow an ACE calculation for each Control Area when their EMS is down (via a calculation using neighbors data). Pulse Accumulator Meters (MWh meters used for hourly accounting). All adjacent control areas shall agree on one official tie line meter. All adjacent control areas shall agree on calibration procedures. All adjacent control areas shall agree on meter error adjustment procedures. Pre-identify alternate source of MWh readings. Integrated AGC tie line readings. Integrated State estimator. Redundant MWh meters. AGC Control Control Area survey for AGC backup system, analog and/or digital, availability and control functionality. Assess survey results and come out with recommendations if required. Control Performance Metrics and Standards Questions have also been raised about current control performance criteria for situations where interconnection frequency remains continuously above or below the schedule frequency for long periods of time as happened on 8/13 for close to eight hours and on 8/14 for more than three hours. Current Policy 9 request Reliability Coordinators frequency remedial action when the interconnection frequency error is in excess of Hz for more than 30 minutes. How Reliability Coordinators are interpreting Policy 9 for this frequency condition?, Does this condition impact system reliability?, Is Policy 9 guide adequate?, how compliance is being monitored?, Will the new Policy 9 guides cover this condition?. AIE is the reference standard for a reliable ACE Reserve Sharing: Make sure each control area can validate changes to its NIs with a journal feature in AGC and accounting systems. Control Performance Compliance: Identify and publish Control Areas not providing data (AIE and/or ACE). AIE Survey Recommendations: Draft

40 Surveys: Pre-Decisional Draft for Discussion Obtain data from Control Areas not providing it. Track down cause of other resource loss nearly coincident with Eastlake 5 (i.e. letters to entities with AIE indicating a possible resource loss). Get explanations from Control Areas with questionable ACE. Get explanations where real-time ACE data varies from 10 minute AIE-reported ACE. Get explanations from Control Areas with AIE > L10 aggravating frequency Recommend running frequency response surveys for 8/14/03. Provide a template on RCIS for collecting unit trip information to facilitate benchmarking of frequency response and investigation of system events Develop a common format data provision and archiving requirement (or standard) to facilitate analysis and performance validation. Forms filled out incorrectly indicate a need to review the clarity of the instructions. Perform a Frequency Response Characteristic survey for the Eastlake generator trip. This is cross check the assumed EI natural frequency bias. Modify the AIE NERC summary report to support more cross checks on the AIE ACE, the 10-minute AIE ACE and the CERTS 1-minute ACEs. Real Time Monitoring Tools Improve current real time monitoring tools for Reliability Coordinators for monitoring ACE and frequency with requirements identified during this investigation. Provide Reliability Coordinators delta flow tools to flag and point to problems. Tuning up Reliability Coordinator monitoring and alarm tools: For unexplained delta frequency Investigate the use of additional real time control monitoring tools to help operators to identify normal operating conditions and unexpected operating conditions. NERC to expedite approval and implementation current Frequency Data Analysis and Storage project (one second resolution redundant data frequency information storage for each interconnection). NERC to expedite Control Areas transfer of data for the NERC AIE Real Time Monitoring application. Draft

41 Pre-Decisional Draft for Discussion Wide-Area Real Time Operations Control Training Major effort and short and long term plans for educating Reliability Coordinators and security personnel in how to use that data in a constructive, or at least effective, manner during normal and emergency operational conditions. Develop training documents to assist: Reliability Coordinators in identifying and tracking down frequency problems. Control Areas in meeting their balancing obligations. Incorporate in Training plans and documents what is a quality ACE? Alert the System Operator when ACE is suspect When do you know ACE is suspect? Real-time: check instantaneous ACE components against state estimator ACE components Hourly: check pulse accumulator ACE components against integrated ACE components Any backup ACE should be computed in hot standby mode so it can be checked continuously. Pre-identify alternate sources of tie line telemetry Assess true independence of alternate sources to prevent common mode failures. Does ACE change correlate to the expected frequency change? Interconnection ACE should be: Compared to ΣACECA. Compared to Interconnection frequency. Provided to each control area so the system operator can observe the same information Reliability Coordinator sees. This data sharing supports better communications. Incorporate in training plans and documents Good Utility Practice Guidelines: For hourly accounting and error reconciliation. Synchronize ramping generation to transactions ramps Draft

42 Pre-Decisional Draft for Discussion Meter and telemetry design for primary and backup service Draft

43 Pre-Decisional Draft for Discussion Data Exploration and Requirements Team Draft Recommendations The type of data required should be standardized in terms of the formats, the sampling rates, time-stamps, storage times, accuracy required along with calibration schedules, and lengths of time it should be stored. The amount of data could be overwhelming so appropriate compression and storage mechanisms might also be candidates for standards. There are standards like COMTRADE that are excellent ones that should be mandatory where appropriate. A method to quickly (within 24 hours) generate the sequence of events should be developed. This would mean that time-stamping topological data (which is not currently done) would need to be implemented. Software could be developed that polled a certain minimum amount of information needed to construct the sequence of events. Once the sequence is known more tailored and detailed additional data could be requested from the most affected areas. This would be analogous to the flight recorder or black box in aviation. Implementing an events recorder approach would also ensure steps could be taken to mitigate the possibility of a similar outage in the near term because a general cause would be known quickly and accurately. Recording equipment should be standardized: Data Format: It is useful to have a standard, required data format. The format should be unambiguous. In the generation section of the NERC data request, the format for the generation outages was mostly clear (see reactive power discussion below under data quality), while the format for the supplying 2-second data for multiple units was not clear. Some chose to place the data for the units in separate files; others placed the data for all units in a single file. The latter proved difficult to analyze using common spreadsheet software (excel). We consider two options for supplying the 2-second data: 1) the data can be supplied in separate files for each unit, 2) the data for all units can be incorporated into a single file. The former is straightforward, the latter requires further consideration. If the data is to be combined in a single file, the first column(s) should contain the time data and subsequent columns should contain unit information. In this case it makes sense to have separate files (or worksheets) for the different types of data (MW, MVARS, Voltage). In the course of an investigation, it will be necessary to study the output of individual plants. To that end we recommend that 2-second data be supplied for each unit in separate files. Data Needs for Quick Analysis: The data that will be most valuable for a quick analysis of a power outage will depend on the cause of the outage. In a slowly-accumulating voltage collapse scenario, voltages, active and reactive loadings and power flows, and component tripping data will be key. For angle instabilities, additional frequency and control signal information will be required. For a general network collapse characterized by a cascading sequence of component failures without unusual initial voltage or frequency deviations, power flows, protection settings, component tripping, voltage and frequency information is required. All types of data are necessary since the type of failure will not be known beforehand. Ideally, all active and reactive injections and power flows, voltage, frequency, and apparatus availability will allow one to describe a sequence of events, and probable causes. A more in-depth analysis will require information about component capabilities and protection settings to determine if the system operated according to design. Interviews with operating personnel and transcripts of operator Draft

44 Pre-Decisional Draft for Discussion interactions will help determine if human decisions affected the blackout in beneficial or detrimental ways. These interviews may also clarify changes in dispatches and breaker operations that may appear anomalous in raw data. There are critical issues related to the sampling of data that deserve note. Of primary importance is that the reported times are synchronized. Otherwise the sequence of events and subsequent analyses may be flawed. Also, the sampling of the data must be fast enough to resolve the true phenomenon. For example, a one-half hertz frequency (or power) oscillation will not be accurately resolved by 30-second SCADA data, or even 6-, 4-, or 2-second sampled data.1 Data from other devices will be required to accurately resolve such fast, dynamic events. A geographical display of the system will greatly facilitate the visualization and analysis of a large failure. To construct such a detailed display necessitates collection of information regarding the location of all relevant equipment. This need not be a task for an investigation, but should be collected and implemented by NERC and appropriate agencies in anticipation of outage studies. Assuming such a visualization tool exists, the display of voltages, power flows, congested interfaces, and power imbalances by area will go a long way to giving a quick picture (movie) of an outage. Data Quality: The time stamps need to be accurate and clearly delineated in the data. Accuracy is required to understand the interactions among many elements. Stating the time with sufficient resolution is needed to study the data. In some of the 2-second data that was submitted to NERC for this blackout study, the time is only shown to the minute. The seconds need to be shown. Concerning generator capabilities, one needs to be clear on how to describe the generator reactive power capability. Certainly the capability at the time of the blackout is important. In evaluating what if scenarios, it would be useful to have a more complete characterization of the unit s MW/MVAR capabilities. It is reasonable to suggest that reliability councils should collect this information on an annual or quarterly basis. 1 Recall that the sampling theorem requires sampling at least twice the frequency of the signal. Slower sampling will result in an observed frequency at lower than the true frequency. Draft

45 Attachment 4 Gerry: these are Vickie VZ s Panel B comments for the record/posting 10:15 11:30 Panel B: Emergency Response 1. What recommendations are there to improve the ability of operators to identify, declare, and effectively respond to emergencies? What training and certification of qualifications should be required? Assessment encompassing more than seasonal studies of typical conditions more widespread determination of simultaneous transfer limits taking into account generating patterns, reactive requirements and voltage collapse points Interactive simulation of low probability, but severe system conditions to recognize disturbance characteristics and to practice response techniques for both disturbance response and restoration Good visualization tools to help with this recognition and to get immediate feedback on the effectiveness of operator actions in real-time Reinforcement of training curriculum and testing to include simulators using actual disturbance scenarios, recognition of disturbance precursors, mitigation of disturbance severity. Include lessons from other areas/regions. Hopefully, we can eliminate the common causes in disturbances because we have effectively transferred learnings from one area to others. 2. What minimum tools do operators (and reliability coordinators) need to recognize and manage system emergencies effectively? Status and alarm features minimum fail-safe criteria for EMS systems to include back-up provisions, functionality monitoring and assessment, and transfer of control to other authorities should the primary and back-up systems fail. Consider 24/7 IT monitoring of EMS tools (separate from system operations staff) Thermal, voltage collapse studies either real-time or offline, but with current system conditions If cascading occurs for some contingencies more quickly than an operator can reasonably respond must have automatic controls in place to act as a safety net. Note: all the studies in the world and regardless of the care taken in assessing the current status and vulnerabilities of the power system, they mean nothing if there are obstacles in the transmission rights-of-way. Transmission lines that trip out before overload render that assessment invalid. 3. Between system operators and reliability coordinators, who has the authority and responsibility to declare emergencies and how should that relationship work? Control Area Operators are the first line of reliability defense should be first to declare an emergency. Should be the entity to take action. Control Areas need to raise the alarm when they re in trouble. However, if they do not, RC s must have sufficient authority

46 and means to carry out actions to preserve reliability. Confirm that authority through regular and rigorous audits. If the RC sees a disturbance in the making authority to declare an emergency and order specific relief. System operators need to comply. If they disagree, comply first, argue later. RC needs enough visibility with enough granularity to assess system problems and determine appropriate response to shortstop a potential cascading problem. RC needs tools sufficient to determine simultaneous limitations on transmission paths (thermal, voltage, transient stability if appropriate). In some areas, SE and Contingency Analysis is necessary but not sufficient. If CA says you have a problem you do if it says you don t have a problem you still might (doesn t provide transient stability limitations). 4. Do operating personnel (and reliability coordinators) have the necessary authority to act in a timely manner during an emergency? Are they empowered to act in an emergency to prevent conditions that could lead to a cascading outage? What can be done to reinforce that authority? Authority to take actions and responsibility to take those actions to prevent a cascading outage. Industry must support operators exercising that authority it will not be obvious if their actions have been successful. Must have and know where collapse points are assessment tools need to include wide area view of current conditions and warning of contingency problems beyond that of nonsimultaneous thermal limits of transmission circuits. 5. Do operating personnel (and reliability coordinators) have adequate system resources (redispatch, reconfiguration, load-shedding, etc.) to take timely corrective action during a system emergency? Is a full range of credible emergencies being assessed during operations planning? Are there adequate resources available to respond to rapidly escalating system emergencies? Not always. If must use TLR 6 must be possible to shed load very quickly to prevent the severity of a disturbance in progress. Need a stack of redispatch options to relieve transmission quickly. Schedule reductions is meant for contingency relief proactively in anticipation of a possible contingency not for emergency action 6. What recommendations are there to improve communications among operating personnel (and reliability coordinators) during emergencies? Standard protocols common terminology Emergency drills simulation with both control area or transmission operators and reliability coordinator See into each other s systems far enough to be effective. Agreement at seams about what facilities will be discussed with relevant CAO and RCs which ones matter

47 7. What recommendations are there to ensure operators (and reliability coordinators) focus on reliability and are not distracted by other activities? Dedicated personnel assessing of status of system possibly leaving the commercial arrangements separate from operation Communicate with each other (within and between control centers) Protocols prioritizing the attention of a control center dispatcher to determine course of action as a disturbance progresses delegate commercial schedules or routine phone calls to others

48 Attachment 5 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE Overall Perspective and Comparison of FE, AEP, MECS and PJM Two of the key parameters in observing the performance and security of an integrated power system are voltage profiles and dynamic reserves as a function of load demands, transactions across the regions, network topology and availability of generators. This section provides a high level comparison of the salient parameters of the major interconnected systems, AEP, MECS and PJM, surrounding the FE footprint to show the general trend of load demands, transaction activities, voltage profiles, reactive power reserves. In addition, the dynamic Var reserves from representative FE generators in and near the Cleveland-Akron area and representative generators of the interconnected systems that are located near the interconnection buses with First Energy are compared to emphasize the importance of providing adequate reactive power locally under normal and contingency conditions. This section also provides the results of P-V and Q-V analyses and compares the impact of external transactions vs. Cleveland-Akron load increases based on the observed system conditions at various critical times on August 14. Comparisons: Figure AA displays the peak load demands for the August 1-14 period. The temperatures measured at the Akron-Canton Airport increase from 78 F on August 11 to August 14. The FE peak load demand increases by 20% from a low of 10,095 MW to a high of 12,165 MW. The load increases in the surrounding systems displays a similar trend. (Under preparation) Figure BB shows the transactions involving FE and all the surrounding systems as either sinks or sources for the August (and the transactions during the peak day in 2003). The comparison shows that the transaction bias on August 14 is not materially different from the proceeding week days. It is high, but not unusually high as compared other days of that week and the peak day in (Under preparation) Figure CC compares the voltage profiles in the south-to-north direction for the four hours (12:00, 13:00, 14:00 and 15:00 EST) for the August period. The voltage profile consists of buses as far south as buses xx in AEP and as north as Monroe in MECS. It displays a V-shape profile with the critical buses such as Star, Hanna, Fox in the Cleveland-Akron area forming a low cluster. The interconnection buses in the surrounding systems and their remote buses display a near uniform profile and are consistently higher than the low cluster in the Cleveland-Akron area. (Under preparation) Page 1of 39

49 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE Figure DD provides a similar comparison on voltage profiles in the west-to-east direction for the same four fours for the August period. It displays characteristics similar to those observed in Figure CC. (Under preparation) Figure EE shows the recorded voltages at the Star 345 kv bus, for the August It shows a wide range, from the peak of near 355 kv to below 340 kv. (Somehow, we will lift this from the EPRI report Figure 13). It shows that the FE 345 kv network is capable of operating at 355 kv if not higher. Figure EEE shows a summary of the reactive power resources in the AEP system on August 14. It displays an effective reserve margin of 3,050 MVAR at around 15:00. Figure FF shows a summary of the reactive power resources in the PJM system on August 14. It also shows a significant amount of reactive reserve at 15:00. Figure GG shows a summary of the reactive power resources in the MECS system on August 14. It displays??? of reactive reserve at 15:00. While a similar plot on the Cleveland-Akron Area is not available, as a proxy, it is instructive to compare the dynamic Var reserves from the representative generators in and near the Cleveland-Akron Area and in the adjacent interconnected systems (AEP, MECS and PJM). (Lee: Help and pre-write this based on what you know. If we can t do this, we will skip this paragraph in the final report.) With respect to changes in network topology in the August period (prior to the Chamberlin-Harding outage), there are no major changes in FE and its surrounding systems that would have a material impact on these interconnected systems. (Need to confirm this). With respect to the changes in the available generations in the August period, the salient changes are those in the Cleveland-Akron Area. On August 14, FE s additional generation outages compared to August 12 consist of the Sammis 3 (180 MW), East Lake 4 (240 MW) and East Lake 5 (600 MW). It is noted that these are generators critical to the supply to the Cleveland-Akron Area. (Need to confirm other CA did not have substantive changes in generation.) Figure HH shows the results of P-V analyses involving south-to-north transactions and west-to-east transactions show minimum voltage variations at the Star 345 kv bus, using the power flow case with East Lake 5 out of service. Power flow analyses show the south-to-north transactions would result in a transfer distribution factor (TDF) in the 15-18% range through the cut-set of the transmission lines into the Cleveland-Akron area, where the west-to-east transactions would result in a TDF of less than 10% through the same cut-set. (Under preparation). Page 2of 39

50 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE Figure II shows the results of P-V analyses involving load increases in the Cleveland- Akron area show significant voltage declines and depletion in dynamic Var reserves in the generators supplying the area. It clearly demonstrates the impact due to load increases in the Cleveland-Akron is much more significant than that due to external transactions not involving FE. (Under preparation) Figure(s) JJ shows the results of V-Q analyses at Star 345, Hanna 345 and Fox 345 for the four key system states at 15:32 (loss of Hanna-Juniper), 15:41 (Loss of Star-S. Canton).. The results show a rapid upward shift of the voltage collapse points above the 0.9 pu level and a rapid depletion of Var reserve at the corresponding buese. (Under preparation.) Figure(s) KK shows the results of V-Q analyses at the surrounding interconnection buses in AEP, MECS and PJM for the same system states. The results consistently show that the voltage collapse points at the surrounding buses are below the 0.85(?) pu level with considerable Var reserves at these bueses. (Under preparation). Conclusions and Observations: From the overall comparison of recorded data in the August period and P-V and V- Q analyses the following conclusions and observations can be made: There are significant increases in the load demands across the regions. As the temperatures increase from 78 F to 88 F with a 20% of load increase from 10,095 MW to 12,165 MW in the FE system. The load increase consists of a significant portion of air-conditioning loads with a significant increase in reactive power consumption. The transactions on August 14 are high, but not unusually high compared to preceeding days or the peak day of the The voltage profiles in the south-to-north direction and the west-to-east direction show a consistent pattern of a V-trough in the Cleveland-Akron Area. The voltage profiles in the surrounding systems are consistently near or above 1.0 pu. The downward changes in voltage profiles in the Cleveland-Akron area are much higher, ranging from xxx kv on August 11 to yyy kv on August 14 at 14:00 pm observed at the Star (or Hanna) 345 kv bus. There are considerable amounts of reactive power reserves in AEP (x Mvar), MECS (Y) and PJM (Z) on August 14 for their own system needs. The dynamic reactive power reserves on August 14 at the representative locations in AEP, MECS and PJM consistently show reasonable levels of dynamic VAr reserve. The FE representative generators such as xx, yy, and zz show depletion of dynamic var reserves as earlier as 14:00 pm on August 14. This also Page 3of 39

51 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE demonstrates the laws of physics on the inertia (or reluctance) of reactive power to a long distance. The recorded bus voltages at Star (or Hanna) range from a high of near 355 kv for extended periods on the previous days. This is contrary to a general belief that the FE network was developed at a nominal voltage of 340 kv while others at kv, and as such the FE maximum operating voltage is generally restricted to 340 kv while others at kv. There is no evidence to suggest that FE cannot maintain a uniform voltage profile consistent with its neighbors from the viewpoint of its equipment s voltage capability. P-V analyses for the conditions on August 14 show that external transactions not involving FE have a minimum impact on the FE voltages. P-V analyses for the conditions on August 14 show that load increases in the Cleveland-Akron area has a significant impact on the voltage declines and depletion of dynamic Var reserves in the generators supplying the area. Q-V analyses for the conditions on August 14 show that the voltage collapse points (theoretical critical voltages) at the key buses in Cleveland-Akron area are near 0.9 and could exceed 0.9 pu. Q-V analyses for the same conditions for the surrounding buses in AEP, MECS and PJM show considerable margins in terms of voltage collapse points and Var margins. Recorded system conditions show no material changes in network topology in the previous days. Recorded system conditions show no major changes in generation availability outside the FE footprint on August 14 compared with previous days. However, the changes in generation in and near the Cleveland-Akron area are significant, 1020 MW consisting of Sammis 3, East Lake 4 and East Lake 5 compared to August 11. In addition, excerpts from FE control room conversation indicates that at least one 50 Mvar capacitor bank and possibly up to 4 and 5 capacitor banks were removed from service for routine inspection. Page 4of 39

52 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE Story Board Questions and Answers Operations planning and resource management (2) Are the industry-accepted methods and practices of operations planning for response to abnormal and emergency conditions adequate, considering the speed at which conditions can degrade on the power system? Is 30 minutes to restore the system back to within normal limits following a contingency that causes an operating security limit violation an appropriate amount of time? The mitigation actions and their effectiveness do not seem to be appropriate to mitigate large magnitudes of power through the flowgates in the required amount of time of 30 minutes. Based on the conversations between the AEP operator and the PJM RC, the magnitude requested for the TLR had never been that high. The use of TLR for most of the normal conditions experienced on the electric grid is acceptable. However, when large magnitudes of power need to be reduced in a short period of time (30 minutes or less), a new process may need to be explored. There are a number of actions to be taken that could be taken during emergency conditions to recover from an OSL: 1) Re-dispatch of generation; 2) Re-configuration of the transmission (may not be an option based on the equipment already out of service); 3) Controller adjustments (counter-schedules) between Control Area operators using AGC generating units and re-dispatch; 4) Utilization of RAS/SPS dedicated for specific outage conditions; 5) Pre-agreed upon measures between flowgate operators 6) Pre-arranged interruptible load shedding (available in less than 30 minutes) 7) Firm load shedding (available in less than 30 minutes) The above actions should be investigated by the operations engineering / grid operation RCs and CAs, and then provided to the operators in operating procedures/guidelines with appropriate training. The dispatchers should have complete authority to take the above actions and any others in order to satisfy the reliability requirements of the reliability organizations. For a transmission interface or path restricted by transient-dynamic stability or voltage stability, the 30-minute period needs to be evaluated. For example, in the WECC, a stability/voltage limited path is 20 minutes to restore the system back to within limits, while a thermally limited path is given 30 minutes to restore. These times may be short, but with agreed upon operating procedures and training between flowgate operators and Reliability Coordinators, these times are reasonable and executable based on WECC s experiences. Page 5of 39

53 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE Schedules and Transfers (1) Were FE and other parties involved in the blackout in conformance with their transmission tariffs on August 14? Was posted TTC and ATC accurate? Was FE operating within its posted TTC for imports? There is no evidence to suggest that any party was not in conformance with their respective tariffs on August 14. The discussion on TTC/ATC calculation as contained in the respective OATT documents in Attachment C is general in nature and is not prescriptive. By definition the TTC and ATC values are forecasts of transmission conditions months, weeks, days or hours into the future and are not operating security limits. Therefore reliability should not be impacted by inaccurate TTC and ATC values. The monthly TTC/ATC values for August 2003 were first determined over a year prior, and are to be updated as anticipated conditions on the network changes and as transmission service reservations are made. The daily TTC and ATC values for August 14, 2003 were first determined 30 days prior to that day. Similarly, these daily values are to be updated/revised as anticipated conditions on the network changes and as daily and longer term transmission service reservations are made. The hourly TTC and ATC values for the afternoon hours of August 14, 2003 were first determined approximately seven days prior and are to be updated as anticipated conditions on the network change and as hourly and longer term transmission service reservations are made. The hourly TTC and ATC values, for the afternoon of August 14 were determined hours, if not much longer, before the afternoon of August 14. When the TTC and ATC values were determined for these hours, forecasted load levels, generation dispatches and transmission/generation outages schedules are used to calculate these values. The TTC/ATC values for the ECAR area are generally determined on a first contingency basis. This means that the calculation of transmission capacity available for further commercial uses, above projected committed uses, (e.g. ATC) assumes that only the single most impactive facility may be forced out of service during the period of requested transmission service. If, during real time operation, more than one transmission facility is in-fact unavailable, load levels are significantly different forecast, generation dispatch is significant different from anticipated, unanticipated conditions are being experienced on other systems, it is likely that local operating procedures, market redispatch (if available) and/or TLR will be required to maintain adequate level of reliability on the system. Therefore, contrary to popular belief, the posted TTC and ATC values cannot reasonably be expected to be anything other than calculated best guess estimates of the transmission capacity available for further commercial activity beyond already committed uses. The TTC and ATC values are commercial values, and, therefore do not directly impact the reliable operation of the transmission system. Real-time monitoring and contingency analyses, among other reliability tools within EMS systems, as well as the NERC IDC, provide the means to the system operators to respond to real time facility loadings and voltages and real time contingency analyses to assure the transmission system is reliably operated to within reliability limited parameters. Page 6of 39

54 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE As August 14 progressed, FE lost the Eastlake generating unit and 345 kv circuits before the events became uncontrollable. Theoretically, the posted TTC and ATC values should have been recalculated as the forecasted conditions changed. However, it is difficult to expect that quickly changing conditions on the network can be reflected into the short term horizon posted TTC and ATC values. As stated earlier, since TTC and ATC values posted on the OASIS are not operating values, reliability should not be impacted by inaccurate TTC and ATC values. (Was FE operating within its posted TTC for imports?) A comparison of the power flows on the flowgate #xx shows it was (within or above, need to obtain data from FE to ascertain this) the posted TTC. While FE has not submitted the 2003 import capabilities and the actual imports representing critical times on August 14, evidence strongly indicates that the FE System could have operated above its System Operating Limits for the conditions experienced on August 14. This is based on the Summer 1998 Import Capability of the (former) Cleveland Electric Illuminating Company, as part of FE data submission to NERC, as summarized below: 1998 Summer Import Import Level Capability of CEI Load Level (CEI) All Transmission facilities In-service Next Most Critical Contingency 3500 MW Up to 2550 MW L/O Hanna-Juniper OK to 2350 MW 3800 MW Up to 2450 MW L/O Hanna-Juniper OK to 2250 MW 4000 MW Up to 2300 MW L/O Hanna-Juniper causes divergence at import levels >2000 MW 4200 MW Up to 2250 MW L/O Hanna-Juniper causes divergence at import levels > 2000 MW L/O East Lake 5 and Perry violates voltage criteria Notes: (1) LS 18 assumed on line (2) The import levels include the CEI portion of Davis Besse (454 MW), Bruce Mansfield (470 MW), Beaver Valley 2 (201 MW), Seneca (351 MW) and anticipated municipal wheeling for a total base import of (3) Voltages at numerous 138 kv and 345 V buses would be below 131 kv and 328 kv (0.95 of nominal 138 and 345 kv) for higher imports. (4) All CEI and CPP capacitors assumed in service and energized, where appropriate. Comments on Notes comparing to August 14 Conditions: CEI load was 4044 MW per FE data submission The difference in installed capacitors between 1998 and 2003 is unknown. The outages of Davis Besse, Sammis 3, East Lake 4 and East Lake 5 totaling 1,903 MW imposed a heavy import requirement to the Cleveland-Akron Area. Page 7of 39

55 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE At the 1998 studied limits of CEI load at 4,000 MW, loss of Hanna-Juniper at levels above 2000 MW resulted in non-convergence. The August 14 conditions are expected to be more onerous than the 1998 study conditions. Voltage criteria violations in the 1998 study would likely be within the current FE voltage criteria of pu pre-contingency and 0.9 post contingency, whereas non-convergence of post-fault power flows is not related to voltage criteria. (2) Were control areas in the Eastern Interconnection conforming to interchange schedules on August 14 and properly tagging those schedules for input to the IDC? What is the process for assessing and approving transaction schedules/tags? Analyze tagged transactions and actual schedules for August 14 to determine their conformance to system operating limits, including reactive constraints. Determine the change in system transfers caused by schedule changes in the period noon to 1600 hours EDT on August 14. How did the schedule/tags over this period compare with previous days and seasons? There is no evidence to indicate that any party was not conforming to the interchange schedules nor properly tagging those schedules for input to the IDC. According to the TagNet data as of 1200 hours on August 14, FE was forecasting net interchange within a range of importing 115 MW to exporting 94 MW. As that afternoon progressed, the net FE interchange for the hours 1300 through 1600 (as captured by TagNet for each of those hours) was importing 157 MW, imprting 533 MW, importing 577 MW, importing 641 MW respectively. It should be noted that FE s Beaver Valley generating station (1640 MW nuclear) is in the DLCO Control Area. Deliveries of the output from this station to the FE Control Area are non-tagged transactions based on previous agreements and therefore are NOT included in FE s net interchange values as discussed above. If the Beaver Valley deliveries were tagged as a point to point transaction, the output would be subject to curtailment under TLR protocols. Being a non-tagged transaction, forced re-dispatch of this unit would be a result of TLR. Not tagging transactions involving control area external resources for network requirements is not atypical. The merits of tagging this schedule (or dynamic scheduling) and similar transactions elsewhere are not quantifiable at this point, but are worthy of further investigation. The specific process used by FE is not known. However, Good Utility Practice would indicate that a process be employed of the schedule approving entity that assures that acceptance of any specific schedule will not result in reducing transmission reliability to unacceptable levels. The acceptance process should include a determination, before the schedule is accepted, if the acceptance of the schedule would be expected to result in any System Operating Limit being exceeded. Based upon this analysis, requested schedules that would be expected to reduce reliability to below acceptable levels, should be rejected. Page 8of 39

56 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE Although the specific tag/schedule analysis employed at FE on August 14 is not clearly understood, it is evident from the TagNet analysis that FE approved tags to further increase the import into the FE system, generally causing greater loadings on the 345 kv circuits that would be critical in the causation of the August 14 blackout later that day. During periods of uncertain conditions on the transmission system, the benefits of any passive approval of any requested schedules should be questioned. For the August time period the area consisting of MECS, IMP, FE, DLCO, and PJM (Erie Companies) was scheduled to import on average 7100 MW within a range of 5500 MW (8/11 at 1400 hours) and 8900 (8/14 at 1600 hrs). Refer to Exhibit 1. Over this same period, First Energy exported an average of over 500 MW within a range of over 1300 MW export (8/8 at 1700 hrs) and over 600 MW import (8/14 at 1600 hours). Also during this time, the transactions between systems north of PJM/east of IMO and the Erie Companies was an average of over 800 MW to the Erie Companies within a range of importing from the Erie Companies of nearly 400 MW (8/8 at 1700 hrs) to exporting over 1900 MW (8/14 at 1600 hrs) to the Erie Companies. As can be concluded from these data, and the data in Exhibit 1, the transactions on the afternoon of August 14 were higher than the previous five business days, but not significantly higher. For example, on July 22, 2002 the Erie Companies had scheduled imports ranging between 8700 MW to over 11,700 MW. Of this amount between 4300 MW and 5300 MW was supplied from companies north of PJM and east of IMO. During this same afternoon, First Energy s transactions ranged from exporting 600 MW to importing nearly 900 MW. On August 22, 2002 transfer levels during the afternoon hours were again similar. The Erie Companies imports ranged from over 6,100 MW to over 10,000 MW. During these same hours First Energy s transactions ranged from exporting over 500 MW to 1100 MW. Therefore, based upon TagNet data summarized above, the transaction levels on August 14, were at high levels, but within previously experienced levels. The transactions of the afternoon were also examined from the perspective of the incremental flows caused by these transactions on the First Energy system using linear analysis, similar to methods used by the IDC to determine flowgate impacts. Based upon this analysis of all transactions impacting the First Energy system for August 14 at 1400 hrs there was approximately 2300 MW flowing into First Energy and approximately 1800 MW flowing from First Energy to other neighboring systems. The transaction impacts on First Energy s Cleveland/Akron 345 kv circuits are minimal. For example, the Harding-Chamberlin 345 kv was approximately 50 MW and the Canton Star 345 KV circuit was calculated to be approximately 200 MW. Of these flows into or out of the FE Control Area, a significant portion of these flows was the result of transactions where FE was either a POR or a POD. For example, on August 14 at 1400 hours the net impact of Fe POR/POD transactions resulted in Page 9of 39

57 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE incremental flows of 1300 MW into FE via FE s interconnections with PJM, AEP, DP&L and MECS and approximately 800 MW out of FE on FE s interconnections with DLCO. However, when considering only the transactions for which FE was neither the POR or POD, the FE Control Area experienced flows, less than 150 MW (actually 137 MW) of loop flow was on the South Canton Star 345 kv circuit is attributed to non-fe transactions. Of the critical 345 kv circuits of August 14, the South Canton-Star was the most impacted by the loopflow. Similar comparisons can be drawn for the values contained in Exhibit 2.Flows on the First Energy interface flows and the calculated flows on critical 345 kv circuits for various time periods is summarized in Exhibit 2. Although on August 14, the FE system experienced notable levels of flows caused on transactions. Even when excluding non-fe transactions, flows through the FE System were not insignificant. However, the impacts of these loopflows on the individual critical 345 kv circuits of August 14 are small by any realistic measure. Based upon these observations it can be concluded that the transaction had minimal impact on critical facility loadings of August 14. It is also worthy to note that as a result of the operation and underlying physics of an interconnected transmission system, every portion/individual system of the transmission will experience loopflows caused by the activities of other parties. As a simple illustrative example, a potion of the output of the Beaver Valley (FE generation physically located in DLCO Control Area) will flow across facilities of other systems, including AEP, IMO, PJM and others in addition to the direct connections between the DLCO and FE Systems. At noon on August 14, the FE scheduled interchange was importing 115 MW. At noon, the FE scheduled interchange for 1600 hours was importing 12 MW. At 1600 hours, for that hour, the FE interchange schedule was importing 641 MW. At noon, the five companies (MECS, IMO, FE, DLCO, PJM) was importing nearly 7400 MW and scheduled to import approximately 7800 MW. At 1600 hours these companies were importing nearly 9000 MW. The transaction bias on August 14 was not materially different from the previous days of that week. Nor were the transaction patterns significantly different from other days of the summer 2003 or the previous summer. On August 14, the transaction magnitudes were high, but not unusually high as compared to other days of that week, other days during summer 2003 or summer (3) Is TLR effective as a transmission loading relief tool if there are mismatches between schedules and tags of the magnitude seen on August 14? What can be done to make it more effective? What alternatives should be considered in the future? With respect to mismatches between schedules and tags, TLR will not identify the full set of transactions to be curtailed. But regardless of the ultimate effectiveness of the TLR procedure, the TLR procedure takes a finite period of time to implement up to well over 1 hour depending upon prevailing system conditions. Since the results of implementing a Page 10of 39

58 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE TLR procedure is not immediate, the TLR procedure cannot be considered an exclusive corrective tool to manage or mitigate system emergencies it may take too long! Therefore, to respond to fast and detrimental changes on the transmission system, operators must have in place plans and practiced procedures to quickly respond to multiple outages that could jeopardize the reliability of the transmission system within the time period required to arrest a potential cataclysmic event on the system. Nothing can replace diligent operators monitoring the conditions on the system and manage evolving conditions as standard practice before events devolve into system emergencies. However, when multiple events are occurring on the transmission system and typical operating practices are not able to adequately maintain reliability, transmission operators must be knowledgeable, trained and authorized to declare a system emergency and take extra-ordinary measures in cooperation with other operators and Reliability Coordinators to prevent a calamity of the transmission system. (4) Although the interim report noted the system was within defined non-simultaneous transfer limits, what simultaneous transfer limits are appropriate and would the system have been within those limits on August 14? What simultaneous transfer limits should be developed and by what process in the future? The interim report noted the system was within defined non-simultaneous transfer limits with respect to individual control areas. However, the inter-regional transfers such as south-to-north and west-to-east were within the simultaneous transfer boundaries established by the interregional study group. One set of inter-dependent system conditions that could have an adverse impact on northeast Ohio for the 2003 summer was identified by AEP and discussed with FE. Both parties acknowledged the issues and agreed to implement mitigation measures in facility and/or operating limit changes as per documents (xx, yy and zz). In the future, if there are simultaneous (inter-dependent) transfer limits within several control areas or within regions identified by a control area/region (e.g. contingencies in one control area/region adversely impacting the neighboring control area/region), then the results should be reviewed as part of a formal Regional process to enable joint development of inter-dependent operating limits. Voltage and Reactive Management (1) Summarize the results of analysis of the voltage conditions on August 14 showing PV-PQ curves for FE and the surrounding region. Did the FE system have adequate capacity and reserves; static and dynamic reactive reserves; and delivery capability? Was the FE system near voltage collapse prior to the outage (pre- and post East Lake 5; Chamberlin Harding; Hanna-Juniper; and Star-S. Canton)? (Monitor redundancy with Chapter IV. B) Page 11of 39

59 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE Overall, voltages in the Cleveland and Akron areas were the lowest in the ECAR east and PJM west regions on August 14 th. Voltages were observed to be higher in neighboring areas in the FE control area. Bus voltages bordering FE in adjacent controls were observed to be higher still. Figure XX and YY (illustrative, awaiting real-time data) shows the voltage profiles across the ECAR and PJM region. Because the sagging voltages occurred in First Energy, with the worst sags in the Cleveland / Akron areas, the following discussion and analysis concentrates on this region. East to West FE Voltage Profile South - North FE Voltage Profile kv :05 3:06 3:32 3:41 3:45 3:59 4:05 kv :05 3:06 3:32 3:41 3:45 3:59 4: HOMER CY WAYNE ERIE W 02AT 02PERRY 02EASTLK 02INLAND 02HARDIN 02JUNIPE 02AVON 02BEAVER 02DAV-BE 02BAY SH 19MON MANSFD 15BVRVAL 02SAMMIS 01WYLIER 05TIDD 05CANTNC 05SCANTO 02STAR 02CHAMBR 02HANNA 02HGHLND 02JUNIPE 02EASTLK 02AVON As designated by the orange shading, generators along Lake Erie around Cleveland and Toledo ran out of reactive power reserves during the 15:05 16:09 time period. Further evaluation of real-time data suggests that for the most part, units just outside of the Cleveland-Akron Cleveland area maintain their reactive reserves throughout this same time period. Of notable interest is the 345 kv connected cluster of plants southeast of Akron: Sammis, Beaver Valley, and Mansfield. As a group, this set of units maintains reactive reserves throughout the event. Therefore, the reactive analysis concentrated on the region shown in the Figure below essentially the Akron / Cleveland load area. The total MW flow through the critical cut-set of 345, 138, and 69 kv lines into the Akron / Cleveland region, shown by the thick solid red line in the Figure, was monitored for PV analysis. Page 12of 39

60 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE P-V analysis stresses a power system by increasing load or transfers in a region. The first set of P-V analysis for south-to-north transfers was constructed by varying transfers from south of TVA to the IMO. Figures 1 and 2 (illustrative, awaiting final runs) below show these P-V curves for the 15:05 and 15:32 cases. The X axis shows the incremental transfers, with 0 representing the NERC base case transfer. The red line, and correspondingly the left Y axis, represents the MWs flowing into the Cleveland-Akron area (reference the thick solid red line in Slide 2). The yellow line plots the Star 345 kv bus voltage. The plot shows that for reasonable variation on transfers, the impact on the Star bus voltage is minimal. This is due to the fact that less than 15% of the south-tonorth transfer flows through the critical cut-set of lines into the Cleveland-Akron area (reference the MW flow into Cleveland-Akron area plotted on the PV curve) South to North Transfers Flow (MW) CEI-Akron MW Flow Qlim Star 345kV Voltage (pu) N-S Incremental Transfers (MW) South to North Transfers CEI-Akron MW Flow Qlim Star 345kV Flow (MW) N-S Incremental Transfers (MW) The P-V curves for west-to-east transfers shown in Figures 3 and 4 (not shown here but will be provided) were constructed by varying transfers from west of FE (Cinergy and Commonwealth Edison) to east of FE (New England). Similar to the behavior of the south-to-north transfers, regardless of the base case used, the impact of the west-to-east Page 13of 39

61 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE transfers on the Star bus voltage is minimal. This is due to the fact that less than 10% of the west-to-east transfer flows through the critical cut-set of lines into the Cleveland- Akron area (reference the MW flow into Cleveland-Akron area plotted on the PV curve). Figures 3 and 4 (not shown but will be provided later) The P-V curves for FE load increases as shown in Figures YY and ZZ (illustrative, awaiting final runs) were constructed by increasing the load in the Cleveland-Akron area, and increasing the generation from the south (TVA). The X axis on these curves show incremental transfers into FE to supply the increasing loads (at constant power factor). It can be observed that close to 70% of the MW transfer from TVA to FE flows through the critical cut-set of lines into the Cleveland- Akron area. A significant voltage decline from xx kv to yy kv was observed at the Star 345 kv bus FE Load Increase Flow (MW) CEI-Akron MW Flow Qlim Star 345 kv FE Load Increase (MW) Voltage (pu) FE Load Increase Flow (MW) CEI-Akron MW Flow Qlim Star 345 kv FE Load Increase (MW) Voltage (pu) Q-V analysis is another technique to assess the voltage stability of a transmission system. Reactive power and subsequently the bus voltage are varied on a selected bus by use of a fictitious Var source with no reactive limits. The nose of the curve where Q/ V approaching zero is the theoretical voltage instability point below which voltage collapse is expected. The right side of the curve with a positive Q/ V is the stable operating regions. The distance between the nose-voltage (also known as the critical voltage) to the initial operating voltage provides a good indication of the voltage margin requirement to ensure the operating voltage is secure. Similarly, the Var reserve on the selected bus also Page 14of 39

62 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE provides a good indication of the status of the power system from the dynamic Var reserve viewpoint, although it cannot be directly correlated with a particular generator or a group of generators nearby. By repeating the Q-V analysis at different key buses, very insightful information for a given system condition can be obtained in terms of the critical bus of the system, the highest critical voltage, voltage margin required to ensure the post-contingency voltage is above the critical voltage and an approximate amount of Var reserve to avoid voltage collapse. It is intuitive that all of the above indicators are inter-related. A series of V-Q curves for the four representative system states (e.g. aa after EL5, bb after Harding-Chamber, cc and dd) for the selective buses in the Cleveland-Akron area and at interite buses in AEP, MECS and PJM are shown in the following Figures 1-6? (illustrative only, awaiting final runs.) The results show that the critical voltages at Harding, Avon, Fox and Star increase rapidly as the contingencies evolves with rapidly decreasing var reserves at these critical buses. At the xx bus, the critical voltage corresponding to xx:yy EST (prior to loss of Sammis-Star) exceeds 0.?? pu. This is a strong indicator that the FE system was near voltage collapse after the loss of Sammis- Star. For the buses in the surrounding areas, a similar trend is displayed but with great resiliency. For example, the xx bus in AEP shows a critical voltage of 0.yy pu at xx:yy EST which is significantly below the actual operating voltage. Harding 345kV Hanna-Juniper Star-S Canton Canton Central-Tidd West Akron 138kV Lines Dale-W Canton Set Qmax = Qgen for Lakeshore and Ashtabula in NERC cases 400 Critical Voltage approx 0.96 pu MVAR Voltage PU Jennings 138kV Hanna-Juniper Star-S Canton Canton Central-Tidd West Akron 138kV Lines Dale-W Canton Page 15of 39 Critical Voltage > 0.96 pu Set Qmax = Qgen for Lakeshore and Ashtabula in NERC cases MVAR

63 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE (2) Review the results of reactive power planning studies for the summer of 2003 to determine the adequacy of reactive supply and reserves including the balance between static and dynamic resources. Assess the sufficiency of sensitivity studies carried out to allow for the uncertainty in reactive power demand and the unavailability of static and dynamic reactive supply resources. Identify the status of plans to provide additional reactive power support. Review the FE engineering justifications for two reactive projects recently completed the transformer at East Lake and the Avon capacitors to determine FE s assumptions regarding system reactive power needs. First Energy noted that to some degree First Energy depends on ECAR regional assessments to identify anticipated reactive requirements and recommended corrective action. 10 A Review of ECAR s 2003 Summer Assessment of Transmission System Performance 03 TSPP 3 dated May 2003 shows the following: 1) The study included a section devoted to Voltage Analysis. The intention of the section was to provide study results through AC analysis which examine the voltage performance of those portions of the ECAR transmission network where potential voltage limitations have been identified through past operating experience or other ECAR assessments. Noticeably absent from the analysis was an assessment of the voltage constrained import capability of the Cleveland region 11 and 13. 2) In the Individual Company Assessment section, First Energy only identified potential overloads for the loss of both Star 345/138 kv transformers and identified operating solutions to mitigate that risk. First Energy did qualify that the assessment was at transfer levels studied, but did not mention any expected voltage limitations. Page 16of 39

64 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE ECAR s response (reference xx)to SPDST s question on voltage analysis in the region is shown below: The ECAR TSPP seasonally performs assessments on its transmission systems and has included voltage analysis (P-V curves) as part of these assessments since These assessments are as comprehensive as time and resources permit, but are not all encompassing, as it is not possible to study all combinations of the possible contingencies that could occur. To make the most of the limited resources, the TSPP expects the ECAR companies to be mainly responsible for providing scenarios that they feel need to be studied in the ECAR seasonal assessments, with the TSPP selecting the worst-case conditions to be included from the submitted voltage scenarios. From 1999 through 2002, five different voltage scenarios covering Northern Ohio have been examined. In the 2001 Summer assessment (02-TSPP-3 report), the Allen Junction voltage in Northern Ohio was monitored while studying south-to-north transfers, which were thought to be the most severe for that area. That same condition was again studied during the 2002 Summer assessment. The P-V analysis done for 2002 Summer, as compared to 2001 Summer, showed great improvement due to the addition of the Lemoyne (IPP) generating facility in the area. Based on this improvement, the TSPP agreed that this scenario could be removed from further consideration, thus redirecting study resources to other more pressing areas for the 2003 Summer assessment. The TSPP is presently reviewing the procedures used to select these voltage scenarios, with changes expected to be made for 2004 Summer. Copies of the TSPP reports noted above can be provided for reference, if needed. Because regional assessments are expected to concentrate more on the security of the regional transmission system, it is good utility practice for a control area to conduct transmission studies that are more detailed than regional assessments. First Energy provided a summary of results of their 2003 Summer Assessment of the expected summer peak load conditions 12. The intention of the study was to provide an indication of how FE system might perform under peak conditions including guidance for FE System Control Center on how to apply operating solutions for various overload and voltage violations. The study did indicate contingencies and potential operating solutions for some local voltage level violations but it did not include an assessment of the voltage constrained import capability of the Cleveland area, which a) with the assumption used in past FE studies of losing two critical generators plus another critical single contingency, would have arguably determined the extent that the available dynamic and static reserves could have facilitated imports into in the region and b) would have been especially relevant with the long term outage of the Davis Bessie plant. Our observation is that it is not practical and indeed unrealistic for any control area to rely on a regional study group to carry out an adequate voltage stability assessment with a wide range of system conditions such as in-service status of generators, lines and capacitors, firm and non-firm transactions and more importantly, a reasonably detailed Page 17of 39

65 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE representation of the sub-transmission network. Voltage assessment as an integral part of deriving system limits must be carried out in-house including a comprehensive set of system conditions with a series of n-1-1 conditions. For NERC report, summarize meetings between AEP and FE on South Canton problem (will be included later.) FE s justification and its study reports for the two reactive projects recently completed: 345/138 kv transformer at East Lake and the Avon Capacitors, are not available. The following observations can be made: The Avon capacitors (one of the two projects) would be very beneficial to the reactive power needs, since due its proximity to the Cleveland load pocket. However, their in-service status on August 14 is not clear as reflected by the FE Control Room conversation: 13:41 Mike Ball to Cliff: I can get those Avon caps back on by 2:00 or 2:30 there was nothing specifically wrong when they took them off. It was in service when they took them off. They were doing some routine checks I ll try to get a crew to go over to Fox and start checking that bank out. Cliff: everything is going down; we lost a unit at EL, and now we have that transformer out After the loss of East Lake 5 and transformer #62 at East Lake (second of two recently completed projects), power flow simulations comparing two scenarios: i) with one 345/138 kv transformer i/s, and ii) with both 345/138 kv transformers i/s, show transformer #62 provide an equivalent of 31 MVar to the Cleveland syustem. Total MW and MVar flows into the transformer(s) at 345 kv Total MW and MVAR flows out the transformer(s) at 138 kv Reactive Losses in the 345/138 kv transformer(s) Difference in reactive losses in the transformer(s) Reactive Losses in the Cleveland Zone Difference in reactive losses in the Cleveland Zone Scenario 1 345/138 kv Transformer #62 at East Lake forced out Scenario 2 Both 345/138 kv Transformers assumed i/s 503 MW, 66 MVar 621 MW, 86 MVar (328 MW, 44 MVar) #61 (293 MW, 42 MVar) # MW, 15 MVar 620 MW, 45 MVar (327 MW, 22 MVar) #61 (293 MW, 23 MVar) #62 51 MVar 41 MVar 10 MVar 1144 MVar 1113 MVar 31 MVar Page 18of 39

66 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE It is noted that at the East Lake substation, units 1 to 4 are incorporated at the 138 kv level while units 5 (600 MW) is incorporated at 345 kv. While the values in the comparison table should not be taken as absolute values, the difference between the two scenarios does provide useful information on the effectiveness of the transformer project, in relation to the capabilities of East Lake 4 and East Lake 5 units that were not available on August 14. The following table provides a general comparison of the two recently completed reactive supply projects: Avon Capacitors (assumed 2 x 50 MVar) 345/138 kv transformer at East Lake East Lake Unit 4 (240 MW) assumed pf at 0.9 lagging East Lake Unit 5 (600 MW) assumed pf at 0.9 lagging 100 MVar 31 MVar 120 MVar 300 MVar (3) Review the reactive power management strategies and operations measures in place to maintain an acceptable voltage profile. Determine if any of these were deployed on August 14. Are the FE criteria for voltage limits adequate? Were voltage and reactive effectively coordinated among systems prior to and on August 14? Was FE reactive reserve planning adequate? Assess the appropriateness of the voltage schedules and profiles in the affected systems and the deployment of reactive resources including the balance of static and dynamic reserves for the load levels and transactions over the period noon to 1600 hours EDT. Compare voltage schedules/profiles of August 14 with previous days and if atypical, determine conditions that made it so. Is reactive reserve planning adequate in general in the region and industry? What recommendations are needed? Based on the information from FE s testimony and FE s control room conversation on operation measures to maintain an acceptable voltage profile, FE did attempt to deploy them on August 14, albeit resulting in aggravating the voltage profile. For example, requesting the East Lake 5 operator to increase the Var output without instructing him to reduce the MW output is not consistent with good utility practice especially the MW output was already at higher than the rated values. Recalling the capacitors that were removed from active operation for routine maintenance was in the right direction but it did not appear to have an extremely high level of urgenecy. FE s criteria on transmission voltages are that normal voltage can range from 0.90 pu to 1.05 pu of nominal during on peak and off-peak conditions. The minimum contingency Page 19of 39

67 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE voltage is 0.90 pu of nominal. The maximum pre-to-post contingency voltage change is 0.05 pu for 345 kv transmission substations, and 0.10 pu for the remaining transmission substations. The maximum allowable peak to off peak is the same as that for the pre-topost contingency voltage changes. Given that critical bus voltages shown in the Q-V analysis under certain operating conditions could be 90% or higher in the Cleveland area, a minimum voltage requirement of 90% pre- and contingencies without comprehensive voltage analyses to ensure the operating voltages are above the voltage collapse points under a range of expected and extreme system conditions is not considered as good utility practice. Therefore, FE s voltage criteria are not deemed to be adequate to ensure system security. In fact, it provides a wrong signal to the system operators that operating at slightly above 0.9 pu or 0.95 pu under peak load conditions would be acceptable. Even by FE s own admission, the voltage level criterion for operating the transmission system is low. Testimony by First Energy stated that the minimum acceptable operating voltage level is 92%, though it is also stated that FE would typically not operate below 90% 4. However, as part of FE s Reliability Coordinator s (MISO) process, the MISO RC or Operation Engineer may contact the control area operator(s) whenever the voltages are outside generally accepted levels (95-105%) and verify the operators have taken steps (adjustments to capacitors, reactors, and generator voltage set points) to try to get the voltages back within the range if possible 4. In comparison, AEP s practice is try to maintain an operational voltage level of 95% 2. AEP s voltage criteria is 0.96 to 1.02 on 765kV and 0.95 to 1.05 for all other voltage levels. PJM operates its 500 kv system in the range of 1.0 to 1.1 pu with an emergency low voltage at 0.97 at which point, load shedding is included as part of the operating measure to avoid voltage collapse. Although the entire FE system may have appeared to be a net VAR exporter during certain periods on August 14 this should not be construed as having sufficient VAR to meet the demands in the FE system. (Need an explanation on why it was a Net var exporter.) Reactive sufficiency should not be evaluated in terms of commercial/ownership boundaries. Reactive power must be made available to locations which require it to support load and bulk power transfers. (Not too sure this paragraph is useful in this context.) In the case of the Cleveland-Akron area which relies on heavy power importer to meet the metropolitan load, adequate reactive supply and reserves in the area are essential. Given that the voltages were lowest in the Cleveland-Akron area, the reactive power capabilities at the East Lake units were depleted prior to xx:yy, and the generators at Sammis, Beaver Valley and Mansfield still had some reactive reserves, this indicates that the installation and utilization of reactive sources to meet the Cleveland-Akron area was sub-optimal. Combined with the facts that a number of capacitors was removed from service on August 14 for routine inspection with the knowledge that forecast peak Page 20of 39

68 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE demand would be higher than the previous days and East Lake 4 unavailable, the evidence suggests that reactive reserve planning was not adequate in the long term planning horizon nor in the near term operation planning horizon. No testimony or documentation that directly links First Energy s policy of assessing an adequate mix of static and dynamic Vars was identified. However, testimony by First Energy stated that they do not operate a reactive reserve directly, but indirectly in terms of schedule 6. The term schedule used in the testimony is assumed to be a reference to generator voltage schedules and/or transmission bus voltage limits. It should be noted that the First Energy testimony shows that Mr. Morgan of First Energy was under the incorrect assumption that FE units along the lake still had dynamic reactive reserves therefore, he was not in a position to assess if the Reactive Reserve in the region were adequate. In fact, as seen in Figure X below, real-time data showed that as the events of the 15:00 hour unfolded, Reactive Reserves in the Cleveland load area were exhausted. The depletion of Reactive Reserves was a local issue FE units to the Southeast maintained dynamic reactive reserves throughout the event. And as seen in Figures Y and Z below, entities interconnected to First Energy had plenty of reactive reserves thus these entities were not having to lean on First Energy for voltage support. Based on the LF, the Cleveland load is about 90% statically compensated (including line charging). The ratio of dynamic to static compensation is about 70% in Cleveland. {Compare this to other FE sub areas and PJM, AEP & MECS) The poor voltage profile in the region, perhaps caused by the depletion of reactive reserves from some units resulted in the need for EL5 to boost reactive output. The excessive over-excitation lead to the unit tripping. The 600MW loss of local Cleveland generation would significantly increase the loadings of circuits importing power. The loss of the associated reactive power and the further depression of voltages would further aggravate the circuit loading, including increased Mvar losses and decreased Mvar reserves. However, it is important to realize that while the depletion of reactive reserves contributed to low voltages in the region, the primary driver of the conditions leading up to and including the tripping of the Sammis Star 345 kv line (which initiated the uncontrolled cascading portion of the blackout) was line tripping. Therefore, while the depletion of Reactive Reserves in the region should have been a glaring clue that the system was under severe stress and ultimately in an insecure state (both thermally and voltage level), the resulting low voltages were an effect instead of a cause. While information regarding FE s generator voltage schedules were reviewed, an assessment to determine the adequacy of those schedules without a detailed study using appropriate software tools (such as Optimal Power Flow) is not possible. A cursory Page 21of 39

69 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE review of the generator voltage schedules shows they are within a normal band of scheduled voltages ( pu? to be confirmed) commonly used by most utilities. Refer to the overall comparison with AEP, MECS, PJM on loads, transactions, voltage profiles in the south-to-north direction, in the west-to-east direction, reactive power reserves, and generator Var reserves with the following illustrative plots. (More plots under preparation. Cleveland / Akron Mvar Reserves (Realtime data used for Selected Questionable NERC Basecase Generator Mvar values) Generator Mvar Reserves Generator Mvar Production Line Charging Bus Shunts AEP (ECAR) Transmission System Reactive Resource Summary 8/14/2003 MVAR 12,000 11,500 11,000 10,500 10,000 9,500 9,000 8,500 8,000 7,500 7,000 6,500 6,000 5,500 5,000 4,500 4,000 3,500 3,000 2,500 2,000 1,500 1, Effective Reactive Resource Reseves 12,233 MVAR Effective Reserves Margin 3,050 MVAR Effective Reactive Resource 00:00 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 09:00 10:00 11:00 12:00 Page 22of 39 13:00 Time (EST) 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 Confidential DOE Investigation Resultant Line Capacitance Fixed Reactors Switchable Reactors Capacitors Condensers Total Capability

70 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE PJM VAR PRODUCING SOURCES, IN RESERVE MEGAVARS :00 15:45 15:30 15:15 15:00 14:45 14:30 14:15 14:00 13:45 13:30 13:15 13:00 12:45 12:30 12:15 12:00 August 14th, 2003 LAG RESERVES CAPACITOR RESERVE Figures xx, yy, zz etc show the voltage profiles in the footprints of FE, MECS, AEP and PJM over the August period. Both voltage profiles, in the south-to-north direction and the west-to-east direction display a V shape with the critical buses in the Cleveland- Akron area forming a cluster at the V-trough. Voltages in First Energy declined progressively from Monday August 11 th to August 14 th. P-V analysis has shown that the effect of transactions on First Energy s voltage profile on August 14 th was minimal. The poor voltage profile in the region on August 14th, perhaps caused in part by the depletion of reactive reserves from some units, resulted in operators calling on the Eastlake 5 unit to boost reactive output. The excessive over-excitation lead to the unit tripping. The 600MW loss of local Cleveland generation would significantly increase the loadings of circuits importing power into the region. The loss of the associated reactive power and the further depression of voltages would further aggravate the circuit loading, including increased Mvar losses and decreased Mvar reserves. Another factor that likely put pressure on the voltage profile in the Akron / Cleveland area was the increase in temperatures as the week progressed. As seen in the figure XX 17 below, peak temperatures in the Akron / Cleveland area steadily increased from approximately 78 degrees Fahrenheit on Monday August 11 th to approximately 87 degrees Fahrenheit on August 14 th. This would result in increased load which would increase flows on First Energy s internal transmission system, thus increasing Page 23of 39

71 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE transmission line series reactive losses. Observation of load data showed that compared to Monday August 11 th, First Energy control area load on August 14 th increased approximately 21.4% - a little higher than the ECAR average of approximately 19.3% 18. Additionally, the increased loading would predominately be in the form of air conditioning load, which has an unfavorable power factor. Both of the aforementioned phenomena s would result in additional reactive losses, which would degrade voltage profiles and tend to deplete dynamic reactive reserves from the remaining units in the Akron / Cleveland region. Degree F Akron Temperatures, Aug 1-14 The voltage profile plots show that the buses voltages in the neighboring control areas shows minimum variation as the loads increased significantly throughout the week due to increasing temperatures. Combined with the reactive power reserves in AEP, PJM and MECS, it can be concluded that reactive reserve planning is generally adequate in the surrounding areas to meet their respective needs. See overall SPDST recommendation list reactive power management. (4) Is the lack of standards on voltage and power factor (for generators and loads) a problem? Was it a contributing factor to the August 14 outage? What recommendations are appropriate for voltage/reactive coordination and standards on voltage and/or power factor? NERC has a number of planning standards and guides related to voltage and reactive planning that provide comprehensive requirements for designing and operating transmission systems with adequate reactive support, yet providing flexibility for the different characteristics of each transmission system. Planning Standard I.D.S1 (System Adequacy and Security, Voltage Support and Reactive Power) requires that Reactive Page 24of 39

72 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE power resources, with a balance between static and dynamic characteristics, shall be planned and distributed throughout the interconnected transmission systems to ensure system performance as defined in Categories A, B, and C of Table I in the I.A. Standards on Transmission Systems. This standard is measured by the requirement that entities responsible for the reliability of the interconnected transmission conduct assessments at least every five years, or as required by changes in system conditions. There are a number of guides associated with this standard, including G2, which recommends that Distribution entities and customers connected directly to the transmission systems should plan and design their systems to operate at close to unity power factor to minimize the reactive power burden on the transmission systems. LYT added comment to John Twitchell s strawman Because NERC Planning Standards addresses the power factor of distribution system loads as a Guide, rigorous conformance would have been followed by more entities if this had been a Standard. NERC Planning Standard II.B.S1 (System Modeling Data Requirements, Generation Equipment) requires that NERC Regions establish procedures for generation equipment data verification and testing. This testing includes the real and reactive capabilities of generating units within the Region. NERC Planning Standard III.C.S1 (System Protection and Control, Generation Control and Protection) requires that All synchronous generators connected to the interconnected transmission systems shall be operated with their excitation system in the automatic voltage control mode unless approved otherwise by the transmission system operator. S2 of this standard also requires that Generators shall maintain a network voltage or reactive power output as required by the transmission system operator within the reactive capability of the units. As discussed earlier, the main contributing factor to the August 14 event is the changes in network topology due to series of line outages. Lack of standards on voltage and power factor for generator and loads does not appear to be a contributing factor, in spite of the fact that higher operating voltages and more dynamic Var reserves would also result in a higher levels of resiliency or robustness of the power system. However, one could speculate that additional standards of this type, or more rigorous implementation of existing standards and guides, would have resulted in a higher voltage profile on the afternoon of August 14 th. A higher voltage profile could have negated the need for additional Var support out of Eastlake 5, and thus loss of this critical unit could have theoretically been avoided. Higher voltage profiles would have also marginally reduced loadings on transmission lines, perhaps adding some more time for the condition to be properly recognized and acted upon by the First Energy operators. It would not be practical to quantify the likelihood that higher voltage profiles as a result of additional load power factor and generator standards would have actually mitigated or lessened the reach of the August 14 th outage. Page 25of 39

73 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE In the Eastern Interconnection, Mvar capability requirements for new generators interconnecting to the transmission system are typically defined by the ISO s or RTO s. There is no known direct guidance from Regional Reliability Councils, NERC or FERC. New generators interconnecting to AEP s system are required in their Interconnection Agreements to have a power factor rating no greater than 0.90 at the generator terminals. GSU tap settings and voltage or reactive schedules are usually recommended to maximize the availability of overexcited reactive power. Information from other transmission owners near the effected region have not yet been received. A good utility practice that has been successful is to require new generators to hae the design and operational capability to maintain an effective power factor at the transmission bus of the interconnecting generator. As an example, the Reactive Policy for Generation Facilities Connecting to the Southern Company Transmission System outlines the amount of reactive power capabilities required from generating facilities connected to the Southern Company transmission system. Specifically, Southern s Reactive Policy requires all new generators to have the design and operating capability to supply 0.33 dynamic Mvars / MW at the highside of the generator step up transformer at the point of interconnection to the transmission system, and absorb 0.23 dynamic Mvars / MW. Enacting proper regional or national standards regarding minimally acceptable power factors for generators and loads could enhance transmission reliability. However, care would have to be taken to ensure that local transmission entities maintain authority to enforce more stringent standards to meet their individual reliability requirements. (5) Review the validity of reactive power demand and reactive capability of generating units used to assess system performance. Determine how these data are validated. Review the assumptions regarding load power factor for system studies to determine if these assumptions are adequate. FE has not responded to the SPDST data request, and no substantial information that could help answer this question could be uncovered. Respondents to the SPDST data request reported that loads are determined and verified through metering of load points. It is likely that FE follows this usual utility practice Though FE did not respond to the SPDST data request, as members of ECAR, they are expected to abide by the requirements set forth in ECAR Document 4 which references NERC Planning Standards, II B, "Generation Equipment," approved September 16, It should also be noted that this document references NERC Planning Standard II B, which is a part of Phase 4 and thus has not been field tested. As such, ECAR s compliance program does not include review of adherence to ECAR Document 4. Page 26of 39

74 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE ECAR Document 4 requires that all generators over 70 MW should be tested for reactive capability at least once every 5 years. This includes the testing of both overexcited and under-excited reactive capability. This philosophy is adopted by entities in ECAR which have thus far responded to the SPDST data request: AEP and METC-Consumer Power. It should be noted that this ECAR requirement is applicable to all generating units in ECAR, which would include IPP plants interconnected to the systems within ECAR. ECAR Document 4 also states that all voltage regulators on units with a nominal capacity greater than 70 MW shall be open-loop tested for functionality and proper operation of all controls and protective relays. More frequent testing may be done if the Generator Owner so chooses. Testing with the unit out of service using artificial inputs, (i.e., terminal voltage, phase current and field current such as from a voltage regulator test set), is acceptable. In addition, the operation of all limiters and coordination of all limiters and protective devices shall be verified with the voltage regulator connected open loop. Finally, all limiters, where possible, shall be verified while in closed loop operation for stability of operation and for proper pickup points. While performing an the open step in voltage test with the generator rotor spinning but not connected to the transmission system would be preferable to obtain a positive indication that all the excitation system inputs are indeed functioning properly, the requirements and minimal testing schedule as set forth by ECAR Document 4 easily meets the good utility practice standard. In conclusion, if FE is in compliance with the requirements set forth in ECAR Document 4, FE s assessment regarding the availability of dynamic reactive resources for the purpose of assessing system performance has followed good utility practice. However, since ECAR s compliance program does not include review of adherence to ECAR Documents 4, there is no independent audit process which verifies that any entity in ECAR (including FE) satisfies the practices set forth in the document. Planning studies were planning studies in effect for August 14 accurate and effective for the conditions seen on August 14? Were generator and load models adequate? Were regional studies sufficiently coordinated among regions? What improvements should be made in the areas of system studies and coordination of studies on a regional basis? FE s 2003 Summer Assessment is the only document available to the NERC Blackout Investigation. Based on our review of this document (which does not includes a detailed description of the FE system conditions, imports or exports), it is concluded that the studies in effect for August 14 are not comprehensive or effective for the conditions seen on August 14 as follows: It likely assumes all the available generation (except Davis Besse and Perry) in service. On August 14, Davis Besse remained o/s, whereas it is not clear whether Page 27of 39

75 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE the Perry unit (1245 MW) has the same effect as over 1,300 MW of generation (East Lake 4, East Lake 5, Sammis 3 and several smaller generators in and near the Cleveland-Akron area) that were out of service. It likely assumes all the installed capacitors were in service. On August 14, a number of capacitors that were in active operation were removed from service for routine inspection. There was no specific mention on the assumed levels of import into Cleveland- Akron area compared to the study report 1998 Summer Import Capability of CEI. The latter shows a number of voltage violations including non-convergence of post-fault power flows and the corresponding restrictions on imports into Cleveland (see table in section xx.). It identified thermal restrictions largely on the 138 kv facilities with no voltage violations on the 345 kv. Unless major transmission reinforcements or new generation installed in the Cleveland-Akron area, the voltage performance of the two studies does not appear to be consistent. On August 14, the Cleveland-Akron area was experiencing voltage problems around noon with East Lake 5 still in service while the FE load of about 11,400 MW was lower than the studied level of 13,209 MW. As indicated later in this report, external transactions not involving FE as sink and source has very little impact on the voltage profile in the Cleveland-Akron area. The modeling of generators and loads is adequate from the entities responding to the NERC data requests. FE has not responded to the data request. Regional studies appear to be sufficiently coordinated among the regions. The following coordination groups are currently in existence that affects the ECAR region MAAC/ECAR/NPCC (MEN), VACAR/ECAR/MAAC (VEM) and MAIN/ECAR/TVA (MET). Each of the regions has representation on these coordination groups, which provide a mechanism to consider input from each of the respective regions for the interregional seasonal and future system studies. In addition, these representatives also provide a link to the regional studies so that concerns from other regions are incorporated into the regional studies. As an example, the ECAR seasonal and future system studies include contingencies of facilities in adjacent regions in their studies and monitor facilities in adjacent regions. In addition to the regional and inter-regional studies, there are studies performed by the RTO s. In June of 2003, the Coordinated Summer Assessment (CSAT) study was completed. This was a study of the 2003 summer period and was a joint study involving representatives from MISO, PJM, AECI, ENTERGY, SOCO, SPP and TVA. For improvements on coordination, see recommendation list. Page 28of 39

76 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE (1) Review the criteria used for identifying deficiencies in the planning timeframe and in setting system operating limits and compare with NERC and regional criteria. Examine where exemptions have been applied by the entities and approved by NERC or Regional Councils. AEP, MECS, MISO and PJM s criteria are consistent with those of NERC and regional councils and all have more stringent voltage criteria than FE. AEP Transmission Planning Criteria (Steady State System Performance PJM Voltage Limits: Transmission System Minimum Bus Voltage Condition EHV 138 kv All facilities in service 95% 95% 1 facility o/s 90% 92% 2 facilities o/s 90% 92% 500 kv 345 kv 138 kv High 550 (1.1) 362 (1.05) 242 (1.05) Normal Low 500 (1.0) 328 (0.95) 219 (0.95) Emergency Low* 485 (0.97) 317 (0.92) 212 (0.92) Load Dump* 475 (0.95) 310 (0.90) 207 (0.9) * Load shed if voltage continues to decay or potential for voltage collapse MISO s Planning Criteria: , Operating: per local CA or transmission providers. FE s existing voltage criteria of pu pre- and post-contingency are not adequate in identifying deficiencies or in setting secure operating limits. The 0.9 pu precontingency as an acceptable operating voltage gives a false sense of system security to the FE system operators especially under peak load and heavy import conditions in which a secure operating voltage should have been within the high range of 1 to 1.05 pu. Defining such a large range of acceptable normal operating voltage without a subset of say 1 to 1.05 pu under more onerous system conditions such as peak load with poor power factors, heavy imports, outages of generators, lines and capacitors etc. is not good utility practice, as evidenced by all neighboring utilities using the range of pu as a normal operating range. In addition, FE s low range would negatively influence the mindset of: Page 29of 39

77 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE System Operators that even under more onerous system conditions there is no urgency to invoke immediate operating measures to secure the system. Engineers engaged in planning and operation planning studies to assess the system more rigorously in sensitivity studies and system impact on the evolving n-1 system states. Since the system appears to be so robust in relation to the 0.9 pu nominal, there is no need to identify new facilities or operating measures to address the system inadequacies. (Note: In the Interim Report, FE s minimum acceptable voltage is 92% from Steve Morgan s testimony, whereas in FE s Form 715 filing to FERC, it is 0.9 pu) Not aware of any exemptions for ECAR members. (Will check with ECAR and NERC later.) (2) Assess the methods and assumptions used in carrying out power system studies in all time frames and compare with good utility practice. In particular evaluate the approaches used in determining the more restrictive of thermal, voltage stability and transient stability limits. Also evaluate the extent to which studies are coordinated with adjacent and neighboring systems Detailed responses from AEP, PJM, MECS and MISO show that they are all consistent with good utility practice and all three limits: transient, voltage and thermal are evaluated to ensue the most restrictive one is the basis for operating limit. Based on the only available document from FE in this area (2003 Summer Assessment), FE s study scope, criteria and identification of operating measures are not adequate. The depth of coverage and criteria are not consistent with those of the former Centerior Energy which developed a set of stringent system criteria to comply with NERC and ECAR criteria as contained in Szwed s direct testimony to FERC in All ECAR companies must adhere to ECAR Document No. 6, which states that companies must coordinate facility connection studies such as IPP proposed connections with all affected companies. Response from companies: AEP AEP met with FE early in 2003 to discuss the concerns that AEP had regarding the loading of AEP s South Canton 765/345 kv transformer. AEP shared results of their internal studies indicating the potential for voltage collapse in northeast Ohio under conditions of high south to north transfers along with multiple contingencies. [AEP file: Summary of Discussions and Interactions with First Energy.pdf ]. AEP participates in regional (ECAR) and inter-regional studies such as MEN/VEM/VAST/MET [AEP file: AEP Response NERC SPDST Data Request doc ]. AEP also coordinates IPP connection Page 30of 39

78 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE studies with impacted neighbors, coordinates transmission interconnection studies, MISO studies and coordinates studies to address concerns at interfaces such as the South Canton study. [AEP file: TP Response to NERC data request of doc ]. FE - FE participates in regional (ECAR) and inter-regional studies such as MEN studies. FE also coordinates IPP connection studies with impacted neighbors. [FE file: FE Oct 1 Response Letter.pdf ]. FE did not make specific reference to any interface studies with its neighbors. ITC - ITC participates in regional (ECAR) and inter-regional studies such as MEN studies. ITC also participated in Michigan Import Study, Michigan-Ontario Transfer Capability Study and the MISO Transmission Expansion Plan study. [ITC file: MECSDATA ITC Response to NERC Additional Request for Data of.doc ]. METC - METC participates in regional (ECAR) and inter-regional studies such as MEN studies. METC also participated in Michigan Import Study, Michigan- Ontario Transfer Capability Study and the MISO Transmission Expansion Plan study. METC coordinates new IPP connection requests with impacted neighbors per MISO process. [METC file MECSDATA METC Response to NERC Additional RFD of doc ] (3) Review system studies for the summer of 2003 to assess the sufficiency and effectiveness of the analyses of system performance. In particular assess whether the studies failed to expose system deficiencies, which if addressed could have materially affected the event. Where deficiencies were identified what was the status of remedial action plans? FE 2003 Summer Assessment [FE File: FirstEnergy03v1.pdf] Used ECAR summer 2003 base case and modeled various combinations of generator outages (Davis Besse and at times Perry) along with single contingency of a transmission element. Does not appear that additional transfer biases were modeled other than the imports required for FE generator outages. Based on studies performed by AEP on concerns with voltage collapse in Northeast Ohio under multiple contingencies and south to north transfer bias, it would have seemed prudent for FE to conduct some sensitivity studies for south to north bias transfers for their 2003 summer assessments. What is not known is whether FE conducted additional studies and not provided them to NERC or whether FE used the results of AEP s studies and conveyed those concerns to their operators. FE did provide a write-up in the ECAR 2003 Summer Multiple Contingency Assessment of ECAR Transmission System Conformance to ECAR Document No. 1. [File 00-TSPP-55, not provided as part of NERC data request.] This study was completed Page 31of 39

79 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE in August FE indicated in their write-up that they did perform internal studies that considered the outage of substation bus sections, outages resulting from breaker failure, corridor outages, and loss of entire substations and generating plants for their 345 kv system. AEP 2003 Summer Assessment [AEP File: System Performance Appraisal Report 03 Summer.pdf] Used ECAR summer 2003 base case as starting point and then added firm transmission reservations scheduled for 2003 summer on the AEP system. Performed single and multiple contingency analysis on this base case plus looked at scenarios of high transfers and load in order to stress various parts of their system. Performed First Contingency Incremental Transfer Capability (FCITC) along with P-V curves to study voltage performance in certain areas. The P-V analysis also looked at multiple contingencies. AEP s study of 2003 summer appears sufficient and effective. MISO 2003 Summer Assessment [MISO Files: Coordinated Summer Assessment CSAT2003SUMMER FINAL.pdf, CSAT_Study_Report_Addendum.pdf] Performed analysis of 2003 summer using 2003 summer base case as starting point along with variety of wide area to area transfer scenarios. Studied voltage performance in two areas: MAPP-WUMS interface and Southern Kentucky area. Primarily studied single contingencies. Concentrated on simultaneous transfer analysis on MISO, PJM, TVA and SPP areas. This study was meant to complement the regional and interregional studies. The scope of future seasonal assessments performed by MISO should be reviewed by MISO and should be coordinated into the regional and inter-regional study processes. ECAR 2003 Summer Assessment [ECAR File: 03-TSPP-3.pdf] Performed assessment of 2003 summer using 2003 summer base along with four transfer bias cases in each general direction south to north, north to south, east to west and west to east. Performed basically single contingency analysis and determined FCITC s for various imports, exports and transfers across ECAR against the base case and transfer bias cases. The combinations included exports from generation clusters in ECAR consisting of non-dispatched IPP s and member company s generation. Voltage performance in selected areas of ECAR was performed using P-V curve analysis. While the thermal analysis appeared to be sufficient and effective, ECAR should evaluate expanding the list of areas to perform voltage analysis. Also, some assessment of multiple contingency analysis should be considered using results from previous ECAR multiple contingency studies as a reference. Page 32of 39

80 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE MECS METC 2003 Summer Assessment [METC File: MECSDATA METC 03- TSPP-3 Appendix.pdf] The above referenced file is a copy of the ECAR Summer 2003 Transmission Assessment previously described above for ECAR. No additional company specific study was provided so it is not possible to determine the sufficiency and effectiveness of METC s study of 2003 summer. MECS ITC 2003 Summer Assessment [ITC File: ITC Ferc final.doc] The above referenced file is a copy of ITC s FERC 715. Part 6 of that document contains an Evaluation of Transmission Planning Performance. That section refers to sensitivity of constraints on the ITC system to transfers but without a specific study it is not possible to determine the sufficiency and effectiveness of ITC s study of 2003 summer. PJM 2003 Summer Assessment PJM Files: MEN_2003S_ReportBody.doc, VEMbody03s-11.doc] The above referenced files refer to the MAAC-ECAR-NPCC and VACAR-ECAR- MAAC inter-regional studies. (Need to review these studies and also need to determine if PJM as a Reliability Coordinator performs its own studies or if they rely on the above inter-regional studies. Need to find copy of PJM OATF study that was referenced in their latest response to Oct 30 data request.) IMO no data received as of this date. Identification of deficiencies and status of remedial actions FE 2003 Summer Assessment [FE File: FirstEnergy03v1.pdf] Operating procedures were identified for those contingencies that resulted in thermal or voltage violations. Some capital projects were identified that would provide relief before the 2004 summer. For certain combinations of multiple generating unit outages, no effective switching procedures were identified. AEP 2003 Summer Assessment [AEP File: System Performance Appraisal Report 03 Summer.pdf] Page 33of 39

81 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE Operating procedures were identified for those contingencies that resulted in thermal or voltage violations. Approvals of granting of transmission service requests were to be limited and use of TLR s were expected to be used to control flow of power on critical circuits. Long-term solutions such as additional capacity at South Canton 765/345 were identified. MISO 2003 Summer Assessment [MISO Files: Coordinated Summer Assessment CSAT2003SUMMER FINAL.pdf, CSAT_Study_Report_Addendum.pdf] Operating procedures were identified for those contingencies that resulted in thermal or voltage violations. ECAR 2003 Summer Assessment [ECAR File: 03-TSPP-3.pdf] Operating procedures provided by the ECAR member companies were identified for those contingencies that resulted in thermal or voltage violations. MECS METC 2003 Summer Assessment [METC File: MECSDATA METC 03- TSPP-3 Appendix.pdf] See ECAR 2003 Summer Assessment above since MECS only provided a copy of the ECAR report. [check with METC to see if they do their own studies.] MECS ITC 2003 Summer Assessment [ITC File: ITC Ferc final.doc] Operating procedures were identified for those contingencies that resulted in thermal or voltage violations. Also identified possible capital projects that are being considered or that have been identified that could alleviate the thermal or voltage violations. [check if they do their own.] PJM 2003 Summer Assessment PJM Files: MEN_2003S_ReportBody.doc, VEMbody03s-11.doc] The above referenced files refer to the MAAC-ECAR-NPCC and VACAR-ECAR- MAAC inter-regional studies. (Need to review these and also find out if PJM as a Reliability Coordinator performs seasonal assessments or if it just relies on the interregional studies. Have not found anywhere a study performed by just MAAC similar to what ECAR does. OATF - check) IMO no data received as of this date. Page 34of 39

82 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE (5) Compare system conditions and assumptions used to study system performance for August 2003 with conditions that actually existed on August 14. How typical was August 14 compared to earlier in the week or a typical summer day? FE 2003 Summer Assessment [FE File: FirstEnergy03v1.pdf] Based on review of the above document, the system loads used in FE s 2003 study were higher than loads experienced on August 14. Transfer bias levels in FE s 2003 study were not as high as those experienced on August 14. No additional transfer bias levels were used other than those modeled in the ECAR summer 2003 base case based on interpretation of data received from FE. Generation outages modeled in FE 2003 study were significantly different than conditions experienced on 8/14. AEP 2003 Summer Assessment [AEP File: System Performance Appraisal Report 03 Summer.pdf] Based on review of the above document, the system loads used in AEP s 2003 study were higher than loads experienced on August 14. AEP performed P-V analysis using transfers to MECS up to a 5,000 MW level and in other scenarios, a transfer up to 5,000 MW into FE was modeled. FCITC analysis was also performed with transfer directions covering all directions that impose stress on critical facilities. While no transfers into IMO were specifically modeled above those in the ECAR 2003 summer base, it appears that the range of transfers modeled in the AEP 2003 study was sufficient to stress critical facilities in AEP and surrounding systems. Follow up with Paul on 5000 number. MISO 2003 Summer Assessment [MISO Files: Coordinated Summer Assessment CSAT2003SUMMER FINAL.pdf, CSAT_Study_Report_Addendum.pdf] (Need to review this study). ECAR 2003 Summer Assessment [ECAR File: 03-TSPP-3.pdf] Performed assessment of 2003 summer using 2003 summer base along with four transfer bias cases in each general direction south to north, north to south, east to west and west to east. These transfer bias cases consisted of an additional 4000 MW bias for each of the directions. For example, the south to north transfer bias case consisted of 1000 MW from VACAR to IMO, 1000 MW from VACAR to MECS, 900 MW from TVA to FE and 1100 MW from TVA to MAIN. FCITC s for various combinations of ECAR imports, exports and transfers across ECAR were evaluated against the base case and four transfer bias cases. The combinations included exports from generation clusters in ECAR consisting of non-dispatched IPP s and member company s generation. (The ECAR Page 35of 39

83 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE report has a table of assumed interchanges in base case. Should we try to match that up with schedules on August 14 th at some hour?) mention FE loadings were lite in base case. MECS METC 2003 Summer Assessment [METC File: MECSDATA METC 03- TSPP-3 Appendix.pdf] The above referenced file is a copy of the ECAR Summer 2003 Transmission Assessment previously described above for ECAR. No additional company specific study was provided so it is not possible to determine the sufficiency and effectiveness of METC s study of 2003 summer. check MECS ITC 2003 Summer Assessment [ITC File: ITC Ferc final.doc] The above referenced file is a copy of ITC s FERC 715. Part 6 of that document contains an Evaluation of Transmission Planning Performance. That section refers to sensitivity of constraints on the ITC system to transfers but without a specific study it is not possible to compare conditions assumed in the study to those on August 14 th. check PJM 2003 Summer Assessment PJM Files: MEN_2003S_ReportBody.doc, VEMbody03s-11.doc] The above referenced files refer to the MAAC-ECAR-NPCC and VACAR-ECAR- MAAC inter-regional studies. (Need to review those studies for comparison to August 14 th. Also need to get copy of PJM OATF study.) IMO no data received yet. Refer to the Overall Comparison of System Conditions in FE, AEP, MECS, and PJM. The August 14 conditions from the load demand viewpoint were significantly higher than those of previous days in the week, but not as high as the summer peak demand recorded or the forecast demand that was including in their respective assessments. Figure BB shows the transactions involving FE and all the surrounding systems as either sinks or sources for the August (and the transactions during the peak day in 2003). The comparison shows that the transaction bias on August 14 is not materially different from the proceeding week days. It is high, but not unusually high as compared other days of that week and the peak day in Since FE s 2003 Summer Assessment did not include any description on external transactions, imports or imports into or out of FE, it is not possible to compare the transaction levels studied by FE with the actual transactions on August 14. Page 36of 39

84 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE (6) Review the extent to which the resiliency or robustness of bulk power systems were assessed by examining system performance for contingencies beyond normal criteria. Review what measures were taken to prevent cascading outages either through minimizing the probability of the event or its impact. FE s Summer Assessment identified the operating measures to deal with thermal overload on 138 kv and sub-138 kv facilities after the first set of n-1 contingencies. In some cases, a statement Perry Gen On line relieves this overload was provided. There were no contingency cases which would result in overload on 345 kv or voltage violations on 345 kv buses. The extent of FE s 2003 Summer Assessment, if its primary intention is to reflect the expected conditions and operating scenarios and operating measures to prevent a cascade starting in the northern Ohio area would not be adequate. (Pending on Ed s finding on FE 2000 Multiple-Contingencies Study filed with ECAR.) 3.Design criteria what design criteria were in effect to prevent a cascade starting in the northern Ohio area and why were these criteria ineffective? Can improvements be made in system design criteria to reduce the risk of future cascading outages? The current FE s voltage criteria of pu pre-contingency, and 0.9 as a minimum acceptable voltage range are not adequate from a design criteria viewpoint to identified effective operating measures to prevent a cascade. The existing operating practice in the FE control room that operators are empowered to carry out manual load shedding to deal with generation contingencies, but not transmission contingencies, are inherently not effective since the severity of system emergencies cannot be differentiated between generation and transmission. For improvements, see Recommendation list. 4.Equipment ratings and system limits should there be standards on setting of equipment ratings and system limits? How can ratings and limits be better coordinated? Need to coordinate this with Wiedman (1)Review the responsibilities and procedures used for setting system operating limits and communicating them to the control room operators. Compare these with good utility practice for prevailing system conditions and changing system conditions. Review the process for ensuring that the system operating limits are updated as system conditions change and events unfold in real time. Page 37of 39

85 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE Based on the information provided, and with few exceptions, the system operating limits are developed and updated in real-time from state-estimator and security analysis solutions. These real-time state-estimators have extensive contingency analysis capability, but perform limited voltage/transient analysis analysis. (PJM and Southern security analysis includes voltage collapse analysis functions). MISO Operations Engineers updates the real-time security analysis model, and the real-time operations personnel conduct the security analyses. MISO also makes adjustments to the real-time Flowgate Monitoring Tool based on information from its member systems. However, if the member systems are not aware of the outage(s), then the Flowgate Monitoring Tool is not updated. The responsibilities between the Reliability Coordinator (MISO) and the Control Area (FirstEnergy) are clear-cut, but lack a checking function that prevents mishandled information from propagating to the next level of reliable interconnected operation. From the information provided, the prevailing system conditions and changing system conditions should be captured by the real-time state estimators, including transmission and generation outages. However, the failure of a state estimator to solve or provide correct system conditions due to data corruption/availability needs to be addressed. In addition, the state estimator lacks the functionalities to provide the operator with the maximum limit of the flowgates, and to determine transient stability/voltage collapse limited conditions. A system operating limit derate look-up table for scheduled/forced outage based on off-line studies for reasonable key element outages, and reasonable combinations of element outages, would be useful in the event of a failure of the state estimator or communication/data sources, and it can also supplement the state estimator with guidelines for the maximum limit on flowgates limited by transient stability/voltage collapse conditions. Good utility operating practices would ensure the following: 1) State-estimators used in real-time are automatically running and performing contingency analysis, with immediate notification to appropriate personnel upon divergence of a simulation or communication/data failure, such as ICCP. 2) A separate back-up system to the state estimator should be available if the primary EMS state-estimator fails, such that the system operating limits will not solely rely on the EMS. 3) Off-line study conclusions for voltage collapse/transient scenarios are documented if the state-estimator does not have the capability of voltage collapse/transient analyses. Page 38of 39

86 SPDST s Input to Technical issues and recommendations per Story Board December 15, 2003 Draft Work in Progress 2003 OUTAGE INVESTIGATION CONFIDENTIAL CRITICAL ENERGY INFRASTRUCTURE INFORMATION DO NOT RELEASE (2)Determine the appropriateness and adequacy of the system operating limits in place prior to and following each generator and/or transmission element outage that occurred on August 14 prior to the start of the cascade. The system operating limits were appropriate and adequate until the second contingency occurred on the FirstEnergy system. Prior to the loss of the Harding Chamberlin 345 kv line, the system satisfied the NERC Operating Manual Policy 2.A.1 with regard to the next worst single contingency. The system began in a secure operating state, but following the Harding Chamberlin 345 kv line outage, the system was in an insecure state following the loss of the next worst contingency. From the information provided, there were no actual changes to the system by FE or MISO to restore the voltages or reduce the line loadings. In addition, a TLR was requested from AEP to PJM, but did not materialize. Once the Harding Chamberlin 345 kv line relayed, the FirstEnergy system was not capable of withstanding the next worst single contingency and satisfying the NERC Operating Manual Policy 5 of threatening the reliability of the Interconnection. Following the loss of the Harding Chamberlin and the Hanna Juniper 345 kv lines, the Star - South Canton 345 kv line loading exceeded the normal rating of the line, but was within the emergency rating. Once the Star South Canton 345 kv line tripped, the Sammis Start 345 kv line exceeded the normal rating, and continued to exceed the normal rating as successive 138 kv lines tripped. As evident by the increasing loading on the remaining 345 kv line, no operator actions were taken. (Refer to November 2003 Blackout Report, Page 40). (3) Determine when on August 14 the system transitioned from a studied and known state to an unstudied and unknown state. Identify limit violations and review what actions were taken. This question is answered in (2) above. Page 39of 39

87 Attachment 6 Modeling Issues In Blackout Analysis Modeling and System Studies Issues Blackout Technical Conference December 16, 2003 Robert W. Cummings Director Reliability Assessments and Support Services Load representation Power Factor overly optimistic Generation representation Optimistic reactive capability Line rating disagreements Interchange transactions Studies not capturing levels of transactions Simultaneous transaction impacts 2 Powerflow Modeling Issues Overly optimistic power factors Incorrect circuit ratings Need to benchmark to system readings periodically what is the power factor served as observed at the kv system Databases to ensure ratings and topology consistency Common Information Model compatible 3 Post-Mortem Modeling Issue Time Synchronization of Telemetry Non-NIST synchronization problems on DFRs and DERs Synchronization at point of origin a must Synchronization at point of receipt insufficient Inconsistent data quality and retention Incomplete / inconsistent periodicity 4 1

88 Line Rating Problems Methodology of calculation Bounded parameters? Quality control on ratings clearance considerations Problems encountered with disagreements in ratings Method to eliminate disagreements between: Planning models Regional / Interregional Studies EMS State Estimators Reliability Coordinators 5 SDX Issues Disagreements with actual conditions omissions, errors in data, non-outaged elements Untimely data entry Incomplete picture not all CAs / RCs entering data To be depended upon by PJM & MISO for real-time calculations Is real time data needed?? 6 Regional / Interregional Studies Need to take all outages, including generation outages More than N-1 contingencies may need to be run severe outage scenarios Need to monitor entire system (not limited to regional interfaces) Zonal analysis regardless of ownership Need to look at wider variety of transactions Study known patterns analyze historical trends both magnitude and direction and simultaneous transfers 7 Northeast Central West & South Imports 8 2

89 Northeast Central Import / Exports 6/1 through 8/ Simultaneous Transfer Studies Imports Exports MW :00 2:00 4:00 Max Imports Max Exports Average 14-Aug 6:00 8:00 10:00 12:00 Hour (EDT) 14:00 16:00 16:00 18:00 20:00 22:00 Eastern Interconnection Interregional Studies MEN Simultaneous transfer nomograms between ECAR, MAAC, and NPCC VEM Sensitivities studied for selected simultaneous transfers MET Only for WUMS Coordinated Summer Transmission Assessment (CSTA) prototype for 2003 MISO, PJM, TVA, SOCO, AECI, SPP, Entergy Full Simultaneous transfer analysis 9 10 STC ECAR to MAAC/NPCC 1500 Real-Time Modeling Issues Situational Recognition System Data Exchange (SDX) Used to update system topology in IDC Used by some to update their topology for system security analysis Inconsistent Line Ratings Transaction modeling Some dynamic transfers not tagged Not reflected in IDC Not reflected in some system security analyses 12 3

90 Analysis Results S. Canton Star Loading Contributions Tagged transactions only played minimal role limited flow impact on overloaded lines The main contributors to flows on overloaded lines were native and network loads and generation dispatch TLR level 5B (including load shedding) would be required to be effective 1,500 MW post South Canton Star 345 kv 2,500 MW by 16:00 EDT 13 Modeling Improvement Issues Load power factor Generation reactive capabilities Improved topology awareness Improved transactional awareness 15 4

91 Attachment 7 Summary August 14, 2003 Blackout Generation Performance By NERC Generation PPCMD Team December 16, Generation outages did not initiate the cascading blackout 2. Earlier unit trips in the Cleveland area contributed to the conditions leading up to the cascade 3. During the conclusion of the cascading failure, generation tripped off in three general categories: 1. Excitation system overload or failure 2. Plant control system action or failure 3. Consequential result of the broken transmission system 4. In addition, some prolonged out-of-step conditions are evident 5. Generation performance may have expanded the boundaries of the resulting blackout 6. To date, little damage has been discovered as a result of the cascade 2 East Lake Location East Lake Unit 5 Trips 3 4 1

92 Initial Generation Trips in Cascade MCV Unit Location ONTARIO 5 6 MCV Combined Cycle Plant Trips PSDR Locations in Ontario 7 8 2

93 Detroit Units Slip Poles 9 Severe Under Frequency Condition 11 View Into Detroit from Lambton 10 Italian Blackout, September 28,

94 Brownstown Substation at 16:10:53 Brownstown Substation at 16:11: Southern Detroit 345 kv Frequency Frequency in Ontario and New York

95 Nine Mile Point Unit 2 Trips Summary 1. Generation outages did not initiate the cascading blackout 2. Earlier unit trips in the Cleveland area contributed to the conditions leading up to the cascade 3. During the conclusion of the cascading failure, generation tripped off in three general categories: 1. Excitation system overload or failure 2. Plant control system action or failure 3. Consequential result of the broken transmission system 4. In addition, some prolonged out-of-step conditions are evident 5. Generation performance may have expanded the boundaries of the resulting blackout 6. To date, little damage has been discovered as a result of the cascade Data Request 19 5

96 Attachment 8 Transmission System Performance Progress as of December 12, EDT to 1606 EDT Relay Performance Complete Five 345 kv lines and numerous 138 kv lines relayed, a majority of them relaying to lockout. Harding Chamberlain 345 kv line tripped by ground relay and locked out. Hanna Juniper 345 kv line tripped by ground relay and locked out - tree contact Star South Canton 345 kv line tripped by ground relay and reclosed then locked out. Numerous underlying 138 kv lines tripped by ground relay with most of them locking out. Cloverdale Canton Central 138 kv line relayed by ground relay and locked out. On this operation, breaker A1 at Canton Central failed or was slow to clear the fault, initiating a breaker failure scheme which tripped the Canton Central 345/138 kv transformer and the Canton Central Tidd 345 kv line. A motor operated disconnect opened automatically allowing the Canton Central Tidd 345 kv line to reclose successfully. West Akron 138 kv bus relayed when Breaker B26 failed to trip on abnormal var flow conditions. Star Sammis 345 kv line relayed by zone 3 relay on overload conditions and locked out (there is no automatic reclosing for a zone 3 time delay trip) EDT to 1609 EDT Relay Performance Complete The cascade moves west and north 138 kv and 69 kv lines in the Star and So. Canton areas trip on heavy power flow Galion-Ohio-Muskingum 345 kv trips due to ground faults and heavy power flow Ohio Central-East Wooster 138 kv line trips singleendedly due to low voltage and heavy loading East Lima Fostoria Central 345 kv trips due to heavy power flow kv lines into Toledo trip on heavy power flow 1609 EDT to 1610:40 EDT Relay Performance Complete Cascade moves into Michigan Argenta-Battle Creek 345 kv trips, recloses and locks out overload, low voltage, widening phase angle Argenta-Tompkins 345 kv trips, recloses and locks out - overload, low voltage, widening angle Battle Creek Oneida 345 kv trips and locks out overload, low voltage, widening angle kv lines in the Argenta area trip on extremely heavy power flow Hampton Pontiac & Thetford-Jewell 345 kv trips and locks out overload, low voltage, widening angle Synchronism is lost in Michigan 1

97 1610:40.3 EDT Relay Performance Remaining Activity Northern Ohio separates from Pennsylvannia Perry-Ashtabula-Erie West trips at Perry due to low voltage, heavy reactive flow toward Erie West and heavy active power flow. Phase 2 report input Continue to work with SOE, Root Cause, Planning and Dynamics teams to Analyze System Performance 16:10:40 to Evaluate transmission line ratings impact on cascade Incorporate public forum, NERC Steering Committee and NERC standing committee recommendations Seek endorsement on recommendations Lessons to be learned by line trips Convert MVA to apparent Z Line trips due to faults Trim trees or preferable remove from EHV rights-ofway Line trips due to low voltage Install undervoltage load shedding; develop standard Line trips due to heavy power flow Evaluate the need for zone 3 time delayed tripping on all EHV lines; develop settings criteria jz R Φ Z app kv θθ 2 MVA 3θ Φ= tan -1 (MVAR / MW) 2

98 Q-II X θ Q-I Zone 3 versus Zone 2 Loadability low load POWER FLOWS IN VARS FLOW OUT φ POWER FLOWS OUT VARS FLOW OUT high load trip point R, jx By similar triangle geometry, zone 2 line loadability is 50% greater than zone 3 if zone 2 is set at 150% of the line and zone 3 is set at 200% of the line length. POWER FLOWS IN VARS FLOW IN POWER FLOWS OUT VARS FLOW IN If Z line = 60 ohms and the load angle is 30 then Zone 3 has a relay trip MVA of 1402 (2350 a.) while Zone 2 has a relay trip MVA of 2103 MVA (3519 a.) R Q-III Q-IV Additional lessons learned Coordinate underfrequency load shedding and underfrequency tripping of generation Emphasize independent NERC and Regional compliance standards and measures for protection systems Develop NERC standards on reporting of disturbances including prescribed common reporting format Coordinate relay tripping with line emergency capability, report and comply Assure and prescribe proper automatic operation of generator excitation systems Topics for Discussion NERC, its Regions, and Reliability Coordinators need to develop protection performance criteria and compliance audits Use of Zone 3 functions on EHV system Install undervoltage load shedding Evaluate line rating methodologies Identify likely break points in the Eastern Interconnection Consider islanding protection systems similar to others in the world Develop disturbance analysis methodology across the Eastern Interconnection Insist on voltage support requirements for generators 3

99 Attachment 9 JOINT U.S.-CANADA POWER SYSTEM OUTAGE TASK FORCE UTILITY VEGETATION MANAGEMENT INITIAL REPORT December 2003 CN Utility Consulting, LLC Phone: rnovembri@cnutility.com FERC Contract: FERC-03AL-30574

100 INITIAL REPORT UTILITY VEGETATION MANAGEMENT BACKGROUND CN Utility Consulting, (an industry firm specializing in Utility Vegetation Management), was asked to perform the following three tasks: 1. Perform field investigations and outage analysis at four suspect locations on transmission right-of-ways maintained by either FirstEnergy (FE) or American Electric Power (AEP). Three of the sites were located in FE s service territory and one site was located in AEP s service territory. 2. Collect and analyze information and data regarding transmission right-of-way vegetation management practices at FE, AEP, and Cinergy, in order to assess the strengths and weaknesses of each Company s vegetation management program. 3. Identify generic best practices for transmission level vegetation management to enhance system performance and transmission reliability. This initial report provides introductory information related to utility vegetation management activities, and findings related to Task 1. Tasks 2 and 3 will be addressed in the final report. INTRODUCTION Utility Vegetation Management (UVM) programs represent one of the largest reoccurring maintenance expenses for electric utility companies in North America. Indeed, keeping trees and vegetation from conflicting with overhead conductors is a critical, and expensive, responsibility of all utility companies concerned about electric service reliability and fire mitigation. The vast majority of work in this multi-billion dollar a year industry is not performed by utility personnel, but rather outsourced to specialized tree and vegetation management contractors. These contractors typically work under the aegis, and direction, of a Utility Company Arborist or Forester who is charged with overall management of the UVM program. A typical UVM program can include all, or some, of the following activities: 1. Tree pruning and removal 2. Vegetation control around poles, substations, and other electric facilities 3. Manual, mechanical, or chemical control of vegetation along rights-of-way 4. Pre and post inspections of required work 5. Tree planting and transplanting 6. Research & development 7. Public education 8. Tree inventories, work management systems, and sundry computerized functions Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

101 INITIAL REPORT UTILITY VEGETATION MANAGEMENT THE PURPOSE OF UVM: WHEN TREES AND POWER LINES CONFLICT It is appropriate to begin with an explanation of why a UVM program is critical to any utility company that maintains overhead energized lines. The two most often cited reasons are as follows: 1. Electric Service Reliability It is generally accepted that the majority of electric power outages occur when trees, or portions of trees, grow or fall into overhead electric power lines. While not as prevalent, outages can also occur when overhead conductors sag into trees due to increased load or due to a change in ambient conditions, e.g., high air temperature or low wind speed. 2. Fires Arcing between any part of a tree and a bare high-voltage conductor has the potential to occur if the physical separation between both is not maintained (arcing distances vary based on such factors as voltage and ambient conditions). If, for example, arcing does occur between a twig and a high-voltage line, there is the possibility that the twig can ignite and fall to the ground. If flammable material is present on the ground, it could cause a fire. While the incidence of fires caused by tree and power line conflicts appears to be relatively low (less than 1% of wildland fires nationally) the potential for large conflagrations does still exist. This problem is particularly pronounced in the west, southwest, and pacific northwest parts of both the US and Canada. TRANSMISSION AND DISTRIBUTION (T&D) UVM ACTIVITIES While this investigation has focused on transmission UVM activities, it is important to note the relationship with distribution UVM operations. This discussion is necessary in that most utility companies have one program that deals with both transmission and distribution UVM activities. Additionally, many lower voltage transmission poles do have distribution circuits located on the same pole. This is typically referred to as under-build facilities. While T&D UVM operations typically share the same administration and oversight, there are general differences in the type of work that is performed. The following is a brief description of the types of work associated with each of these UVM programs. Distribution UVM By far, distribution UVM activities comprise the largest part of an electric utility s efforts in managing trees and vegetation near power lines. At many utilities, the distribution part of a program may utilize 80-90% of utility funding and resources for managing vegetation. This does not however mean that distribution UVM is any more, or less, important than transmission work. It is only a by-product of having significantly more miles of distribution lines (and exposure), than there are transmission lines. Distribution programs typically prune more trees than they remove, and the costs (on a per tree basis) are higher than equivalent work on transmission lines. This is primarily Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

102 INITIAL REPORT UTILITY VEGETATION MANAGEMENT due to the fact that most distribution tree pruning or removal is done in front of someone s home, and on community streets. Distribution UVM work is more visible to the public, and as such, requires more upfront notification, coordination with agencies, and a greater amount of personal and public education prior to commencing the work. Transmission UVM The primary difference between T&D UVM work can be summed-up as follows. The vast majority of Transmission rights-of-way have documented provisions allowing the utility to clear and maintain the vegetation in order to provide safe and reliable electric power. These easements, in essence, give the utility a greater amount of control over the landscape, than what is experienced adjacent to distribution lines. In the latter case, little, if any, documentation exists giving the utility the right to perform whatever UVM work is required to maintain the distribution lines free of vegetation. Given the greater rights associated with transmission UVM work, it is common to see less pruning and more removals related to transmission work than are typically seen in a distribution program. The unit costs are also typically lower for transmission UVM work than are experienced in distribution activities (fewer customers and landowners to negotiate with). These documented rights also result in greater use of mechanical and chemical UVM tools on transmission rights-of-way. This includes mechanical mowers and the wider use of appropriate herbicides. Laws and regulations Our final report will include a review of current laws and regulations that either mandate or influence transmission UVM activities. The following is a brief discussion of these requirements based on our initial investigation findings. With the exception of California 1, there are no statewide Commission promulgated mandatory clearance requirements. In other words, we found no evidence that mandatory clearances between vegetation and high-voltage lines must be maintained at all times. The most often referenced requirement for UVM work is found in the NESC Rule 218, which has been promulgated in most states. The Rule itself requires: General: Trees that may interfere with ungrounded supply conductors should be trimmed or removed. NOTE: Normal tree growth, the combined movement of trees and conductors under adverse weather conditions, voltage, and sagging of conductors at elevated temperatures are among the factors to be considered in determining the extent of trimming required. 1 California s General Order 95, Rule 35 requires utility companies to maintain specific clearances between vegetation and energized conductors. These clearances are dependent on and vary by voltages. Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

103 INITIAL REPORT UTILITY VEGETATION MANAGEMENT Where trimming or removal is not practical, the conductor should be separated from the tree with suitable materials or devices to avoid conductor damage by abrasion and grounding of the circuit through the tree. At Line Crossings, Railroad Crossings, and Limited-Access Highway Crossings: the crossing span and the adjoining span on each side of the crossing should be kept free from overhanging or decayed trees or limbs that otherwise might fall into the line. This industry dominant rule appears to be very general in nature, and vague on specifics. As such, we do not believe it provides adequate instruction to utility companies in maintaining power lines clear of vegetation. We are however aware of a current industry effort to improve this NESC Rule and will provide additional information regarding this effort in the final report. In addition to the NESC, there are requirements for UVM work found in various model fire codes such as the Uniform Fire Code (UFC) and the Urban-Wildland Interface Code (UWIC). Both codes have been adopted on a state and local basis in various parts of the US. FINDINGS AS THEY RELATE TO THE AUGUST 14 TH OUTAGE A review of available documentation and field investigations at FE and AEP suggest that four of the line outages were, in fact, caused by conflicts between high voltage transmission lines and vegetation. The following contains an overview of our findings of FE and AEP line outages, and identified contributing and/or mitigating factors. We will begin with a brief discussion of conductor sag, and follow this up with our investigation protocol and findings. Conductor Sag A key consideration during the design, construction, and maintenance of transmission lines is that of conductor sag. The height of transmission conductors does not remain static once it is installed, due in part to such factors as temperature and wind velocity. Temperatures typically increase during the summer months requiring additional power to accommodate air conditioning load. As the load increases over transmission lines, the temperature increases and the conductors, typically aluminum, expand. The effect of this expansion is a lengthening of the conductors. This, in effect, causes them to sag closer to the ground. The presence of wind also acts to influence conductor sag. Wind, at high enough velocity, provides a cooling effect on the conductors. This cooling of the conductors reduces the amount of sag that would be encountered on a calm day. Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

104 INITIAL REPORT UTILITY VEGETATION MANAGEMENT The following graphic provides an example of how conductor height can vary depending on load and wind velocity. Field Investigation Protocol The investigation team consisted of Richard Dearman (TVA), Saeed Farrokhpay (FERC), and Stephen Cieslewicz and Robert Novembri of CN Utility Consulting. The investigation consisted of a review of prepared responses and documents provided by FE and AEP, field visits to three suspect locations at FE and one suspect location at AEP, and interviews with FE and AEP contract personnel. Appropriate photographs, GPS readings, measurements and calculations were made at each location. General Findings Overgrown trees, as opposed to excessive conductor sag beyond design, appear to be the cause of these faults. Each of these lines were predisposed to fault under system sag conditions well within normal operating parameters. Incremental increases in amperage and temperature caused an incremental sag increase on the Stuart Atlanta (AEP) line causing it to fault and lock out due to contact with vegetation. Incremental increases in amperage and temperature increased the sag on the Star South Canton (FE) line causing it to fault and reclose due to contact with vegetation. This line tripped three additional times over a period of 11 4 hours before locking out. Incremental increases in amperage and temperature increased the sag on the Chamberlin Harding (FE) line causing it to fault and lock out due to contact with vegetation. Again, incremental amperage and temperature increases, escalated by the loss of the Chamberlin Harding line caused further incremental sag increases on the Hanna Juniper (FE) and it faulted and locked out due to contact with vegetation. Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

105 INITIAL REPORT UTILITY VEGETATION MANAGEMENT We have field evidence of tree contact at three locations. At the fourth location, Hanna Juniper, the tree was removed before we arrived, but the fault was visually (time/date) confirmed during the occurrence and pictures of the tree before it was removed support the visual observation. We also have a revised calculated fault location, provided by FE, for the Star South Canton line that matches the location of the confirmed tree fault. While conductor sag may have contributed in a small way to these events, the direct cause of these incidents can be attributed to overgrown trees.. FE Documented Easements and Cycle The following is language taken from a typical FE easement document that describes the actual rights of the utility regarding what can be pruned or removed in these particular rights-of-way. The easement and rights herein granted shall include the right to erect, inspect, operate, replace, relocate, repair, patrol and permanently maintain upon, over, under and along the above described right of way across said premises all necessary structures, wires, cables and other usual fixtures and appurtenances used for or in connection with the transmission and distribution of electric current, including telephone and telegraph, and the right to trim, cut, remove or control by any other means at any and all times such trees, limbs and underbrush within or adjacent to said right of way as may interfere with or endanger said structures, wires or appurtenances, or their operations. FE claims a 5-year cycle for transmission lines (all required vegetation work completed in a five year period for all circuits). It appears that had these rights (as documented above) been fully exercised, these trees could possibly have been removed in prior cycles. It should be noted that a 5-year cycle is consistent with industry standards, and the phenomena of not fully exercising easement rights (as they pertain to transmission rights-of-way) is common in the industry. An evaluation of AEP s stated cycle and documented rights will be included in the final report. Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

106 INITIAL REPORT UTILITY VEGETATION MANAGEMENT FAULT CHRONOLOGY, OBSERVATIONS AT FIELD SITES AND COMMENTS Stuart Atlanta (345 kv) AEP 14:02:00.0 Line trips and locks out. No calculated fault location provided by AEP for this outage. Evidence of tree contact was observed between towers 222 and 223. Conductor height of north phase measured 39 feet at tree location and point of contact. Center phase measured 41 feet and south phase measured 47 feet. Trees and brush were felled on or after August 14 th. Debris was left on site and inspected. Two ailanthus trees showed evidence of significant fault current damage and were debarked. One measured 2.5 diameter at ground line, and the other measured 6 diameter at ground line (GPS E N). Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

107 INITIAL REPORT UTILITY VEGETATION MANAGEMENT Both trees estimated to be 30 to 35 feet tall. Other trees in the area showed evidence of fault current damage as well. The following readings were provided by AEP at the approximate time conductor height measurements were taken. Time: 10:00 EDT Date: 10/22/03 Temperature Reading: 55.6 Wind Speed: N/A Conductor Height: 39 Loading: Stuart 950 amps Atlanta N/A Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

108 INITIAL REPORT UTILITY VEGETATION MANAGEMENT Star South Canton (345 kv) FirstEnergy 14:27: Line trips and recloses (both ends). 15:38: Line trips and recloses (both ends). 15:41:33.43 Line trips and recloses (both ends). Retrips at South Canton. 15:42:07.0 Line recloses at South Canton, retrips and locks out. Line already open at Star. Calculated Fault Location (revised): 20.5% from Star Substation, Span Inspected conditions at structure and right-of-way toward (between 6.7% and 8%). No vegetation conflicts observed in this area. Did not review (9.1%). Inspected tree conditions at structure (20.5%) between towers and Trees and vegetation were felled on or after August 14 th. Debris and tree parts were inspected on site. Conductor height measured Tree height measured at 30 feet, although we could not verify location of the stump, or missing section of tree (GPS E N). Obvious significant fault damage to clustered trees. Charred limbs, and de-barked by fault current. Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

109 INITIAL REPORT UTILITY VEGETATION MANAGEMENT Topsoil in the area of trunk was disturbed, discolored and broken up at site. This would be indicative of a higher magnitude fault or multiple faults. Tree to structure measured Fourteen year-old tree in the middle of the right-of-way was removed. The following readings were provided by FE at the approximate time conductor height measurements were taken. Time: 14:14 EDT Date: 10/16/03 Temperature Reading: 47 Wind Speed: 1 mph (Wadsworth, OH) Conductor Height: 44 9 Loading: Star 836 amps South Canton N/A Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

110 INITIAL REPORT UTILITY VEGETATION MANAGEMENT Harding Chamberlin (345 kv) FirstEnergy 15:05:41.0 Line trips and locks out. Calculated Fault Location: 11.3% from Chamberlin Substation, Tower No evidence of vegetation at calculated fault location (11.3%). See photo below. At 17.7%, between towers and we inspected vegetation. Trees and brush were felled on or after August 14 th. Conductor height measured at 46 7, tree height measured at 42 (GPS E N). Locust tree showed evidence of fault current damage. Tree damage indicated a lower level of fault current. Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

111 INITIAL REPORT UTILITY VEGETATION MANAGEMENT Burn marks were observed at 35 8 up tree. Portions of the tree had been removed from the site making it difficult to determine exact height of contact, implying that the height is a minimum, and likely 3-4 feet higher than verifiable. Other vegetation along the right-of-way measured between two and five inches in diameter at ground line. The following photo depicts a tree located in the right-of-way that was over six years old, as indicated by the growth rings. The following readings were provided by FE at the approximate time conductor height measurements were taken. Time: 11:58 EDT Date: 10/16/03 Temperature Reading: 47 Wind Speed: 2 mph (Wadsworth, OH) Conductor Height: 46 7 Loading: Chamberlin 405 amps Harding 400 amps Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

112 INITIAL REPORT UTILITY VEGETATION MANAGEMENT Hanna Juniper (345 kv) FirstEnergy 15:32:03.0 Line trips and locks out. Conductor height measured 48 9 at fault location. No evidence of tree debris at site. Walnut tree stump measured 14 diameter at ground line (GPS E N). Subsequent clearing left trees and brush that, in our opinion, could have been removed as indicated by the photo below on the left. This photo was taken in the same span as the fault occurred. Other trees were pruned and left in the right-of-way as part of a landscaped area as indicated by the photo on the right. This photo was taken within one span of the fault. Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

113 INITIAL REPORT UTILITY VEGETATION MANAGEMENT The contract foreman who witnessed the event on August 14 th was interviewed. He described the fault and provided a definitive time/date stamp for the incident. Per FE field personnel, the schedule for completing work on this circuit had been advanced by one year (we have not yet verified this assertion and will need to confirm with follow-up requested documents). Surrounding trees were 18 in diameter at ground line and 60 in height (not near lines). Other locations at this site had numerous (estimated 20+) trees in this right-of-way. South phase, where contact occurred, is lower than center phase due to construction design. Subsequently, FE provided photographs that clearly indicate that the tree was of excessive height. The following readings were provided by FE at the approximate time conductor height measurements were taken. Time: 09:31 EDT Date: 10/16/03 Temperature Reading: 44 Wind Speed: 3 mph (Wadsworth, OH) Conductor Height: 48 9 Loading: Hanna 900 amps Juniper 970 amps Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

114 INITIAL REPORT UTILITY VEGETATION MANAGEMENT Columbus Bedford (345kV) Cinergy 12:08:40.0 Line trips and locks out. 18:23:00.0 Line returned to service. Just prior to submitting this initial report we had the opportunity to perform an initial review of a transmission fault experienced on the Cinergy system on August 14 th in Indiana. While it does not appear that the fault was connected to the blackout, this situation does provide a very good example of the obstacles placed in front of utilities that are attempting to manage vegetation near overhead power lines. We will discuss these issues in greater detail in the final report. In the interim, we do believe that a brief description of what occurred on the Columbus Bedford circuit is an appropriate. Based on discussions with Cinergy, this transmission line fault occurred as a result of tree contact in one span of the Columbus Bedford circuit. See photo below. Apparently work on this span had been halted various times by the owner of the property. The owner of the property had severely limited the ability to achieve necessary clearances, and apply subsequent herbicides to control future growth. While Cinergy does, in fact, have documented rights to perform this work (documented easement), this landowner has successfully halted work from proceeding on several occasions. This included the homeowner obtaining a court granted temporary injunction halting work by Cinergy. Note: the required work was finally completed on October 9, 2003 as depicted in the photo below. Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC

115 INITIAL REPORT UTILITY VEGETATION MANAGEMENT We bring this up to illustrate that there are many hurdles every utility company must face when trying to maintain lines clear of vegetation. In this particular case, it was a landowner that halted work. In other cases we are aware of, it can be local, state, or even federal agencies that hinder progress. In our final report we will include a detailed discussion of the types of obstacles that utilities face when trying to complete required UVM work in a timely manner. PROGRAM ASSESSMENTS A complete assessment of each of the three programs reviewed will be provided in the final report and will be based on a thorough review of the UVM program and activities at FirstEnergy, AEP, and Cinergy. Joint U.S. Canada Power System Outage Task Force Utility Vegetation Management Program Review December CN Utility Consulting, LLC