2007 FRCC AGENT OPERATIONAL AUDIT REPORT

Transcription

1 2007 FRCC AGENT OPERATIONAL AUDIT REPORT Audit Date: November 8 th, 2007 FLORIDA RELIABILITY COORDINATING COUNCIL December 5th, N. Westshore Blvd., Suite 1002, Tampa, Florida Phone * Fax

2 Table of Contents 1.0 Executive Summary Page Introduction Page Purpose and Scope Page Agent Audit Task Force Membership Page Operational Audit Process and Procedure Page Observations and Discussions Page Recommendations Page 8 Appendix A Final Agenda Agent Audit Task Force Audit Page 10 Appendix B On-site Audit Participants Page 12 2

3 1.0 Executive Summary The Agent Audit Task Force (AATF), a group of FRCC member, sector representatives was established and charged with performing an operational audit of the FRCC Agents implementing the functional requirements defined within the FRCC Security Process for the FRCC Bulk-Power Electric System. During this audit process, the 2007 AATF found no issues or concerns related to the FRCC Agents performance of their Reliability Coordination functional roles and fulfillment of responsibilities as the Real-Time Operating Function, the Operations Planning Function and the State Capacity Emergency Coordination Function for the FRCC Region. This conclusion is based on the following: The AATF did not receive or investigate any complaints or inquiries regarding Agent performance. The AATF reviewed previous Agent Audit Reports as well as the most current NERC Readiness Audit Report (Reliability Coordinator) and found that recommendations and opportunities for improvement identified in the past have been appropriately addressed. The AATF performed an on-site audit and review of the Agent facilities and personnel. The previous experiences between AATF members and the Agents. It was evident to the task force that both Florida Power and Light (FPL) and Progress Energy Florida (PEF) have performed their FRCC Agent responsibilities in an effective and commendable manner. This was illustrated in the latest NERC Reliability Coordinator Readiness Audit Report which cited 2 Potential Examples of Excellence and 13, Positive Observations along with the Agents responsiveness to recommendations received from both the NERC Readiness Program and the 2005 Agent Audit Task Force report. The recommendations resulting from this 2007 Agent Audit are primarily directed to the FRCC organizational documents supporting the Reliability Coordinator functions as well as the FRCC s role as an Eastern Interconnection Reliability Coordinator, functioning in the new era of mandatory and enforceable NERC Reliability Standards. The task force has included 11 recommendations that the FRCC should consider in order to improve and enhance its Regional processes. While there is no priority implied in the order of the recommendations, the task force believes that several of the recommendations are more important in terms of documenting the existing Reliability Coordinator function hierarchy that exists within the FRCC Region, namely, the update of the Security Process, update of the agent agreements and execution of an OPC Agreement. Other recommendations are included as enhancements of documentation and clarifications of processes that exist today. The task force believes that the FRCC should evaluate all of the recommendations, and prioritize them for implementation based on available resources and reliability requirements

4 2.0 Introduction The Security Process for the FRCC Bulk-Power Electric System (Security Process) was created to ensure the reliability of the FRCC Region and defines the roles and responsibilities of the FRCC Security Coordinator (SC) along with the Operating Entities within the FRCC. The FRCC organization is the Security Coordinator for the FRCC Region and utilizes agents (Agents) to carry out the functional roles and responsibilities of the Security Coordinator as defined in the Security Process. Over the past several years as NERC (North American Electric Reliability Corporation) has evolved its original policies into a set of Reliability Standards, the terminology describing the functional role of the Security Coordinator has been revised to Reliability Coordinator. For the purposes of this report, the terms Security Coordinator and Reliability Coordinator will be used interchangeably. In the past, in order to ensure that the Security Process was applied and implemented in a manner consistent with the FRCC s goals of no undue discrimination, the FRCC developed the FRCC Monitoring Procedures for Agents (Monitoring Procedure) which provides the mechanism for a requested review of Agent performance as well as the periodic establishment of an Agent Audit Task Force (AATF) to audit Agent performance. The procedure which was adopted in April of 1998 by the FRCC Board of Directors and last revised in December of 2003, requires an operational audit of each FRCC Security Process Agent, every two years. The last Agents operational audit was performed on March 11 th, The 2007 operational audit was scheduled for November 8 th. 3.0 Purpose and Scope The purpose of the FRCC Monitoring Procedures for Agents is to ensure that all parties operating in the FRCC Region, including non-frcc members, have confidence that the FRCC Agents have acted in a manner consistent with the FRCC s goals of no undue discrimination in the application of the Region s Security Process functions as defined in the Security Process for the FRCC Bulk-Power Electric System. The primary scope of the operational audit included the following activities: Review of the FRCC Security Process for required revisions Request for comments on performance of FRCC Agents from entities operating in the FRCC Region Pre-audit Agent questionnaire developed by the task force Input from the adjacent Reliability Coordinator Pre-audit documentation review An on-site audit to review Agent s facilities and internal procedures supporting its Agent functions The AATF did not perform an on-site visit of the Progress Energy Florida (PEF) facilities during this audit. Since PEF only performs the Operations Planning function,

5 and will be transitioning that responsibility and function over to Florida Power and Light (FPL) starting in January 2008, the audit team determined it would be more efficient to have the PEF personnel travel to the FPL facilities for interview purposes. The task force determined that there was significantly more benefit to the audit process to preview the Operations Planning Function transition plan, PEF to FPL, on-site with FPL personnel assuming the function. The FRCC Security Process identifies three primary functions of the Security Coordinator. These functions are the Real Time Operating Function, the Operations Planning Function, and the State Capacity Emergency Coordination Function. Currently the FRCC Security Process Agents are listed below: Real-Time Operating Function Florida Power and Light Company Operations Planning Function Progress Energy Florida * / Florida Power and Light Company State Capacity Emergency Coordination Function Florida Power and Light Company * Through the end of Agent Audit Task Force Membership The Monitoring Procedures for FRCC Agents requires that each FRCC industry sector utilizing the functions performed by the FRCC Agent(s) shall be appropriately represented on the task force. The Task Force was initially made up of representatives from all five (5) of the FRCC member sectors. Due to unforeseen circumstances, the Investor Owned Utility (IOU) Sector was not represented at the onsite audit of the Agents. Based on the lack of any complaints, review requests or expressed concerns with agent performance since the last operational audit, the AATF members present, determined that the participating members were adequately representative to perform the on-site agent audit and that additional steps would be taken to ensure IOUs would have the opportunity for input to the process. The 2007 AATF members are as follows: Eric Senkowicz, Joe McKinney FRCC Staff - Chair Florida Municipal Power Agency Non-Investor Owned Utility Wholesale Sector - 3 -

6 Herman Dyal Richard Gilbert Doug Jensen Ron Donahey * Donna Howard Clay Electric Cooperative Load Serving Entity Sector Lakeland Electric Generating Load Serving Entity Sector Northern Star Generation Services Suppliers Sector Tampa Electric Company Investor Owned Utility Sector FRCC (Observer) * Mr Donahey was unable to participate in the on-site audit but as the FRCC Operating Reliability Subcommittee (ORS) Chairman and for the purposes of this report, is being included as a subject matter expert on agent performance and also as a proxy representative for the Investor Owned Utility Sector which was unable to provide a member representative to the Task Force. 5.0 Operational Audit Process and Procedure Each member of the task force was required to execute a combined FRCC Confidentiality Agreement for Security Coordinator Audit with the FRCC and the Agent. Notification was made to all FRCC members of the pending audit in order to solicit comments and concerns regarding the performance of each FRCC Security Process Agent. A letter was sent to the members of the FRCC Board of Directors along with the Standing Committees on October 11, 2007 requesting comments from FRCC members as well as any other market participants regarding the performance, conduct and non-discriminatory activities of each FRCC Security Process Agent. The previously completed FRCC Agent Operational Audit Report, dated, May 15, 2005 as well as the Monitoring Procedures for FRCC Agents are publicly posted on the FRCC website and provide access and direction to all market participants, regardless of FRCC membership status, on addressing Agent performance issues within the FRCC Region. As a result of the notification, the AATF did not receive any comments, concerns or requests for review from any FRCC member, non-member or market participant regarding Agents performance issues. In light of this, the AATF team began a document review which included the following: Security Process for the FRCC Bulk-Power Electric System NERC Reliability Coordinator Readiness Audit Report, dated March 8-10, 2005 FRCC Agent Operational Audit Report, May 15, 2005 FRCC Operating Reliability Subcommittee Recommendations / Comments to the FRCC Agent Operational Audit Report, dated April 18, 2007 RC Agent updates on previous NERC Readiness Audit recommendations

7 Subsequently, the AATF developed a pre-audit questionnaire requesting specific information which illustrated how the Agent performs certain aspects of the Real-Time Operating Function as well as the State Capacity Emergency Coordination Function. The pre-audit correspondence also included a request to PEF, to have personnel available during the on-site audit at the FPL facility to address the Operations Planning Function as well as a request to FPL to provide a transition plan detailing how FPL will assume responsibility for the Operations Planning Function starting in The AATF also solicited input regarding Agents performance from the only adjacent Reliability Coordinator to the FRCC, the Southeastern RC. It was noted by the adjacent RC that the two RCs maintain sufficient monitoring capabilities to ensure potential reliability violations are identified within their general interface areas and that the RCs have adequate procedures in place to share information regarding potential, expected, or actual critical operating conditions that could negatively impact each other. No specific issues or concerns were noted by the adjacent RC. The AATF met for a coordination conference call meeting prior to the on-site meeting. During this call the team developed additional questions and requests for clarifications based on the pre-audit questionnaire responses and document reviews. The AATF provided a final data request so that the Agent could be prepared to provide clarifications during the on-site audit. Finally, the AATF met as a team, prior to the onsite audit to review and finalize the audit agenda and logistics and review an audit guide that details some of primary aspects of the Security Process. The guide was developed to provide the AATF a quick reference sheet and as an aid to the auditing process. The on-site audit took place on November 8 th, 2007, at the FPL control center in Miami, Florida. As noted earlier, the PEF Operations Planning function representative was present at the on-site audit location. 6.0 Observations and Discussions The AATF Chair appreciates the efforts of all involved. Both the AATF members as well as the Agents were very accommodating and essential in helping the AATF achieve their purpose. In spite of a compressed and aggressive schedule, and the limited number of task force members, the AATF was able to perform the requisite audit, on-schedule, while minimizing impacts to those involved with the process The AATF also recognizes and expresses its appreciation to the FRCC agents for their openness, honesty, and responsiveness in answering some of the AATF questions. This served as both, a useful review of the Security Process for the Agents, as well as a good training and information sharing opportunity for AATF members. In light of some of the discussions regarding mandatory Reliability Standards and the potential obsolescence of the Agent Audit process, it was noted at the conclusion of the on-site audit, that the overall Agent audit process had, in fact been a beneficial process to the

8 overall reliability of the FRCC and should be continued in the future, regardless of the developing Regional Compliance Programs. The on-site audit began with an orientation presentation by FPL. FPL summarized the roles of the various personnel in the control center as well as how those personnel interact with the rest of the FRCC operating community. Next, a question and answer session with key Reliability Coordination infrastructure support personnel reviewed and clarified Agent questionnaire responses as well as addressed emerging AATF questions and concerns that dealt with several areas, including facilities and training. The AATF was then provided a tour of all the facilities and infrastructure that support the FRCC Reliability Coordinator functions. The tour included the control room, various critical infrastructure protection systems as well as the computer and communication infrastructure for not only FPL but the FRCC RC personnel, as well. One of the concerns from previous audits was the fact that FPL provides its own Backup capability for the Reliability Coordinator function. This concern was expressed in several recommendations from past RC audits and reviews. As a result of this, the AATF opted to focus in on this area by providing the Agent with a specific scenario that would require initiation of Back-up RC function, and review the current plans and capabilities in place. The AATF offered up a scenario and the Agent responded accordingly. The AATF witnessed the transition and within minutes the FRCC RC applications could be monitored in a completely remote location from the control center while personnel would have been dispatched to the permanent back-up facilities. Next, the AATF met independently to discuss any other areas of concern which may have been discovered during the morning s audit sessions and to begin developing any recommendations for the Agents. The afternoon session began with the PEF Operations Planning overview and was followed by a presentation of the Operations Planning transition plan from PEF to FPL. The plan included introduction of FPL personnel, transition efforts to date, planned parallel OPC functions during December with FPL assuming full responsibility of the Operations Planning function starting in January. The AATF initiated a second question and answer session and requested clarification on a few responses and RC processes. Finally, the AATF met independently to discuss conclusions of the task force as well as any recommendations that the task force concluded were appropriate to be provide to the Agents. During this meeting, the task force developed an exit briefing which was provided to the Agents prior to completion of the audit. The presentation reviewed the AATF audit process and scope used for this particular audit, a preliminary list of recommendations that may be included in the AATF final report and the next steps of the Agent audit report development and review cycle at FRCC. A tentative report development schedule was provided to the AATF and Agents and the on-site portion of the audit was concluded As a result of all pre-audit document review and activities as well as the conclusions of the on-site audit, the AATF found no problems or concerns related to the FRCC Agent s performance of their functions and fulfillment of their responsibilities. It was apparent to the task force that both FPL and PEF take their FRCC Agent responsibility

9 seriously and work to perform the functions in the most efficient and effective manner possible. As part of the audit scope and preparation was the review of the previous Agent Audit report as well as the previous NERC RC Readiness Audit report, the task force was pleased to note that all of the issues and recommendations identified in the past reports have been, or are being addressed by both the Agents and the FRCC Operating Reliability subcommittee

10 7.0 Recommendations The audit reviewed both; the agents performance of the Security Coordination functions, and the FRCC Security Process. Procedurally a review of the Security Process is required to determine if any changes to the Security Process are warranted in support of FRCC s reliability goals. With the evolution of mandatory and enforceable Reliability Standards and FRCC s role and obligations as an Eastern Interconnection Reliability Coordinator, review of both the Security Process and the Monitoring Procedure are timely and critical elements of the recommendations of this agent operational audit. Overall, the task force found no deficiencies in the agent s performance; however, the task force presents the following recommendations to the FRCC members in order to enhance and improve the quality of the reliability processes of the Region. 1. Update and revise, as appropriate, the FRCC Security Process for the FRCC Bulk-Power Electric System document, to ensure it adequately addresses the NERC Reliability Standards requirements for Reliability Coordinators. The revision should also specifically address the following: a. Replace Security with Reliability b. Replace or re-define Operating Entity 2. Retain the Agent Audit process and update the FRCC Monitoring Procedures for Agents document to reflect the revisions in item 1 above. Revisions should also consider using the monitoring procedure as the first step to developing an internal FRCC Reliability Coordinator compliance program that addresses mandatory RC Reliability Standards as well as a more formal, internal RC, continuous improvement program for the Region. 3. Review and revise Reliability Coordinator training criteria as required. a. Review should document how appropriate Standards of Conduct training requirements are achieved by the Agents. 4. ORS should update and revise as necessary, the Security Coordinator Qualifications handbook document. The revisions should address any specific training requirements for FRCC Reliability Coordinators as well as training requirements on the next-day Reliability Coordinator. 5. Update Agent Agreements to reflect current Reliability Standards terminology. a. Including specific training criteria within the agreements should be considered. 6. Execute an Operations Planning Function Agreement with FPL to reflect the transition of the function over to FPL. 7. Real-time Reliability Coordinator function should fully implement dynamic line rating transfers. All Transmission Owners line ratings should be provided to the

11 RC applications in a consistent and uniform manner that does not rely on manual updates by RC EMS support personnel. 8. Review RC communication requirements to registered entities and how to incorporate contact information into the RC and host BA / TOP processes (ie. IPPs). 9. Review the IPP data (outage and capacity) information and how data is incorporated into, or impacts the OPC process. 10. Although the task force was unable to view the progress on the FPL Document Management system being developed (planned completion, spring 2008), the task force recommends that ORS track implementation of the planned system to ensure that any impacts on RC documents and personnel are appropriately addressed (ref. NERC Readiness Audit Report, March 8-10, 2005, Recommendation 1). 11. ORS should update, and revise as necessary, the FRCC IROL list. The revisions should take into account the current developments occurring at the NERC level regarding the definition of IROLs as well as IROL methodologies

12 Appendix A Final Agenda Agent Audit Task Force Audit (on-site portion of audit process)

13 Final Agenda Agent Audit Task Force Audit Wednesday, November 7, :00 p.m. AATF pre-audit meeting to finalize agenda and formulate audit plan Review with AATF major points of Agent Audit Review AATF audit guide / checksheet Select back-up scenario List any specific AATF issues to be included in audit 8:00 a.m. Introduction of AATF / FPL Review logistics Additional Information requests Thursday, November 8, :30 a.m. Overview and walk-through tour of facilities Position descriptions / roles of personnel Relationships of positions RC Communications to FPL RC real-time function Review Statewide overview RC next-day planning function Review of communications infrastructure Review of EMS infrastructure Review of RC Modeling functions 10:30 a.m. RC Back-up Control Center Drill scenario 12:00 p.m. Lunch 1:00 p.m. Review of OPC functions PEF OPC Representative OPC Transition plan FPL OPC plan 2:30 p.m. AATF discussion on issues / open items / additional information 3:00 p.m. FPL additional question session 3:30 p.m. AATF conclusion summary session 4:00 p.m. Exit briefing with FPL and PEF

14 Appendix B 2007 FRCC Agent Audit Participants Herb Evelyn Jeff Gooding Don McInnis Pedro Modia Andy Pankratz Gilbert Perez Art Pique Frank Prieto Juan Quintana Ritha Ramos Bill Slater Kaveh Tarighy Dick Wielandt Miguel Yanes Manager, FPL Computer Operations Manager, FPL Training & Certification Manager, FPL Operations Engineering Director, FPL Power Supply Manager, FPL System Operations Load Dispatch Supervisor FPL System Operator FPL Manager, FPL System Operations Transmission 2008 SCEC Agent (FPL) Supervisor FPL 2007 OPC Agent (PEF) Supervisor FPL System Operator FPL 2008 OPC Agent (FPL) FRCC AATF members (present on-site) Eric Senkowicz Herman Dyal Joe McKinney Richard Gilbert Doug Jensen Donna Howard FRCC Staff - Chair Clay Electric Cooperative Load Serving Entity Sector Florida Municipal Power Agency Non-Investor Owned Utility Wholesale Sector Lakeland Electric Generating Load Serving Entity Sector Northern Star Generation Services Suppliers Sector FRCC (Observer)