PER System Personnel Training ERO Auditor Workshop. Pete Knoetgen, Director of Training September 20, 2012

Transcription

1 PER System Personnel Training ERO Auditor Workshop Pete Knoetgen, Director of Training September 20, 2012

2 Agenda Purpose of the standard Requirements and compliance approach from RSAW Frequently asked questions 2 RELIABILITY ACCOUNTABILITY

3 Learning Objectives Summarize the purpose and the three main requirements of PER from memory. Describe common elements of a systematic approach to training. Given the PER-005 RSAW, be able to describe the evidence an auditor would review during the conduct of a compliance audit. When asked, be able to discuss responses to frequently asked questions. 3 RELIABILITY ACCOUNTABILITY

4 PER-005 Reliability Standard Strengthen operator training for reliability PER Purpose To ensure that System Operators performing real-time, reliability-related tasks on the North American Bulk Electric System (BES) are competent to perform those reliabilityrelated tasks. The competency of System Operators is critical to the reliability of the North American BES o Use of a systematic approach to training o Verification of a System Operator s ability to perform the tasks o Recurring emergency operations training 4 RELIABILITY ACCOUNTABILITY

5 PER-005 RSAW Development Piloted new approach for developing RSAW RSAW provides methodology to assess compliance with the requirements of the Reliability Standard; should not be treated as a substitute for the Standard or viewed as additional requirements Specific facts, circumstances always assessed for compliance Input and comments from: Personnel Subcommittee (Operator Training SMEs) Standards Drafting Team Industry at large Regional Entity and NERC review Posted at 5 RELIABILITY ACCOUNTABILITY

6 Implementation R1 systematic training program April 1, 2013 R2 task capability verification April 1, 2013 R3 32 hours of emergency training April 1, 2011 R3.1 use of simulation technology April 1, RELIABILITY ACCOUNTABILITY

7 PER-005 R1 - SAT R1.1 Develop and Review Task List R1.2 LOs and Materials R1.3 Deliver Training R1.4 Evaluate Training Program R2 Capability Verification R2.1 New Task Verification R3 EOPs Training Hours R3.1 Simulation If, Then.. 7 RELIABILITY ACCOUNTABILITY

8 R1. R1. use a systematic approach to training to establish a training program for the BES company-specific reliabilityrelated tasks performed by its System Operators and shall implement the program. M1. have available for inspection evidence of using a systematic approach to training to establish and implement a training program, as specified in R1. 8 RELIABILITY ACCOUNTABILITY

9 R1. Compliance Approach Approach for R1 An Entity should have the following: A training program for BES company-specific reliability-related tasks established using a systematic approach to training. A description of the systematic approach to training (SAT) process the entity uses. Training program that includes Initial and continuing training. Training staff that is competent in knowledge of both system operations and instructional capabilities. Evidence of implementation of the training program, such as training records. 9 RELIABILITY ACCOUNTABILITY

10 R1. FAQ Q - What is meant by a systematic approach to training? A - A systematic approach to training is a widelyaccepted methodology that ensures training is efficiently and effectively conducted and is directly related to the needs of the position in question. (FERC Order No. 742 para 25). 10 RELIABILITY ACCOUNTABILITY

11 Q - Is there an approved systematic approach to training model? R1. FAQ A - There is not a specific model of SAT required by the standard. However, the Department of Energy s Systematic Approach to Training methodology described in the DOE Handbook (DOE-HDBK , Training Program Handbook: A Systematic Approach to Training) is a particularly good and relevant model to use, BUT IS NOT REQUIRED to be used. andard.html 11 RELIABILITY ACCOUNTABILITY

12 What is SAT? Systematic Approach to Training Planned, documented, repeatable, logical, structured Method, process, management, tactic, attitude Instruction that achieves desired levels of performance Used for performance based training Focused on what the performer needs to do o Knowledge, skills to perform a task to meet a desired outcome Provides a plan or roadmap that describes the training process to achieve performance 12 RELIABILITY ACCOUNTABILITY

13 Common Elements - ADDIE Analysis- identifying training/performance requirements. The major output of the analysis phase is a task list. Design - provides a blueprint for developing training. The output is learning objectives. Develop - incorporates the results of design activities. The outputs are completed training materials and training aids. Implementation - conducting the training, including testing knowledge and skills. The output is trained personnel. Evaluation - periodic review of training program and feedback on the effectiveness of training. The outputs are decisions to improve the training program. 13 RELIABILITY ACCOUNTABILITY

14 PER-005 and ADDIE A - Create a list of BES company-specific reliabilityrelated tasks (R1.1) D - Design learning objectives (R1.2) D - Develop training materials based on the task list (R1.2) I - Deliver the training (R1.3, R3); verify operator s capabilities to perform each assigned task (R2) E - Conduct an annual evaluation of the training program to identify changes to the training program (R1.4) 14 RELIABILITY ACCOUNTABILITY

15 R1. FAQ Q What is a real-time reliability related task? A The standard does not define a real-time reliability related task. Entities must identify tasks performed by their system operators that can impact their system s reliability. An entity could consider real-time operator tasks that are associated with NERC Reliability Standards, company processes or operating procedures that impact the BES, or Functional Model activities. Generic System Operator task lists, while not required to be used, are available for reference in the PER-005 archives under Version 1 at Training.html 15 RELIABILITY ACCOUNTABILITY

16 R1. FAQ Q Is continuing training required by PER-005? A Yes. FERC Order 742 paragraph 34 states: continual training is a fundamental part of a systematic approach to training and an enforceable requirement under PER-005-1, we find that any systematic approach to training, including the systematic approach to training mandated by Reliability Standard PER-005-1, would entail continual training to refresh system operators knowledge and to cover any new tasks relevant to the operation of the bulk power system. 16 RELIABILITY ACCOUNTABILITY

17 R1. FAQ Continuing training is required: To provide 32 hours of emergency operations training for each operator. To train and verify capability to perform any new or modified tasks. As determined by the entity to refresh system operator knowledge. Entities should include in their training program how they systematically determine and deliver continuing training on reliability related tasks for their operators. DOE Guide to Good Practices for Continuing Training HDBK may be useful in determining continuing training. 17 RELIABILITY ACCOUNTABILITY

18 R1. FAQ Q Are instructors included in the systematic approach to training? A Yes. FERC Order No. 742 Paragraph 38, Commission determination: the commission concludes that the current requirement for each training program (that training staff must be identified and that such staff must be competent in both knowledge of system operations and instructional capabilities) is inherent in any systematic approach to training that a registered entity would use to meet this requirement, and thus is an enforceable component of R1. 18 RELIABILITY ACCOUNTABILITY

19 R1.1. R1.1. create a list of BES company-specific reliabilityrelated tasks performed by its System Operators. M1.1 have available for inspection its companyspecific reliability-related task list, with the date of the last review and/or revision, as specified in R RELIABILITY ACCOUNTABILITY

20 R1.1. Compliance Approach Approach for R 1.1 Develop/provide a list of company-specific real-time reliability-related tasks performed by its System Operators. The training program should include an entity defined method for selecting or determining a reliabilityrelated task. 20 RELIABILITY ACCOUNTABILITY

21 R1.1. FAQ Q What are common characteristics of a task statement? A A task statement should state the action required to be performed by an operator. There are many approaches to developing task statements. Common elements of a task statement may address the following: Condition Action Criteria for Performance A task is an identifiable, observable and measurable piece of a job that serves as a unit of work. A Task is a discrete unit of work. It is an action to contribute a specified end result to the accomplishment of an organizational objective. It has an identifiable beginning and end which is a measurable component of a specific job. MRO Application Guide 21 RELIABILITY ACCOUNTABILITY

22 Task Examples Initiate manual load shedding to prevent imminent separation from the Interconnection, voltage collapse, or other adverse consequence. Direct balancing authorities to take actions to mitigate an IROL. Following the activation of automatic load shedding schemes, restore system load as appropriate for current system conditions and in coordination with adjacent systems. Deploy reactive resources to maintain acceptable voltage profiles. Manually calculate ACE. Prepare for a capacity emergency by bringing on all available generation. 22 RELIABILITY ACCOUNTABILITY

23 R R update its list of BES company-specific reliability-related tasks performed by its System Operators each calendar year to identify new or modified tasks for inclusion in training. 23 RELIABILITY ACCOUNTABILITY

24 R Compliance Approach Approach for R1.1.1 Verify the entity has the following: Reviewed the Task List each calendar year. Document the review by including such items as a revision date or a table of revisions. An updated task list that includes any new or modified tasks identified in the review. 24 RELIABILITY ACCOUNTABILITY

25 R FAQ Q What constitutes a modified BES company-specific reliability-related task? A A change to a task that requires different performance, or additional knowledge, should be considered a modified task and requires an update to the task list. If a change to a task is, for example, simply rewording for clarification, and it does not affect the task performance or knowledge requirements, it should not be considered as a task modification that would require training. 25 RELIABILITY ACCOUNTABILITY

26 R1.2. R1.2. design and develop learning objectives and training materials based on the task list created in R1.1. M1.2. shall have available for inspection its learning objectives and training materials, as specified in R RELIABILITY ACCOUNTABILITY

27 R1.2. Compliance Approach Approach for R1.2 Verify the entity has the following: For reliability-related task training that has been delivered, ensure it contains: Learning objectives based on the Task List. Training materials containing the learning objectives based on the Task List. Note: The entity training program should address design and development of training materials consistent with this requirement. Learning objectives and training materials should be developed when training on the task is being planned for delivery. 27 RELIABILITY ACCOUNTABILITY

28 R1.2. FAQ Q How do I show learning objectives are based on the task list? A Training materials, such as lesson plans or presentation materials for a task should include the learning objectives. A task list with associated learning objectives, sometimes called a task-to-training matrix is a simple, direct method for linking tasks, objectives and training materials. 28 RELIABILITY ACCOUNTABILITY

29 R1.2. FAQ Another method is to construct the tasks as learning objectives. Determine the following for the task: Condition Action Criteria for successful performance This is also the format for a terminal objective. The knowledge and skills that are needed to perform the task would be enabling objectives. 29 RELIABILITY ACCOUNTABILITY

30 R1.3. R1.3. shall deliver the training established in R1.2. M1.3 shall have available for inspection System Operator training records showing the names of the people trained, the title of the training delivered and the dates of delivery to show that it delivered the training, as specified in R RELIABILITY ACCOUNTABILITY

31 R1.3. Compliance Approach Approach: Verify the entity has the following: Delivered training designed and developed as required in R1.2. Documented the delivery in training records with names, dates of delivery, title of training. Note: Training on a task is not required to be delivered to a System Operator who is documented (R2) to already be capable to perform that task, unless it is identified by the entity for continuing training. 31 RELIABILITY ACCOUNTABILITY

32 R1.3. FAQ Q Does training need to be delivered for every task by the implementation date? A Re-training on a task is not required for a System Operator who is documented at least one time, to already be capable to perform the task. Training should be delivered systematically when needed: For task performance verification For continuing training to refresh knowledge and skills as determined by the entity s SAT process in their training program 32 RELIABILITY ACCOUNTABILITY

33 R1.4 shall conduct an annual evaluation of the training program established in R1, to identify any needed changes to the training program and shall implement the changes identified. M1.4 have available for inspection evidence (such as instructor observations, trainee feedback, supervisor feedback, course evaluations, learning assessments, or internal audit results) that it performed an annual training program evaluation, as specified in R1.4 R RELIABILITY ACCOUNTABILITY

34 R1.4. Compliance Approach Approach for R1.4 Verify the entity has the following: An annual (see CAN 0010 definition of annual ) training program evaluation. Identify needed changes, if any. Implemented or be in the process of implementing the needed changes. If not implemented, is following its plan for implementation. 34 RELIABILITY ACCOUNTABILITY

35 Q What s needed to do an annual evaluation? R1.4. FAQ A The purpose of conducting an annual evaluation is to determine if any changes are needed to the training program. Consider several sources for feedback on effectiveness of the training program, including: Trainee feedback Supervisor feedback on operator/trainee performance Instructor and training observations Course evaluations Internal audit results Other measures of operator performance BES events DOE Guide To Good Practices - Evaluation Instrument Examples HDBK may be helpful in conducting annual evaluations. 35 RELIABILITY ACCOUNTABILITY

36 R1.4. FAQ Q When do I have to implement any needed changes? A The entity should determine when a needed change should be implemented based on its significance. Changes that are not promptly implemented should be scheduled, tracked, or documented in some systematic way that indicates it is following a plan for implementation. 36 RELIABILITY ACCOUNTABILITY

37 PER-005 R1 - SAT R1.1 Develop and Review Task List R1.2 LOs and Materials R1.3 Deliver Training R1.4 Evaluate Training Program R2 Capability Verification R2.1 New Task Verification R3 EOPs Training Hours R3.1 Simulation If, Then.. 37 RELIABILITY ACCOUNTABILITY

38 R2. R2. verify each of its System Operator s capabilities to perform each assigned task identified in R1.1 at least one time. M2. have available for inspection evidence to show that it verified that each of its System Operators is capable of performing each assigned task identified in R1.1, as specified in R2. This evidence can be documents such as Training records showing successful completion of tasks with the employee name and date; Supervisor check sheets showing the employee name, date, and task completed; or The results of learning assessments. 38 RELIABILITY ACCOUNTABILITY

39 R2. Compliance Approach Approach for R2 Verify the entity has: A list of company-specific reliability-related tasks performed by the System Operators (R1.1). A list of applicable System Operators with their assigned tasks. At least one time, verified each System Operator s capability to perform each assigned task. 39 RELIABILITY ACCOUNTABILITY

40 R2. FAQ Q How should an operator s capability to perform a reliability related task be verified? A Each System Operator s capabilities to perform each assigned real-time reliability-related task identified in R1.1 must be verified at least one time. Verification should be based on performance criteria. Documentation should include: System Operator s name Date verification was conducted Real-time reliability-related task that was verified Evidence of the System Operator s capabilities Memos to the file stating, for example, that an operator has x years on shift and is capable of performing required tasks, is not systematic and most likely not adequate without additional task specific documentation. Guide to Good Practices for On-the-Job-Training, DOE-HDBK may be helpful in conducting task verifications. 40 RELIABILITY ACCOUNTABILITY

41 R2.1. R2.1. Within six months of a modification of the BES company-specific reliability-related tasks, verify each of its System Operator s capabilities to perform the new or modified tasks. 41 RELIABILITY ACCOUNTABILITY

42 Approach for R2.1 R2.1. Compliance Approach Verify each System Operator s capabilities to perform new or modified tasks, as identified in R1.1.1, within six months of a documented new or modified BES company-specific real-time reliability-related task. Document the task verification using methods as in R2. If a change to a task is, for example, simply rewording for clarification, and it does not affect the task performance or knowledge requirements, the change is not considered a modified task. 42 RELIABILITY ACCOUNTABILITY

43 PER-005 R1 - SAT R1.1 Develop and Review Task List R1.2 LOs and Materials R1.3 Deliver Training R1.4 Evaluate Training Program R2 Capability Verification R2.1 New Task Verification R3 EOPs Training Hours R3.1 Simulation If, Then.. 43 RELIABILITY ACCOUNTABILITY

44 R3. R3. At least every 12 months provide each of its System Operators with at least 32 hours of emergency operations training applicable to its organization that reflects emergency operations topics, which includes system restoration using drills, exercises or other training required to maintain qualified personnel. M3. have available for inspection training records that provide evidence that each System Operator has obtained 32 hours of emergency operations training, as specified in R3. 44 RELIABILITY ACCOUNTABILITY

45 R3. Compliance Approach Approach for R3 Verify that the entity has: Provided each operator with 32 hours of emergency operations training at least every 12 months. Defined every 12 months in the training program May use the annual definition in CAN-0010 as the basis for delivering training. Should not have a pattern of intervals significantly greater than 12 months. Training that is applicable to it s organization. For personnel who became System Operators after the entity provided its annual emergency operations training, the System Operator must fulfill the requirement for 32 hours of training within 12 months. 45 RELIABILITY ACCOUNTABILITY

46 R3. FAQ Q What is the definition of emergency operations training and what topics would be acceptable? A For reference, the NERC Glossary of Terms defines Emergency as Any abnormal system condition that requires automatic or immediate manual action to prevent or limit the failure of transmission facilities or generation supply that could adversely affect the reliability of the Bulk Electric System. The NERC System Operator Certification Manual Appendix A Recognized Operator Training Topics includes a list of recognized emergency operations training topics for credential maintenance, which may be used as a guide. The training should be applicable to the entity s organization and include restoration using drills, exercises or other training methods. 46 RELIABILITY ACCOUNTABILITY

47 R3. FAQ Q How often do operators need emergency operations training? A At least every 12 months. The 12 months should be defined by the entity, and can use annual methodology. For personnel who became System Operators after the entity provided its annual emergency operations training, the System Operator must fulfill the requirement for 32 hours of training within 12 months. 47 RELIABILITY ACCOUNTABILITY

48 R3.1. R3.1. Each that has operational authority or control over Facilities with established IROLs or has established operating guides or protection systems to mitigate IROL violations shall provide each System Operator with emergency operations training using simulation technology such as a simulator, virtual technology, or other technology that replicates the operational behavior of the BES during normal and emergency conditions. M3.1 Each shall have available for inspection training records that provide evidence that each System Operator received emergency operations training using simulation technology, as specified in R RELIABILITY ACCOUNTABILITY

49 R3.1. Recommended Approach R3.1 applies to the entity based on whether the entity has one or more of the conditions in the requirement: Operational authority or control over facilities with established IROLs, or Established operating guides to mitigate IROL violations, or Protection systems to mitigate IROL violations. If none of the above applies, R3.1 IS NOT APPLICABLE. 49 RELIABILITY ACCOUNTABILITY

50 R3.1. Compliance Approach If any of the above applies to the entity, each System Operator must be provided emergency operations training using simulation technology. Verify that the entity has: Documented the training in Training Records for each System Operator. Documented that the training provided used simulation technology such as a simulator, virtual technology, or other technology that replicates the operational behavior of the BES during normal and emergency conditions. Ensure some training has been conducted as required by R3.1; however, there is no minimum number of hours required using simulation technology. Emergency operations training conducted to meet R3.1 may be counted toward the hours required under R3. 50 RELIABILITY ACCOUNTABILITY

51 R3.1. FAQ Q Is a company specific simulator required to meet R3.1? A No, use of a company specific simulator or simulators specific to an operator s own system is not required. FERC Order 742 para 55: PER-005-1, Requirement R3.1 does not require the use of simulators specific to an operator s own system. While the Commission continues to feel there is value in using custom simulators, we acknowledge that NERC and industry have determined that it is not necessary 51 RELIABILITY ACCOUNTABILITY

52 The purpose of PER-005 is? Do you know? How often do SO s need emergency ops training? A S.O. completes 32 hours of EO training on January 31, He next completes 34 hours of EO training on March 9, Is this a non-compliance? Why? Joe is designated a Transmission Operator on June 1, His company had an established task list and a performance checksheet for each task is in his training record from The current task list has not changed. Does Joe need to be trained again on each task in 1Q 2013? 52 RELIABILITY ACCOUNTABILITY

53 Do you know? ABC Transco TOP serves 700 MW of load and has an operating staff of 10. Their task list contains 8 tasks. XYZ TOP has identified 45 tasks. Is there a problem with ABC s task list? Why or why not? Joe is fully qualified on the TOP desk. A new EMS system will be installed on June 15, Does Joe need any training? Does anything need to be verified? By what date? Nancy is designated a S.O. on shift beginning Thanksgiving Day By when does she need to participate in 32 hours of EO training? 53 RELIABILITY ACCOUNTABILITY

54 PER-005 R1 - SAT R1.1 Develop and Review Task List R1.2 LOs and Materials R1.3 Deliver Training R1.4 Evaluate Training Program R2 Capability Verification R2.1 New Task Verification R3 EOPs Training Hours R3.1 Simulation If, Then.. 54 RELIABILITY ACCOUNTABILITY

55 Summary The purpose is to ensure that System Operators performing real-time, reliability-related tasks on the North American BES are competent to perform those reliability-related tasks. Approach training as an entity developed, systematic process. Specific facts, circumstances always assessed for compliance. 55 RELIABILITY ACCOUNTABILITY

56 Questions? 56 RELIABILITY ACCOUNTABILITY

57 Internal Controls Exercise Purpose of exercise is to reinforce internal control concepts using PER-005 and CIP Breakout Groups O&P groups and CIP groups Smaller teams within each discipline Use fictional scenario about a registered entity s compliance activities to assess internal controls Each team to develop five responses: Internal controls, classification (prevent, detect, correct), and COSO component Opportunities for additional internal controls or enhancements Potential audit tests for internal controls Summarize your assessment of the entity s control environment How would entity internal controls affect audit scope 57 RELIABILITY ACCOUNTABILITY

58 Internal Controls Exercise We ll be available to answer questions as you work. Reconvene to discuss responses. Teams selected to lead discussion for each response. 58 RELIABILITY ACCOUNTABILITY

59 BREAKOUT ROOMS Main Classroom Elev. Lobby Rest rooms EXIT Kitchen Area Front Desk Reception Area 59 RELIABILITY ACCOUNTABILITY

60 Breakout Timeline Internal Controls Exercise 11:50-12:00 Form breakout teams and room pre-assignments. 12:00-1:00 1:00-2:00 2:00-2:30 2:30-3:00 Lunch Teams work on exercise. Reconvene; teams report results. Wrap-up, lessons learned. 60 RELIABILITY ACCOUNTABILITY

61 Internal Controls for NERC Standards Compliance Group Exercise PER-005-1, System Personnel Training Instructions Read the NERC standard and requirements along with the narrative example of an entity s internal control environment related to the standard provided below. Ignoring all other NERC standards and requirements beside PER-005-1, execute the following procedures on the provided worksheets: 1. Identify the specific internal controls described, the COSO component of internal control (Control Environment, Risk Assessment, Control Activity, Information and Communication, and Monitoring), and the control classification (i.e. preventive, detective, corrective). 2. List opportunities for either additional internal controls that are not discussed in the narrative or to enhance the internal controls identified in Step Identify audit procedures for the Compliance Enforcement Authority (CEA) to test the effectiveness of the entity s controls identified in Steps 1 and Assuming all the internal controls identified were favorably tested by the auditor, assess the entity s internal control environment using one of the maturity definitions provided. 5. Based on the results of the internal controls testing, what are some ways the auditor may want to adjust the scope of compliance testing and why? PER Standard Purpose and Requirements To ensure that System Operators performing real-time, reliability-related tasks on the North American Bulk Electric System (BES) are competent to perform those reliability-related tasks. The competency of System Operators is critical to the reliability of the North American Bulk Electric System. R1. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall use a systematic approach to training to establish a training program for the BES company-specific reliability-related tasks performed by its System Operators and shall implement the program. R1.1. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall create a list of BES company-specific reliability-related tasks performed by its System Operators. R Each Reliability Coordinator, Balancing Authority and Transmission Operator shall update its list of BES company-specific reliability-related tasks performed by its System Operators each calendar year to identify new or modified tasks for inclusion in training. R1.2. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall design and develop learning objectives and training materials based on the task list created in R1.1. R1.3. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall deliver the training established in R1.2.

62 R1.4. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall conduct an annual evaluation of the training program established in R1, to identify any needed changes to the training program and shall implement the changes identified. R2. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall verify each of its System Operator s capabilities to perform each assigned task identified in R1.1 at least one time. [Violation Risk Factor: High] [Time Horizon: Long-term Planning] R2.1. Within six months of a modification of the BES company-specific reliability-related tasks, each Reliability Coordinator, Balancing Authority and Transmission Operator shall verify each of its System Operator s capabilities to perform the new or modified tasks. R3. At least every 12 months each Reliability Coordinator, Balancing Authority and Transmission Operator shall provide each of its System Operators with at least 32 hours of emergency operations training applicable to its organization that reflects emergency operations topics, which includes system restoration using drills, exercises or other training required to maintain qualified personnel. [Violation Risk Factor: Medium] [Time Horizon: Long-term Planning] R3.1. Each Reliability Coordinator, Balancing Authority and Transmission Operator that has operational authority or control over Facilities with established IROLs or has established operating guides or protection systems to mitigate IROL violations shall provide each System Operator with emergency operations training using simulation technology such as a simulator, virtual technology, or other technology that replicates the operational behavior of the BES during normal and emergency conditions. Description of Entity s Internal Controls for Compliance with PER Requirements XYZ utility is a NERC registered Reliability Coordinator, Balancing Authority and Transmission Operator. They have one secure control center that houses all operational staff. All three operational functions are staffed continually by a total of three operators working 12 hour shifts. XYZ has 12 system operators that staff the shifts. In order to ensure that XYZ has competent staff, they have always required their operating personnel to hold an active NERC RC certification. The responsibility for arranging the training necessary to maintain their NERC RC has been historically placed in the hands of the operators themselves, and XYZ did not play an active role in the training of its operating personnel. However, in the third quarter of 2012, in order to comply with the pending PER NERC requirements becoming effective in 2013, XYZ developed a documented training program. The program was developed using an ad hoc working group that met with all operating personnel and XYZ management to determine the necessary tasks for XYZ s reliability, balancing, and transmission operating personnel. Once the tasks were determined, the working group was tasked with the development of a training program that encompassed all the needed skills and related training requirements. In order to gain experience in developing training, the working group ranked all tasks by complexity and began creating programs beginning with those tasks that were deemed least complex. Each task was assigned a fixed and equal period of time for training development to ensure that all tasks had developed training by the implementation date of PER PER-005 2

63 The program follows the requirements outlined in PER Once a year the ad hoc working group reconvenes to re-evaluate XYZ s operator training program by conducting or repeating interviews, updating the list of critical tasks, and updating the training program to reflect the changes required. If edits are made to the training program document, a revised document replaces the legacy one, but XYZ does not track revisions to the policy. All operators were provided a copy of the training program via upon its completion and the official copy was filed with an administrative assistant assigned to the operation s group. XYZ develops training materials for each task, as mandated by its program, and each of its operators are required to attend scheduled classes, including an emergency operations workshop, on an annual basis. The administrative assistant keeps training records and a log of the operators that completed the required training. The administrative assistant reminds the operators of their training requirements and notifies them of their need to make up training when they miss classes. XYZ is a small utility and does not have an independent Internal Compliance Program (ICP). One engineer in the reliability coordination group is assigned the role of NERC Compliance Engineer (along with his other responsibilities as a reliability coordinator) and verifies XYZ s compliance with all NERC standards. That individual will verify XYZ s compliance with PER for its annual self-certification for compliance with the standard and upcoming compliance audits performed by its Regional Entity. Verification consists of the Compliance Engineer performing a mock audit on all of PER where a sample of operators will be selected to determine if they have the requisite training and certification. The NERC Compliance Engineer updates XYZ management regarding compliance topics on a quarterly basis. PER-005 3

64 Participant Identification and Classification of Entity Internal Control Activities and Opportunities for Additional Internal Controls 1. Internal controls noted in narrative. No. Internal Control Description Classification (prevent, detect, correct) COSO Component 1 Requiring operators to have active NERC-RC certification Prevent CA/CE 2 Training of all system operators Prevent CA/CE 3 Working group annual evaluation of training program Prevent M 4 Maintenance of a written training policy Prevent CA 5 Admin Assistant monitoring of training program Detect CA 6 Compliance Engineer mock audit of PER-005 Detect CA, M 7 Quarterly compliance update to management by engineer Detect IC, CE 2. Opportunities for additional internal controls or enhancements of internal controls identified in Step 1. No. Internal Control Description Classification (prevent, detect, correct) COSO Component 8 Should prioritize training based on risk of task to BES Prevent RA 9 ICP should have a degree of independence Detect M/CE 11 Enforceability of training requirements by admin Prevent CE 12 Training policy revisions should be documented Prevent CE 13 More frequent monitoring by ICP Prevent CE/IC 14 Operators should sign policy acknowledgement Prevent CE/IC/CA 15 More proactive related to pending NERC Standard Prevent CE 3. Participant Identification of Tests of Internal Controls No. All All Description of Tests of identified internal controls to be applied by CEA Obtain/create documentation of all XYZ processes related to PER-005 to obtain understanding of controls by speaking with SMEs and obtaining process walk-througs. Review NERC events, minutes of management meetings where compliance is discussed, mock audit reports performed by XYZ, org charts, and compliance mission statements. 5 Test administrative assistant s monitoring of operator training. 7 Verify that management updated provided as described in narrative by obtaining meeting minutes. 1,2,5 Sample operators to check for RC certification and training requirements. 3 Evidence of working group s annual policy review (i.e. revision history, s, presentations). 4 Review training policy against NERC requirements. PER-005 4

65 4. Participant Overall Assessment of Entity Internal Controls Based on the internal controls that the registered entity has in place as described in the narrative, provide an assessment of the registered entity s control environment related to the PER Do not consider any opportunities for additional controls or your suggestions for improving deficiencies, when making your assessment. It should be based only on the information described in the narrative. When you are ready to make your assessment, circle one of the following designations that best describes the entity s control environment related to the example standard. Be prepared to justify your assessment. 1 Instances of results, mainly ad hoc framework 2 Repeatable actions with some documentation 3 Defined programs with procedures and documentation 4 Managed effectively with metrics and controls 5 Optimized for performance and efficiency 5. Adjustments to Audit Scope Based on Registered Entity s Internal Controls 1) Since all operating task, regardless of complexity, were assigned equal time (and the fact that XYZ did not even begin their compliance initiative until the 3 rd quarter of 2012) for training development, CEA may question the adequacy of training for the more complicated tasks. 2) Since administrative assistant may not have sufficient authority to enforce operator training requirements, CEA needs to provide extra focus on whether operators are adequately trained. 3) Hasty processes associated with creation of training policy such as no revision history, maintenance by administrative assistant, no signatures, lack of approvals may cause CEA to more closely evaluate whether document covers all applicable requirements. PER-005 5