NERC Internal Controls Evaluations

Transcription

1 NERC Internal Controls Evaluations Common Practices, Approaches, and Other Control Ideas April 11, 2017

2 Introductions Archer Energy Solutions acquires compliance division of Utility System Efficiencies Panelists o Richard Shiflett o Bob Dintelman 3

3 Objectives Identify what internal controls are and why they are needed Discuss risk thresholds and risk mitigation Discuss the types and characteristics of controls Discuss key controls Discuss a defense-in-depth approach for controls Provide a controls evaluation example for COM R5 4

4 What is an Internal Control? A process effected by an entity s oversight body, management, and other personnel that provides reasonable assurance that the objectives of an entity will be achieved: Operations Effectiveness and efficiency of operations Reporting Reliability of reporting for internal and external use Compliance Compliance with applicable laws and regulations Taken from United States General Accounting Office Standards for Internal Control in the Federal Government 5

5 Benefits of Internal Controls Why would an entity want quality internal controls? 6

6 Risk and Internal Controls Identify risks and determine risk acceptance levels or thresholds 7

7 Risk and Internal Controls Internal controls help mitigate risk exposure Risk profiles drive nature and complexity of internal controls 8

8 Nature of Internal Controls Internal controls can range in nature and complexity o ID cards, fences, locks, Virtual Private Network (VPN), or fireproof files o Independent verification of processes and deliverables o Authorization of employee time cards 9

9 Basic Types of Controls Preventive o Aimed at preventing any errors or irregularities from occurring which may have negative effects o Example: Documented process requiring development and maintenance of training schedule Detective o Designed to discover any errors or irregularities which may have occurred o Example: Documented process requiring periodic review to identify any required training not completed as scheduled, as well as training not completed per reliability standard requirements. - Quarterly review of completed training records to identify individuals who have not completed training by the required deadline. - Documentation and utilization of an event review and root cause analysis process to determine cause and affects surrounding an unwanted event 10

10 Basic Types of Controls Corrective o Corrective controls restore the system or process back to the state prior to a harmful event o Example: An entity may implement its restoration plan for a computer system from backup tapes after evidence is found that someone has improperly altered the data 11

11 Control Characteristics Examples of how controls may be characterized: Less Assurance Manual Can be overridden No management oversight Simple Performed by junior or inexperienced personnel Single control High level Control tests a sample Occurs after the fact More Assurance Automated Cannot be overridden Has management oversight Complex Performed by experienced personnel Multiple or Layered Controls Detail or transactional level Control tests entire population Occurs in real time 12

12 Key Controls What is a key control? A key control is a control that, if it fails, means there is at least a reasonable likelihood that a material error would not be prevented or detected in a timely basis. In other words, a key control is one that is required to provide reasonable assurance that material errors will be prevented or timely detected. 13

13 Key Controls Example An entity has a list of 25 controls that it feels addresses a risk area identified o Five controls occur at the end of the entire process and confirm that the other 20 controls have done their work and that there are indeed no remaining problems or other errors. Without the 20 earlier controls there would be a huge number of errors coming through and the five final checks would be little comfort Focusing on the five final controls, that usually found nothing requiring correction, might be enough for the key control review if the controls are designed and implemented correctly. 14

14 Key Control Factors Factors that help uncover possible key controls Likely points of failure How controls rely on each other o Look at interaction between controls o Individual controls may not address the risk o Some controls prevent other control failures 15

15 Controls Defense in Depth Preventive, Detective, and Corrective Layered controls supporting and enhancing key controls Control output visibility 16

16 Controls - Defense in Depth What is the right amount of controls? 17

17 Controls Evaluation Example COM R5 Each Balancing Authority, Reliability Coordinator, and Transmission Operator that issues an oral two-party, personto-person Operating Instruction during an Emergency, excluding written or oral single-party to multiple-party burst Operating Instructions, shall either: Confirm the receiver s response if the repeated information is correct (in accordance with Requirement R6). Reissue the Operating Instruction if the repeated information is incorrect or if requested by the receiver, or Take an alternative action if a response is not received or if the Operating Instruction was not understood by the receiver. NERC Standard COM

18 Controls Evaluation Example COM R5 Entity provides its communications protocols document Operator consoles have visual reminder to use 3 part communication The entity has implemented a detailed and technical initial training program for system operators, and retrain periodically Operators use 3 part communication for all information exchanges All operator communications are recorded Shift supervisor regularly listens to the recordings to verify 3 part communication Feedback to operators on improving 3 part communication 19

19 Controls Evaluation Example COM R5 Preventive o Communications protocol document o Operator visual reminder o Initial and continual training of operators o Use of 3-part communications throughout Detective o Review of audio recordings by supervisor o Communications protocol document may have detective controls present Corrective o Feedback to operators for improvement 20

20 Controls Evaluation Example COM R5 Key controls identified o Communications protocol documentation o Review of audio communications by supervisor Characteristics o A mix of automated and manual controls, but largely manual o No indication of management oversight o Controls are relatively simple and performed by experienced personnel o Not clear if supervisor review of audio recordings are sampled or not o No mention made of communications during Emergencies 21

21 Controls Evaluation Example Evaluation COM R5 o Request evidence that controls are present o Grade may range from Partially Implemented to Largely Implemented o Recommend entity include some form of management oversight and/or notifications based upon Operating Instructions issued either as part of Emergencies or otherwise. 22

22 Q & A Please post your questions to the Q&A area of the webinar. If you would like, you may us directly at: Richard Shiflett r.shiflett@archerenergysolutions.com Bob Dintelman b.dintelman@archerenergysolutions.com 23

23 Page 1 of 1 Scenario Instructions Prior to conducting this exercise, students should know what the three types of controls are, how to identify key controls, and evaluate controls according to the NERC guidance document. By performing a controls analysis in this scenario, the students should: Identify risks to the entity associated with the scenario Determine the controls that ABC employs Identify the types controls utilized Identify key controls Evaluate and justify the evaluation of the controls set Documents included as part of the scenario are: ABC Controls Answer Sheet (Spreadsheet) ABC Controls Scenario ABC Controls Scenario Answer Key The students should be given the scenario document and the spreadsheet to record the answers to the questions contained at the end of the scenario document. The answer key is used to assess the answers. Please note that a bonus question is provided that may or may not be used for open discussion on possible recommendations for controls implementation level improvements. info@archerenergysolutions.com SE Sunnyside Road Suite 292 Clackamas OR 97015

24 Risk Factor (1) Internal Control Identified (2) Rationale (3) Type of Control (4) P, D, or C Key? (5) "Key" Selection Support (6) Level of Assessment FI, LI, PI, NI or M (7) Implementation Rationale (8) info@archerenergysolutions.com SE Sunnyside Road Suite 292 Clackamas OR 97015

25 Page 1 of 2 Scenario ABC Company ABC Electric Company (ABC) is a medium sized integrated electric utility operating in the US with over 2.5 million customers. With approximately 10,000 employees, ABC has an installed net generation capacity of 4200 MW. While ABC is moderate in size when compared to many corporations, as an electric utility, the technology infrastructures are very complex incorporating real-time operator control systems, communication systems supporting the delivery of electricity to the customers. This environment, coupled with the key responsibility to operate and maintain the critical electric grid infrastructures, sets the stage for the need of a robust operating environments. ABC Electric Company uses state of the art status indication in their control center staffed with well trained, certified operators to avoid the risk of an operator making a mistake. ABC has an Automatic Voltage Regulator (AVR) status indication so that an alarm alerts its Transmission Operator s Control Center indicating an AVR status change from Automatic to Manual of a particular generating unit, thus providing notification to the TOP of an AVR status change within 30 minutes as required by Reliability Standard VAR-002. However, the GOP alarm did not update appropriately for a 24-hour period. ABC, as a GOP, self-reported a possible violation of VAR R3. Unfortunately, generator operator G.I. Jane was expecting the AVR status to be updated and it was not. G.I. Jane, as the GOP, should have been aware that the AVR was not changing since it often changed during that particular season. This fact is covered in ABC s Operator Training material. Further investigation into G.I. Jane s training record revealed that she missed this training. Somehow ABC missed this during their quarterly review of completed training records to identify individuals that have not completed training by the required deadline. ABC has an automated tracking tool that notifies the individual of scheduled training, reminds individuals to complete the training, and notifies management that training has not taken place prior to the training deadline so management can take appropriate action, but G.I. Jane ignored this reminder. Furthermore, ABC had a 3rd party rate their capabilities in a Management System to Minimize Human Factor Issues. The 3rd party rated this capability Fully Implemented. ABC provided the following evidence of its controls: - GOP training program that includes discussion of the quarterly review process - Screen capture of AVR alarm on SCADA - Procedure identifying the seasonal change in AVR status - Reports from automated tracking tool listing operator training and an example notification - Report from 3 rd party rating capabilities of Management System to Minimize Human Factors info@archerenergysolutions.com SE Sunnyside Road Suite 292 Clackamas OR 97015

26 Page 2 of 2 Questions: 1. What are the risk factors in the scenario? (Fill in column 1) 2. For the identified risk factors, what are the internal controls that you identified in the scenario? (Fill in answer in column 2). 3. For each identified internal control, briefly describe the rationale for the control. Explain how the internal control is meant to mitigate risk. (Fill in column 3) 4. For each identified internal control, determine whether the control is preventative (P), detective (D), or corrective (C). (Fill in column 4) 5. Review each possible control identified and determine whether the control is a key control. (Fill in columns 5) 6. For each key control identified, include a brief explanation on why you considered the control to be key. (Fill in column 6). 7. For the family of controls associated with VAR-002-3, determine the level of implementation. Indicate whether the controls are fully implemented (FI), largely implemented (LI), partially implemented (PI), not implemented (NI), or missing (M). (Fill in column 7). 8. Briefly explain what factors you considered to determine level of implementation. (Fill in column 8). Bonus Question: What controls recommendations would you provide to ABC to improve the level of implementation? info@archerenergysolutions.com SE Sunnyside Road Suite 292 Clackamas OR 97015

27 Page 1 of 2 Scenario Answer Key Questions: 1. What are the risk factors in the scenario? (Fill in column 1) The risk factors may be those taken from the NERC guidance or developed ad hoc. Risk factors may include human performance (error), training, voltage stability, and others. 2. For the identified risk factors, what are the internal controls that you identified in the scenario? (Fill in answer in column 2). 3. For each identified internal control, briefly describe the rationale for the control. Explain how the internal control is meant to mitigate risk. (Fill in column 3) 4. For each identified internal control, determine whether the control is preventative (P), detective (D), or corrective (C). (Fill in column 4) Below is a list of controls from the scenario, the rationale, and the type of control. Internal Control Identified (2) Rationale (3) Type of Control (4) [P, D, and/or C] ABC has an alarm generated for AVR status changes Reduce the likelihood that an AVR status change notification to the TOP is missed. P and D ABC has periodic system operator training System operator personnel receive training on current procedures with regards to voltage regulation status. P Quarterly reviews of training records are performed Ensure that system operators are receiving required training before deadlines are met. P, D, and possibly C Automated tracking tool for scheduled training that provides notifications and reminders Ensure that system operators are receiving required training before deadlines are met. P and D 3rd party assessment Determine if existing controls are sufficient C 5. Review each possible control identified and determine whether the control is a key control. (Fill in columns 5) 6. For each key control identified, include a brief explanation on why you considered the control to be key. (Fill in column 6). Internal Control Identified (2) ABC has an alarm generated for AVR status changes ABC has periodic system operator training Key? (5) Y Y "Key" Selection Support (6) Without the presence of the alarm, TOP personnel would need to rely upon verbal notification from the GOP which is apparently nonexistent. Without system operator training, personnel would likely not be aware the AVR status alarm and may likely go unnoticed. info@archerenergysolutions.com SE Sunnyside Road Suite 292 Clackamas OR 97015

28 Page 2 of 2 Internal Control Identified (2) Quarterly reviews of training records are performed Automated tracking tool for scheduled training that provides notifications and reminders 3rd party assessment Key? (5) Y N N "Key" Selection Support (6) Failure of performing the quarterly review may result in operators not receiving required training. The tool assists operators and management in the administration of the training, but without it does not raise the likelihood of failure significantly. Controls that are solely corrective cannot be key controls. 7. For the family of controls associated with VAR 002 3, determine the level of implementation. Indicate whether the controls are fully implemented (FI), largely implemented (LI), partially implemented (PI), not implemented (NI), or missing (M). (Fill in column 7). 8. Briefly explain what factors you considered to determine level of implementation. (Fill in column 8). Columns 7 and 8 of the spreadsheet should evaluate the controls as Partially Implemented (PI). ABC has several preventative and detective controls that were documented, namely the training program, the AVR status alarm, and the automated tracking tool. However, the quarterly records review was not documented and there is a lack of internal corrective controls. The 3 rd party assessment appeared to be a one off control and did not provide anything substantial regarding the processes associated with the AVR status. As added training, the students may be queried on what recommendations they would provide to ABC in order to improve the finding. info@archerenergysolutions.com SE Sunnyside Road Suite 292 Clackamas OR 97015