Brent Read Compliance Engineer - Enforcement. Root Cause Analysis for Commonly Violated Requirements October 23, 2013 Compliance User Group

Transcription

1 Brent Read Compliance Engineer - Enforcement Root Cause Analysis for Commonly Violated Requirements October 23, 2013 Compliance User Group

2 Summary Methodology Root Causes and Effective Solutions for: o FAC R1, R2, and R3 o PRC R3 o COM R2 Additional Resources 2

3 Purpose Share common root causes as observed by WECC Share effective solutions Prevent occurrence/recurrence of these violations o Learn from known root causes o Solutions based on effective plans and WECC s experience 3

4 Methodology

5 Methodology WECC analyzed all O&P Violations reported between 1/1/2011 8/31/2013 Each violation was individually assessed & the root cause was analyzed Effective solutions were developed based on plans submitted by entities and WECC s experience in these standards 5

6 Methodology WECC narrowed the analysis to 3 Standards o FAC R1, R2, & R3 Formerly FAC R1 o PRC R3 o COM R2 6

7 Root Causes

8 Root Causes - Importance Identify the Root Cause o Always evaluate why the violation occurred o Involve all related staff in the process o One option, out of many, for root cause analysis is Toyota s 5 Whys methodology Use the Root Cause to mitigate and prevent future recurrence of the violation 8

9 Root Causes - 5 Whys Example

10 Root Cause Categories 4 Categories of Root Causes o Administrative Programs, Processes, or Procedures Company ABC s training program doesn t identify the training staff 10 o Logical Cyber Assets, Relay Settings Company ABC s relay settings do not match it s Protection System design

11 Root Cause Categories o Physical A Physical Security Perimeter (PSP) does not have a way for Company ABC to monitor physical access o Human Personnel, Unintentional Changes A Relay Technician incorrectly entered maintenance and testing information into Company ABC s PRC- 005 tracking program 11

12 Effective Solutions Determination

13 Effective Solution Determination Directly related to root causes Common themes for the standards o Identification o Oversight o Collaboration 13

14 FAC R1, R2, R3

15 FAC R1, R2, R3 Historical Root Causes Administrative o Company ABC s Facility Rating Methodology did not include all devices o Company ABC s Facility Rating Methodology does not include ambient conditions o Company ABC has inadequate supporting documentation 15

16 FAC R1, R2, R3 Effective Solutions

17 17 FAC R1, R2, R3 Effective Solutions Specifically identify equipment in the Facility Rating Methodology o Implement Asset Management software to identify and track all equipment in the Facility Rating Methodology Keeps inventory of all equipment Stores facility ratings for the equipment Eases ordering new and replacement equipment Creates a way to monitor changes and initiate change control procedures

18 FAC R1, R2, R3 Effective Solutions 18 o Continuously review the list of identified equipment in the Facility Rating Methodology Utilize multiple business units in the review Use field personnel to validate the location and type of equipment as part of inspection or maintenance programs o Create and implement change control procedures for when equipment is replaced or modified Create a process flow and checklist to track changes made to equipment

19 FAC R1, R2, R3 Effective Solutions Periodically review the Facility Rating Methodology o Incorporate multiple business units in the review process Determine if the most limiting element is identified and accurate Determine if the methodology needs to be expanded to include additional equipment Ask what each group can do to improve the procedure 19

20 FAC R1, R2, R3 Effective Solutions Maintain a current library of the applicable manufacturers ratings, industry standards or verification documentation used to ensure the methodology is consistent with the requirements. o Incorporate reviewing and updating the documents in FAC-008 library as part of the change control procedures. 20

21 PRC R3

22 PRC R3 A GOP or TOP shall coordinate new protective systems and changes as follows. o R3.1. Each GOP shall coordinate all new protective systems and all protective system changes with its TOP and Host BA. o R3.2. Each TOP shall coordinate all new protective systems and all protective system changes with neighboring TOPs and BAs. 22

23

24 Why is Coordination Important? Together we can quickly recover from, or prevent abnormal and emergency conditions. 24

25 Why is Coordination Important? Without coordination, who knows what might happen 25

26 PRC R3 Historical Root Causes Administrative o Company ABC lacked documentation to show coordination of all protective system changes o Entity ABC lacks procedures for coordinating protective system changes Human o Company ABC s personnel failed to coordinate the protective system changes with neighboring entities 26

27 PRC R3 Effective Solutions

28 PRC R3 Effective Solutions 28 Specifically identify protective system equipment and schemes o Implement Asset Management software to identify and track all protective system equipment and schemes o Continuously review protective system equipment Incorporate verification of equipment, settings, and schemes as part of the maintenance and testing program

29 PRC R3 Effective Solutions All protective system changes must be coordinated o This allows the potentially affected entities to: Review the changes and determine the impact to their systems Work with the initiating entity if the changes present problems Implement any changes to their own protective systems 29

30 PRC R3 Effective Solutions Communication is two part; Coordination is three part o Correspondence should include: Proposed changes are provided by the initiating entity to the potentially affected entity The potentially affected entity acknowledges receipt of the changes and conducts necessary reviews The potentially affected entity approves the changes The initiating entity implements the changes and notifies the potentially affected entity 30

31 PRC R3 Effective Solutions Entity A needs to change protective systems; Entity A notifies Entity B of the proposed changes Entity B acknowledges receipt of the changes; Entity B analyzes proposed changes; Entity B approves changes and notifies Entity A Entity A implements changes and notifies Entity B; Entity B acknowledges the changes 31

32 PRC R3 Effective Solutions Develop procedures for making protective system changes, including coordination of those changes o Create an internal workflow o Identify the who, what, when, and how for making changes to protective systems 32

33 PRC R3 Effective Solutions In your procedures for making protective system changes: o Identify how you determine the need for changes to protective systems o Create an internal communication protocol between all appropriate personnel for protective system changes 33

34 PRC R3 Effective Solutions o Create an external communication protocol Include steps for coordinating protective system changes with other entities Include contact information for other entities o Provide notification of implementation of changes to other entities 34

35 PRC R3 Effective Solutions Continuously review the procedure for making protective system changes o Incorporate multiple business units in the review process Include engineering, field personnel, compliance personnel, and any other groups involved in making changes to protective systems Ask what each group can do to improve the procedure 35

36 PRC R3 Effective Solutions Keep good documentation o Maintain documentation in a single place o Dates and titles matter o Create clear mapping for documentation Document names and dates should match records o You should be able to easily connect the dots between protective system changes and associated documentation 36

37 PRC R3 Effective Solutions 37 Conduct continuous training for protection system change coordination procedures and protocols o Review company procedures and protocols o Review the list of entities that need to be included in coordinating the changes o Ensure Protection System personnel understand the NERC Standard requirements o Always ask if there are ways to improve the process

38 COM R2

39 COM R2 Each RC, TOP, and BA shall issue directives in a clear, concise, and definitive manner; shall ensure the recipient of the directive repeats the information back correctly; and shall acknowledge the response as correct or repeat the original statement to resolve any misunderstandings. 39

40 COM R2 Historical Root Causes Administrative o Company ABC failed to have a proper procedure for identifying and communicating directives Human o Company ABC s operators failed to adhere to communication procedures. 40

41 COM R2 Effective Solutions

42

43 Three Part Communication 43 A communication protocol where: o Information is verbally stated by initiating party; o The receiving party repeats the information back correctly (not necessarily verbatim) to the initiating party; and then o The initiating party either: Confirms the information repeated by the receiving party is correct, or Reinitiates the communication until the information repeated by the receiving party is confirmed to be correct.

44 COM R2 Effective Solutions Require use of three part communication for all communication, not just directives Communication Group Example o Any Volunteers? 44

45 COM R2 Effective Solutions Keep a log of all instances in which three part communication is used, including o Date of communication o Time the communication was issued/received o Entity issuing/receiving the communication 45

46 COM R2 Effective Solutions 46 Conduct reviews of voice recordings using three part communication o System Operators conduct weekly peer review of recordings o Monthly supervisor reviews of voice recordings o Monthly compliance team review of voice recordings o Quarterly training ream review of voice recordings o Create a forum for feedback

47 COM R2 Effective Solutions What are the reviewers looking for? o Was three part communication required and used? o Can the communication procedures be improved? o Are there any lessons to be learned? 47

48 COM R2 Effective Solutions Create a checklist to aid in reviewing the voice recordings o Was three part communication required? o Was the information clearly stated by the initiating party? o Did the receiving party repeat the information back correctly to the initiating party? o Did the initiating party confirm the information repeated by the receiving party is correct? 48

49 COM R2 Effective Solutions Periodically review communication procedures and protocols o Incorporate multiple business units in the review process o Ensure consistency o Identify when three part communication should be used 49

50 COM R2 Effective Solutions Conduct continuous training on communication protocols o Use simulations o If applicable, combine operators from multiple desks/business units (TOP, GOP, BA) o Include review of NERC standards related to communication(i.e. COM R2 and TOP R3) 50

51 Additional Resources

52 Additional Resources The 5 Whys o C_5W.htm Cause Analysis Methods for NERC, Regional Entities, and Registered Entities o Methods-Version1-2-Oct10.pdf 52

53 Additional Resources NERC RSAWs o Standard-Audit-Worksheets-(RSAWs).aspx 53

54 Summary Methodology Root Causes and Effective Solutions for: o FAC R1, R2, and R3 o PRC R3 o COM R2 Additional Resources 54

55 Questions? Brent Read (801)