EFFECTIVE ROOT CAUSE ANALYSIS

Transcription

1 EFFECTIVE ROOT CAUSE ANALYSIS David Tooth CEngFIMechE Copyright 2011 Sologic, LLC. All Rights Reserved. 1

2 What RCA is NOT... A search for a Single Root Cause Root Cause!) A search for a Quick Fix! A search for who is to BLAME! (THE 2

3 What is EFFECTIVE RCA? A robust/objective analysis any problem in any discipline. An evidence based process. A search for effective/sustainable solutions. A process that is applicable both to negative and positive events. A business improvement tool. 3

4 Things you will hear after an incident Root Cause: Employee Failed to read the instructions Employee was not paying attention Employee didn t follow procedure Associated Solutions: Stress the need for Employees to read instructions Communicated importance being alert at all times Retrain on procedure How Effective are these Solutions?? 4

5 A Different Approach is Needed Seek alternatives to the tired, old solutions: Retrain, Reinforce, Re-communicate People will always make mistakes The Key is to understand WHY the mistake was made and address deeper seeded causes. Understand their thought process at the time. We also need to look for and document the causes in the work processes, tools, environment, systems and culture. 5

6 Root Cause Analysis- A closer look Root cause resolution What does that mean to you? Root Cause is heard frequently Government News media Executives..What root cause are they normally focused on? Fault? Blame? Human error? Effective Problem Solving moves beyond blame and punishment Finger pointing becomes a thing of the past Understand the cause and effect relationships for the problem Can t rely on common sense Proactive 6

7 RCA--its most basic form Problem Cause & Effect Relationships Solutions 7

8 Premise WHEN THINGS GO WRONG.. Copyright 2011 Lyncsolve Publishing, LLC. All Rights Reserved. 1 8

9

10 RCA Process Sologic s 5 Steps 1 Gather and Manage Data 2 Create the Problem Statement 3 Analyze Cause and Effect 4 Generate Solutions 5 Produce the Final Report 10

11 RCA Process Sologic s 5 Steps 1 Gather and Manage Data 2 Create the Problem Statement 3 Analyze Cause and Effect 4 Generate Solutions 5 Produce the Final Report 11

12 Step 1: Gather and Manage Data D A T A Gathered from the incident, large amount, value unknown INFORMATION Relevant evidence & causes KNOWLEDGE Individual solutions WISDOM! Systemic solutions Source: Ackoff, R L From Data to Wisdom. Journal of Applied Systems Analysis 12

13 Importance of Evidence Evidence validates (or invalidates) causes Supports conclusions Reliable evidence: Eliminates personal bias, speculation, hidden agendas and politics Leads to more accurate interpretation of the problem and causes Leads to more effective solutions 13

14 Root Cause Analysis- 5 Steps 1 Gather and Manage Data 2 Create the Problem Statement 3 Analyze Cause and Effect 4 Generate Solutions 5 Produce the Final Report 14

15 Problem Statement Improvements Focal Point is now the link between the Problem Statement and Cause & Effect Chart Better describes the teams choice of a starting point Impact is qualitative and quantitative impact on the goals and objectives of the Organisation. Actual and Potential Impact are both addressed Potential impact is often just as important to capture, if not more Potential impact can create link for formal risk assessment 15

16 Sologic Problem Statement Focal Point: 100 out-of-spec units shipped to client When: Received by customer; April 1, 2011 (after first run on new CNC mill) Where: Part 154; Rec d by customer Y, produced at supplier Z on mill M-104 Impact: Actual Potential Actual $ Safety: None Greater safety risk due overtime to rework parts. Quality: Revenue: Cost: Actual Total: 100 units outside of spec width limits. Next NCR results in supplier downgrade = 40% more inspection required. 10% concession on contract price Rework costs Expedited shipping Investigation costs Could have missed the entire production run of 1,000 shipments. Could lose entire contract, potential to bid on future contracts. $100,000 Potential costs could have been much higher. Frequency: First quality escape by supplier Z with customer Y $67,000 $2,000 $3,000 $172,000 Focal Point: 100 OOS units shipped to customer 16

17 Step 2: Problem Statement Focal Point: The focus of the investigation, a.k.a. the problem When: Date: The date (or date range) of the problem Time: The time and duration of the problem Unique: Any unique timing aspects of the problem Where: Facility: Start broadly, such as with Facility System: Narrow down to a specific location or process Component: Pinpoint the exact location of the problem. Unique: Any unique location aspects of the problem Impact: Safety: What was the safety impact of the problem? Environmental: What was the environmental impact of the problem? Revenue: What was the impact on money flowing INTO the organization? Frequency: Cost: What was the impact on money flowing OUT of the organization? Other? How many times has the problem occurred? This is a value multiplier Modify as needed this section is flexible. Different problems will experience different impacts. Copyright Copyright Sologic, Lyncsolve LLC. All Publishing, Rights Reserved. LLC. All Rights Reserved. 17

18 Root Cause Analysis- 5 Steps 1 Gather and Manage Data 2 Create the Problem Statement 3 Analyze Cause and Effect 4 Generate Solutions 5 Produce the Final Report 18

19 Step 3: Analyze Cause & Effect There is always more than 1 cause for everything Transitory Cause Changes! Effect Caused By < AND > Non- Transitory Causes Conditions at the time of change 19

20 Cause Types Cause Type Transitory: Non-transitory: Changes Energy transfers Change in state/status External force applied Operation performed Trigger/Catalyst/Decision point Often the last cause to present Objects Description Tangible: Hardware/Systems, Environment Intangible: Goals, standards, procedures, rules, laws, training, specifications Properties/Attributes Status Construction materials, design intent, color, other attributes Quantity Velocity Position/location relative to other objects 20

21 Transitory & Non-Transitory Transitory Upset container Transitory causes represent a point of change. In this case, the water was not spilled until the container was upset. This represents a change from contained water to spilled water. Focal Point Spilled water Non-Transitory Water in container Non-Transitory Open-top container Non-Transitory causes are the players in an event. In this case, the water in the container represents a status of the container (and the water) at the time of the event. The status of a cause is more likely to change over time. The cause open-top container is a non-transitory property of the container. Properties are generally stable and resist change. 21

22 Simplified Causal Logic The Sologic process builds charts by asking the following questions: Question 1: What causes this effect? Builds the analysis horizontally Starts in the present and works towards the past Question 2: Every time this cause occurs, does it always result in this effect? Builds the analysis vertically Identifies combinations of causes Functions via logical and/or relationships 22

23 Question 1: What caused this effect? Focal Point Lost productivity Transitory Transitory Transitory Transitory Exchange server outage Server lost power Water leaked into data center Toilet leak above data center How does this compare to a timeline? 23

24 Question 2: Every time this cause occurs, does it always result in this effect? Non-Transitory Server requires power source Non-Transitory Data center not waterproof Focal Point Lost productivity Transitory Transitory Transitory Transitory Exchange server outage Server lost power Water leaked into data center Toilet leak above data center Non-Transitory 24 hour availability requirement Non-Transitory Water contact opens breaker 24

25 Sources of Cause People Procedures Hardware Environment People Strengths in one area compensate for weakness in others Environment Procedures Hardware 25

26 Root Cause Analysis- 5 Steps 1 Gather and Manage Data 2 Create the Problem Statement 3 Analyze Cause and Effect 4 Generate Solutions 5 Produce the Final Report 26

27 How do Solutions work? By changing, eliminating or controlling causes! When causes are eliminated, you break the causal chain When you break the causal chain, you are eliminating precursor events 27

28 Example: Solutions Non-Transitory Server requires power source Non-Transitory Data center not waterproof Focal Point Lost productivity Transitory Transitory Transitory Transitory Exchange server outage Server lost power Water leaked into data center Toilet leaked above data center Non-Transitory Non-Transitory 24 hour availability requirement Water contact opens breaker 28

29 Example: Solutions Non-Transitory Server requires power source Non-Transitory Data center not waterproof Focal Point Lost productivity Transitory Transitory Transitory Transitory Exchange server outage Server lost power Water leaked into data center Toilet leaked above data center Non-Transitory 24 hour availability requirement Install back-up power supply (UPS) Non-Transitory Water contact opens breaker 29

30 Example: Solutions Non-Transitory Server requires power source Seal data center, update specification requirements, examine other data centers for similar failure modes Non-Transitory Data center not waterproof Focal Point Lost productivity Transitory Transitory Transitory Transitory Exchange server outage Server lost power Water leaked into data center Toilet leaked above data center Non-Transitory Non-Transitory 24 hour availability requirement Water contact opens breaker 30

31 What is a good Solution? Effective Eliminates the problem Breaks the causal chain Is there solid evidence to support? Easy to Implement + Return on Investment Avoids potential negative impacts 31

32 23-May-13 32

33 23-May-13 33!!!

34 CAPA Solution Type Corrective Actions (CA) Characteristics Eliminate the risk of a problem recurring Applies to problems that occurred in the past Reactive Preventive Actions (PA) Eliminate the risk of a problem that has not yet occurred Addresses problems that may occur in the future Proactive May be found in other departments May be the result of dynamic analysis (also called systemic cause analysis) 34

35 Turning the corner from: Reactive to Proactive Copyright 2011 Sologic, LLC. All Rights Reserved. 35

36 Reactive vs. Proactive Problem Solving Reactive Performing RCAs on past problems RCA trigger criteria aligned with business goals Proactive Identifying and eliminating common cause from past incidents (Dynamic Analysis) Performing RCA on hypothetical problems Identifying and eliminating the failures in your protective systems 36

37 Traditional Reactive RCA Focal Point Pump 102 Down 24 hours Seal leaking Excessive Seal runout Pressure In pump Excessive Shaft runout Seal unable to Handle runout Bearings worn Shaft req. in Spec brgs = Caused By Leak is HF HF in process Continues Not able to Run w/ leak HF is lethal HF Chemistry Stop Copyright 2011 Sologic, LLC. All Rights Reserved

38 Traditional Reactive RCA Focal Point Pump 102 Down 24 hours Seal leaking Excessive Seal runout Pressure In pump Excessive Shaft runout Seal unable to Handle runout = Caused By Leak is HF HF in process Focal Point Mill down For 36 hours Not able to Run w/ leak Continues. HF is lethal HF Chemistry Copyright 2011 Sologic, LLC. All Rights Reserved. 38

39 New Focal Point Reliability Losses in Akron plant Systemic Cause RCA -1 st Step Pump 102 Down 24 hours = Caused By Seal leaking Excessive Seal runout Pressure In pump Leak is HF Excessive Shaft runout Seal unable to Handle runout HF in process Mill down For 36 hours Not able to Run w/ leak Continues. HF is lethal HF Chemistry Copyright 2011 Sologic, LLC. All Rights Reserved. 39

40 New Focal Point Pump 102 Down 24 hours Reliability Losses in Akron plant Fan 250 Down 10 hours Systemic Cause RCA -1 st Step Mill down For 36 hours Seal leaking Continues. Not able to Run w/ leak Continues. Excessive Seal runout Pressure In pump Leak is HF HF is lethal Excessive Shaft runout Seal unable to Handle runout HF in process HF Chemistry Copyright 2010 Apollo Associated Services, LLC Copyright 2011 All Rights Sologic, Reserved. LLC. All Rights Reserved. 40

41 New Focal Point Reliability Losses in Akron plant Pump 102 Down 24 hours Fan 250 Down 10 hours Mill down For 36 hours Copyright 2011 Sologic, LLC. All Rights Reserved. Cause / Effect Cause / Effect Cause / Effect A Cause / Effect Cause / Effect? A Cause / Effect B C Cause / Effect Cause / Effect Cause / Effect C Cause / Effect Cause / Effect Cause /? Effect Systemic RCA- -Common cause Common Cause! If no action is taken, Cause A will continue to show up in the future!? B A 41

42 New Focal Point Reliability Losses in Akron plant Pump 102 Down 24 hours Fan 250 Down 10 hours Mill down For 36 hours Copyright 2011 Sologic, LLC. All Rights Reserved. Cause / Effect Cause / Effect Cause / Effect A Great reduction in Impact of Focal point! Future Problems Prevented!! Cause / Effect Cause / Effect? A Cause / Effect B C Cause / Effect Cause / Effect Cause / Effect C Cause / Effect Cause / Effect Cause /? Effect Systemic RCA- Proactive!! Implement Solution for A? B A 42

43 Another way to be Proactive w/ C&E Primary Effect Technical Causes XXXXX YYYYY = Caused By Unexpected Failure Protective System did not prevent or detect AAAAA BBBBB Systemic Causes Copyright 2011 Sologic, LLC. All Rights Reserved. 43

44 Root Cause Analysis- 5 Steps 1 Gather and Manage Data 2 Create the Problem Statement 3 Analyze Cause and Effect 4 Generate Solutions 5 Produce the Final Report 44

45 Reporting What Information is Important? Problem Statement Summary Solutions for causes, assignments & due dates Cause and Effect Chart Contact Name and Team Members 2 to 4 pages usually is enough 45

46 RCA Programs at work 46

47 Best Practices for Enterprise Problem Management View RCA as a program not a tool Implement RCA as early as possible Create specific RCA program goals that are aligned with business goals 47

48 Aligning RCA Performance Business Goal Maintain production levels above 70% KPI RCA Program Goal Reduce production delays related to IT service interruptions by 50% KPI Number of days production fell below 70% Number of production delays related to interruptions in IT service 48

49 Case Study--RCA Results w/in Aerospace IT Time Reduced time to close RCAs by 42% over first 5 years Effectiveness 100% effective on solutions Cost RCA program reduced overall cost to operate IT services 49

50 Activities Averages RCA Results w/in Aerospace IT EPM Annual Activity Volume Year 1 Year 2 Year 3 Year 4 Year RCAs Avg Duration Avg Activty per analyst Number of analysts for each yr

51 Suggestions for Getting RCA Started Establish basic Program Infrastructure Develops goals, training plan, KPI s, action tracking, etc Turnkey Workshop Deploy RCA Facilitator training Target 1 facilitator/10 employees RCA s/year/facilitator max at the start Establish Threshold Criteria RCA Software Makes development of RCA report easier and quicker Has RCA structure built in Results in a much higher quality RCA w/ more effective solutions Causelink demo available at: 51

52 Best Practices for Enterprise Problem Management Key RCA performance indicators: Percent effectiveness of solutions Cumulative savings Time to complete each RCA Number of RCA completed/facilitator 52

53 Thank you Questions please? Presentation title May 13 53

54 1 Major Incident 10 Losses 6,500 Work Orders/Repairs 20,000 Defects

55 55

56 The importance of clear instructions: A new fuel tanker arrives at a location, somewhere in the Middle East, and the HSE Manager tells the fleet supervisor to ensure that it is clearly labelled- Diesel Fuel and No Smoking in Arabic. This is what he got. Presentation title May 13 56

57 57

58 Why RCA? Robustness evidence based. Unbiased only the facts! Consistency. Promote Teamwork. Harness local knowledge. Focus on solutions (not blame). Identify actual causes. 58

59 Why RCA? Effective learning. Systemic issues. Effective solutions not the dreaded re- suffix! Review/Re-train/Revise/etc. Provides a platform for effective action tracking/audit. Easily scalable process minor to major. Identify soft issues. 59