Root Cause Failure Analysis: Fact or Fiction? Part 2

Transcription

1 Challenge of Incident Investigation The RCFA Inductive Process, evaluating the risk due to the degraded PSM elements, is a systematic method in incident investigation to analyze the integrity of current PSM system as defined as policies and procedures. PSM system should ensure that changes do not result in operations outside of established safety parameters. This need to become essential component of a plant s process safety system as degradation and non compliance can occurs daily in a hydrocarbon processing plant. Hydrocarbon Processing Plant Complexity The Incident Investigation Team challenge is in obtaining knowledge and understanding of how work and decision making is carried out within current organization, of the current PSM management system and PSM management system effectiveness. Investigation team challenge is to assess: Operation Changes: The operating plant environment is changed on the daily basis. Although highly automated hydrocarbon processing plant are subject to frequent operation parameter changes as well as equipment failures. There could be thousand of alarms that go on and are left unattended in one day of operation. In addition, hydrocarbon processing plants are becoming more complex and tightly coupled between process units and operating plants. And interconnections between plants and functions are frequently not automated. Equipment Modification: Hydrocarbon processing plants are exposed to flux of changes implemented by the personnel involved in redesign, operation and maintenance. Large hydrocarbon processing complex can issue thousands upon thousands maintenance work orders per year as well as to be involved in large number of abnormal operating scenarios! Frequently employees are overwhelmed by magnitude of changes and issues. Technological Advances: Process changes, improvements, as well as technological advances result in an increase in competence requirements. As the equipment complexity increases, understanding of the equipment implications relative to maintenance and operating change can becomes critical element for safe operation. A characteristic feature of the complex hydrocarbon processing plan is that their complexity often exceeds employee grasp of how facilities are designed and how should they be operated and maintained. Redundancy: Current high prices of crude oil have put pressure on production and processing capability of existing facilities. In order to meet targets availability, equipment & system redundancy is implemented. A high level of redundancy is easily justified for increasing reliability, and possibly safety. Reliance on redundancy may lead to decreased emphasis on PM, monitoring, operating envelopes, as well as other safety engineering techniques. In practice, redundancy may cover up, or mute, operating, maintenance and design errors and prevent them from becoming visible until something catastrophic occurs. The Investigating Team challenge is to identify causes, ineffective management system, that drive human performance by comparing to the PSM system benchmark, i.e. plants with pacesetting HSE and economic performance. Example: Pump and Electrical Motor Failure Application of the RCFA Inductive Process considering risk of the degraded PSM elements will be illustrated by failure analysis of Starhill Platform Crude Transfer Pump. The example is illustrative and has no resemblance to any actual installation, facility, country or company. The pump was installed in 1988 and was operating with expected reliability till year Six pump and motor failures have occurred in the last four years, four have required replacement with a factory new motor at a cost of US$ 150,000 each. Failures are characterized by failed bearings, a bent rotor shaft and damage to rotor and stator from heavy rubbing contact. Norcan Reliability Engineering Proprietary Do Not Copy Page 1 of 15

2 Figure 13: Example Investigation: Pump and Electric Motor Failure Sequence of Events: Pump and Electrical Motor Failure The Investigation Team has developed the Sequence of Events for Pump Failures at Starhill Platform, describing timing, who is involved, details of the sequence, outline / definition of performance less than expected for that sequence, agent involved in this change (PSM procedure or inadequate facilities), and consequence of performance less than expected. There are two distinctive periods that can be observed from the sequence of events, expected operation from 1988 till 2000, and high failure rates after year Norcan Reliability Engineering Proprietary Do Not Copy Page 2 of 15

3 Figure 14: Sample Sequence of Events The Investigation Team using the Sequence of Event, Fig. 14, has identified the change, performance less than expected and procedure / standards / practices that used control this step. Focus of investigation is a period of high failure rates. Each Event in the Sequence, Fig.14, are evaluated for Active, Condition and Supervision and PSM System failure. The Investigation Team Analysis of the period after year 2000 has probed deep enough to uncover issues at Starhill platform. Issues that have surfaced have been created due to inadequate design, fabrication, construction as well as poor operation or maintenance. Everything happens for reason and those need to be fully understood to prevent reoccurrence Deviation Statement Physical Failure Mechanism: Pump and Electrical Motor Failure The Investigation Team has identified considering Physical Failure Mechanism that change in operating condition involving Water, Sand and Slugs Carry Over was a significant factor that warrants further investigation of this event. The following is the outline of the Investigation Team consideration of Physical Failure Mechanism: Norcan Reliability Engineering Proprietary Do Not Copy Page 3 of 15

4 Figure 15: Outline of Physical Failure Mechanism Norcan Reliability Engineering Proprietary Do Not Copy Page 4 of 15

5 Deviation Statement Steps in the Sequence of Events: Pump and Electrical Motor Failure The Investigation Team has defined in each Event in the Sequence of Events: Details of Event, Change (Performance Less than Expected), Agent of Change and Effect of Change. Figure 16: Assessment of Step in the Sequence of Events Scope of Investigation Active, Precondition and Supervision, Degraded PSM system: Pump and Electrical Motor Failure The Investigation Team has analyzed the Operating Condition involving Water, Sand and Slugs Carry Over by considering Active, Precondition and degradation of PSM elements. Investigation is performed by application of qualitative techniques defining assessment dimension with rating for completeness and effectiveness considering PSM system degradation. RCFA Inductive Process, considering PSM degradation, is a technique backed up by web based software with quite extensive questionnaire involving assessment of completeness and effectiveness of each PSM dimension. The following, Fig.17, is outline of the RCFA Inductive Process management system investigation Menu: Norcan Reliability Engineering Proprietary Do Not Copy Page 5 of 15

6 Figure 17: Scope (Menu) for the Event Assessment The Investigation Team directed the investigation for each a sentinel event identified for analysis. The members of the Investigation Team were trained in the RCFA Inductive Process techniques and its goals, as the tendency to revert to personal biases is strong. Multiple investigators and management interference can allow triangulation or corroboration of major findings and increase the validity of the final results. Norcan Reliability Engineering Proprietary Do Not Copy Page 6 of 15

7 PSM Element Completeness: Pump & Electrical Motor Failure Investigation Team utilizes criteria s to assess the completeness and effectiveness of each element of PSM. Those criteria s require assignment of ranking in values from 1 to 4. PSM Elements Effectiveness: Pump & Electrical Motor Failure The following, Fig. 18, is the outline of effectiveness scoring: PSM ELEMENTS EFFECTIVNESS ASSESSMENT Leaders Focus on Setting PSM Direction and Strategy PSM Element implementation is per Individual ability based Competence Assessment Each Work Group has PSM Element Implementation defined by Competence Assessment Each PSM Element Implementation Group has a Designated Leader Effective PSM Goals used by Work Groups Required PSM Competencies are Defined for all Positions PSM Roles for Operations and Mtce based on Operator leadership All PSM Implementation Groups are Organized Similarly at each Site Process Shift Teams are Highly Skilled on PSM elements Applicable Yes Current State Maximum Score Effective Coordination between Shift Teams and Day Work relative to PSM elements Single Accountability for PSM Performance Measures (KPI) Criteria for Assessing PSM Elements: Effectiveness Figure 18: Criteria for Effectiveness Assessment: Pump & Electric Motor Failures PSM Elements Degradation: Pump & Electrical Motor Failure The Investigation Team used the software with build in methodology to assess PSM elements completeness and effectiveness as related to each step of the Sequence of Events. The Investigation Team assessment of PSM Elements as related to the Pump Failures at Starhill Platform, Sequence of Events, Event 02, Period of Frequent Failures, is: PSM Mechanical Integrity Assessment: Pump and Electrical Motor Failure The Investigation Team following RCFA Inductive Process questionnaire has examines the content of the knowledge base that should be available to operator and its lack of completeness and effectiveness. The Investigation Team assessment of Sequence No.2 relative to PSM Mechanical Integrity is partially shown below: Norcan Reliability Engineering Proprietary Do Not Copy Page 7 of 15

8 Evaluate Mechanical Integrity Figure 19: PSM Mechanical Integrity Assessment Norcan Reliability Engineering Proprietary Do Not Copy Page 8 of 15

9 As per assessment of the Investigation Team, there is no Defect (Failures) Control at Starhill platform. The operator and mechanics were not provided baseline mechanical integrity data and data on past repair was not available, Fig. 19 The Investigation Team has provided very low rating on operator s awareness of equipment date base, limitations, PM program and equipment maintenance data. The equipment maintenance data base can include Equipment Maintenance and Reliability Strategy, Reliability Targets, Operating Limits, Material of Construction, specification sheets, time consumed, impact, and costing. Schedule, Annual Preventive Maintenance calendar need to be available and displayed for operators, with work instructions. Data should be made available of values derived using calibrated instruments as compared to Design Specification. Agreements on deviations to schedules and any resultant impact should be documented and provided to operators. The Investigation Team has provided a low rating for Operator involvement in PM and Operational database, understanding of Mean Time Between Failure information for increasing On-time service hours, knowledge of base for Predictive Maintenance, Reliability Engineering data for equipment and components, Plant Life Extension studies, Remnant Life Analysis and Sparing philosophy. The Investigation Team has provided a low rating for testing and inspection programme, Engineered Controls, Vents, Drains, Flare Lines, Elbows, Bends, Turbulent Spots, Spring hangers, Pipe supports, Foundation Settlements, Stack Top Segments, Critical Isolation Valves and Non-Return Valves. The Investigation Team has provided a low rating for inspection techniques used to ensure integrity, reference to appropriate Inspection Standards, Individual Competency, Instrument and Result interpretation. PSM Training Assessment: Pump & Electrical Motor Failure The Investigation Team has assessed the PSM Training, as per questionnaire on Fig. 20. Assessment of training is essential to understanding of cases leading to failure in Sequence No.2 Norcan Reliability Engineering Proprietary Do Not Copy Page 9 of 15

10 Evaluate Training. Figure 20: PSM-Training Norcan Reliability Engineering Proprietary Do Not Copy Page 10 of 15

11 The Investigation Team has provided a low rating for Training, no clear Goals and Objectives, not based on Task analysis, Skills Gap analysis, personnel evaluation criteria, with a structured calendar, participant profile, venue selection for ambience, Trainer choice, duration prescribed, subject topics relevant to the participants. Employees at Starhill platform were overwhelmed by magnitude of changes and issues. The constraint at Starhill Platform is employee understanding of processing technology and the systems in which facilities design, operation and maintenance is embedded. The Investigation Team has provided a low rating for Levels of Training as published both for budgetary purposes as well as progress evaluation. This should be treated as an on-going activity preferably with a specified budget and an absentee list. A characteristic feature of the Starhill Platform is that the complexity exceeds employee grasp of how facilities are designed and how should they be operated and maintained. In complex, automated systems, such as Starhill Platform, management system accidents are predominant. Completes and Effectiveness of PSM Elements: Pump & Electrical Motor Failure The Investigation Team has provided completeness and effectiveness rating for all PSM elements relative to failure analysis of Starhill Platform Crude Transfer Pump. The Investigation Team by using the RCFA Inductive Process proprietary software and methodology has summarize, Fig. 21, a degradation assessment of all PSM element as related to Event 02 in the Sequence of Events. The Investigation Team has provided rating for the degraded PSM element to prioritize elements where a departure from standard procedures or specifications in PSM results in non-conforming processes or where there have been unusual or unexplained events which have the potential to impact on production, system integrity or personal safety. The Investigation Team rating for PSM degradation is summarized below: PSM Evaluation Summary Results Figure 21: PSM Evaluation Summary Score Risk Assessment due to Degraded PSM System: Pump & Electrical Motor Failure The Investigation Team rating for PSM degradation relative to Events in the subject Incident Investigation, as summarized in Fig.21, are evaluated for the risk due to degraded PSM system. Norcan Reliability Engineering Proprietary Do Not Copy Page 11 of 15

12 The Investigation Team assessed the risk due to degraded PMS, Fig. 22, by using the PSM system completeness and effectiveness variance, shown in Fig.21, of analyzed steps in Sequence of Events. The Investigation Team risk rating is: Risk associated with noncompliance to PSM element for Event 02 of the Sequence of Events is CRITICAL, High Variance and High Consequence, Risk associated with that hazard need to be reduced minimum variance to PSM Investigation Team to determine appropriate ways to eliminate or control the hazard, variance to PSM. The investigation results need to ascertain that deviation to PSM is a common cause of accidents, by managing PSM system we are managing potential incidents. 1 2 Figure 22: RCFA Inductive Process Risk Metrics Norcan Reliability Engineering Proprietary Do Not Copy Page 12 of 15

13 Defining Improvements to Reduce Risk: Pump & Electrical Motor Failure The Investigation Team RCFA has defined improvements, Fig. 23, and prioritizes risks associated with PSM degradation and followed by application of resources to minimize, monitor, and control the probability and/or impact of failures / incidents or to maximize the realization of opportunities. Figure 23: RCFA Inductive Process Root Cause and Improvements Action Norcan Reliability Engineering Proprietary Do Not Copy Page 13 of 15

14 The Investigation Team has provided recommendations dealing with fundamentals of PSM in employee involvement, hazard analysis, operating procedures, training and mechanical integrity. The recommendations define improvements and manage risk due to PSM degradation include transferring the risk to another party, avoiding the risk, reducing the negative effect of the risk, and accepting some or all of the consequences of a particular risk. For the Investigation Team recommendations prioritization is used, whereby the risks with the greatest loss and the greatest probability of occurring are handled first, and risks with lower probability of occurrence and lower loss are handled in descending order. Implementing Improvements: Pump & Electrical Motor Failure Investigation Team recommendations incorporate fundamental culture change to avoid resistance: Fundamental revisit to PSM implementation to offset manager s resistance to change existing structures. Fundamental revisit to PSM implementation to offset labor force resistance due to fears of additional work, job security, layoffs! Way Forward: Pump & Electrical Motor Failure The Investigation Team will uncover what they are looking for. If there is a common belief that accidents take place due to faults in the manufacturing management system, they will probably focus on deteriorated PSM system. However, if the shared belief is that fast consensus is preferable, that brainstorming is the way to go. Safety and effectiveness vary significantly in the industry. Similar plants can spend third in maintenance cost to achieve pacesetter safety, availability, reliability and product quality. Hydrocarbon Processing Industry needs to have an appreciation of a benchmark, how the stellar performance is accomplished by the best in industry. For hydrocarbon processing plants with poor safety and operating effectiveness, plant output does not correlate with resources utilized: the resources are consumed, but the resource consumption stubbornly fails behind the potential accomplished by pacesetters. Poor performing hydrocarbon processing plants feature operating and maintenance teams not equipped to solve safety and reliability problems, poor availability, operating equipment beyond design limitations, poor repair, safety and maintenance effectiveness, ineffective contractor policy, large and ineffective spare material inventories and etc. Way forward for hydrocarbon processing industry could be to promote a benchmark model for PSM effectiveness and use this as a guide for PSM assessment and incident investigation. List of References: [1] FMRIMS@Root, [2] Swain, A.D. & Guttmann, H.E., Handbook of Human Reliability Analysis with Emphasis on Nuclear Power Plant Applications. 1983, NUREG/CR-1278, USNRC. [3] Humphreys, P. (1995). Human Reliability Assessor s Guide. Human Factors in Reliability Group. [4] "The Human Factors Analysis and Classification System (HFACS)," Approach, July - August Accessed July 12, [5] Reason, J.[1990] Human Error. Cambridge University Press [6] Standards Association of Australia (1999). Risk management. North Sydney, N.S.W: Standards Association of Australia. ISBN X. [7] United States Environmental Protection Agency (April 2004). General Risk Management Program Guidance. United State Environmental Protection Agency. [8] OSHA PSM ; Process safety management of highly hazardous chemicals Norcan Reliability Engineering Proprietary Do Not Copy Page 14 of 15

15 Namik Kosaric is a Canadian Professional Engineer with experience with Norcan Reliability Engineering (current), PETRONAS, Bahrain Petroleum Company and ESSO Petroleum Canada in reliability improvements and maintenance cost reduction, mechanical design, project engineering and technical support of Oil Refineries and Oil Production Facilities. For the last 8 years in PETRONAS Namik Kosaric was responsible for providing technical and knowledge leadership in development, coordination and implementation of plant reliability and integrity improvements and program to PETRONAS OPU s to improve and support the overall Petroliam Nasional Berhad objectives. In BAPCO, Namik Kosaric, pioneered and implemented a root cause failure analysis of lost profit opportunities and chronic failures using a multi-disciplinary teams to improve plant reliability, availability, safety and to ultimately reduce operating costs. Significant cost savings were achieved as a result of over 200 completed investigations. For 23 years in ESSO Petroleum Canada, Namik Kosaric has made significant contribution worldwide in reliability improvements, design, projects and maintenance cost reduction in upstream and downstream facilities. Norcan Reliability Engineering Proprietary Do Not Copy Page 15 of 15