Functional Safety Assessments of Safety Controls, Alarms, and Interlocks

Transcription

1 Functional Safety Assessments of Safety Controls, Alarms, and Interlocks How efficient are your functional safety projects? Eloise Roche, Monica Hochleitner, and Angela Summers SIS-TECH Solutions, LP Houston, TX Keywords: Human Error Assessment, Safety Management System, Periodic Inspection, Safety Interlocks, Safety Alarms, Safety Instrumentation Systems Abstract The execution of and return on investment (ROI) for safety controls, alarms, and interlocks (SCAI) projects can be negatively impacted by human error, such as inadequate design, installation, testing, maintenance, and operation of the automation systems. These human errors can be reflected throughout a site, expanding the potential losses with each new functional safety project. Administrative controls in the form of functional safety assessments (FSA) are needed to identify and correct these systematic errors. The FSA plays a significant role in assuring that each SCAI project deliverable is generated in an efficient and timely manner. The purpose and content of each FSA stage is reviewed and execution timing for each stage is suggested with the objective to focus project resources to maximize FSA benefit. Finally, a case study illustrates how lack of effective and timely FSA contributed to the occurrence of a costly and catastrophic loss event. 1 Introduction Functional safety is achieved through implementation of effective process control and safety systems that manage the process safety risk remaining after inherently safer design practices have been applied to the process design. Functional safety projects can implement a wide variety of safeguards, including safety controls, alarms, and interlocks (SCAI). Successful execution of SCAI projects requires significant resource investment and typically involve hundreds of engineering hours to brainstorm hazard scenarios, identify functional safety gaps, and determine the best means to reduce risk. To begin, a clear understanding of the functional safety assessment (FSA) is needed. A common terminology error is to interchange "functional safety assessments" and "functional safety audits". These terms are not synonymous, though their interpretation is often dependent on what needs to be accomplished on a particular site by a particular assessor. Fundamentally, the functional safety assessor is looking at the specific details associated with each device and function within the SCAI system to determine if the system is achieving the required performance as installed, operated, inspected,

2 maintained, and tested. In contrast, the functional safety auditor is examining site records and performing spot checks to determine whether personnel are following the functional safety procedures and practices. Nevertheless, there is often significant overlap in the personnel who perform these reviews and in the specific information being reviewed. It is not uncommon for a functional safety audit to be performed in concert with the FSA Stage 4. For more information on functional safety audits, refer to "How Effective are Your Safety Controls, Alarms, and Interlocks? The Importance of Functional Safety Auditing." [5] Several industry practices have been published to address the lifecycle management of instrumented safeguards, such as SCAI or more specifically safety instrumented systems (SIS). ANSI/ISA [9] is acknowledged by OSHA as a recognized and generally accepted good engineering practice (RAGAGEP) for SIS [10,11]. ANSI/ISA is the USA adoption of IEC [1], which is widely referenced even in countries that do not have a specific process safety regulation. IEC lists specific assessment requirements targeting the identification and elimination of SIS failures and human errors. Until recently, the guidance for other instrumented safeguards had been much more general and typically was expressed in terms of meeting the core attributes of independent protection layers (IPLs). Now, ANSI/ISA [2] and draft ISA [3] provide requirements for the functional safety management of SCAI. These standards [1,2,3,9] unanimously consider the FSA an essential requirement for assuring the risk reduction claimed in the hazard and risk analysis (H&RA). The skills needed are so similar that it is usually an efficient practice to evaluate both SIS and the other SCAI as part of the same activity. All FSA are alike in a number of key elements. All require proactive safety planning to support effective execution. All require written procedures addressing how the assessment is carried out. Each FSA needs to involve at least one competent assessor that is independent of those supporting (designing, operating, maintaining, etc.) the SCAI being assessed and is provided access to the information covered by the assessment scope. Finally, all assessments require timely and satisfactory resolution of recommendations. 2 The Stages of FSA The mere existence of standards for instrumented safeguards is not enough. Project teams must be sufficiently competent in the fundamentals behind the requirements of these standards that they correctly interpret and apply the standards. Project staffing plans and schedules must account for the associated reviews and deliverables if the functional safety project is to be executed in an efficient manner. Any deficiencies in timeliness or quality of a SCAI deliverable have a significantly higher cost impact the later the deficiencies are corrected (Figure 1). FSA are intended to verify that each SCAI system is in conformance with the applicable standards and its safety requirements specification (SRS) long after the project startup and handover to operations. FSA are an on-going activity that periodically assesses how well the SCAI are meeting the needs of the operating facility. Instrumentation, application programs, and procedures are subject to changes throughout the operation and maintenance phase. If the reality of operation and maintenance is different from the design assumptions, the risk of loss events can be substantially greater than desired. FSA are essential to the sustainability of the risk management plan and to preserving the intended ROI of the project.

3 Figure 1: Relative Cost of Design Changes as a Function of Project Phase (adapted from original) [4] The project execution plan typically establishes the timing for stages 1 through 3. Stage 4 occurs repeatedly throughout the installation life, while stage 5 is conducted as part of modification and decommissioning. Table 1 provides suggested timings and the purpose for each FSA stage, which builds upon previous stages. A robust FSA program prevents propagation of systematic errors from one project phase to the next. Deferring earlier assessments can result in increasingly costly rework. Table 1 - Purpose of Relative Timing of FSAs Stage Project Task Timing Purpose 1 H&RA Independence and Risk Reduction Limit Review Immediately after initial H&RA has been performed and SCAI functional specification documented Ensure risk reduction gaps were covered and save money during project 2 Detailed Design Review Immediately after detailed design is complete and before purchasing, programming, and installation begin Ensure design achieves required risk reduction and save money during project 3 Functional Safety portion of pre-startup safety review (PSSR) [REQUIRED] After installation, precommissioning and validation are complete and all procedures developed but BEFORE hazard is present Ensure safety

4 4 Operation and Maintenance Review [REQUIRED] Periodically Confirm performance of installed system 5 Proposed Change Review [REQUIRED] Prior to implementing a proposed modification Determine which assessments apply to the proposed change 3 Content of FSA Each FSA stage relies upon a different set of input documentation and assessment items to determine the acceptability of the SCAI. In practice, FSA are generally performed with detailed checklists or other support tools designed for use by a competent assessor. Detailed FSA procedures help ensure that the applicable requirements are addressed and that there is clearly documented justification for each finding, whether positive or negative. 3.1 H&RA Review: Independence and Risk Reduction Limits (FSA Stage 1) FSA stage 1 determines before a significant amount of engineering resources are spent that the proposed risk management plan is compliant with standards and that adequate SRS have been developed. Stage 1 focuses on the H&RA and resulting safeguard documentation, leaving detailed design reviews of the hardware for later stages. One frequent exception is a high level review of the risk reduction claimed for each selected logic solver, especially programmable logic solvers which tend to be long-lead items and are often selected and even purchased well before the SRS is completed. The potential impact on project cost and schedule of an error in the logic solver specification can be substantial. Key input documents Early stage P&IDs or Process Flow Diagrams Early stage control narratives Initial zone and conduit diagram for networks connected to process control and SCAI systems Initial H&RA documentation, including identification and classification of each SCAI function associated with loss events SRS for all SCAI systems Logic solver installation and configuration constraints per safety manual List of typical assessment items Frequency assumed for process control (a.k.a., Basic Process Control System) failures Risk reduction claimed for each instrumented safeguard, e.g., safety control, alarm, interlock, SIS Total risk reduction claimed for instrumented safeguards for each demand cause Maximum risk reduction claimed on each logic solver Physical and functional independence of each SCAI from demand cause and from other safeguards identified for each loss event Degree of completion of the SRS for each instrumented safeguard Consistency of P&ID (or Process Flow Diagram) and control narrative with H&RA and SRS details Initial evaluation of security vulnerabilities

5 3.2 Detailed Design Review (FSA Stage 2) Stage 2 is conducted during the latter part of detailed design. A significant amount of project automation resources and the project schedule are put at risk if the bulk of instrument purchasing, implementation, or application programming work are initiated before the SCAI design is verified to be technically correct and unambiguous. As the process equipment and process control design nears completion, a review of the earlier H&RA work is recommended in order to identify any new consequences and demand causes. For best efficiency, the resulting updates to the H&RA should be completed before the Stage 2 FSA begins. Key input documents Procedure and results for FSA Stage 1 Updated H&RA documentation with related support data (e.g., logic solver constraints), including status on recommendations Updated SRS, including response time, test interval, preventative maintenance, inspection, and repair plans, bypass control plan, compensating measures, and application program safety requirements Final P&IDs Instrument design drawings and diagrams Updated security network and zone diagram for networks connected to process control and SCAI systems Detailed process control and SCAI application program narratives/logic diagrams/cause & effect matrices List of typical assessment items Recommendations from FSA Stage 1 have been satisfactorily addressed Consistency of SRS with any H&RA changes made since FSA Stage 1 Sufficient independence between each SCAI and the demand cause(s) for any loss event, including likelihood of common cause, common mode, and dependent failures Instrument selection justification Achieved hardware fault tolerance (HFT) SIL Verification results SRS (including application program requirements specification) is complete and unambiguous Consistency of final P&IDs, instrument documentation, and automation design with updated H&RA and SRS Description of the maintenance interfaces and physical equipment needed to support initial validation, the required test interval, and the assumed mean time to restoration (MTTR) Security countermeasure level achieved Operation and maintenance representatives have confirmed all H&RA, SCAI allocation, and SCAI design assumptions 3.3 Functional Safety portion of PSSR (FSA Stage 3) FSA Stage 3 is explicitly mandated by both international standards for instrumented safeguards and by process safety management regulations. Stage 3 is completed after all equipment is installed and the SCAI operation is validated per the SRS, but before the startup of the process equipment under protection. In addition, SCAI functional defects found during the assessment must also be resolved, or adequately managed compensating measure put in place, prior to startup. Otherwise, management of change must approve any increase in operational risk that may be unacceptable per the facility's risk criteria. Stage 3 is generally performed as part of the Site Acceptance Test (SAT).

6 Key input documents Procedures and results of FSA Stage 1 and Stage 2, and documented resolution of findings Final H&RA and SRS documentation Final P&IDs with as-built markups Final instrument drawings with as-built markups Final application program design drawings Instrument commissioning and verification documentation Application program verification documentation (e.g., Factory Acceptance Test and SAT detailed records) Validation procedures and documented results Training records of operations and maintenance personnel SCAI operations and maintenance procedures (e.g. operation manual) Assessment results for any design, development, or production tools used for safety lifecycle activities List of typical assessment items Recommendations from FSA Stage 1 and 2 are satisfactorily addressed and no new issues introduced by any changes made in H&RA or SCAI design since FSA Stage 2 Consistency of SRS with respect to any changes in the H&RA made since FSA Stage 2 Verification and validation activities completed with acceptable results, defects resolved, or adequate compensating measures Security countermeasure verification completed with acceptable results or defects resolved All personnel trained on SCAI operation and expected operator and maintenance actions Complete implementation of SCAI procedures As-built updates, after any verification and validation defect corrections, made to SCAI documentation 3.4 Operation and Maintenance Performance Review (FSA Stage 4) FSA Stage 4 is performed periodically during the life of an instrumented safeguard to ensure that the installed systems are providing the risk reduction assumed in the H&RA. This requires the examination of detailed reliability parameters for each SCAI system. Depending on the timing documented in the functional safety plan, the stage 4 FSA and the functional safety audit may be performed at the same time. Key input documents Procedures and results for FSA Stage 1-3, including resolution of findings Current H&RA and SRS documentation Current verification and validation records Demand and spurious trip event records Bypass event log Inspection, test and repair records Management of Change (MOC) records with identified impact to SCAI requirements Training records for individuals responsible for SCAI lifecycle activities List of typical assessment items Documented proof that safety management and verification requirements remain met Documented proof that maintenance (i.e., testing) and operation (e.g., bypass management) activity requirements are being met Actual SCAI demand rate, SCAI device failure rates, and SCAI spurious trip rates are consistent with the assumptions used during design Documented assessment of competence of those involved in the SCAI lifecycle activities 3.5 Proposed SCAI Change Impact Review (FSA Stage 5) FSA Stage 5 exists to ensure that the functional safety impact of any proposed change to the SCAI is understood prior to proceeding with the change and that the appropriate safety lifecycle activities are conducted during change planning and execution. Changes triggering stage 5 include those that increase the number or type of demand causes or that impact the allocation of risk reduction to the instrumented safeguards.

7 Assessors should have access to previous FSA Stages 1-4 documentation results for the safety function being modified or demolished. If these do not exist, the extent of the impact of the proposed change may be very difficult to isolate. This may trigger additional reviews in order to ensure the modification to the safety function will result in acceptable risk reduction and be sufficiently compliant with the current standard. For example, the portions of FSA Stages 1-3 that are relevant to the modified SCAI may need to be performed, instead of a more limited verification and validation change management scope that may be considered appropriate for a minor change to a well-documented safety function. Key input documents Procedures and results for FSA Stage 1-4, including resolution of findings Current H&RA and SRS documentation Current verification and validation records Current P&IDs, application program, and instrument design documentation Change proposal documentation/justification List of typical assessment items Assessment of functional safety impact Identification of safety lifecycle stage that this particular change management activity must begin at Change authorization and close out Evidence of change testing 4 Potential Consequences of Insufficient FSA The following case study illustrates how taking shortcuts and not resolving deficiencies in a timely manner directly contributed to the sequence of events that led to a catastrophic event. This paper does not attempt to address all the contributing causes documented within the official incident report, but instead focuses on deficiencies more directly related to functional safety management of SCAI. 4.1 Managing change. Case Study: Institute, 2008 A runaway chemical reaction occurred on August, 2008, inside a 4,500-gallon pressure vessel in the Methomyl process unit at Institute, West Virginia. This incident resulted in two fatalities, eight injuries, plant personnel evacuation and 40,000 nearby residents shelter-in-place for over three hours. Residences, businesses, and vehicles up to 7 miles away were damaged by overpressure. Traffic on local roads and the interstate highway was disrupted for hours. Business losses included $5.8 million USD in fines and lawsuit settlements to date, in addition to the destruction of facility equipment and loss of production Case study summary The methomyl process was being started up with a new residue treater and a new distributed control system (DCS) after an outage of several months. DCS checkout and standard operating procedure updates were incomplete, and operations had not been trained on the process control modifications resulting from the DCS change. The partial checkout revealed field equipment and instrumentation problems, but these had not been repaired. A solvent-only run required to verify piping integrity and control system functions, perform instrument calibration and control loop tuning, and test safety functions, as well as to prepare downstream equipment (including the residue treater) for operation per facility procedures, had also not been done. Despite these known deficiencies, the decision was made to start up the plant. Routine sampling revealed methomyl concentration in upstream equipment was over twice the

8 operating limit, as a result of significant operational problems, but the startup was continued. The vessel immediately upstream of the residue treater could not be sampled to understand the cause of the upset. On the night of the incident, methomyl-containing solvent was pumped into the residue treater without this vessel being pre-filled with clean solvent and heated to the required minimum operating temperature specified in the standard operating procedure. Figure 2 shows a simplified version of the process schematics. A minimum recirculation flow interlock that would ensure adequate clean solvent quantity and mixing should have prevented this, but this interlock was left bypassed by DCS programmers. A further safety interlock preventing feed to the residue treater without required treater temperature was bypassed, which had become an accustomed practice among some operators. Finally, operations failed to confirm the concentration of methomyl in the residue treater prior to starting automatic control of this unit. When operators started recirculation flow, temperature increased normally. Operators did not take startup sample readings of the residue treater, as required by the procedure. Eventually, the contents reached the critical decomposition temperature. The recirculation flow ceased abruptly because of an incorrectly configured and untested split range loop. Even if the loop performed correctly, however, it would not have been able to compensate for the rapid temperature rise resulting from the runaway decomposition reaction. Figure 2: Institute Process Schematics [6] A high pressure alarm sounded, and the DCS indicated a reading above the maximum safe operating pressure and rising. In reality, there was insufficient time remaining for operations to investigate the alarm and then retreat to a safe distance. The emergency vent system was overwhelmed by the evolving gas from the runaway decomposition reaction of methomyl, due to an initial concentration approximately 20 times the assumed design limit. A few minutes later, the residue treater violently exploded. The estimated energy of the explosion was equivalent to about 17 pounds of TNT. Air monitors to detect toxic chemicals near the Methomyl Unit were not operational, preventing investigators from determining the potential exposure of responders to chemicals released during the incident Case study conclusion The US Chemical Safety Board (CSB) report explicitly stated "effective process safety management inspection and audits are designed to expose potentially dangerous flaws in company

9 operation and procedures, and to ensure equipment are safe to operate. This was not done before the Methomyl process unit start up" [6]. Modifying instrumentation, an application program, or a procedure that is a component of a safety control, alarm, or interlock is certainly a change that most facility personnel should recognize as having a potential safety impact, triggering multiple opportunities for inspection. Table 2 depicts where various defects in the implementation should have been identified through FSA. Table 2 Relationship between FSA and key instrument and control gaps Functional Safety Defect Lack of sampling capability in immediately upstream vessel from prior H&RA recommendations Inadequate MOC, including incomplete control system checkout, calibration, tuning, and related procedure updates Inadequate training prior to startup of modified controls on new DCS Inadequate PSSR [8] Bypass passwords regularly left logged in by supervision during startup Applicable Assessment(s) FSA-1 FSA-3, FSA-4, and FSA-5 FSA-3 FSA-3 FSA-3 and FSA-4 Minimum recirculation flow safety interlock left bypassed by DCS programmers FSA-3 Minimum residue treater temperature safety interlock bypassed by operations Pressure alarm ineffective for a runaway reaction event Pattern of deviations from operation procedures (including safety procedures) during start ups FSA-3 FSA-2 and FSA-3 FSA-4 In this case, the local leadership was involved in the functional safety deviations. It is for this reason that "There must also be a strong ally in upper management to support the auditing process that will be required to ensure that the EGS [engineering guidelines and standards] are used." [7] 5 Conclusion As the process industry has evolved, automation has taken a crucial role in both the normal control of the facility and in the protection of the personnel, environment, and facility assets. The sustainability of safe automation is essential to the successful, productive and safe operations of these plants. Automation MOC, alarm management, verification and validation, functional safety assessments, and bypass controls are widely acknowledged as critical management systems to safe operation. Evidence is provided in many process safety event reports, which describe how flaws in local management systems contributed to the incident. Robust execution of FSA can help identify and correct such systematic failures,

10 reducing the chance that failures will contribute to loss events and preserving limited project resources from costly rework. To be effective, these reviews require sufficiently independent and competent personnel and strong support from senior management to ensure that recommendations are resolved in a timely and effective manner. As with any of the safe automation management system practices, defects in FSA execution should be detected through routine functional safety audits. 6 References [1] IEC Functional safety: Safety instrumented systems for the process industry sector - Part 1-3. IEC Geneva: IEC. [2] ANSI/ISA Identification and Mechanical Integrity of Safety Controls, Alarms and Interlocks in the Process Industry, ANSI/ISA Research Triangle Park: ISA. [3] ISA. [draft Rev 5] Functional Safety: Safety Controls, Alarms, and Interlocks for the Process Sector, ANSI/ISA Research Triangle Park: ISA. [4] CCPS. [draft] Guidelines for Safe Automation of Chemical Processes. New York, NY: Center for Chemical Process Safety. [5] Hochleitner and Roche "How Effective are Your Safety Controls, Alarms, and Interlocks? The Importance of Functional Safety Auditing." 12th Global Congress on Process Safety, Houston, TX, April 11-13, [6] CSB Investigation report - Pesticide Chemical Runaway Reaction and Pressure Vessel Explosion at Bayer CropScience. Report I-WV. Washington, D.C.: U.S. Chemical Safety Board. [7] Raney, Glenn and Angela Summers. [1999]. Common Cause and Common Sense, Designing Failure Out of Your Safety Instrumented Systems (SIS), ISA Transactions, 38, pages [8] CCPS Guidelines for Performing Effective Pre-Startup Safety Reviews. New York, NY: Center for Chemical Process Safety. [9] ANSI/ISA Functional Safety: Safety Instrumented Systems for the Process Industry Sector - Part 1: Framework, Definitions, System, Hardware and Software Requirements, ANSI/ISA (IEC Mod). Research Triangle Park: ISA. [10] OSHA /29/ Use of ANSI/ISA S Parts 1-3 (IEC MOD) to comply with OSHA's Process Safety Management standard. id=25164 [11] OSHA /11/ Recognized and Generally Accepted Good Engineering Practices in Process Safety Management Enforcement. id=30785