Office for Nuclear Regulation

Transcription

1 ONR INSTRUCTION ONR Inspection and Use of Licensee Safety Performance Indicators (SPls) Document Type: Unique Document ID and Revision No: ONR Instruction ONR-OPEX-IN-002 Revision 3 Date Issued: February 2013 Review Date: February 2016 Approved by: A Lindley Director, Civil Nuclear Reactors Programme Record Reference: Trim Folder (2013/67691) Revision commentary: Updated into new template and reviewed for continued accuracy TABLE OF CONTENTS 1. INTRODUCTION PURPOSE AND SCOPE POLICY LEGAL FRAMEWORK AND RELEVANT GOOD PRACTICE ADVICE TO INSPECTORS REFERENCES ANNEX 1: SPI FRAMEWORK... A12 ANNEX 2: SPI FRAMEWORK STRATEGIC INDICATOR PURPOSE STATEMENTS... B13 Crown copyright. If you wish to reuse this information visit for details. You may reuse this information (excluding logos) free of charge in any format or medium, under the terms of the Open Government Licence. To view the licence visit write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or psi@nationalarchives.gsi.gov.uk. Some images and illustrations may not be owned by the Crown so cannot be reproduced without permission of the copyright owner. Enquiries should be sent to copyright@hse.gsi.gov.uk. For published documents, the electronic copy on the ONR website remains the most current publically available version and copying or printing renders this document uncontrolled. Comments on this guide, and suggestions for future revisions, should be made and recorded in accordance with ONR's standard procedures. Comments made from outside ONR should be sent via onrenquiries@hse.gov.uk.

2 1. INTRODUCTION 1.1. Many of the licence conditions attached to the standard nuclear site licence require, or imply, that licensees should make arrangements to comply with regulatory obligations under the conditions. ONR inspects compliance with licence conditions, and also with the arrangements made under them, to judge the suitability of the arrangements made and the adequacy of their implementation. Most of the standard licence conditions are goal-setting, and do not prescribe in detail what the licensees' arrangements should contain; this is the responsibility of the duty-holder who remains responsible for safety. To support inspectors undertaking compliance inspection, ONR produces a suite of guides to assist inspectors to make regulatory judgements and decisions in relation to the adequacy of compliance, and the safety of activities on the site. This inspection guide is one of the suite of documents provided by ONR for this purpose. What are SPls? 1.2. The term "safety performance indicators" (SPls) refers to observable measures that provide insights into safety (OECD, 2008). SPls are also called Key Performance Indicators (KPls), Process Safety Performance Indicators (PSPls) and Nuclear Safety Indicators. The Health and Safety Executive (HSE, HS(G)254) and the Organisation for Economic Co-operation and Development (OECD, 2008) have developed guidance on the process for the development of Safety Performance Indicators, whilst the International Atomic Energy Agency developed a model defining SPls for nuclear power plants (Tecdoc 1141, IAEA, 2000) that can be considered relevant good practice. Why are SPls useful in High Hazard Industries? 1.3. Safety Performance Indicators are a way by which the nuclear industry can demonstrate the efficacy of their risk controls through monitoring the effectiveness of their management system and its associated risk controls. The intent is to provide information on safety, motivate people to work safely and contribute towards performance improvement (HSE, 2001) Investigations and reports into major accidents in high hazard industries have recommended the use of safety performance indicators in the prevention of future incidents e.g. the investigation following the explosions at the Buncefield Oil Storage Depot recommends that: 1.5. The sector should agree with the Competent Authority on a system of leading and lagging performance indicators for process safety performance. This system should be in line with HSE's recently published guidance on Developing process safety indicators HSG254.' (Buncefield Major Incident Investigation Board, 2008). The "Baker Panel Report" following the 2005 Texas City Oil Refinery Explosion makes similar recommendations (BP US Refineries Independent Safety Review Panel, 2007). ONR-OPEX-IN-002 Revision 3 Page 2 of 17

3 2. PURPOSE AND SCOPE 2.1. To provide guidance to Inspectors in the on the use of Safety Performance Indicators (SPls) by Licensees for the management of health and safety and for use by the ONR to support regulatory activities Inspectors should: Ensure the licensee has a robust SPI process as part of its overall arrangements for monitoring nuclear safety performance (Plan-on-a-Page (POAP) 2.1: Influence Improvements). Consider whether SPls selected are appropriate and effective during relevant interventions (POAP 2.2: Ensure Compliance). Review the Licensees analysis of performance as part of normal regulatory business, discuss with the licensee as appropriate & consider any issues arising as part of the intervention planning process (POAP 2.5: Make Balanced Judgements). 3. POLICY SPI Project in ONR 3.1. The IAEA published guidance on Operational Safety Performance Indicators for Nuclear Power Plants (IAEA Tecdoc ). It presents a suite of indicators designed to provide a comprehensive picture of all aspects of safe nuclear plant performance. HSE's Nuclear Directorate, in consultation with the UK nuclear industry, tailored this framework for broader application. A project was established, overseen by a joint industry/regulator Steering Group, to roll the framework out across the industry. The project entailed licensees submitting SPI data to the Regulator. However, the intention is to move to 'normal business' which focuses on the licensees own analysis and use of SPI information A specific subgroup of the Safety Directors Forum is taking the Safety Performance Indicators initiative forward. This group is overseeing the move to 'normal business' and reviewing use of the SPI framework. ONR supports this forum as a means of creating a culture of continuous improvement and sustained excellence in operation. This is expanded in ONR's Plan-on-a-Page 2.4 as follows: 'it is our role to engage with the industry strategically to influence them so they create an excellent safety and security culture and have the processes and measures in place to demonstrate compliance and assurance' Within ONR, an inspector in the Civil Reactors Programme (Licensing and Organisational Capability) provides central support for SPI activities. Another provides support to the CNRP Programme. Framework for Nuclear Safety Performance 3.4. Key attributes for nuclear safety described in the framework are: ONR-OPEX-IN-002 Revision 3 Page 3 of 17

4 Sustained excellence of operation - this is related to the operation of the plant, and is linked to physical operations including programme delivery, status of plant and events. The rationale is that normal operations should be adequately and appropriately controlled and that sustained excellence of operation is safer. - these indicators refer to the processes in place for safe operations, including safety management, emergency management and operational risk control. Indicators are clustered around the protection of the public, workers and the environment. - these indicators covers the implementation of management systems, which critically depend on individual's attitudes to safety, striving for improvement, and leadership and management The framework has a hierarchical structure with overall and strategic indicators described against these three key attributes of nuclear safety set out above. Licensees should develop site-specific strategic indicators that will provide useful information for the management of their hazards and activities, and they should provide indicators across the breadth of the model. Furthermore, ONR recognises that many licensees will need to develop more detailed and specific nuclear process safety indicators and there may be value in following the established process set out in HS(G)254 (HSE, 2006) The key attributes and overall indicators are shown in Figure 1. Annex 1 shows the full model, whilst Annex 2 provides a list of the purpose statements for all of the strategic indicators which will help Licensees develop their own SPls. NUCLEAR SAFETY PERFORMANCE INDICATOR FRAMEWORK NUCLEAR SAFETY PERFORMANCE Sustained excellence of operation Control of hazards Positive safety culture ATTRIBUTES OF NUCLEAR SAFETY PERFORMANCE Programme delivery Events Public Workers Environment (Radioactive waste management) Leadership Capable Organisation Decision Making Learning From Experience OVERALL INDICATORS STRATEGIC INDICATORS Figure 1 - Nuclear Safety Performance Indicator Framework 3.7. Licensees can also use opportunities such as Periodic Reviews of Safety Cases (LC15) and Hazard Identification & Risk Evaluation Reviews under REPPIR to help the identification and development of site specific performance indicators. 4. LEGAL FRAMEWORK AND RELEVANT GOOD PRACTICE 4.1. Legal requirements for management of safety are well established (e.g. HSW; MHSWR; 1999). Regulation 5 of MHSWR sets out the requirement for effective monitoring arrangements which is developed further in the ACOP. Safety Performance Indicators are one way of monitoring health and safety and form part of the suite of measures for effective self-monitoring. ONR-OPEX-IN-002 Revision 3 Page 4 of 17

5 4.2. Relevant legal requirements outlined above are supplemented by a significant amount of guidance which establishes good practice for measuring performance particularly IAEA Tecdoc 1141 and HS(G)254 'Developing Process Safety Indicators' ONR's internal guidance recognises the role of Safety Performance Indicators in providing assurance that the Licensees' arrangements for compliance with Licence Conditions and Safety Assessment Principles (SAPs) are effective (HSE, 2006), as well as providing a demonstration of the overall effectiveness of management controls. Within the 'Leadership and Management for Safety' SAPs it states, "Leading and lagging indicators should be used to monitor performance over time to track the effectiveness of the control of risks" In relation to the Management Prospectus (MP),T/AST/072 suggests that licensees should describe the "approach to the use of safety performance indicators... intended to promote nuclear safety". Additionally, Nuclear Installations Licence applicants are expected to submit a MP and one area that it is expected to cover is "details of performance indicators to monitor health and safety effectively" Other ONR guidance acknowledges the role of SPls, e.g. the 'Function and Content of the Nuclear Baseline' guidance (T/AST/065) highlights the importance of leading and lagging measures of performance, and states that Licensees should consider the likely outcomes of both positive and inadequate delivery of nuclear safety. The Nuclear Industry's Code of Practice on the Nuclear Safety Baseline and Management of Change also advocates use of performance indicators to: a) monitor ongoing capability of the organisation and, b) monitor the impact of organisational change 4.6. Furthermore, SPls are acknowledged in ONR's inspection guidance (e.g. INS/008 and G/INS/008), and are listed as a source of information for monitoring Licensees performance, and as an input to intervention planning. 5. ADVICE TO INSPECTORS SPls and Safety Management - use of SPI framework 5.1. Licensees should develop and use a suite of indicators that demonstrate the adequacy of their organisational arrangements for the management of safety and control of risks. Licensees may not differentiate between SPls and their often more extensive set of Performance Indicators. However, ONR would expect Licensees to use the SPI Framework Model as they develop indicators to provide assurance that the risk controls are adequate and appropriate. Licensees should be able demonstrate to ONR that they can map their SPls back to the Framework for Nuclear Safety However, it is important to acknowledge that SPls are only one means of obtaining feedback on the adequacy of their risk and safety management controls, and they need to be integrated and analysed together with knowledge and information from other sources used within an organisation (e.g. operational feedback, incident ONR-OPEX-IN-002 Revision 3 Page 5 of 17

6 investigations, audit outcomes, safety culture surveys etc). It is important to understand how licensees monitor their own performance and respond to indicators. SPls are one element of the Licensees' organisational learning process and, as such, should be used to support a holistic view of safety governance and learning Inspectors can follow the Troubleshooting Guide - Table 1 below, when considering the adequacy of the Licensees SPls. It is important to avoid the pitfall highlighted by Hopkins (2009): "(don't) manage the indicator itself rather than the phenomenon of which it is supposed to provide an indication" Table 1: Troubleshooting SPls Indicators must Have an explicit link to nuclear safety Be clear and concisely worded Allow the information to be presented clearly Indicators must not Be ambiguous Be easily manipulated Be a box ticking exercise Be clear what good and bad performance looks like Lead to 'perverse' behaviours, e.g. spurious target chasing Allow the setting of meaningful thresholds Have a clear link to risk controls Be specific and state what is to be measured and how Lead to misunderstanding and misinterpretation Allow lowering a measure's quality to increase its quantity e.g. inspection metrics, e.g. substituting visual inspections for more invasive techniques. Be an opaque process, where metric owners and data providers do not understand what they are collecting or why Be a mix of qualitative and quantitative measures Document the rationale for inclusion of measures, and its links to the ONR SPI framework Be viewed as a mechanistic and regulatory process Be used if there is no link between safety management, risk control and performance indicators Lead to appropriate actions Evolve appropriately as a suite of indicators Be static Be focused solely on 'poor' areas but tackle apparently 'good' areas too to avoid complacency ONR-OPEX-IN-002 Revision 3 Page 6 of 17

7 Assessing the Licensees SPI process 5.4. ONR would expect Licensees to have a defined process for the development, collection, use and management of safety performance indicators. The process described below is one method, and is consistent with the ONR SPI framework of indicators, however alternate approaches can be adopted Key steps for the development of a successful SPI Process are defined within HSG254 (HSE, 2006) and the process is as follows: Step 1 - Establish the organisational arrangements (e.g. a champion, a team); Step 2 - Decide upon the scope and consider what can go wrong and where; Step 3 - Identify the risk control systems in place to prevent major accidents; Step 4 - Identify appropriate indicators including leading and lagging; Step 5 - Establish the data collection and reporting system; Step 6 - Review Whilst the steps set out above describe the process of selecting indicators it is important to establish how and where they will be reported i.e. the relevant decision making bodies. Sufficient analysis of the data (and trends) should be included in reports to enable action to be taken to deal with emerging concerns Inspectors should consider the following issues concerning the SPI process: Key Issue Expectation Is there a clear link between the SPls and the hazard profile of the site? How are SPls used within the organisation - and at what levels/functions? How are key messages reported to the Executive level? Is it clear how the SPls map onto the ONR SPI framework? Is there evidence that the SPI data has been verified, checked or controlled for correctness? The licensee should have systematically considered nuclear safety hazards and the risk controls in place to control and mitigate when identifying specific performance indicators. SPls should form an aide to decision making at difference levels (and functions) within the organisation. Risks of aggregation should be recognised and key messages should not be lost during aggregation? Whilst the Licensee may not use the framework structure for performance reporting they should be able to show how their indicators map back to & demonstrate that they have covered all the strategic indicator areas. The framework is also useful for when indicators are reviewed. Obviously, poor data quality could lead to misinterpretation of the true situation and drive the wrong action. SPI definitions need to be clear and unambiguous. ONR-OPEX-IN-002 Revision 3 Page 7 of 17

8 Key Issue Expectation What confidence and weight do they place on the SPls? Is there a danger of overreliance? Is there evidence that monitoring of SPls has resulted in actions to improve performance? How are conflicts between SPls and other measures resolved? Can managers explain how SPI data is interpreted? SPls can only be part of the picture and should not be relied upon as the only way performance is monitored. The licensee should recognise the limitations of SPls and be able to manage this. Clearly, this is the aim & the licensee should be able to describe examples. Care needs to be taken to avoid any 'knee jerk' reactions which in themselves may create problems. There is often a scrutiny meeting to weigh up information from difference sources. Sometimes issues are flagged even though an indicator may not show an adverse trend. The licensee should be interpreting and interrogating data & and considering alternative explanations. Have additional measures been identified to target areas of concern? A considered approach should be taken to use indicators when tackling specific problems or issues. Short-term indicators can prove very useful. Is the risk of perverse behaviour, for example chasing a target rather than quality, acknowledged? Have measures been put in place to manage this? Are SPls perceived as an integral part of the Licensees' management system rather than something required by the regulator? Are SPls periodically reviewed to confirm their ongoing efficacy Care should be taken that the introduction of indicators does not have an adverse effect on safety and that management of the indicator does not become an end in itself. Due in part to the history of the SPI project and the requirement for reporting data to ONR, there is sometimes the perception that indicators are for the regulator. Clearly, this is not the case and any misconception should be corrected. It is important the overall suite of SPls is kept under review and individual SPls changed if they are not providing useful information. It may be that if, for example, numbers reported are always so low as to glean any useful information then a leading indicator may prove more effective. SPIs and Other Inspection Activities 5.8. Safety Performance Indicators can only reflect the health of the underlying systems they are set up to monitor. ONR would expect licensees to have arrangements in place to provide assurance that the management system is working as intended. Internal assurance programmes are important and provide valuable information to enable intelligent interpretation of SPI data. ONR-OPEX-IN-002 Revision 3 Page 8 of 17

9 5.9. Specific safety performance indicators can also be explored during other inspection including Licence Condition Inspections, where specific indicators could be interrogated. Licensees should be able to explain: To what does the SPI provide an indication? Why was the SPI selected? What are the inbuilt assumptions? How are decisions made about abandoning SPls? How are alternatives identified? How are long-term trends be identified? How SPls are used with other measures of organisational feedback? That some SPls are project specific and used for short term monitoring? How the chosen SPls are used? Who reviews them? What actions have been taken as a result? SPIs and Investigation As identified by a number of major accidents, SPls are suggested as a mechanism by which companies can prevent the recurrence of an incident. Similarly, within the ONR following an incident, SPI information can be considered. For example, Were early warning signs shown in the SPls? If these had been acted on could the incident have been prevented or been less severe? Did the information show maintenance backlogs, missed inspections etc? Has the SPI process been reviewed in light of the incident to avoid oversights in future? Could any SPls be identified to avoid a recurrence? Could SPls be used to monitor the effective implementation of recommendations and remedial measures? ONR would expect that SPls be reviewed as part of the licensees' own investigation process. Use of SPIs by ONR Safety Performance Indicators can provide useful information for ONR inspectors. They can be used during inspection, assessment and investigation activities. ONR inspectors should agree which reports will be routinely submitted to the ONR as part of normal regulatory business. Reports should be submitted monthly or as agreed with site inspectors. Performance reports will provide a basis for discussion during site visits e.g. Regulatory Forums; Level 3 RIFs. Annual SPI review meetings should normally be held and mid-year meetings where further ONR intervention is required Some Licensees' use the SPI Reporting and Input Tool (SPIRIT), a software package, which is managed by the ONR Information will also be discussed internally within the ONR as appropriate, for example, at Regulatory Review Meetings and will provide one input to planning regulatory activities for the various licensees and their intervention plans. ONR-OPEX-IN-002 Revision 3 Page 9 of 17

10 5.15. Summary of Actions: ONR would expect Licensees to use the SPI Framework Model as they develop indicators to provide assurance that the risk controls are adequate and appropriate. Licensees should be able demonstrate to ONR that they can map their SPls back to the Framework for Nuclear Safety (see para 18). ONR would expect Licensees to have a defined process for the development, collection, use and management of safety performance indicators (see para 21). ONR would expect licensees to have arrangements in place to provide assurance that the management system is working as intended (see para 25). ONR would expect that SPls be reviewed as part of the licensees' own investigation process (see para 27). ONR inspectors should agree which reports will be routinely submitted to the ONR as part of normal regulatory business. Reports should be submitted monthly or as agreed with site inspectors (see para 28). Annual SPI review meetings should normally be held and mid-year meetings where further ONR intervention is required (see para 28) In conclusion, properly constituted SPls are considered by ONR to be an important part of the suite of measures in place to demonstrate that the licensee has an effective safety management system. Further Information: Chris Flanagan (SPI Project Manager) Organisational Capability and Nuclear Site Licensing Civil Nuclear Reactors Programme Extn 4359 Simon Yates (CNRP Support) Civil Nuclear Reactors Programme Cheltenham VPN REFERENCES 1) BP US Refineries Independent Safety Review Panel (2007) The report of the BP US Refineries Independent Safety Review Panel 2) Buncefield Major Incident Investigation Board (2008) 3) Hopkins, A (2009) Failure to Learn: the BP Texas City Oil Refinery Fire. CCH press, Australia 4) HSE (1997) Successful health and Safety management. HSG65 5) HSE (2001) A guide to measuring health and safety performance 6) HSE (2006), Developing Process Safety Indicators. A Step-by-Step Guide for Chemical and Major Hazard Industries. HSG254 7) HSE, Safety Assessment Principles for Nuclear Facilities, 2006 Edition, Revision 1, January 2008 ONR-OPEX-IN-002 Revision 3 Page 10 of 17

11 8) HSE INS/008 and G/INS/008, Intervention Planning and Guidance on Intervention Planning, February ) HSE, Nuclear Site Licence Conditions 10) IAEA INSAG-13, Management of Operational Safety in Nuclear Power Plants, International Atomic Energy Agency, October ) IAEA-TECDOC-1141, Operational Safety Performance Indicators for Nuclear Power Plants, International Atomic Energy Agency, May ) ONR Plan on a Page 13) NRC (2008) Davis-Besse Reactor Pressure Vessel Head Degradation: Overview, Lessons learned and NRC Actions Based on Lessons Learned. NUREG/BR- 0353, Rev1 Not used. 14) OECD (2008) Guidance on developing safety performance indicators related to Chemical Accident Prevention, Preparedness and Response: Guidance for industry. 15) T/AST/065 Function and Content of the Nuclear Baseline 16) T/AST/072 Function and Content of a Safety Management Prospectus ONR-OPEX-IN-002 Revision 3 Page 11 of 17

12 ANNEX 1: SPI FRAMEWORK NUCLEAR SAFETY PERFORMANCE INDICATOR FRAMEWORK NUCLEAR SAFETY PERFORMANCE Sustained excellence of operation Control of hazards Positive safety culture ATTRIBUTES OF NUCLEAR SAFETY PERFORMANCE Programme delivery Status of plant Events Public Workers Environment (Radioactive waste management) Leadership Capable Organisation Decision Making Learning From Experience OVERALL INDICATORS Operating performance Decommissioning performance Maintenance Plant modification Incidents on the site Challenges to safety systems Safety plant t f i Radiological protection Conventional health & safety Radioactive waste Contaminated land Leadership Behaviour Backlog of safety issues Staffing levels Competence Challenge function Procedural effectiveness Learning from experience Safety case management STRATEGIC INDICATOR S Construction/ commissioning performance Plant condition Configuration risk Challenges to safety Attitudes and Values Emergency preparedness Safety plant O t f i * Human performance spans this framework and licensees should be able to describe how this is reflected in their choice of indicators ONR-OPEX-IN-002 Revision 3 Page 12 of 17

13 ANNEX 2: SPI FRAMEWORK STRATEGIC INDICATOR PURPOSE STATEMENTS Sustained excellence of operation Programme delivery Operating performance Compare the actual plant performance to that planned. Sustained excellence of operation Programme delivery Decommissioning performance Compare progress with decommissioning preparations and actual progress to that planned. Sustained excellence of operation Programme delivery Construction / commissioning performance Compare actual progress with construction and commissioning of new plant to that planned (excludes plant for radioactive waste management and decommissioning). Sustained excellence of operation Status of plant Maintenance Effectiveness of the maintenance of nuclear safety related plant (planning and execution) Sustained excellence of operation Status of plant Plant modifications Effectiveness of plant modification process. Sustained excellence of operation Status of plant Plant condition To monitor the condition of nuclear safety related plant. Sustained excellence of operation ONR-OPEX-IN-002 Revision 3 Page 13 of 17

14 Events Incidents on the site To monitor all events including near misses. Public Challenges to safety systems To monitor challenges and effectiveness of nuclear safety barriers, to help ensure that these are maintained. Public Safety plant out of service To monitor the unavailability of nuclear safety plant, to help ensure that it is minimised. Public Configuration risk To monitor the ability to manage equipment out of service so that the risk is controlled at all times. Public Emergency preparedness To monitor the ability to protect the public if a radioactive release occurs. Workers Radiological protection To monitor the ability to protect workers from ionising radiation. ONR-OPEX-IN-002 Revision 3 Page 14 of 17

15 Workers Conventional health & safety To monitor the ability to protect workers from conventional health and safety hazards. Workers Challenges to safety systems To monitor challenges to worker safety systems, to help ensure that these are minimised. Workers Safety plant out of service To monitor the unavailability of safety plant to protect workers, to help ensure that it is minimised. Environment (Radioactive waste management) Radioactive waste To monitor the management of radioactive waste on-site, including forward planning, progress with waste treatment (in particular, passivation and hazard reduction) and waste minimisation. Environment (Radioactive waste management) Contaminated land To monitor progress with planning, characterisation, control and remediation of radioactively contaminated land. Leadership Leadership behaviour Attention taken by leaders to nuclear safety Leadership ONR-OPEX-IN-002 Revision 3 Page 15 of 17

16 Backlog of safety issues Ability to respond to nuclear safety related issues in a timely and appropriate manner Capable organisation Staffing levels Management of resource levels, both current and future, consistent with nuclear safety needs Capable organisation Competence Ensure people assigned nuclear safety duties are (technically) competent Decision making Challenge function Frequency and effectiveness of management response to challenge Decision making Procedural effectiveness Preparation of and adherence to procedures that affect nuclear safety Decision making Attitudes and values Behaviours and attitudes of individuals align to the organisational values Learning from experience Learning from experience Effectiveness of translating experience capture into improvement ONR-OPEX-IN-002 Revision 3 Page 16 of 17

17 Learning from experience Safety case management Effectiveness of producing and maintaining a fit-for-purpose Safety Case Human Performance - is not now allocated a specific strategic indicator Human performance spans this framework and licensees should be able to describe how this is reflected in their choice of indicators ONR-OPEX-IN-002 Revision 3 Page 17 of 17