Key Performance Indicators (KPIs) for staff fatigue management systems

Transcription

1 Key Performance Indicators (KPIs) for staff fatigue management systems Jeremy MAWHOOD Office of Rail and Road (ORR), Manchester, United Kingdom Abstract. To manage staff fatigue, organisations need a comprehensive fatigue risk management system (FRMS). One difficult FRMS element is fatigue KPIs - metrics for evaluating fatigue controls effectiveness, identifying hotspots and incipient control failures, and tracking trends over time, between roles, sites etc. This paper: outlines an approach for deriving fatigue KPIs; suggests example fatigue KPIs for common railway fatigue problems; and identifies guidance from GB s rail regulator and other sectors which may be useful when devising or reviewing fatigue KPIs. Keywords. Fatigue; FRMS; KPI; indicator. 1. Background Staff fatigue is a significant source of risk across industries including the rail sector. There is extensive guidance to support the industry s efforts on implementing a comprehensive fatigue risk management system (FRMS), as advocated by the GB rail regulator, the Office of Rail and Road (ORR), the Railway Safety and Standards Board (RSSB) and similar bodies in other industries (ORR, 2012; RSSB, 2012; RSSB, 2015). Although different industries and authors describe FRMS elements slightly differently, they all in effect outline an iterative risk control loop, consisting of elements covering: Policy; Organising; Planning and implementing; Measuring; Auditing and Reviewing these repeating stages are often abbreviated to the acronym POPMAR (Mawhood and Dickinson, 2012). As part of the measurement and monitoring activities in this control loop, ORR advises that an organisation s FRMS should include a system for evaluating and reporting the overall effectiveness of fatigue controls (ORR, 2012). Metrics or key performance indicators (KPIs) should be established, to help track the effectiveness of the FRMS over time and between roles, sites etc. The organisation should monitor these metrics regularly, looking for signs of fatigue hotspots where controls may need strengthening, and assessing trends over time. However, in its dealings with GB rail organisations over recent years, ORR has observed that rail organisations use of fatigue KPIs would greatly benefit from an improved approach to fatigue KPIs. 1.1 Common issues encountered with fatigue KPIs in GB rail During ORR s inspection and investigation work, fatigue KPI issues encountered include: An absence of any fatigue KPIs; Reliance on a single, simplistic KPI, usually number of working hour exceedances

2 rather than a wider suite of KPIs measuring the effectiveness of wider FRMS components; Reliance on old, excessively tolerant working hour limitation exceedances. Better than nothing, but liable to give a falsely re-assuring picture if the limits being exceeded are, in light of more recent fatigue research, too tolerant; Using assessments of planned working patterns, without also considering the - often very different - actual patterns worked; Using excursions above excessively tolerant company thresholds for bio-mathematical fatigue tool scores, again with the risk of false re-assurance. Benefits and potential problems from using bio-mathematical fatigue tools are briefly outlined in ORR (2012) and RSSB (2012) guidance, and more fully discussed in RSSB Project T1083, (RSSB, 2016). See ORR (2016b) for a summary of some points to take into account if using GB rail s most commonly used bio-mathematical fatigue tool, the Fatigue and Risk Index; Inadequate monitoring of fatigue KPIs by senior managers. Overall, ORR concluded that many rail organisations fatigue KPIs, where they existed, were often not sufficiently risk-based, and that companies were not making best use of KPIs in their fatigue risk control loops. ORR decided there would be benefits in surveying the literature on fatigue KPIs and distilling learning points, which may help organisations devise and make better use of risk-based fatigue KPIs. 2. Literature survey A brief literature survey was conducted using the themes of fatigue KPIs, SPIs (Safety Performance Indicators) and other management system indicators for fatigue. The terms/acronyms KPI and SPI are often used interchangeably. Strictly, SPIs are the safety subset of an organisation s wider suite of KPIs the latter s scope extends beyond safety to other business goals. Some key sources identified during the survey are identified below, though it must be stressed that the literature survey was not exhaustive. 2.1 Generic KPI/SPI guidance Generic guidance on KPIs/SPIs is given in the Organisation for Economic Co-operation and Development s guidance for chemical accident prevention (OECD, 2008). The Health and Safety Executive have provided a step-by-step guide on developing SPIs for chemical and similar major hazard industries (HSE, 2010). The principles of this guidance, although aimed primarily at the chemical industries, are transferable to the railway industry, and RSSB have produced a brief How to guide on measuring safety performance in the railway industry (RSSB, 2013), and a more detailed publication on devising and using suitable metrics (RSSB, 2014). 2.2 Fatigue-specific KPI/SPI guidance The increasing emphasis in recent years on controlling fatigue risks by means of a comprehensive FRMS has led to various high-hazard industries, key amongst them the petro-chemical and aviation sectors, promulgating sector-specific guidance on devising and implementing fatigue

3 KPIs. Perhaps the most comprehensive menu / picklist of possible fatigue KPIs is contained in guidance for the oil and gas industries (IPIECA, 2012). This proposes devising indicators both for fatigue contributors and for corresponding fatigue controls, listing desired outcomes from the controls with corresponding critical control elements, suggested leading and lagging indicators and metrics, and assessment questions. Much of the content could be readily adapted by railway organisations to their own operations, provided careful consideration is given to selecting only relevant content, and tailoring suggestions to their own work and risks blindly adopting suggestions derived elsewhere is clearly risky. The aviation sector has also been an early adopter of the FRMS approach, and suggested approaches on KPIs for fatigue are outlined in guidance from the International Air Transport Association (IATA, 2014), Gander et al (2014) and the Civil Aviation Authority (CAA, 2015). 3. Drawing up guidance for railway staff fatigue KPIs ORR distilled key content from the identified literature and its experience of inspecting and investigating fatigue controls, and drafted an information document on fatigue KPIs. This was used to structure a workshop on the topic held in May 2017 at the Chartered Institute of Ergonomics and Human Factors Conference, was subsequently made available on ORR s website (ORR, 2017), and has been brought to the attention of all GB mainline passenger and freight train operating companies. Initial feedback appears positive, with organisations finding it useful. The following provides a summary of the information document s content. 3.1 Deriving fatigue KPIs A generic process for deriving KPIs is detailed in RSSB's Measuring Safety Performance (RSSB, 2013 and 2014). Questions to help derive KPIs are summarised in Figure 1 below. Fig.1. Questions to help derive performance indicators (RSSB)

4 Good fatigue KPIs help measure the presence and effectiveness of fatigue defences, providing an early warning of weaknesses in fatigue controls. In selecting fatigue KPIs, the guidance advocates concentrating on those elements which are: (a) most critical to controlling fatigue risk, and (b) most vulnerable to degradation. 3.2 Types of indicators The literature on KPIs uses various systems for categorising different types of indicators. Some categorise indicators as either leading or lagging indicators, whilst others distinguish between activity and outcome indicators. However, there is no hard and fast distinction between activity and outcome indicators, and similar grey areas exist between leading and lagging indicators in reality these terms describe points towards opposite ends of a spectrum. Nevertheless, in devising a suitable mix of fatigue KPIs, it can be helpful to consider the following broad descriptions. Activity indicators - essentially measure whether barriers (fatigue control layers) are in place to avoid adverse fatigue outcomes. Outcome indicators - measure whether the fatigue control layers (activities) are having the desired effect on fatigue risk. In devising KPIs for fatigue it can be helpful to think of three kinds of outcome indicators: o Results, for example the results of fatigue competence assessments; o Precursors, for example working patterns which contain features likely to increase fatigue (e.g. the fatigue factors in ORR, 2016a) or produce higher predicted bio-mathematical fatigue tool scores; o Incidents, for example accidents or near misses where fatigue appears to have contributed, or reports of fatigue concerns from staff. 3.3 Dual assurance It is generally best to gain dual assurance, by using a mix of both Activity indicators and Outcome indicators as outlined in Figure 2 below. Fig.2. Types of indicators (RSSB)

5 ORR s information sheet stresses that tailoring fatigue KPIs to an organisation s own operation is vital, carefully targeting the most critical and vulnerable fatigue control elements, and involving staff and their representatives such as trade unions. Honesty is needed, requiring careful thought about what metrics would give the best handle on how well current fatigue controls are working. If for instance the company s biggest problems from fatigue are due to variable, short notice shifts, there is a need to think through how the company can best measure both the extent of the problem (outcomes) and the presence and effectiveness of relevant controls (activities). In order to assess the effectiveness of fatigue controls for different parts of the operation, companies will probably need several layers of indicators of varying detail (granularity): Individual depots or management units are likely to require a suite of fine grain, more detailed tactical indicators, whereas; Coarser, summary indicators will be needed for more senior, strategic management decisions. Quality is more important than quantity of KPIs, and it may often be possible to bundle together several lower level indicators into suitable higher-level summary indicators. For instance, good scheduling software may allow the bundling together of many fatigue factor indicators (ORR, 2016a) into one or a few summary indicators for a strategic picture, whilst facilitating ready breakdown and analysis by different users tactically, according to need. 3.4 Calibrating KPI thresholds The guidance emphasises that calibrating thresholds for KPIs is important, so that the organisation can assess the effectiveness of controls and detect trends and changes without being repeatedly swamped with alerts. However, it is also pointed out that if an organisation is using risk-based thresholds and is swamped with alerts, this suggests serious weaknesses in current fatigue controls, which should be urgently investigated and addressed. Contrary to what ORR has found with some organisations, it is no good the organisation merely making the thresholds less demanding - leading to false re-assurance - unless they also address the reasons for so many alerts being generated. 3.5 Features of good fatigue KPIs/SPIs Organisations are advised, when devising fatigue KPIs, to consider the features of good performance indicators listed in Fig 3 below (from RSSB, 2013):

6 Fig.3. features of a good performance indicator (RSSB) 3.6 Examples of fatigue KPIs The information document outlines some possible example fatigue KPIs, devised with common railway fatigue problems in mind, and these are summarised in the Appendix to this paper. The document also collates extensive further examples of fatigue KPIs from other industries, but these are too numerous to reproduce here. The information document stresses that the example KPIs suggested are illustrations of possible approaches, and that whilst some may be suitable for adapting to a particular organisation s own operation, each organisation should consider its own main sources of fatigue risk and devise a suitable mix of corresponding indicators. 4. Conclusion Weaknesses in many GB railway organisations use of indicators / metrics for measuring the effectiveness of their fatigue controls prompted the regulator to survey approaches in the risk management literature to fatigue KPIs. An information document was produced, summarising an approach for devising fatigue KPIs which are more risk-based, and collating in one place examples of possible fatigue KPIs which organisations may find useful when devising KPIs to measure the effectiveness of their own fatigue controls. 5. Appendix Some example railway fatigue KPIs Working pattern indicators (good scheduling software can help identify these): % of (planned & actual) rosters assessed for selected fatigue factors (see Fatigue factors guidance ORR, 2016a) (an activity indicator) % or number of shifts involving selected fatigue factors as below (outcome indicators):

7 Time of day factors: Number or % of shifts covering 00:00 and 05:00 (night shifts) Number or % of shifts starting between 05:00 and 07:00 (early shifts) Number or % of shifts starting before 05:00 (very early shifts) Duty length factors: Number or % of shifts starting before 05:00 & over 8h long Number or % of shifts which are day shifts over 12h long Number or % of shifts which are night shifts (covering 00:00-05:00) & >10h long Number or % of shifts which are earlies (starting before 05:00) & >10h long Recovery time factors: Instances of less than 2 days rest after a block of consecutive night shifts (00:00 and 05:00) Instances of less than 2 days rest after a block of consecutive early starts (<07:00) Instances of more than 13 consecutive shifts without a 48h break Intervals between duties factors: Instances of less than 12h rest in any 24h period for day shifts Instances of less than 14h rest in any 24h period for night shifts Instances of only one day rest after night shifts o Cumulative fatigue factors: Instances of more than 4 consecutive 12h day shifts Instances of more than 4 consecutive nights (covering 00:00-05:00) in a rotating pattern Instances of more than 4 consecutive early shifts (starting between 05:00 and 07:00) in a rotating pattern Instances of more than 3 consecutive night shifts (covering 00:00-05:00) over 8h long Instances of more than 6 consecutive night (covering 00:00 to 05:00) or early shifts (starting between 05:00 and 07:00) in a permanent pattern Instances of more than 12 consecutive day shifts Instances of more than 7 consecutive 8h shifts Instances of more than 55 hours worked in a 7 day period o Circadian phase shift (body-clock adjustment) factors: Number or % of shifts involving a backward rotating pattern Number or % of shifts involving a rotating pattern of about a week Number or % of shifts where successive shift start times vary by more than 2 hours Number or % of shifts which are a first night shift

8 Other examples of working pattern indicators: o % of (planned & actual) rosters assessed using bio-mathematical fatigue tool (activity) o % of shifts (planned & actual) with predicted bio-mathematical tool scores exceeding company guidelines. o % of shifts identified as exceeding company fatigue rules which were fatigueassessed by a person competent in fatigue management before authorisation or refusal (activity) o % of shifts more than x mins longer than originally planned (outcome) o Number or % of shifts covered by overtime or rest day working (outcome) o Number or % of shifts where staff had less than x days notice of a shift / changes to a shift (reduces ability to plan and obtain sleep) Combined work and journey (door-to-door) time indicators: o % of mobile worker shifts where door-to-door times were checked against company rules by a supervisor beforehand e.g. using Googlemaps or similar journey planning software (activity) o No or % of mobile worker shifts with combined door-to-door time over company s limit e.g. >12h (outcome) Fatigue competence indicators: o % of staff carrying out safety-critical staff work who have been assessed as competent in fatigue o % of staff who design working patterns (or authorise changes) who have been assessed as competent in the company s fatigue assessment and control processes (activity). o % of fatigue refresher training completed against programme (activity) Fitness for duty indicators: o % of shifts where fitness-for-duty (e.g. recent sleep & time awake check) confirmed (activity) o % of shifts where staff reported <6h sleep in previous 24h, or <12h sleep in previous 48h o % of safety critical staff who have been screened for sleep disorders (activity) Contractual chain indicators: o % of safety critical contractors whose fatigue controls have been reviewed against client s fatigue management expectations (activity) o % of safety critical contractors whose fatigue controls have been reviewed against client s fatigue management expectations and assessed as satisfactory (outcome) Fatigue reporting indicators: o Number of fatigue concerns reported (outcome) o % of reported fatigue concerns investigated by joint fatigue group (activity)

9 References Civil Aviation Authority (2015.) CAP 1267: EASA Flight Time Limitations Guidance ORO- FTL-110. Retrieved Aug 4 th, 2017 from Gander, P.H., Mangie, J., Van den Berg, M.J., Smith, A.A.T., Mulrine, H.M. and Signal, T.L. (2014). Crew fatigue safety performance indicators for fatigue risk management systems. Aviation, Space and Environmental Medicine, Vol. 85 No.2 Feb Retrieved Aug 4 th, 2017 from tors_for_fatigue_risk_management_systems Health & Safety Executive (2010). HSG254: Developing process safety indicators. A step-bystep guide for chemical and major hazard industries. HSE Books, Sudbury. Retrieved Aug 4 th, 2017 from International Air Transport Association IATA (2014). Fatigue safety performance indicators (SPIs): a key component of proactive fatigue hazard identification. Retrieved Aug 4 th, 2017 from IPIECA (2012). Performance indicators for fatigue risk management systems guidance document for the oil and gas industry. Mawhood, J.A. and Dickinson, C.E. (2012). Rail staff fatigue the GB regulator s perspective on managing the risks. In N. Dadashi, A. Scott, J.R. Wilson and A. Mills (Eds.), Rail Human Factors: Supporting reliability, safety and cost reduction (pp ). Retrieved Aug 4 th, 2017 from Office of Rail and Road (2012). Managing Rail Staff Fatigue. Retrieved Aug 4 th, 2017 from Office of Rail and Road (2016a). Good practice guidelines Fatigue Factors. Retrieved Aug 4 th, 2017 from data/assets/pdf_file/0008/24758/fatigue-key-performanceindicators.pdf Office of Rail and Road (2016b). Points from RSSB Projects T1083 regarding the Fatigue and Risk Index. Retrieved Aug 4 th, 2017 from data/assets/pdf_file/0004/23683/points-from-rssb-project-t1083- regarding-the-fatigue-and-risk-index-november-2016.pdf Office of Road and Rail (2017). Fatigue Key Performance Indicators. Retrieved Aug 4 th, 2017 from data/assets/pdf_file/0008/24758/fatigue-key-performanceindicators.pdf

10 Organisation for Economic Cooperation & Development (OECD, 2008), Guidance on developing Safety Performance Indicators related to chemical accident prevention, preparedness and response. Retrieved Aug 4 th, 2017 from Railway Safety and Standards Board (2012). Managing fatigue a good practice guide. Retrieved Aug 4 th, 2017 from Railway Safety & Standards Board (2013). Measuring Safety Performance a quick how to guide. Retrieved January 2017 from Railway Safety & Standards Board (2014). Measuring safety performance. Retrieved Aug 4 th, 2017 from Railway Safety and Standards Board (2015). Fatigue and its contribution to railway incidents special topic report. Retrieved Aug 4 th, 2017 from Railway Safety and Standards Board (2016). Preparing rail industry guidance on biomthematical fatigue models (T1083). Retrieved 4 th Aug 2017 from