Business Continuity, Crisis Management & Resilience Engineering in Socio-Technical Systems

Transcription

1 Business Continuity, Crisis Management & Resilience Engineering in Socio-Technical Systems Diarmuid Moran BSc, PhD, CEng, MIEI Bowline Risk Management Ltd, Currently consultant on a new build Nuclear Project

2 The Story & Topics The Story of the Crisis Event Graded Risk Approach & Business Impact Assessment Disaster Recovery / Back-up Site (mirroring / failover) / Business Continuity Plan (BCP) Safety in Design, Loss & Risk Management CMT & Resilient Engineering FRAM & Learning from Experience Plan, Prepare, Practice

3 Thursday 18 th Nov 1999 The next 3 slides were prepared for manager briefings in the telecom company just after the new & first H&S Mgr appointment

4 Why support improved health and safety performance? People have increasing expectations for better health, safety and protection in all aspects of life - new legislation reflects this Management is required to make that happen Good Management is Good Health & Safety COMPANY MISSION TEXT DELETED

5 Two elements will ensure world class health & safety Reporting of accidents and incidents - elimination of minor incidents will significantly reduce potential for major accidents A behaviour based approach can be used to establish a strong safety culture this covers: Feedback, Right Design, People & Consequences, Measurement, Conversation, Participation, Synergy & Teams

6 For help or guidance: The recently appointed Company Health & Safety Manager is Diarmuid Moran - based in the Risk Management Dept. Diarmuid would be delighted to make a 15 minute presentation at any of your Team Meetings New Incident, Accident & Injury report forms are available on etc Call 1- xxxxxxxxxx Diarmuid previously worked as EHS Manager in Wyeth and Seagate Technology (Wafer Fab) and completed a PhD including the Radiation & Health Physics arising from the Chernobyl accident.

7 Friday 19 th Nov 1999 A strange situation arose in Dublin city centre that Friday afternoon offices and agencies stopped working! Within an hour ~ xx million of our customers couldn t make phone calls

8 Network alarms activated, Smoke and fire alarms key exchange, Engineer Folks drove quite fast, Extinguishers were employed, A flood was discovered, Power was switched off, A Crisis was underway

9

10 Xxxxxxxxxxxxxxxxxxxx Xxxxxx xxxxxxxxxxxx Folks worked though the night People said we are lucky it started at 3pm as come 4.30pm any Friday.. most technical resources would be away and would not have been contactable Saturday 20 th Nov 1999 Irish Independent

11

12 Systems reconfigured back-ups reloaded, but xxxxx & other systems xxxxxxxxx

13 Slide deleted

14 Slide deleted

15 Selection of Notable Events Y2k, 9-11, ---- drills, Foot & Mouth Disease, Swine flu & similar, Strike, Fraud, Regulatory IT Virus & Denial of service attack, Fire, Crash, Fatality, Robbery.

16 Business Continuity, Crisis or Incident Management & Resilience Engineering in Socio-Technical Systems

17

18 OHS & Risk Management Parallels From Jan Hudsal

19 Parallels Graded Approach Hazard, Risk =f(p,c), Mitigation, Transfer/Avoid, Contingency, Exploit Cost of BCM Benefit of measures (cost of loss v cost of mitigation) Worst case, expected case & reasonably practicable Business Impact Assessment (critical systems) Project Risk Management (time, cost, quality) Supply Chain Management Duty Holder

20 Differences between Robustness, Agility / Flexibility and Resilience events have minimal impact BAU continues certain events do impact and outputs are recovered but maybe different a major event impacts but outputs are quickly restored The time & anticipation factor - later

21 Modified Perrow diagram characterisation of complex systems framework shows where socio-technical / resilience engineering impacts and factors arise

22 The Story & Topics The Story of the Crisis Event Graded Risk Approach & Business Impact Assessment Safety in Design, Diversity & Risk Management Business Continuity Management I/CMT & Resilient Engineering FRAM & Learning from Experience Plan, Prepare, Practice

23 Safety in Design & Procurem nt Life Cycle Design and Planning Design for needs and functionality eg MTBF Design for operation Design for security / capacity Design for ease of maintenance Design for environmental performance Design for decommissioning Design for economics and efficiency Design for future proofing / IPR mgmt & control Life time records. Human factors & Non-Linear coupling or Functional Resonance

24 Redundancy Principle N+1, N+2, N+N, N+3 N + N

25 Diversity Principle ESB + Battery Eircom + BT Fibre + Wireless Front + Back

26 Separation Principle

27 Banking Crisis Banter Senior Mgmt Crisis managers? Various levels / lack of robustness in sector (lack of testing, accountability and intelligent audits) Staff now resilient after 8 years Certain agile and flexible systems such as new products and systems

28 Business Continuity Management Roles and Activities» Planners / Implementers / Systems» Test & Exercise» Discipline Experts & Threat Specialists» Managers / Oversight» Auditors & Reviewers» Documentation and Sustaining» Business Impact Assessment / Analysis» BC Strategy» Risk Analysis Could mirror entire organisation Apply good succession planning & Cross training / Org change / CPD

29 BCP and Crisis Management Crisis Management (strategic) Business Process Failure Extortion Market Crash Fraud Etc Technology Restore Business Continuity Management Business Process Restore Site Restore Environment Restore Etc Other Crisis Strategies Software Restore Technology Disaster Recovery Hardware Restore Network Restore Data Restore Etc Other Continuity Treatments Is there a BC plan for the CMT / ERT? Have they a back-up venue, PPE & communications system

30 BCP Process Project Initiation and Management Risk Evaluation and Control Business Impact Analysis Critical Functions Risk Reduction Measures Define Resource Requirements Business/ Work Area Facilities, people logistics, SLA Switches Comms Hardware etc Plan Design Data Input Written Plans Team Structure Plan Structure Developing Business Continuity Strategies Continuous Improvement Awareness & Training Maintaining & Exercising Emergency Response Business Resumptio Disaster n Recovery Implement Continuity Strategy, Insurance & Risk Reduction Where is crisis management? Gather Contact Information

31 all bases may not be covered So Crisis Management is the non standard procedure undertaken by individuals or groups to attempt to save the day and mitigate an incident or event This is a Resilience approach / objective It is applicable in a socio-technical system Federal Reserve on 9-11, Chesley Sullenberger US Airways

32 Crisis (Incident) Management Crisis Management Roles / Training Emergency Response CM Planners Implementers / Managers / Experts Crisis Management Team CM Communications CM External co-ordination Incident Command Site control Scenario & Practices

33 Hope you are enjoying. Slide deleted

34 Prepare / Practice / Learn from Experience Exaggerate Safety in complex systems needs to learn from millions of successful (variable) outcomes Experience and team wisdom Exaggeration Culture of understanding / peer systems Culture of learning / leadership Example - Nuclear safety culture

35 Non-linear and Safety II Prof Erik Hollnagel Complexity too many requirements ETTO is the principle that there is a tradeoff between efficiency on one hand, and thoroughness (such as safety assurance and human reliability) on the other. In accordance with this principle, demands for productivity tend to reduce thoroughness while demands for safety reduce efficiency Prof James Reason, Human error: models and management - although he does conclude that High reliability organisations are the prime examples of the system approach. They anticipate the worst and equip themselves to deal with it at all levels of the organisation.

36 From others inc E Hollnagel Safety Level Global Leadership in High Tech, Health, Safety, Design and Risk Management

37 Resilience system that deals with recognition, compensation, return to BAU 4 key qualities Learning Responding Monitoring Anticipating Developing the Resilience & or V s Safety compliance culture is an interesting aspect. EG. Incident >>> call the ERT / FA / CMT: >>>> Q. Does ERT/FA/CMT comply with xxx or do they have flexibility?

38 Functional Resonance Analysis Method (FRAM) Functional resonance is the detectable signal that emerges from the unintended interaction of the normal variabilities of many signals Excludes the human error factor and gives equality to the successful outcomes FRAM treats variability rather than safe v s error Example a rescue mission fails.. Do we blame?

39 Functional Resonance Analysis Method FRAM (FR Accident Model!) Relationships coupling & principles Model useful where there are a number of root causes. ie socio-technical aspects

40 FRAM ily group of researchers The Principle of Equivalence of Successes and Failures FRAM and Resilience Engineering acknowledges that things go right and wrong in basically the same way. The Principle of Approximate Adjustments Many socio-technical systems are intractable. The conditions of work therefore never completely match what has been specified or prescribed. Individuals, groups, and organisations normally adjust their performance to. The resulting performance variability is the reason why things go right, but also the reason why things go wrong. The Principle of Emergence The variability from multiple functions may combine in unexpected ways, leading to consequences that are disproportionally large, hence produce a non-linear effect. Both failures and normal performance are emergent rather than resultant phenomena, because neither can be attributed to or explained only by referring to the (mal)functions of specific components or parts. The Principle of Functional Resonance.This can be described as a resonance of the normal variability of functions, hence as functional resonance. The resonance analogy emphasises that this is a dynamic phenomenon, hence not attributable to a simple combination of causal links.

41 Example of the Comair Flight 5191 Paper identified findings not found in the NTSB analysis of the crash which had 49 victims. The NTSB explained the accident as being caused mainly by the failures of the humans at the sharp end. Erik Hollnagel, Shawn Pruchnicki, Rogier Woltjer, Shawn Etcher. Analysis of Comair ight 5191 with the functional resonance accident model. 8th International Symposium of the Australian Aviation Psychology Association, Apr 2008, Sydney, Australia. 8 p. <hal > HAL Id: hal Submitted on 10 Aug 2011

42 Software System Available through the FRAM model Tools esonance.com/ Example from the Model as shown in the paper

43 FRAM tool identified the accident was not caused by a sequence of pilot mistakes. Paper suggests safety best served by management of performance and controlling sources of variability. Linate Airport disaster occurred on 8 / 12 / 2001 in fog, a MD-87 airliner carrying 110 people collided on take-off with a Cessna business jet carrying 4 people, 118 people were killed. Milan courts found four persons guilty for the disaster. Air-traffic controller xxxxxx was sentenced to eight years in prison. XXXX xxxxxx, former head of the air traffic control agency, received sentences of six and a half years. In the appeal trial (July 2006), another four people were sentenced. (Ref Wikipedia) Consider askthepilot.com untold-concorde-story

44 Business Continuity, Crisis Management, Resilience Engineering in Socio-Technical Systems Dr Diarmuid Moran