SESSION 405 Tuesday, November 3, 10:00am - 11:00am Track: Industry Insights

Transcription

1 SESSION 405 Tuesday, November 3, 10:00am - 11:00am Track: Industry Insights Major Incident Management: From Inhibitor to Enabler Nicole Skibinski Manager, Service Assurance, Spectrum Health nicole.skibinski@spectrumhealth.org Session Description Follow Spectrum Health on its year-long journey of transforming a chaotic, homegrown process into a true incident command system. Attendees will learn about the Major Incident Lifecycle Management Model, command-and-control structures that can integrate seamlessly with the National Incident Management System (NIMS), best practices in communication, KPIs that demonstrate success, and opportunities to reduce the risk of critical IT system failure. (Experience Level: Intermediate) Speaker Background Nicole Skibinski leads the service assurance team at Spectrum Health, which is responsible for the operational processes that support the availability, quality, and continuity of the IT services provided to the organization. Leveraging her experience in the manufacturing environment, Nicole has been able to apply tools and techniques in unique ways to define, document, and improve IT services.

2 Major Incident Management: The Journey from Inhibitor to Enabler Nicole Skibinski Introduction Agenda Major Incident Management Journey The beginning What we did The end What we learned What is next Questions 2

3 Who is Spectrum Health? A not-for-profit health system, based in West Michigan. Comprised of 12 hospitals, 180 ambulatory and service sites, 1300 physicians and advanced practice providers, and a health plan. West Michigan s largest employer with 22,500 employees, including 800 IT staff. In 2015, named one of the nation s 15 Top Health Systems by Truven Health Analytics. 3 Who am I? Manager of the Service Assurance Team. Responsible for the operational processes that support the availability, quality, and continuity of the IT services provided to the organization. Contact Information nicole.skibinski@spectrumhealth.org Phone:

4 What is Major Incident Management? The process provides structure and support to IT efforts during critical infrastructure, business service, or clinical technology impacting Incident incidents. The goal is to maintain continuity of operations, restore service as soon as possible, provide consistent communication, and make informed and timely decisions. Major Incident 5 THE JOURNEY Inhibitor Enabler 6

5 Journey Timeline July 2013 Aug Sept. Dec Jan. Mar April 2014 May 2014 June 2014 Oct Major Incident Kick-off Work Session Process Design & Documentation Communication and Training Plan Test Plan Go-Live Monitoring & Continuous Improvement 7 The beginning Major Incident Management 8

6 Who we met Who s Why My When does critical working will this What Who s should we on business this service keep and system be what tell responsible? customers? is are happening? down restored? they AGAIN! doing? Customer Business Mgr. Incident Mgr. Helpdesk Mgr. C-Suite IT Staff 9 What we saw 10

7 What we did Developed a new process. Ensured IT leadership commitment and involvement. Aligned framework with the business and had business user involvement. Established commitment to the new process and to continuous improvement. Defined meaningful metrics. Act Check Plan Do 11 The Framework and its Benefits The framework is aligned with the guidelines provided by the National Incident Management System (NIMS). Benefits: Same framework as used by the Hospital Incident Command System. Enabled a coordinated response through a defined command and control structure. Established a common process for managing resources and maintaining a manageable span of control. Ensured integrated communications and facilitated a process that provided a clear understanding of what needed to be accomplished. 12

8 Command and Control Structure The Incident Command Organization (ICO) is the command and control structure for the organizational response and the resource engagement required to manage a major incident. Each role is assigned to a specific individual with clear responsibilities and reporting relationships. Not based on organizational structure. For each major incident, the entire ICO will engage. The ICO is only maintained to the size necessary to manage the major incident. Roles may be released or engaged as determined appropriate. 13 The ICO IT Commander Executive Liaison Liaison Officer Information Systems Support Infrastructure Section Chief Clinical Applications Section Chief Site Section Chief Business Applications Section Chief Infrastructure Function Leads Clinical Applications Function Leads Site Function Leads Business Applications Function Leads Functional Staff Functional Staff Functional Staff Functional Staff 14

9 Major Incident Management Lifecycle Incident Identification Incident Analysis Major Incident Declaration Restoration Operations Recovery Operations Continuous Improvement Closed 15 Incident Identification and Analysis Detection of an abnormality in service performance or availability and the determination of scope and impact. Increased call volume with reports of similar symptoms into the Helpdesk. Alerts generated by monitoring tools. Identification of issues by higher level support teams. 16

10 Major Incident Declaration An incident of significant scope, complexity, and/or impact to the organization is escalated to a major incident. Activation of the major incident management process, the ICO, and all communication protocols. Roles and Responsibilities IT Commander: Makes decision to declare a major incident. 17 Restoration Operations All efforts to return the service to an unimpaired condition. At the conclusion of this stage, the system can be accessed by end users. 18

11 Restoration Operations Roles and Responsibilities IT Commander: Establishes overarching objectives and strategy and ensures consistent messaging. Executive Liaison: Lends support and assists in communicating to the organization. Liaison Officer: Primary point of contact between the IT ICO and another ICO. Advises on the impact to clinical or business workflow and facilitates communication within the external ICO. Section Chiefs: Manages the IT response under the assigned section and prioritizes workflow around continuity of business operations and system restoration. Function Leads and Functional Staff: Implements the approved action plan and other tasks as assigned. 19 Communication Protocols - Tools Text Paging: Primary communication method used for the major incident process (notification, updates, restoral). Conference Lines: Primary communication method used to facilitate resource coordination, collect and analyze information, make decisions, and maintain situational awareness through all levels of the ICO. System Status: Used to inform all staff of IT system status. Major Incident Workspace: Used to communicate all information related to a major incident. Contains real-time major incident status, resource assignment, a log of all major incident activities, and all other documentation pertinent to the major incident. 20

12 Communication Formal vs. Informal Formal Communication Utilizes the prescribed communication channels of the ICO. Information that changes the direction of the incident or is related to the incident action plan must be conveyed using formal communication. Facilitated by Command Staff Briefings: These occur upon declaration and after a defined operational period. Purpose is to review completed objectives, define objectives for next operational period, and evaluate the communication plan. Informal Communication Information gathering and sharing within and between the ICO Sections, Function Leads, and Functional Staff. 21 Recovery Operations All efforts to return the service to normal operation. May last for an extended period of time or may not be required. At the conclusion of this stage, the system is fully functional. Roles and Responsibilities Similar to Restoration Operations but only roles as required. 22

13 Continuous Improvement and Closure All efforts to identify and remediate issues that caused or contributed to the service degradation or outage. SBAR (Situation, Background, Assessment, Recommendation) Failure Impact Analysis (FIA) Action Items (< 120 days) At the conclusion of this stage, all assigned action items identified in the FIA are complete and the major incident is closed. 23 Continuous Improvement and Closure Roles and Responsibilities IT Commander: Overall management of a major incident throughout the entire lifecycle. Major Incident Process Manager: Facilitates the FIA, tracks action item status, and manages all documentation. Service Owner: Creates SBAR, participates in FIA, allocates resources for action items. Functional Staff: Participates in FIA and completes action items. 24

14 Failure Impact Analysis (FIA) Based upon the Failure Mode and Effects Analysis (FMEA) used in manufacturing. A systematic technique used to identify failures and analyze the consequences of the failures. Purpose is to document and implement actions to eliminate or reduce failures. For major incidents, the FIA is reactive and the scope is limited to the failure which resulted in the incident. 25 FIA Severity, Occurrence, Detection Failures are classified based on: Severity: An assessment of the seriousness of the effect of the failure to the customer or technology environment. Occurrence: The likelihood or frequency the cause of failure is projected to occur. Detection: An assessment of the probability that the detection controls will identify the failure and/or the cause of failure. Severity, Occurrence and Detection are determined for current state and future state (after completion of identified action items). 26

15 The end Major Incident Management 27 What we saw 28

16 What we heard Time and time again I am impressed with level of overall professionalism that the Very Great deliberate I Had look ICO the forward has people enabled to in the response to major action communication! technology was taken issues. The respect of participating necessary on in roles Got with and down concise clear to communication the between the call. FIA. our communication. problem operational fast. teams is nothing short of amazing. 29 What we learned Leadership commitment and involvement Proven framework that also aligned with business processes Robust communication and training plan Thorough documentation Dedicated staff response and all hands on deck approach Defined communication protocols Complete FIA and action items for every major incident Commitment to continuous improvement of the process Analyst dedicated to the major incident process Report metrics monthly Send out a survey to major incident participants 30

17 Where to from here? The journey is not over! Implement improved communication protocols Utilize a new tool for documenting all information related to a major incident Consider alternatives for FIA and action item tracking Reevaluate ICO Section assignments Continue to review metrics and survey results; implement identified process improvements Determine how the ICO integrates with other processes Begin proactive FIAs (Problem Management) 31 Questions 32

18 Thank you for attending this session. Please don t forget to complete an evaluation form!