Mock Disaster Exercises How to Test True Capability

Transcription

1 Mock Disaster Exercises How to Test True Capability Richard Long, Senior Advisory Consultant, MHA Consulting 2018 MHA CONSULTING. ALL RIGHTS RESERVED. July 11, 2018

2 COMPANY BACKGROUND KEY FACTS 19-year proven track record of applying industry standards and best practices across a diverse pedigree of clients. 19 Years in operation. 20 Average years industry experience. SENIOR LEADERSHIP MHA Consulting s senior team has an average of over 20 years of industry relevant experience in the areas of Business Continuity, Disaster Recovery, and Project Management. CAPABLE Comprehensive suite of services. GLOBAL Diverse, global client base. SAAS Compliance and risk tools. Richard Long, Practice Leader & Senior Advisory Consultant Phoenix, Arizona A simple mission: Ensure the continuous operations of our clients critical processes. 60% of revenue comes from Business Resiliency, 30% from IT Disaster Recovery, and 10% from SaaS tools. SaaS Tools: BIA On-Demand, Compliance Confidence, Residual Risk MHA CONSULTING. ALL RIGHTS RESERVED. 2

3 DIVERSE, GLOBAL CLIENT BASE SERVICES HEALTHCARE EDUCATION FINANCIAL INSTITUTIONS CONSUMER PRODUCTS INSURANCE TRAVEL & ENTERTAINMENT GOVERNMENT/UTILITY 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 3

4 ROBUST SUITE OF SERVICES ASSESS THE CURRENT ENVIRONMENT RECOVERY STRATEGIES & SOLUTIONS RESPONSE & RECOVERY PLANS EXERCISES MAINTAIN & IMPROVE Current State Assessment Policy & Standards Business Impact Analysis Threat & Risk Assessment BCMMETRICS TM BIA On-Demand (BIA OD ) BCMMETRICS TM Compliance Confidence (C 2 ) Business Recovery Strategies & Solutions Data Center Recovery Strategies Crisis Management Business Recovery IT Disaster Recovery Training & Awareness Mock Disaster Exercises Plan Functional Walkthroughs Alternate Worksite Exercises Update Recovery Plans Update Current State Assessment Update Business Impact Analysis & Threat Assessment Third Party Assessments BCMMETRICS TM Residual Risk (R 2 ) 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 4

5 WHAT IS A MOCK DISASTER EXERCISE? a simulation of an unplanned disruption that requires participants to identify the actions and steps they would take to successfully respond, assess impacts, activate resources, and recover in a timely manner. MDE 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 5

6 THE SESSION APPROACH & METHODOLOGY EXERCISE TYPES The RIGHT people. The RIGHT planning. Increasingly complex exercises to heighten skills. Properly training participants to maximize learnings. Conduct maturity-appropriate exercises. Assess and document the results. Resolve the findings and update the process. Tabletop Exercise/Structured Walkthrough Test Walkthrough Drill/Simulation Test Functional Drill/Parallel Test Full Interruption/Full Scale Test 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 6

7 TRENDS & THE FUTURE TRENDS IMPACTING PLANNING NEXT EVOLUTION OF PLANNING Few organizations have best of class capabilities. Management places no emphasis on training/exercises. Believe events will never happen to them. Attention spans, availability, accountability, participation. Refuse to exercise at the highest levels. Don t incorporate external agencies and partners. Lack of willingness to use a standard IM process. Leaders can t lead the team. Scenario agnostic it s about the impacts. Test real-time capabilities in unplanned exercises. Fully integrate external agencies and partners. Certify IM team members on a regular basis. Conduct virtual reality exercises. Mini and full exercises. Use Checklist Manifesto approach MHA CONSULTING. ALL RIGHTS RESERVED. 7

8 THE BIG PICTURE Risk Resilience 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 8

9 DEALING WITH THE HUMAN FACTOR THE HUMAN FACTOR It s simple, you must have the right people in the right seats. Don t pick people simply based on title; must consider fit and capability. Remember the team takes on the personality of its leaders. Train your Incident Management leaders on how to run a team. Be prepared for political, hierarchy, generational, and cultural issues. Keep structure simple and easy to understand. Teams who work together more often are proven to be more successful. Remember attention spans are short; train appropriately MHA CONSULTING. ALL RIGHTS RESERVED. 9

10 PLANNING THE EXERCISE EXERCISE MUST NOT BE TAKEN LIGHTLY DOCUMENT ALL EXERCISE PLANS CONSIDERATIONS Scenario complexity increases planning time Exercise planning team experts Maturity of the team being exercised Time available for the exercise Areas of weakness to address from past exercises Resources required (people, process, technology) Exercise type Budget PLANNING DOCUMENT Purpose Scope Objectives Assumptions Risks Resources (People, Places, Things) Action Item Lists RESOURCES DHS Exercise Planning Guidelines 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 10

11 WHAT TYPE OF EXERCISE? 01 Consider the maturity of the team; are you and your team members beginners, intermediates or advanced? 02 What key objectives are you trying to achieve with the exercise? CONSIDER Build on increasingly complex exercises to achieve success over time (Tabletop, Walkthrough, Functional, etc.). It s okay to use the same exercise type more than once to build confidence and success, but move on when you have met your objectives Sometimes, you have to go back to a less complex exercise to reinforce the key objectives you are trying to achieve before you can move forward again. Remember the mission is to build the capability of your team - not to make them fail MHA CONSULTING. ALL RIGHTS RESERVED. 11

12 APPROPRIATE EXERCISE TYPE FOR TEAM EXERCISE TYPE Tabletop Exercise/Structured Walkthrough Test Novice/Days/Risk Low Walkthrough Drill/Simulation Test Novice to Moderate/Days to Weeks/Risk Low Functional Drill/Parallel Test Moderate to Advanced/Months/Risk Moderate to High Full Interruption/Full Scale Test Advanced/Months/Risk High 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 12

13 EXERCISE THE TEAM AND PLAN USE INCREASINGLY COMPLEX DR EXERCISES EXERCISE TYPE Standalone Test the ability to recover the system/application on its own. Integrated Test the ability to recover the system/application with one or more of its critical upstream/downstream dependencies. Business Process Test the ability to recover the system/application as part of a complex business process with many systems/applications involved from end to end. Complexity Increase complexity over time to heighten ability to validate your recovery plan and strategy for critical systems/applications MHA CONSULTING. ALL RIGHTS RESERVED. 13

14 PERFORM MULTIPLE EXERCISES EXERCISE TYPE Perform a single large integrations-based exercise annually. Perform limited application/department exercises monthly/quarterly. Tabletop on new app or changed technologies. Emergency/Crisis Exercises: Corporate IT Coordinated 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 14

15 THE SCENARIO THE SCENARIO SHOULD TELL A STORY CONSIDERATIONS It s about getting them to reach new capabilities. Ensure the scenario elicits the actions you want to happen. Use a relevant scenario that makes sense for your company. Do your research on the scenario. Doesn t have to be complex to be successful. Use pauses and breaks. OUTPUT Best possible exercise to heighten team capabilities by providing them with points of reference should the real event occur MHA CONSULTING. ALL RIGHTS RESERVED. 15

16 THE SCENARIO BUILDING THE SCENARIO SCENARIOS Determine if natural, man-made or technological event. Build summary of the simulated event. Align objectives to the scenario. Build injects and timeline that tell the story. Use quality over quantity of injects. Identify actions to be taken based on injects. Validate scenario with your planning team. Consider doing it (or a portion of it) in real time. RESOURCES DHS Exercise Planning Guidelines 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 16

17 FACILITATING THE EXERCISE A GOOD FACILITATOR CONSIDERATIONS Choose a primary facilitator. He or she must be good at leading the exercise, but not directing. Complex exercises may require multiple facilitators. Must keep it on time and on point. Be able to make decisions to adjust the exercise as needed. Ensure its success. REMEMBER If you are NOT a good facilitator, find someone who is to lead your exercises. Even a good exercise will fail with a bad facilitator MHA CONSULTING. ALL RIGHTS RESERVED. 17

18 CONDUCT THE EXERCISE CONSIDERATIONS Have fun! Start on time and end on time. Keep it on track based on your timeline. Adjust based on progress made or lack of it. Be prepared to add or delete injects on the fly. Don t be afraid to stop and do a RESET. Breaks should be short. Be prepared for real life emergencies. Let participants take longer if they are being successful or stop them if they need to move on. Focus on identifying issues, not binary results (success or failure). Debrief at the end MHA CONSULTING. ALL RIGHTS RESERVED. 18

19 DOCUMENT THE EXERCISE DOCUMENT THE RESULTS OF THE EXERCISE POST-EXERCISE DOCUMENT Purpose Scope Objectives Assumptions Successes and Opportunities Participating Resources (People, Places, Things) Action Item Lists with Responsible Parties/Due Dates RESOURCES DHS Exercise Planning Guidelines 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 19

20 USE THE RESULTS HOW TO USE THE RESULTS OF THE EXERCISE Adjust the Strategy - May require modifications to the recovery strategy based on results and needs. Adjust Recovery Procedures and Documentation - A major benefit is updating and ensuring functional procedures. Review and update coordination between plans - Review and validate the integration and validity of BC plans while recovery occurs. Budget - Honest results help in determining both BC and DR budgets and approval from management MHA CONSULTING. ALL RIGHTS RESERVED. 20

21 INCIDENT PRIORITIES & OBJECTIVES Life Safety Incident Stabilization Property Preservation Restoration of the Business 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 21

22 A.P.I.E. Use the Assess, Plan, Implement, and Evaluate approach to size up the incident and define a plan of action. A.P.I.E. consists of: (A)ssess What is the current situation? (P)lan What steps are needed to address the current situation? (I)mplement What resources do we need to assign to execute steps to address the current situation? (E)valuate How well did we execute the plan and what needs to be addressed? 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 22

23 EXAMPLE SCENARIOS 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 23

24 SCENARIO #1 YOU ARE A SCHOOL DISTRICT INCIDENT MANAGEMENT TEAM 5 elementary schools 2 junior high schools 1 high school Each school has its own Emergency Response Team Incident Command System is new to the district Command Center is at the district office School wide emergency notification system School resource officers available at each school 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 24

25 SCENARIO #1 SCENARIO ACTIONS A commercial tanker truck carrying ammonia hydroxide was diverted off the freeway to local roads due to an accident. The tanker truck was making a left turn when a car cut in front of it and crashed on its side; passenger car is under the tanker truck. Police and fire are on the scene. A hazmat team from a nearby city is en route; ETA is 30 minutes. Fire Incident Commander advises Dispatch there are 3 schools downwind within 2 miles of the scene (elementary, junior high, and a private K-8 school) with a total of around 500 students. Fire Incident Commander has not detected leakage from the tanker at this time. ESTABLISH command. ASSESS the strategic risks to the school district. PLAN your approach to address the strategic risks. IMPLEMENT your approach. EVALUATE your performance MHA CONSULTING. ALL RIGHTS RESERVED. 25

26 SCENARIO #2 YOU ARE A MOBILE PAYMENTS COMPANY INCIDENT MANAGEMENT TEAM Headquartered in downtown San Francisco 1,000 employees Multi-tenant facility BCM program is new to the organization Critical operations have been identified Can work from home but no dedicated backup sites Data centers are in colocation sites Primary call center is at headquarters no backup 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 26

27 SCENARIO #2 SCENARIO ACTIONS Protestors against the North Dakota pipeline have taken control of your building overnight. They have come to protest against the Bureau of Land Management, headquartered in your building. They got past security and up to the BLM office using stolen access cards. The building is closed to all tenants until this is resolved; could be days. Police have been contacted. Protestors are inside and outside the building. ESTABLISH command. ASSESS the strategic risks to the organization. PLAN your approach to address the strategic risks. IMPLEMENT your approach. EVALUATE your performance MHA CONSULTING. ALL RIGHTS RESERVED. 27

28 SCENARIO #3 YOU ARE A GLOBAL REAL ESTATE COMPANY INCIDENT MANAGEMENT TEAM Headquartered in downtown Austin, Texas 3,000 employees Global real estate offices Critical operations have been identified Crisis Team has been implemented but not tested Data center is in headquarters Primary call center is at headquarters no backup 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 28

29 SCENARIO #3 SCENARIO ACTIONS The company is holding its annual global corporate get together for its employees and real estate agents. There are over 7,500 people in attendance at a downtown conference center 10 miles from headquarters. CNN is reporting that a gunman entered the conference center and began shooting during a keynote speech. Police and fire are on-scene attempting to address the situation. On-scene employees are reporting the CEO has been shot along with at least 15 to 20 others. Your team has been assembled to address the situation. ESTABLISH command. ASSESS the strategic risks to the organization. PLAN your approach to address the strategic risks. IMPLEMENT your approach. EVALUATE your performance MHA CONSULTING. ALL RIGHTS RESERVED. 29

30 SCENARIO #4 YOU ARE A MOBILE PAYMENTS COMPANY INCIDENT MANAGEMENT TEAM Headquartered in downtown San Francisco 1,000 employees Multi-tenant facility BCM program is new to the organization Critical operations have been identified Can work from home but no dedicated backup sites Data centers are in colocation sites Primary call center is at headquarters no backup 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 30

31 SCENARIO #4 SCENARIO ACTIONS At approximately 6:00 AM today, the IT Help Desk began handling several calls related to Alert Errors being received as employees were attempting to log in to the network and various Point of Sale (POS) systems. Although trouble-shooting efforts began immediately upon notification of these Alerts Errors, the cause is unknown and has been escalated to the ISERT. Based on the Severity Level, the ISERT has escalated to the Oversight Committee. ESTABLISH command. ASSESS the strategic risks to the organization. PLAN your approach to address the strategic risks. IMPLEMENT your approach. EVALUATE your performance MHA CONSULTING. ALL RIGHTS RESERVED. 31

32 SCENARIO #4 - INJECT SCENARIO ACTIONS The ISERT is reporting that a disgruntled (IT) employee who thought he might be terminated (and eventually was) developed a program that would change passwords on critical SQL databases on a specific date and time. It appears the script may not be on the backups so it will be safest to restore at the Alternate site with the 7/10 backup. Any data that was entered into the POS Systems on 7/11 is no longer available. Although attempts have been made to contact the (terminated) employee he has directed all inquiries to his attorney. At this point, this issue has been referred to Legal for further review and action. At a minimum, the ISERT is reporting that the POS Systems will remain off-line until noon on Thursday 7/12. ESTABLISH command. ASSESS the strategic risks to the organization. PLAN your approach to address the strategic risks. IMPLEMENT your approach. EVALUATE your performance MHA CONSULTING. ALL RIGHTS RESERVED. 32

33 SCENARIO #5 YOU ARE A SCHOOL DISTRICT INCIDENT MANAGEMENT TEAM 5 elementary schools 2 junior high schools 1 high school Each school has its own Emergency Response Team Incident Command System is new to the district Command Center is at the district office School wide emergency notification system School resource officers available at each school 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 33

34 SCENARIO #5 SCENARIO ACTIONS A man has been shot at a local convenience store and the shooter has evaded capture. Local elementary school is placed on lockdown. A mother of 3 is shot dead at her home; her 3 children attend elementary school, junior high, and high school. News outlets are reporting that a close friend of the dead mother stated that her ex-husband had threated to come back to kill her and her boyfriend, and to get his 3 kids back. Two school principals report a black Impala slowly driving by the schools and asking children if they know Billy, Cindy, or Joanne. ESTABLISH command. ASSESS the strategic risks to the school district. PLAN your approach to address the strategic risks. IMPLEMENT your approach. EVALUATE your performance MHA CONSULTING. ALL RIGHTS RESERVED. 34

35 FINAL THOUGHTS THINGS TO THINK ABOUT Get the RIGHT humans in the right seats. Focus on making incremental gains over time. Build on increasingly complex exercises that heighten performance. Create an easy to use Incident Management process think Checklist Manifesto. It s not about the number of injects or length of the exercise but the quality. Perfect is the enemy of the good. Focus on the outcome, become target focused, and concentrate on the ability to successfully recover MHA CONSULTING. ALL RIGHTS RESERVED. 35

36 SUMMARY TEST TRUE CAPABILITY Human Factor. Plan Your Exercise. Select the Right Exercise Type. Build an Appropriate Scenario. Find the Right Facilitators. Conduct the Exercise. Document the Exercise and Update Your Process MHA CONSULTING. ALL RIGHTS RESERVED. 36

37 THANK YOU Richard Long, Senior Advisory Consultant MHA CONSULTING, INC. (888) (602) MHA CONSULTING. ALL RIGHTS RESERVED.