Testing Your Operational Readiness with Outside Agencies. Tom Clark, CBCP Director, IT Infrastructure Continuity Services Liberty Mutual Group

Transcription

1 Testing Your Operational Readiness with Outside Agencies Tom Clark, CBCP Director, IT Infrastructure Continuity Services Liberty Mutual Group

2 Operational Readiness Capability of an organization to continually perform the functions for which it is designed Assessed according to internal standards: the difference between an organization s actual capability and its absolute potential Measured in terms of how soon an organization can reach its peak operational capacity

3 Testing Operational Readiness Every organization must test its operational readiness plans regularly to continually prepare for any disruption. Once a certain level of operational readiness has been reached within a corporation, it makes sense to continually improve its continuity capability by interjecting as much realism as possible into the exercise scenario.

4 Testing Operational Readiness Annual large scale exercises are recommended so the operational readiness of the organization can be tested. Although employees never know exactly how they will act or perform in a disaster, exercises allow them to learn specific skills that will improve the likelihood of success in a real event.

5 Exercise Effectiveness How can a real event be simulated so that all employees know their roles and responsibilities during an incident and have the opportunity to identify gaps in the continuity process? Involving outside agencies is one great method

6 How? Involve outside agencies in the planning process and the actual exercise Why? Testing Operational Readiness with Outside Agencies In a true emergency situation, employees will need to know the roles of various external agencies Employees must know exactly what is expected from external agencies The processes for communicating with external agencies should be well defined and understood

7 Our Process: Results Criteria for Success Conduct Exercise Preparations Exercise Planning Build on Existing Relationships Establish Relationships

8 How We Established Relationships Met regularly with other local businesses Attended association meetings and emergency management conferences ACP local chapter (Association of Contingency Planners) NEDRIX (NorthEast Disaster Recovery Information X-change ) State emergency management conferences

9 How We Established Relationships Meeting with Local Businesses Our organization is part of a group of local businesses that meet regularly for lunch and learn sessions. Every participating business works with the others to develop strategies for working together during an incident or a crisis. This may include something as simple sharing a parking lot, or even office space.

10 How We Established Relationships Local businesses share with each other the dates and times of exercises and drills. As an example, A drill occurred between the local hospital and the local S.W.A.T. team. Our security personnel and employees were made aware of what was going on next door.

11 How We Established Relationships During meetings with local businesses, members of state and local agencies are invited as guest speakers. We have found that this platform serves both parties well. In one hour or less, the speaker can get his or her message out to several representatives from multiple local businesses.

12 How We Established Relationships Speaking at one meeting, a local fire chief provided recommendations regarding employee safety. He identified the types of areas within a building that could potentially benefit from having floor plans and escape routes posted. Each business shared its strategy for establishing an Emergency Operations Center if there was an event.

13 How We Established Relationships We exchanged information with the fire department regarding roles and responsibilities during an emergency. This fire department, shared it s strategy for establishing an Emergency Operations Center (EOC) to handle emergencies during a major storm such as a hurricane or nor easter.

14 How We Established Relationships Members from the local police department have also participated in these meetings. They collected written data from each business: Who would be a primary contact and backup for the business during an event. How can these contacts be reached? Does the business have an emergency response team? Are there hazardous materials on site? How many employees are located in each facility?

15 Other Topics Discussed with Local Agencies: What are the expectations once local agencies are on the scene? They will likely assume total responsibility of the event. How would they like us to communicate with them? What are their emergency management procedures?

16 Attending Conferences Many of our employees attend conferences throughout the year that many outside agencies also attend. For example, at a recent NEDRIX conference, FEMA had representatives in attendance. During the NH Emergency Preparedness Conference in 2008, our Incident Commander met a former division commander for a local police department. He is now a Deputy Sheriff with another local department.

17 Attending Conferences Because of our new relationship, the deputy sheriff was invited to participate in our event. He introduced us to colleagues that would like to take part in the exercise. This resulted in the local Deputy Fire Chief and the State Emergency Preparedness Coordinator for the Department of Homeland Security attending.

18 Building on Existing Relationships Our organization has been actively building relationships with various local and state agencies including: Local Police Departments Local Fire Departments County Sheriff Departments State Police NH Emergency Management Agencies

19 Keys to Building on Existing Relationships When meeting with outside agencies, we focus on: Better understanding their capabilities in a crisis Discussing how they handle incidents Defining their expectations of our organization during an event

20 Our Relationship with the Fire Department We invited the local fire department to assess our facility and provide training to our employees. Employees were given detailed fire extinguisher training, as well as basic guidelines for knowing when to attempt to put out a fire and when not to.

21 Our Relationship with the Fire Department The assessment of the facility gave the fire department a chance to see the layout and structure of our building. Employees were also provided with general information regarding structural fires and how that information related specifically to our facility.

22 Our Relationship with the Police Department The local police department has provided awareness programs to our employees. These programs include topics such as: Refuse to become a victim Violence in the workplace

23 Our Relationship with the Police Department The phone system at our facility is set up in such a way that to call outside of the building, employees must first dial 9. This resulted in the police department responding to several false 911 calls when employees also had to dial 1 and an area code. Our company and the local police department worked together to developed a strategy for preventing false calls in the future.

24 Local Emergency Response Team (LERT) Liberty Mutual Group has adopted the Incident Command System (ICS), a nationally recognized structure originally designed in the 1970 s to combat wildfires. In addition, our organization has an Initial Assessment Team (IAT) which is made up of executive management. This team has the responsibility of determining if the event requires activation of the LERT.

25 Local Emergency Response Team (LERT) The Incident Command System (ICS) is comprised of five teams: Command Operations Logistics Planning & Intelligence Finance

26 Planning Our Operational Readiness Exercise Operation Safehouse Determine exercise goals: Scope Objectives and how we measure our success Participants and the description of their roles

27 Exercise Type and Scope A exercise involving enough complexity to test the operational readiness of our processes, people and technology Create an exercise that describes a disruptive event that would require concise communication between our company and various outside agencies

28 Exercise Goals and Objectives Involve multiple company physical facilities in an event that requires the utilization of their existing business continuity plan to manage the event Assess the ability of the team to manage a prolonged event. Assess the effectiveness of the communication between the team and outside agencies. Assess the ability of the entire team to activate plans effectively.

29 Exercise Goals and Objectives Assess the size and composition of the team. Assess the ability of the team to provide employee and stakeholder communications, press releases, and customer notifications. Assess the ability of the Incident Commander (IC) to manage the Emergency Operations Center (EOC )

30 Participants Local Emergency Response Team (LERT) in Portsmouth, NH Local Emergency Response Team (LERT) in Kansas City, MO Corporate Emergency Response Team (CERT) at our Boston Headquarters Local police departments Local fire departments State emergency management agencies

31 Preparing for the Exercise Choosing the scenario Involvement of and support from Senior Management Selecting the design team Selecting the simulation team (SIM Team) Coordinating the exercise with local agencies

32 Choosing the Scenario We researched the scenarios of past exercises and the lessons learned from those events We looked at current events to assist in the development of the scenario The basic concept for the scenario was developed five months prior to the exercise.

33 Influential Current Events: Alloy Fabricators of New England, Inc. Randolph, Massachusetts in April 2008 One dead and one injured Atlantis Plastics Henderson, Kentucky in June 2008 Six dead and one injured

34 Economy Other Influential Current Events The large decline in the stock market during September 2008 was the trigger for our assailant s rampage. He was a temporary contractor with access to the company Data Center, whose contract had not been renewed. His frustrations resulted in reactions designed to cause a great deal of damage. Weather Tropical Storm Omar was also used as a factor in the scenario to increase the complexity. While Omar was not geographically close to the facility in NH, part of the exercise was to assess how our company could handle a second incident at another facility while a primary Data Center was disrupted.

35 Senior Management Involvement The success of the exercise depends heavily on the involvement of and support from senior management Only the CIO was informed of the plan. The Initial Assessment Team (IAT): Knew that the simulated event was going on, but not the details Knew that they would be called Had time scheduled on their calendar in advance

36 Senior Management Involvement To increase the realism of the exercise, a design team wrote scripts for senior management to use during the event These scripts provided other teams with realistic actions during an event such as this.

37 Selecting the Design Team To increase the complexity of the scenario, the Design Team was created to address realistic and potential gaps in processes, people, and technology We selected five Subject Matter Expert s (SME s) with expertise relative to the type of event selected.

38 The Design Team The areas represented by our subject matter experts: Disaster Recovery Data Center Facilities Information Security Physical Security User Support Center (USC call center)

39 Creating the SIM Team As the exercise date approached, members were added to the design team to create the Simulation Team (SIM Team). The new members were not given any details of the scenario prior to the exercise. They were given their roles and responsibilities the day before the exercise. The SIM Team simulated calls to the Emergency Operations Center (EOC) They had pre-scripted roles to play throughout the event

40 Coordinating with Outside Agencies The participating members of the outside agencies assisted the design team in the preparing the scripts that would be used during the exercise They worked with the SIM team to create realistic inputs and outputs to provide a true representation of interactions with outside agencies.

41 Coordinating with Outside Agencies Members from outside agencies participated in role playing during the exercise. In addition, members of outside agencies were present in the EOC and observed the team s actions and reactions during the exercise.

42 Preparations In preparation for this exercise, we invited the NH State Police to visit our Portsmouth facility and give a presentation regarding violence in the workplace. The NH State Police took pictures throughout the building and offices to help identify safe and non-safe areas for our employees during a violence in the workplace type of event.

43 Preparations The NH State Police developed and delivered an assessment report to senior management with suggestions on what our employees should do in the event of an active shooter scenario. The assessment report reflected the types of areas that employees should avoid during an active shooter situation and how to make themselves less of a target.

44 Scenario of Operation Safehouse The day began at 8:00 AM with the LERT attending training in a company facility in Dover, NH Dover, NH is 15 miles north of the Data Center in Portsmouth, NH In addition, Tropical Storm Omar was heading towards Miami, FL

45 Scenario of Operation Safehouse Suddenly, an incident occurred at the Portsmouth Data Center. At 9:00 AM, a software contractor, with access to the Data Center, took hostages, shot some employees, and detonated an explosive device damaging equipment. It is suspected that additional explosive devices are in the Data Center and throughout the rest of the building.

46 Scenario of Operation Safehouse At 10:00 AM the shooter shot and killed himself. At 10:15 AM local authorities rescued the hostages, secured the body of the assailant, and declared the facility a crime scene. The Portsmouth Data Center was non-functional and the Disaster Recovery Plans had to be activated in the Kansas City, MO Facility

47 Scenario of Operation Safehouse The second part of the exercise moves forward two days Due to the incident: One employee was killed by the assailant Ten employees were injured by the assailant due to shots fired Recovery activities have been in progress in Kansas City, MO

48 Scenario of Operation Safehouse Tropical Storm Omar has turned into Hurricane Omar It is expected to directly hit Miami, FL After two days, local authorities released the Data Center back to our security and facility teams to conduct a damage assessment.

49 Conducting the Exercise Morning Session The exercise began with a simulated radio broadcast: - Breaking News - Reports of shots fired and hostages taken at local business in Portsmouth NH

50 Conducting the Exercise Morning Session A simulated phone call came in from the Data Center facilities manager notifying the Incident Commander (IC) of the situation. The building had been evacuated, and the police department had arrived on scene. Tactical units were enroute. Not all employees had been accounted for. Reports of gunfire had been made by some employees.

51 Conducting the Exercise Morning Session The Incident Commander (IC) activated the IAT and established a conference bridge to brief the IAT on the situation. The determination of the IAT was to activate the LERT The Incident Commander then reached out to the Corporate Emergency Response Team (CERT) in Boston to advise them of the situation and activation of the Portsmouth LERT.

52 Conducting the Exercise Morning Session The LERT developed a series of short-term objectives focused around five key areas: People Facilities Technology Mission-critical activities at risk Communication.

53 Conducting the Exercise Morning Session The Incident Commander gave a short briefing to the entire Local Emergency Response Team (LERT) and then work began. Teams worked together to gather data and decide on action items moving forward The team developed an Incident Action Plan to determine both the operational and support activities to address the incident

54 Conducting the Exercise Morning Session The Incident Action Plan Addresses What do we want to do? Who is responsible for doing it? How do we communicate with each other? What is the procedure if someone is injured?

55 Conducting the Exercise Afternoon Session: The second half of the exercise was two days after the shooting. We had not been able to gain access to any part of the company facility in Portsmouth, NH since the incident. After a lengthy search, the NH State Police Bomb Squad found no other explosive devices.

56 Conducting the Exercise Afternoon Session: The building was finally released back to us and teams have begun the damage assessment process. The damage assessment identified several pieces of equipment to be replaced in order to restore the Data Center. The Disaster Recovery Plan implemented production at the alternate DC in KC which will be operational for at least 30 days.

57 Conducting the Exercise Afternoon Session: Where possible, employees have been using VPN to work from either their home or other company locations. Additional staff required to support production at the alternate Data Center in Kansas City, MO have been deployed. Due to the traumatic incident, it was difficult to obtain specific technical resources qualified to support production.

58 Conducting the Exercise Afternoon Session: Business units have been attempting manual workarounds while waiting for systems to be restored. Call center call volumes have dramatically increased due to the additional complexity of Hurricane Omar.

59 Outside Agencies and the Role They Played Several members from Outside Agencies were located in a separate room with the SIM team to initiate and receive simulated telephone calls to and from the EOC to create more realism Others representatives from Outside Agencies were positioned in the EOC to observe the interaction of our company team members

60 Outside Agencies and the Role They Played Examples of role playing telephone calls: Who is the Incident Commander? This is the Fire Department, we need to inform you that all power to the Data Center will be shut down due to concerns of electrocution. This is the police department, we need a list of all employees that have entered the building this morning as well as copies of any floor plans and video surveillance available at this time.

61 Outside Agencies and the Role They Played Examples of role playing telephone calls: We are searching the building and do not have access to certain areas, who can assist us in gaining access? This is the police dept we need to speak to someone from human resources. We need to gather any information available on the assailant. This is the fire department. Based on the condition of the Portsmouth facility, you will need a certificate of occupancy before your employees are allowed to return to work.

62 Outside Agency Observer Comments Deputy Chief of the local Fire and Rescue Department Your Organization displayed an amazing commitment to business continuity for your companies customers, and also caring for its employees during this exercise. Your organization is clearly a seasoned company in emergency planning and crisis management, having perfected the corporate Incident Command System (ICS) after many years of practice. As a citizen of this town and as someone who is insured through your company for auto and homeowner s insurance policies, I am very impressed and feel that no matter what happens in our world that your company will go on and deliver. Thank you for inviting me into your exercise and allowing me to observe.

63 Outside Agency Observer Comments The Local Deputy Sheriff Based on my experience and training in both law enforcement and emergency management I would characterize my overall impression of your company s LERT Command Section during this exercise as outstanding. This overall impression is based on the Command Sections obvious grasp of their role in the ICS and their acceptance of responsibility for dealing with the complexity of the challenges presented during the exercise.

64 Criteria for Success Has there been sufficient cross training in roles and responsibilities for all team members? Are the communication processes clearly defined? Can every role be filled with more than one person? Are any applications expected to be up and running that may not actually be available in certain circumstances?

65 Criteria for Success Do employees understand the scope of the Disaster Recovery process. Is there a documented process for every team member? Are roles associated with a single employee? What happens if that person is not available?

66 Criteria for Success Using the LERT manual, could someone that has never participated in an exercise reasonably be able to perform some duties if needed? How will status reports be given? Who will give them? How will phone calls (both incoming and outgoing) be handled?

67 Our Results What Worked Using the Incident Command System made communications with outside agencies run much more effectively than in previous exercises It was evident that the increased amount of training prior to this exercise was of great value Teams anticipated action items and started working immediately using the LERT manual

68 Our Results What Worked The various teams (Command, Operations, Logistics, Planning & Intelligence and Finance) assigned one person to handle messages between teams to improve communications This exercise was a great opportunity for the Kansas City, MO team to actively participate in the exercise

69 Our Results Areas for Improvement Employees began to slip out of their roles towards the end of the day Some roles were not as clearly defined as they could have been For example, the absence of two employees caused one team confusion regarding how to perform their specific tasks

70 Our Results Areas for Improvement Some teams were assuming that certain applications were readily available when in fact they were not Action planning meetings took longer than expected Improve the clarity of communications

71 Our Results Areas for Improvement Provide employees training on the Disaster Recovery Plan Improve the clarity of hand-off procedures for the shift changes during the exercise Add additional methods of communication such as whiteboards and overhead projectors

72 Our Results Areas for Improvement Increase the level of involvement with outside agencies Participate and observe outside agency exercises to gain insight into their process and procedures Some of our employees will play victims in a large scale readiness exercise simulated by a local County Emergency Management Agency

73 Ultimately, our goal is to improve our emergency preparedness by working together with outside agencies in our community to guarantee the safety of our employees.