Enterprise-wide Business Continuity and Disaster Recovery Planning. Presented by Kelley Okolita

Transcription

1 Enterprise-wide Business Continuity and Disaster Recovery Planning Presented by Kelley Okolita

2 Don t get caught without a plan

3 Gloom and Doom My job and yours is to preach Doom and Gloom

4 Planning, not panic In order to make sure we plan for potential risks

5 What to plan for Loss of site Loss of technology Loss of people Organizational Crisis

6 Obtaining Management Commitment Management should have a documented responsibility to the company in the development and testing of a viable Business Recovery Plan Without management commitment to this project, it will fail Require Management sign off and approvals at each major milestone of the project

7 Phased Approach Emergency Notification List Vital records backup and recovery Risk and Business Impact Analysis Strategy Development Alternate Site selection and planning Plan Development Testing, maintenance, update

8 Building a Team Business Resumption Plan Emergency Notification Emergency Response Recovery

9 Emergency Notification Identify the different types of recovery you will plan for Identify who would have the authority to declare a disaster depending on the scenario Identify who would be part of the recovery effort Build your notification list based on this information

10 Vital Records Do you know: Where they are? What is included in them? How to get them? Who is authorized to retrieve them? How long it will take to retrieve them? Where to have them delivered? How long it will take to restore them?

11 Risk Analysis Identify Business Risks Estimate Probabilities of Risk Occurring Identify Mitigating Factors Implement or Initiate Additional Controls Where Possible Risk analysis tells us where to spend our mitigating dollars

12 Risk Assessment Elements of Risk Threats Assets Mitigating Controls

13 Risk Definitions Financial Strategic Organizational Technology Operational Legal/Regulatory Risks that are an inherent part of the business environment and have an effect on business objectives and performance Risks that are part of a unit s environment relating to people, culture, organizational structure and values that can impact overall organization effectiveness Risks associated with the use of systems and technology, including availability, capacity integrity, operational support, functionality systems integration and change management Risks relating to enforceability of contracts, interpretation of laws, compliance with law and impact of regulation Liquidity, Cap & Funding Credit Market People Process Events Inability to raise debt or equity capital as needed for shortterm liquidity or long-term growth, as well as uncertainty in pricing or sales of assets or liabilities Exposure to loss relating to a change in the credit-worthiness of a counter-party, collateral, customer or partner that may impact the counterparty s ability to fulfill its obligations under a contractual agreement The uncertainty in the future market value of a portfolio of assets and / or liabilities The risk of loss resulting from people The risk of loss resulting from inadequate or failed processes The risk of loss resulting from an unexpected event That interferes With normal business

14 Protecting People and Workspaces Access Control/Key Management Alarm Monitoring Floor Warden/Evacuation Drills Background Investigations Workplace Violence Programs Landscape Design Lighting Cameras Visitor procedures Backup Power systems Facility design/facility sighting

15 Protecting Information Information Security policy and procedures Privacy Policy Firewalls Intrusion Detection Strong Passwords Controlling access to information/standard Access Definitions Vendor Management Secure offsite storage Proprietary Waste Disposal Virus Protection and Response

16 Protecting Reputation Strong Governance Media trained Communication Plans Internal and external audits Operational Management Recoverability Code of Ethics

17 Business Impact Analysis Identify Business Functions Determine Impact of incident Estimate Business loss Determine Recovery Timeframes Gather Requirements for Recovery Business Impact Analysis tells us how long we have before we need to be back in business

18 Impact of a Disaster A disaster may impact... Your Paycheck Company Reputation Customers Ability to meet regulatory requirements

19 Recovery Strategies for Business Recovery strategies will be driven by the recovery timeframe of the function. Recovery options might include the following: Self-service - A business can transfer work to another of its own locations which have available facilities Internal Arrangement - Training rooms, cafeterias, conference rooms, etc... may be equipped to support business functions. Reciprocal Agreements - Other business units may be able to accommodate those affected. This could involved the temporary suspension of non-critical functions at the business units not affected by the outage. Dedicated alternate sites - Built by your company to accommodate critical function recovery. External Suppliers - A number of external companies offer facilities covering a wide range of business recovery needs. No arrangement - for low priority business functions it may not be cost justified to plan to a detailed level. The minimum requirement would be to record a description of the functions, the maximum allowable lapse time for recover, and a list of the resources required.

20 Recovery Strategies for Technology Dual Data Center The applications are split between two geographically dispersed data centers and either load balanced between the two centers or hot swapped between to the two centers. Internal Hot site this site is standby ready with all necessary technology and equipment necessary to run the applications recovered there. External Hot Site - This strategy has equipment on the floor waiting for recovery but the environment must be re-built for the recovery. These are services contracted through a recovery service provider. Warm Site A leased or rented facility that is usually partially configured with some equipment, but not the actual computers. It will generally have all the cooling, cabling and networks in place to accommodate the recovery but the actual servers, mainframe etc equipment are delivered to the site at time of disaster. Cold Site A cold site is a shell or empty data center space with no technology on the floor. All technology must be purchased or acquired at the time of disaster.

21 Documenting the Plan When an emergency happens, everyone needs to know what to do next: GATHER, ASSESS, DECIDE, MOBILIZE, COMMUNICATE, and RECOVER The plan document needs to address all of this

22 Plan Components Purpose, Objectives and Assumptions Human Resources how you will take care of the people Finance How you will track and pay for recovery expenses Communications How you will communicate with stakeholders Recovery Strategies how you will recover Recovery Management how you will manage the over all recovery effort Declaration Procedures how a disaster is declared Offsite Storage procedures how to get your stuff back Alternate site location and directions Command Center locations Seat Assignments in the alternate site Recovery Priorities for Business operations and technology Logistics how you will manage logistical support for recovery (travel, food) Detailed recovery procedures

23 Exercising the Plan Types of Exercises Call Notification Walkthrough Actual/Simulated Comprehensive Set Exercise Objectives Develop an Exercise Plan Conduct the Exercises Document Exercise Results Plan Maintenance

24 Transition from Project to Program Business Continuity needs to become part of the culture of the organization Every single employee needs to know the answer to the question Use Business Continuity tools and practices to manage everyday events

25 Program Requirements Deliverables Emergency Notification List Business Functions/ Resource Requirements Business Resumption Plans with sign-off Training & Awareness Vital Records Program Technology Reviews Call Exercise Walk-Through Exercise Simulated Or Actual Exercise Compact Exercise Systems Loss Test Technology Recovery Exercises Due date Quarterly Semi-Annually Annually Quarterly On-going Annually Semi-Annually Annually Semi-Annually Annually Annually Semi-Annually

26

27 Technology Recovery Status Platform or Application RTO Last Tested Procedures Current & Offsite Recovery met RTO End User Validated Batch Cycle Recovered Successfully Mainframe 24 hours 02/13/2009 NA Peoplesoft 24 hours 02/13/2009 NA Accounting System 36 hours 02/13/2009 Website 12 hours 02/13/2009 NA

28 Imbedding in the Culture Event Management Process Facility Events Technology Events Workforce Impairment Events Information Security Events Crisis Leadership

29 Event Management Contingencies start with Event Management If you do not properly manage Events, all the other Risks may occur Event Management is about Communication and Response Event Management needs to be practiced

30 Goals of Event Management Single Source of Information Triage Rapid Escalation Consistent Problem Management Rumor Control Make sure everyone who needs to know does Allow the problem solvers room to solve Playbook which documents key roles and responsibilities Nearly everyone wants this and agrees that it is needed.

31 Process of Event Management Central reporting location Standing conference bridge Automated notification system Assessment Teams built by event type Team for very location Team for every primary application Assessment team contacted first Escalation needs decided at assessment based on event level Who else needs to know/who else needs to help

32 HERO Team Members Crisis Management Team Event Owner Technology Services Facilities/Real Estate Claims Info Security Regional Management LOCAL RVP, REGIONAL PRESIDENT TSC Contingency Planning/Crisis Team Leader Human Resources Event Management Team Event Manager Business Operations/Leader Technology Services Finance HR/Real Estate Claims Legal/Compliance Corporate Communications Contingency Planning Risk and Insurance Key Business Stakeholders - Event Communicator Call Center Mgmt Agency Services Mgmt Business Continuity Planners Business Leads

33 Communications by Event Level Event Level Management Teams Response Teams 0-1 per year 9/11 Catastrophic loss of Main office Katrina Executive Management Team Event Management Team (Technical, Business, Customer) Make Big Decisions Manage, Translate & Communicate Response Teams (Customer) Response Teams (Technical & /Business) Receive Communications & Act Investigate, Resolve & Update Crisis Management Team Investigate, Resolve, Update 3-6 per year Core system outages Facility fire Event Management Team (Manage Business Impact and Response) Crisis Management Team (Own fixing the problem) Response Teams - Customer (Communicate) Response Teams - Technical/Business (fix the problem) per year Power outages Severe weather Water leaks Outward facing application issues 100 s per year Server failure Internal Application issue Hurricane that doesn t hit Crisis Management Team (Own fixing the problem) Response Teams - Technical/Business (fix the problem) Response Teams - Technical/Business (fix the problem)

34 How ready are you? Pretend you are evacuated right now and you are standing in your assembly area and Facilities tells you that you can not work in the building for at least the next 2 weeks, Do you know what to do next? Does your staff? Business Continuity Planner HERO Initial Response Plans Alternate sites Pandemic Planning Testing

35 Step 1 Identify your team Identify your team and make certain you know how to reach them in an emergency

36 Step 2 Identify your vital records Identify vital records Procedure manuals forms vendor lists contact lists customer lists contracts source documents

37 Step 3 Identify Your Business Functions Identify the business functions for your functional areas Perform risk and business impact analysis for each function Establish the recovery time for your business functions Identify minimum staff requirements Identify Interdependencies

38 Step 4 Identify your desktop requirements Minimum desktop configuration Application connectivity Voice Requirements phones Fax Modems Print Requirements Proprietary software running on desktop

39 Step 5 Define Recovery Strategy Develop recovery strategy for business functions and technology based on the recovery priority

40 Step 6 Internal Site Survey Survey existing sites Identify equipment/phone services Identify desktops to be used for contingency Identify staff to be displaced or moved to off shift

41 Step 7 External Site Recovery Prepare RFP which includes all requirements Identify essential vs. nice to have Receive proposals from vendors Compare for requirements and costs Visit sites identified as potential vendors Select vendors

42 Step 8 Internal Systems Identify all platforms and applications supported by internal systems group Identify recovery priority for each application Identify recovery strategy which meets the business requirements Develop recovery procedures for critical applications

43 Step 9 Document Plan Pull the information together into a plan document and distribute

44 Step 10 Train your staff Everyone should know the answer to the question : If you couldn t get back in your building today, what would you do next?

45 Step 11 TEST, TEST, TEST Event Management tests Alternate site tests Test alternate site

46 Don t be the one taken by storm!! It could happen anywhere...

47 Don t get caught without a plan