Course Materials DISASTER RECOVERY AND BUSINESS CONTINUITY:

Transcription

1 Course Materials DISASTER RECOVERY AND BUSINESS CONTINUITY: Tom Williams Centurion Business Continuity Strategy Manager Jack Henry & Associates Northville, Michigan August 13, 2015

2

3 Disaster Recovery and Business Continuity Tom Williams Centurion Business Continuity Strategy Manager

4 Agenda Identifying the Disaster Risk Mitigation Profile for your bank The FFIEC Guidelines on Business Continuity Planning Centurion s Ten steps to Business Continuity Planning Centurion Disaster Recovery Suite of Services Q&A

5 How would you answer the following questions regarding your bank s BCP? How effective is our BCP? Is it an Enterprise Wide plan or just an I/T Plan? Is the plan tested and maintained on a regular basis? Will the plan meet the examination criteria?

6 The most important question; Will our plan get us through a serious Disaster Event?

7 Top 5 Reasons why banks do not have an effective BCP? 1. Let s just do enough to satisfy the auditors and examiners. 2. The myth that there is a plan in place that will work. 3. It costs too much money. 4. The disaster will never strike. 5. We are to busy with other projects that are of a higher priority.

8 Identify your Bank s Disaster Risk Mitigation Profile? Business Continuity Program No Business Continuity Program LOW RISK MODERATE RISK HIGH RISK BSA/ AML Intern al Fraud Each organization should continually strive to move toward the Low Risk area

9 The FFIEC Guidelines on Business Continuity Plan Updates Recovery Center Testing Tabletop Exercises Mock Drills Business Impact Analysis Disaster Impacts Prioritization Recovery Windows Recovery Strategies Resource Requirements Test / Maintenance Program Risk Assessment Enterprise Wide BCP Emergency Plan Crisis Management Plans IT & Business Unit Plans Develop Business Continuity Plan Threats Frequency Duration Forewarning 7

10 Centurion s Ten Steps to Business Continuity Planning 5. Determine Recovery Strategies 4. Conduct A Business Impact Analysis 3. Review Current Business Continuity Plan 2. Develop BCP Development Team 1. Obtain Management Support

11 Centurion s Ten Steps to Business Continuity Planning 10. Obtain Board and Regulatory Approval 9. Initiate Plan Maintenance Process 8. Test Recovery Plan 7. Document the Business Continuity Plan 6. Conduct a Risk Assessment

12 Step 1 Obtain Management Support Familiarize yourself and management with the FFIEC Business Continuity Guidelines Conduct research to determine plan deficiencies Compare your plan to different plans Talk with other FI s about their BCP approach Read articles to educate management on Best Practices for BCP in the financial sector Go to BCP / DR conferences for education Have your plan reviewed by BCP experts Conduct a mock disaster drill with your senior team

13 Step 1 Obtain Management Support Risk Measurement The correlation between the Level of Risk Mitigation and the Level of Recovery. Risk Mitigation Level Level of Recovery

14 Step 1 Obtain Management Support Disaster Level of Readiness Vs Cost of Recovery Enterprise Wide BCP tested and maintained at all levels High Cost of Recovery Level of Commitment Enterprise Wide BCP Medium IT Plan/Hot Site Only No preparations Low

15 Step 2 Develop BCP Development Team Select an Executive Owner Select a BCP Manager / BCP Coordinator Select a participant from each business unit Include BCP responsibility in job description

16 Step 2 Develop BCP Development Team Establish a project plan for plan completion Plan development tasks / responsibilities / timelines / success factors / critical decision points Departmental interviews / management reporting Escalation process and status meeting dates Determine the following: Plan development methodology (Internal or Outsourced) How plan information will be stored and accessed Word Processor or BCP Software Tool Plan structure

17 Step 3 Review Current Plan Plan Elements In Plan Not In Plan Evacuation Plan in place and tested regularly Succession / Escalation Plan Alternate Work Locations Identified for IT & Departments Critical Processes & Functions Identified / Prioritized Recovery Time Frames Identified at the Functional Level Risks Identified and Prioritized Critical Documentation Identified Resource Requirements Identified Emergency Phone Numbers for Internal & External support Recovery Teams Identified Recovery Tasks Identified for Personnel for I/T & Departments Manual Procedures Documented for I/T & Departments

18 Step 4 Conduct a Business Impact Analysis BIA 8. Document Contingency Procedures per function 7. Determine Alternate Locations to restore function 1. Identify the Processes & Functions per Business Unit The Business Impact Analysis Process 2. Determine the impact if the process / function is interrupted 3. Determine the Recovery Time & the Recovery Point Objectives 6. Determine the Resource Requirements 5. Determine the Recovery Strategies for each function 4. Prioritize to determine the Mission Critical Processes

19 Step 4 Conduct a BIA Determining Impacts IMPACTS Lost Revenue Accounting Records Fines and Penalties Vital Account Records Lost Financial Records Customer Service Work Flow Quality Life and Safety Public Opinion Social Issues Employee Morale Employee Stress

20 Step 5 Determine Recovery Strategies based on the BIA Core System Check Imaging Report Retrieval Document Imaging ATM/ Card Processing Internet Banking Fedline Voice Response Statement Printing Internal Network, i.e. Voice Communications Telephone Banking Call Center Operations Employees Facilities

21 Step 5 BIA Regulatory Expectations Prioritizing Critical Business Functions Source: FFIEC IT Examination Handbook, Business Continuity Planning, March 2008, Appendix F, p. F-3

22 Step 5 BIA Prioritizing Critical Business Functions Can we recover our technology infrastructure from a disaster? What is our Recovery Time Objective (RTO) for our core? What is our RTO for our server environment? What is our Recovery Point Objective (RPO) for core? What is our RPO for our server environment? 20

23 Step 5 Determine Recovery Strategies Cost Vs. Level of Commitment Technology Infrastructure Lower Cost Higher Minutes RPO=near zero, RTO <1min, Automatic Server/Workload/Network/Data SYSPLEX RPO=Near zero, RTO <1Hr. to 4 hours, Automatic Server/Workload/Network/Data Automatic Site Switch RPO=Near Zero, RTO <1Hr. to 4 hours, Manual Disk or Tape Data Mirroring RPO > 15 min. RTO= 4+ hours, Manual PiT or SW Data Replication. RPO=4+ hours, RTO=8 to 24 hours, Manual Point-in-Time Backup to Tape / Disk Data Base Log Replication & Host Log Apply at Remote RPO<24 hours RTO = 8-24 hours Electronic Tape Vaulting Hours Continuous Availability- Disaster Avoidance RTO=Days, RPO>24 hours Tape, HW ATOD Days Multi-Site Failover / Fallback Traditional Tape Recovery

24 Step 5 BIA Recovery Solutions Align recovery strategies to the BIA Critical Business Functions RTO Resources Function Max Allowable Downtime Applications & Systems Core processing Critical Min. to hrs. SilverLake, CIF 20/20 ios Business Continuity Program Item processing Critical Min. to hrs. 4 Sight Wintel Document Imaging Urgent 24 hrs Synergy Wintel Online banking Urgent 24 hrs NetTeller N/A Telephone banking Critical Min. to hrs. italk Wintel Mobile banking Urgent 24 hrs NetTeller N/A Bill pay Urgent 24 hrs ipay Solutions N/A Check printing Urgent 24 hrs SilverLake, CIF 20/20 ios Credit card processor Urgent 24 hrs jhapassport Wintel OS

25 Determine Recovery Strategies based on the BIA Core Applications Network-Internal External Connectivity Network Files Check Imaging Report Retrieval Document Imaging Communications Branch Capture Statement Printing Internet Banking Voice Telephone Banking ATM/ Card Processing Fedline Voice Response Central Capture

26 In-house Processing Considerations Responsible for the restoration of the following: System Recovery of Core System Network Recovery Server Recovery ATM / Debt Cards Check / Document Imaging Internet Banking Telephone Banking Wires, etc. Voice Recovery Equipment setup & Re-configuration

27 Out-Sourcing Processing Considerations Responsible for the restoration of the following: Connectivity to the Core Processing Site Network Recovery Server Recovery for services that are in-house Internet Banking Telephone Banking Wires, etc. Voice Recovery Equipment setup & Reconfiguration - Facilities Ensuring that the bank has a plan to deal with a disaster that strikes the processing center

28 System / Application Recovery Strategies Traditional Media Device Backup Tape USB Hard Drive CD Virtualization / Replication Electronic Vaulting Disaster Avoidance / High Availability

29 Recovery Strategies Full HA INFRASTRUCTURE $$$ Tape Recovery Disk Recovery San Replication 72 HRS 24 HRS 4 HRS 0 HRS RECOVERY TIME OBJECTIVE

30 Traditional Recovery Solutions Customer Primary Site Yellow Hammer Switch Core Tape 4 Sight IVR SAN

31 Traditional Tape Recovery Solutions RPO = Last EOD Tape Backup RTO = 24 hrs - 72 hrs Core DR Center Switch Customer Primary Site MPLS 4 Sight IVR SAN Branch 1, 2, or 3 Switch Switch Yellow Hammer Core Tape 4 Sight IVR SAN Network SAN

32 Traditional Tape Recovery Solution RPO = Last EOD Tape Backup RTO = 24 hrs - 72 hrs Core DR Center Switch MPLS 4 Sight IVR SAN Customer Primary Site Branch 1, 2, or 3 Switch Switch Yellow Hammer Core Tape 4 Sight IVR SAN Network SAN

33 Traditional Tape Recovery Solution RPO = Last EOD Tape Backup RTO = 24 hrs - 72 hrs Core DR Center Switch Customer Primary Site MPLS 4 Sight IVR SAN Branch 1, 2, or 3 Switch Switch Yellow Hammer Core Tape 4 Sight IVR SAN Network SAN

34 Centurion Hosted High Availability Recovery Strategy

35 RPO = Last Transaction RTO = ~15 min. RTO = ~30 min. High Availability Recovery Solution CENTURION HOSTED HIGH AVAILABILITY FOR CORE Customer Primary Site MPLS Riverbed Switch 4 Sight IVR Core DR Center SAN Branch 1, 2, or 3 Yellow Hammer Riverbed Switch Switch Core Tape 4 Sight IVR SAN Network SAN

36 High Availability Recovery Solution HIGHER AVAILABILITY FOR CORE RPO = Last Transaction RTO = ~15 min. RTO = ~30 min. Customer Primary Site MPLS Riverbed Switch 4 Sight DR Site Core SAN IVR Branch 1, 2, or 3 Yellow Hammer Riverbed Switch Switch Core Tape 4 Sight IVR SAN Network SAN

37 Vaulting Defined Vaulting is an online data backup solution that enables financial institutions to store critical data at a secure, off-site location. Vaulting mitigates inherent risks associated with physically transporting tapes from one location to another. Complies with related regulatory requirements, and provides financial institutions with a superior audit trail for all backed-up data.

38 Benefits of Electronic Vaulting Eliminates need for onsite tape. Scheduled automatic backups. Off-site secured protection. Immediate restores. Comply with new and impending regulations. Gain a competitive advantage. Restores are quick, available and assured. Initial full backup to secure data center. Pro-active monitoring and management.

39 Cloud Based Recovery Strategy Vendor Provided Vendor Hosted High Availability Vendor owns hardware. Customer replicates data to hardware in Real Time environment using Mimix or other replication software. Customer data located off-site 24/7 in vendor secured facility. In a disaster situation customer eliminates the need to travel to restore system.

40 CELR Internet Diagram Centurion Enterprise-Level Recovery for Network Centurion RPO = Min to Hours RTO = <24 hours Switch SAN Primary Site Secondary Site Appliance SAN Switch Switch Network 4 Sight Core Network SAN

41 Physical vs Virtual Servers The agent can protect Physical servers. The agent can protect a Virtual server. The agent can protect the Hypervisor. (Vmware)

42 Vault Entire Server vs Select Folders Depending on your recovery plan. Core Dir Other jha apps Exchange Domain controller

43 Bandwidth Requirements CELR travels through your standard internet connection. Speed required will depend on amount of GBs protected. Typically a Minimum of 10mb up.

44 Recovery Solutions Centurion Enterprise-Level Recovery for Network On-site Appliance RPO = Min to Hours RTO = Min to Hours Recovery at LAN Speed Primary Site Appliance Switch SAN Network 4 Sight Core

45 On-Site Media Agent The OMA will give you fast local recovery. Speeds up the backup of data off the server.

46 Recovery Solutions Centurion Enterprise-Level Recovery for Network Centurion Standby virtual infrastructure for testing and recovery Switch SAN Primary Site Appliance SAN Switch Switch Riverbed Network Secondary Site Network 4 Sight Core Network SAN

47 Recovery Options Recovery options: Back to the original server Centurion A new VM in your environment Virtual Server Recovery environment (VSR)

48 Virtual Server Recovery (VSR)

49 What is VSR A virtual server environment that we can configure to match your needs.

50 RTO & MAD What are your Recovery Time Objectives. What is your the Maximum Allowable Downtime.

51 Communications The Bank will need to supply a router in Branson. At the time of use you will open a VPN tunnel to Branson. Centurion will open the VLAN to the servers we have configured for your use.

52 VSR Recovery Diagram Disaster recovery support of virtually every component of operational infrastructure Switch Core Centurion SAN Primary Site 4 Sight Network Secondary Site Appliance SAN Switch Switch Network 4 Sight Core Network SAN

53 Vendor Hosting of Server Environment Diagram

54 Step 6 Conduct a Risk Assessment Natural Quake Tsunami Fire Lightening Tornado Typhoon Epidemic Technological Intentional Utility Failure Cyber Attacks Air Crash Reputational Hazmat Espionage Contamination Terror Mumbai System Failure Threats Proximity Crisis Food Tampering Economic Strike Riot

55 Step 6 Conduct a Risk Assessment Probability of Occurrence Applicability Geography History/Current Events Severity Forewarning Speed - Duration Mitigation Prevention

56 Step 6 Conduct a Risk Assessment

57 Step 7 Document the BCP Step 4 Draft Plans Generated Emergency Management Plan (Per Facility) Crisis Management Plans Information Systems Recovery Plan Business Unit Recovery Plans Executive Summary Plan Testing & Exercise Guide 55

58 Step 7 Document the BCP Table of Contents

59 COPE: Centurion s Online Business Continuity Expert Business Continuity Planning Tool Based on Best Practices of the Financial and Business Continuity industry. A web based business continuity plan built on an SQL server platform utilizing a relational database. Built in house by JHA software developers. Fosters plan ownership at the business unit level. Designed solely for financial institutions. Access plans for planning purposes, testing, maintenance, and plan execution from any web browser.

60 COPE Business Unit Teams

61 COPE Departments

62 COPE Business Functions

63 Step 7 Team Responsibilities Crisis Management Teams Business Units Recovery Teams Management Administrative Damage Assessment Information Systems Business Functions

64 Step 7 Recovery Team Organization Chart Crisis Management Teams Information Systems Management Team Pandemic Response Planning Business Unit Recovery Teams Administration CIG, CFG Asset Management CIG, CTC Trust Services Deposit Operations Facilities Management Administration Support Asset Management Client Support Trust Administration Trust Operations Bookkeeping Electronic Services Item Processing Operations Scanning Human Resources Compliance Human Resources and Training Internal Audit Information Technology IT Internet Branch Clear Sky Branch Lending Recovery Team Commercial/Consumer Loan Operations Mortgage Loan Operations

65 Step 7 Crisis Management Teams Management Crisis Team Administrative Crisis Team Damage Assessment Crisis Team Information Systems Crisis Team Team Leader: Alternate Team Leader: Team Members: Team Leader: Alternate Team Leader: Team Members: Team Leader: Alternate Team Leader: Team Members: Team Leader: Alternate Team Leader: Team Members:

66 Business Unit Recovery Teams Accounting Accounting Administration Chief of Staff Legal Administration Chief of Staff Legal Deposit Ops Loan Ops TM Ops Fraud Card Services Commercial Documentation Deposit Operations Fraud Investigations Loan Operations Treasuary Management Operations

67 Business Unit Recovery Teams ERM BSA Compliance Internal Audit Bank Secrecy Compliance Enterprise Risk Management Internal Audit Human Resources and Project Management Human Resources Project Management IT Risk Management Network Applications Application Support IT Risk Management Network Services

68 Business Unit Recovery Teams Lending Credit Agricultural Lending Commercial Finance Commercial Lending Consumer Loans Collections Credit Special Assets Loss Share Treasury Management Marketing Marketing

69 Plan Execution/Recovery Timeline E V E N T O C C U R S Crisis Management Phase Relocate & Restore Phase Recover Business Functions Phase Rebuild & Return Phase

70 Plan Execution/Recovery Timeline E V E N T O C C U R S Crisis Management Phase Relocate & Restore Phase Recover Business Functions Phase Rebuild & Return Phase Evacuation & Safety Damage Assessment Communications Disaster Declaration

71 Plan Execution/Recovery Timeline E V E N T O C C U R S Crisis Management Phase Relocate & Restore Phase Recover Business Functions Phase Rebuild & Return Phase Evacuation & Safety Damage Assessment Communications Disaster Declaration Notifications Mobilization Relocation Restore

72 Plan Execution/Recovery Timeline E V E N T O C C U R S Crisis Management Phase Relocate & Restore Phase Recover Business Functions Phase Rebuild & Return Phase Evacuation & Safety Damage Assessment Communications Disaster Declaration Notifications Mobilization Relocation Restore Resume Services

73 Plan Execution/Recovery Timeline Crisis Management Phase Relocate & Restore Phase Recover Business Functions Phase Rebuild & Return Phase Evacuation & Safety Damage Assessment Communications Disaster Declaration Notifications Mobilization Relocation Restore Resume Services Rebuild/Repair

74 Step 7 Document the BCP Establish a Recovery Time Line 0?? Hours TBD 7 2 Recovery Time

75 Step 7 Process and Resource Recovery Function Description: Accept and post deposits Recovery Window: Within 24 Hours Describe the business tools and systems needed to complete the activity. 2.1 Information Systems Core Processing Data on Network (External) (Internal) Internet Other EZTeller 2.2 External Data Exchange System System Connections to third parties via modem or other data file exchange. Connection to: File Name/Contents: MS Other Word (Name) 2.3 PC Software MS Access MS Excel MS Internet Explorer 2.4 Specialty Software Examples are teller/platform systems, loan processing systems, A/L management, etc. Name/Description: Supplied by: Jack Henry EZTeller

76 Step 7 Process and Resource Recovery Function Description: Accept and post deposits Recovery Window: Within 24 Hours 2.5 General Office Equipment Adding Machine Copier Fax Machine Phone Typewriter Other: 2.6 PC Workstations & Peripherals PC Workstation Printer Other Computer Hardware (Name): 2.7 Specialty Equipment Examples: Proof, image capture, sorters, etc. Separate multiple entries with a comma (,). Description: Model: Supplied by: Forms Used Electronic (Name): Paper (Name): Reports Used System Generated Manually Externally Supplied (Name): Prepared Supplied by: Files/Documents Name of File/Document: Format: Electronic Physical: Location: External Services Examples: Credit Reporting, Appraisals, Legal or Accounting

77 Step 8 Test the BCP Testing Benefits Identify weaknesses and exposures in the plan Identify backup and cross training requirements Provide training for team members & vendors Establishes credibility and authority to the plan Improves self confidence thru rehearsals Validates contract subscriptions with vendors Meet regulatory compliance requirements Authenticate recovery tasks and timelines

78 Step 8 Test the Plan Test Gap Analysis Modifications Identify Scope Set Test Objectives Conduct Test Test Plan Identify Resource Requirements Determine Test Budget Identify Schedule Options Identify Participants

79 Step 8 Analyze Test Results Conduct a Post Test Review Meeting Review results with team members and observers Identify items to be re tested for next test Document test results for management & auditors Amend Business Continuity Plan as required Update Change Management Program Schedule next test and set test objectives

80 Step 8 Test the BCP Testing Checklist Areas to be tested Activity Recovery of individual application systems by using files and documentation stored off site. Reloading of system tapes and performing an IPL by using files and documentation stored off site. Ability to process on a different computer. Ability of management to determine priority of systems with limited processing. Ability to recover and process successfully without key people. Ability of the plan to clarify areas of responsibility and the chain of command. Effectiveness of security measures and security bypass procedures during the recovery period. Ability of users of real time systems to cope with a temporary loss of on line information. Ability of users to continue day to day operations without applications or jobs that are considered noncritical.

81 Step 9 Initiate Plan Maintenance Process Have all updates coordinated by BCP owner Establish Plan Ownership at the department level Make department managers responsible for updating their business unit plans Update plan annually for smaller organizations and biannually for larger organizations using the FFIEC guidelines. Integrate plan into the Change Management Process Develop a plan update status report & report results to senior management

82 Step 9 Initiate Plan Maintenance Process Maintenance Cycle: Maintenance Phases: One Year 12 Months 6 Phases 2 Months per Phase Phase 1: Locations, Personnel, Recovery Teams, Internal Notifications Phase 2: Business Functions, Processes, Resources Phase 3: Vendors, External Notifications Phase 6: Conduct Plan Exercises, Changes from Exercises Phase 5: Update Documentation, Prepare for Tests and Exercises Phase 4: Schedule Plan Exercises, Application Recovery Procedures 80

83 Step 10 Obtain Board and Regulatory Approval Has policy been determined on how to manage and control identified risks? Have knowledgeable personnel and sufficient financial resources been allocated to properly implement the BCP? Has the BCP been independently reviewed? Are employees trained and aware of their BCP roles? Is the BCP regularly tested on an enterprise wide basis? Has the board reviewed the test results and improvement plans based on the test results? Is the BCP continually updated to reflect the current operating environment?

84 Identify your Bank s Disaster Risk Mitigation Profile? Business Continuity Program No Business Continuity Program LOW RISK MODERATE RISK HIGH RISK BSA/ AML Intern al Fraud Each organization should continually strive to move toward the Low Risk area

85 Next Steps 1. Ensure you have Executive support for the BCP. 2. Have your BCP reviewed by BCP Experts. 3. Conduct a Mock Disaster Drill using your BCP. 4. Determine if outside expertise is required to improve your plan, or if the work will be done internally. 5. Ensure that your BCP is structured at the department level. 6. Build / improve your plan and test it regularly

86 Centurion Business Continuity Planning Services Business Continuity Plan Development Deluxe Business Continuity Plan Development Option Remote Business Continuity Plan Development Option Review of your current BCP / DR Plans Mock Disaster Drills / Training on BCP / DR Business Continuity Strategic Planning Session Business Continuity Executive Webinars (Free)

87 Centurion Suite of Services

88 Contact Information Tom Williams Centurion Business Continuity Strategy Manager GSB Faculty Instructor

89 Questions