BP3: Decomposing the Crisis/ Incident Management Timeline

Transcription

1 BP3: Decomposing the Crisis/ Incident Management Timeline Eric Staffin, MBCI, CISSP VP and Global Head, Product & Infrastructure Risk Management Investment & Advisory Doug Weldon, FBCI, CBRM VP, Product & Infrastructure Risk Management Investment & Advisory

2 Presentation Outline Introductions Quiz Standards Help, but. What is the Incident Management Timeline? What are the Anchor Specifications of the Timeline? The More Comprehensive Set of Specifications! The More Fully Articulated Timeline Summary and Questions

3 QUIZ The Product The product is an automated system for generating financial information to customers. Some specifications include: Needs to recover from any operational interruption and be fully functional in 5 minutes or less When the product is recovered, its databases must be exactly as they were at the point of the interruption there must be no data loss or corruption The product is read only and the databases are updated nightly (changes once per 24 hours) What is the RPO for this product?

4 QUIZ Some Help Definitions of Recovery Point Objective (RPO) from various Authoritative Sources: The maximum amount of data loss an organization can sustain during an event. (DRJ Glossary) The tolerance for lost data measured in time. (BRCCI) The Point to which information must be restored to enable an activity to operate once it is resumed. (Good Practices Guidelines of the BCI Section 2) Represents the amount of data that can be lost without severely impacting the recovery of operations or the point in time in which systems and data must be recovered (e.g., the date and time of a business disruption). (FFIEC Business Continuity Handbook)

5 QUIZ Some Help What is the RPO for this product? RPO = 0 hours RPO = 5 minutes RPO = 24 hours None of the above Answer?

6 Incident Management Timeline What is it? And why look at it? Is it really this simple? Let s look at a typical approach and then dig into the timeline

7 Timeline Example Copyright Sentryx (

8 Specifications Apply to All Critical Resources these recovery objectives require management to determine which essential personnel, technologies, facilities, communications systems, vital records, and data must be recovered and what processing sequence should be followed so that activities that fall directly on the critical path receive the highest priority (FFIEC Business Continuity Handbook) People Technology Data Facilities The Key Resources Multiple Inter dependent RTOs may be Integrated Into the Recovery Specifications of a Process/Product!

9 The Oil Rig Customers Customer #1 Customer #2 Customer #3 Customer #4 Customer # 5 Customer # 6 Products & Services Prod #1 Prod #2 Svc #1 Svc #2 Prod #3 Prod #4 Svc #3 Svc #4 Prod #5 Svc #5 Processes Process #1 Process #2 Process #3 Process #4 Process #5 Sites Site #1 Site #2 People Site #3 Site #4 Process Platforms & Resources Technology Facilities Data Suppliers Supplier #1 Supplier #2 Supplier #3 Supplier #4 Supplier #n Copyright Vigilant Services Group (

10 Why All the Specs? ( Promoting the Art & Science of Business Continuity Management Worldwide The BCI) BCM is not simply a management exercise, it is an engineering discipline that requires technical engineering and human engineering to look at people, data and systems as an ecosystem that must be understood and in balance with: Enterprise Objectives Risk Appetite of the Enterprise Customer Requirements Legal/Regulatory Requirements and Compliance with other Codes of Practice including Standards This is NOT Your Father s RPOs/RTOs Anymore!

11 The Anchor Specifications Maximum Tolerable Period of Disruption (MTPoD BS25999) the point in time after a significant interruption after which an organization s viability will be irrevocably threatened if product and service delivery cannot be resumed Maximum Tolerable Downtime (MTD Sentryx) or Business as Usual Time Objective (BTO Vigilant Services Group) the point in time after a significant interruption at which the product or process returns to a business as usual state in consideration of work queues, lost data, etc. (operating as if nothing happened) Maximum Allowable Downtime (MAD FFIEC Business Continuity Handbook) the point in time after a significant interruption at which the product or process can no longer be inoperable NOT RPOs/RTOs at All!

12 Policy Management with Anchors Since most failure scenarios are NOT smoke and rubble, planning for efforts to be under the MAD and/or MTPoD thresholds is critical Organizational policy must be upheld to ensure that no new solutions can be brought to market without an explicit articulation of MAD and MTPoD Economics of design follow MAD and MTPoD Worst case scenarios follow MTPoD SLAs usually follow MAD Profitability and likely to recommend metrics follow penalties and SLA performance

13 The Anchor Specifications Independent Variables MAD, BTO/MTD, and MTPoD are anchor specifications because they may be driven by external mandates: Customer requirements such as SLAs Regulatory requirements such as FINRA Enterprise Policy such as quality/reliability guidelines Anchor specifications may/may not vary by scenario: Data corruption Pandemic Act of God ISSUE these various standards/guidelines do not provide engineering level specifications with which to design solutions! RPOs/RTOs are Dependent Variables!

14 Operational vs. Business as Usual MAD + WRT = MTD/BTO Work Recovery Time (WRT Sentryx) is the difference in time between full operational recovery and return to business as usual WRT includes: Recovering all lost data beyond RPO specification of lost data as required Processing accumulated workloads down to normal queuing levels Processing any accumulated manual processing of workloads Other activities as required to return to normal performance levels Does NOT imply return to business as usual operating environment! ISSUE Who is responsible for the recovery of lost work/data? SLAs are often Based on MAD, not MTD

15 In Practice Is One Anchor Enough? Should the Maximum Tolerable Period of Disruption (MTPoD) only be applied to smoke and rubble scenarios? If not, is it enough to consider MTPoD (by itself) to define our optimal resiliency solution levels?

16 When MTPoD is the ONLY Anchor! Incident Duration (Hours) Commercial Exposure ($ Millions) MTPoD = $15MM Total Product Revenue $40MM Exposure ($ Millions) Duration (Hours) Single Incident with Duration X MTPoD

17 Ok So They Can Both Be Anchors Now What!? Incident Duration (Hours) Commercial Exposure ($ Millions) MAD = $2.5MM MTPoD = $15MM Total Product Revenue $40MM MTPoD Use historical incident management data to predict the likelihood of a disruptive event occurring adjust resiliency and recoverability levels ASAP!! Exposure ($ Millions) Duration (Hours) Single Incident with Duration X Cumulative Incident Duration Cumulative Commercial Exposure

18 MAD & MTPoD Are Now Anchors!! Incident Duration (Hours) Commercial Exposure ($ Millions) MAD = $2.5MM MTPoD = $15MM Total Product Revenue $40MM Exposure ($ Millions) Optimal resiliency solutions MUST incorporate single incident and cumulative incident exposure levels to ensure that MAD and MTPoD thresholds are NOT exceeded. Anchors MUST be established in the definition and planning phases of the SDLC propositions cannot go to market if anchors aren t LOCKED in place. Duration (Hours) Single Incident with Duration X Cumulative Incident Duration Cumulative Commercial Exposure

19 We re Good Right?? Remember FFIEC and the Oil Rig: Technology Facilities Data People Let s take a closer look at RTOs, MAD and RCOs and then we can incorporate everything into the IM timeline.

20 RTO vs. MAD MAD is the time required to achieve full operational recovery Recovery Time Objective (RTO) represents the maximum allowable downtime that can occur without severely impacting the recovery of operations or the time in which systems, applications, or business functions must be recovered after an outage (FFIEC) RTO: Has historically applied to the time required to recover a specific resource Does not appropriately describe the recovery of a full process, function, or product that has a number of supporting and dependent resources ISSUE Supporting Process/Product Resources may have interdependencies requiring prioritization Multiples RTOs may apply to define a single Process/Product MAD

21 Now We re Really Ready! We ve got our anchors, MAD & MTPoD, We ve got multiple RTOs We ve identified critical WRTs We understand BTO Now let s see how this looks on the Incident Management timeline.

22 Timeline Decomposition Lifecycle Approach RPO Interruption Incident Starts This line, with the additional milestone red and green dots, represents the full Lifecycle of the Incident Incident Ends Permanent Restoration

23 Timeline Decomposition Phase 1 RPO Warning Signs Interruption! Automated Failover Mitigation Fails Last backup of data at RPO point in time If we built monitoring we might now see early signs of a problem in advance of the interruption Incident begins! High and Continuous Availability Systems may failover automatically but don t forget about the backlog!! Now everyone knows!!! The problem has been detected (did you factor this time into the RTO??) Diagnosis phase begins the clock is still ticking!! Great we know what s wrong problem officially diagnosed. Attempted mitigation begins crisis management efforts may avert/reduce the need for recoveries Mitigation Fails (did we consider this when we sold the RTO to management??) Did we think about MAD and the possibility that it could take time to get to this point!!??

24 Timeline Decomposition Phase 2 Backlog Grows Warning Signs Interruption! Automated Failover Mitigation Fails Manual Failover or Recovery Recovery Time (RTO) YES! we knew this could happen and factored it into the design (always thinking about MAD) This part we know executing the failover plan is trivial since we ve proceduralized the process and tested it enough to repeat CONSISTENTLY But we also know that many of the high likelihood scenarios could impact more than just one product or infrastructure Or that the products/processes we are recovering may require the recovery of a number of components in sequences defined by dependencies So, since we know that there isn t just one RTO have we accounted for all of them?

25 Timeline Decomposition Phase 2 RTO Breakout Backlog Grows Warning Signs Interruption! Automated Failover Mitigation Fails Manual Failover or Recovery Recovery Time (RTO) Validation Dependent RTOs must be fully articulated for facilities, people, data, and people Remediation efforts for recovery may need to include smoke testing or QA/QC Validate system restoration plan and ensure that there is an orderly process for restoring data with or without other services being exposed to customers or downstream systems must be noted in RTOs for each service/component Revalidate based on dependencies across all other related infrastructures Isolate critical ingredients and reduce SPOFs that have more than single impact Why Test? Validate what you know in what timescale + What you don t know > dependencies

26 Timeline Decomposition Phase 2 Backlog Grows Warning Signs Interruption! Automated Failover Mitigation Fails Manual Failover or Recovery Recovery Time (RTO) Validation Product at RCO & RPO YES we knew this could happen and factored this into the design (always thinking about MAD) This part we know executing the failover plan is trivial since we ve proceduralized the process and tested it enough to repeat CONSISTENTLY Good thing we factored in the RTOs for each of the supporting and attendant resources without this, we might have blown by MAD! Dependencies cleared Serial and parallel activities will be completed here just like we planned it!! Validation begins smoke testing is still critical this isn t a drill and the risk of support or product failure is REAL if test plans aren t carried out efficiently Good news the product has been identified as fully functional at RPO loss of data and the RCO is validated What is RCO and why is it important?

27 Recovery Capacity Objective (RCO) RCO (source: Vigilant Services Group) is the capacity level to which each Process/Product resource is recovered at the RTO point in time for that resource RCO has NOT been identified in any standard/guideline to date RCO: Can only be determined based on a detailed understanding of the workload backlogs that can accumulate over time during the interruption of a Process/Product RCO may actually be greater than normal processing capacity if workload backlogs accumulate at a high rate relative MAD and MTD/BTO ISSUE If RCO is not adequately specified, WRT may be severely protracted beyond the point of MTPoD! Multiples RCOs may apply to define a single Process/Product MAD

28 Timeline Decomposition Phase 3 Backlog Grows Warning Signs Interruption! Automated Failover Mitigation Fails Manual Failover or Recovery Recovery Time (RTO) Validation Product at RCO & RPO Work Recovery Time (WRT) Return to BAU Failover Capability Restored Failback Procedures Failback Capability Permanent Restoration How are we doing still under MAD any SLA exposure or did we factor that into the design as well? Remember our Work Recovery Time (WRT) discussion Recovering all lost data beyond RPO specification of lost data as required Processing accumulated workloads down to normal queuing levels Processing any accumulated manual processing of workloads Other activities as required to return to normal performance levels Does NOT imply return to business as usual operating environment! Looks solid and we ve reached another green dot we have returned to Business as Usual (BTO) Was this the first outage in the measurement period check SLA and MTPoD thresholds Next step is to restore the failover capability and return the primary data center to available service levels Execute failback procedures invoked (if necessary) Validate failback success Permanent restoration of services achieved!

29 More Fully Articulated Timeline

30 One Major Incident Can Kick Off Many Timelines Product Data Center 1 Product Data Center 2 Customer Data Center PROD 1 PROD 2 PROD 3 PROD 4 PROD X PROD Y C PROD A C PROD B INCIDENT RTOs MAD MTPoD Dependencies can exist at the resource/component level or the full product level! INCIDENT DEPENDENCY RTOs MAD MTPoD

31 Summary RTOs and RPOs are the legacy specifications of BCM dating back to our origins in the time of mainframes (late 70s) More specifications are required to fully articulate the requirements not only of today s more complex technology architectures but also of the other supporting resources (people, data, facilities) and their complex interrelationships We must tie back to standards to anchor our requirements specifications, but must interpret standards level definitions to support engineering level development Bottom line we must mature our approach to developing BC/DR operational capabilities