Is Your H/A Truly High Availability. Richard Dolewski

Transcription

1 Is Your H/A Truly High Availability Richard Dolewski

2 What is a Disaster? ANYTHING! That stops your business from functioning & that cannot be corrected within an acceptable amount of time. 2

3 The Value of Systems Availability Competitive value Increased End-user & Business Productivity Ongoing improvement in customer support & service Positive Business Image Reduce Outages = 3

4 The Value of Systems Availability Its value is unique to your business Is "IT" mission critical to your business? Is "IT" a competitive advantage? Do you know how much an hour, a day, or a week of application downtime costs your business? 4

5 Dynamic Business Needs Drive HA 24 x 7 Availability: 24/7 Customer support 24/7 Services 24/7 E-commerce 24/7 Manufacturing Service level agreements Compliance Decreasing maintenance windows Unplanned downtime 5

6 The Value of Systems Availability Does your business have a Business Continuity & Recovery Plan? Is it a true business plan, or an IT plan? Who owns the plan? How often has it been tested? 6

7 How much money would your company lose if a major outage occurred? 7

8 Disaster Recovery THE GOAL No Business tolerance for DOWNTIME Critical systems & Networks continuously available Business interruption measured in hours & minutes rather than DAYS. 8

9 High Availability 9

10 Business Continuity Hierarchy Business Resiliency High Availability Disaster Recovery Transaction Integrity Risk Management Disk Protection Backup 10

11 High Availability Mirrors business critical applications to another iseries/i5 Disaster Recovery from days/hours to minutes Reduce Downtime from System Maintenance Eliminate Downtime from Daily Tape Saves Move Workloads Off Production System Audit Requirements fulfilled and verified 11

12 Decreasing Cost of Bandwidth Since 1999, each MB of communications bandwidth has dropped over 90%! Allows most companies to economically connect to offsite backup i5/iseries for disaster recovery protection 12

13 Decreasing Cost of HA Software Full-featured HA software is priced significantly less today IBM s OS/400 Remote Journaling for data transport dramatically reduced development costs 13

14 High Availability Addresses both planned & unplanned outages Multiple systems are kept synchronous When one is down the Hot Backup takes over. Ensure no planned or unplanned outages Recommended level of availability management for systems running mission critical applications. 14

15 Remote Journaling There are two ways to move data to target Machine: Proprietary Local Journal Harvest & Send Or Remote Journaling 15

16 High Availability Caution Flag BULL**** METER 16

17 Disaster Recovery Planning I have H/A Therefore I do not need a DR Plan! 17 17

18 No Disaster Recovery Plan Guarantees: Confusion Lack of direction Conflict Lost Customers 18

19 Disaster Recovery Planning Cannot be approached casually The Plan must be... Well organized Action Oriented Comprehensive Objective: Total service Delivery in a timely manner 19

20 Continuous Availability I replicate my Data Base to another Partition Its another form of H/A! 20 20

21 Continuous Availability Production Partition Backup Partition High Availability Software 21

22 Why is Continuous Availability not H/A! It does eliminate the need for planned shutdown of production system Objective is to to allow users 24 hour access to the production system Normally only key Production application libraries are mirrored in this approach 22

23 Why is Continuous Availability not H/A! Typically no full system mirror recovery capabilities Does this solution eliminate ALL backup outages on the production partition? 23

24 Point of Failure Production Partition Backup Partition High Availability Software 24

25 H/A - Side by Side I fully replicate my H/A Both boxes are in the same computer room! 25 25

26 Building Address: 222 Cross my Fingers Drive Primary Node Backup Node Send & Receive Apply Users 26

27 H/A Side by Side Reasons for this approach. Cost to build a second data center Environmental No other sites in the corporation Communication costs - latency Staffing 27

28 Issues with Side by Side Single point of failure Same power grid Same CO for communications No alterative in a Building Disaster 28

29 H/A - Side by Side New Building Address: 222 Un-Employment Lane 29

30 How Far is Really Far Enough 3 Miles? 20 Miles? 100 Miles Generic Guidelines adapted from the Army Err on the side of Caution Costs Issues and Transport Time 30

31 Big Guy Little Guy My Target machine is 50% smaller because in a Disaster it should not really matter!! Its only for a short time! 31 31

32 Does Size Matter? 5500 CPW 2400 CPW 32

33 Performance Profile of Target Will the end users accept poor response time? Will your customers understand its takes twice as long to turn around an order? SLA Contractual commitments 33

34 Performance Profile of Target Switching back to the bigger system may become mandatory versus when convenient A second outage 34

35 Application Assessment I mirror all my Production Libraries Nothing is hard coded! At least I do not Think so

36 Common Findings Mirrored System data integrity in serious question Data in-consistencies beyond application. Numerous application support model requirements missing or out of sync. Solution questioned by Management 36

37 Requirements Definition Ensure the target Server is in a consistent, known, usable state which minimizes application impact Provide a technological compliant solution in the current operating infrastructure Provide fully documented procedures for recovery process any time of day. 37

38 Workshop Review present Production model Identify Consistent Breakpoints in current model for recovery Define Recovery model using H/A Software Provide H/A education Define Scheduler Requirements Define Report Distribution Define Messaging Model 38

39 Your to Do list Obtain 3 rd party license keys for applications Special product restart procedures Identify any special switching requirements for ERP packages (SAP, JDE, BPCS) if multi-tier switching is being done. 39

40 Your to Do list Check home grown applications for hard coded IP addresses or system names. Communication file retransmissions Application Interfaces Application Security 40

41 IBM Side of the Fence Is my Target = Source I thought the H/A product took care of That! 41 41

42 IBM System Libraries Maintain the same Operating system level Maintain same CUM Groups & PTF s Any IBM applications installed on the source must be installed on the target. 42

43 IBM System Libraries Maintain the IBM job scheduler Entries Does your H/A product support all IBM LICPGM s Mirror user profiles only, not *Q profiles. SBSD, JOBD, OUTQ, JOBQ, DIRE, etc) 43

44 IBM System Libraries Libraries QUSRSYS & QGPL IFS, DLO and Spool Files Have the latest *SAVSYS available in case you need to update the system. 44

45 Communications Can I get to the Target System Oh my carrier and Network folks take care of that! 45 45

46 Communications Are all LIN/CTL/DEV descriptions on the target Review device switching for attached servers PCs, printers, Twin-ax and external feeds. Frame Relay PVC s 46

47 Communications Firewalls VPN Alternative? Configure multiple IP Interfaces for switching on the same network segment. 47

48 Communications DNS change may be required if on a different subnet. Dual desktop emulator icons for the target system may be required if DNS/interface switching cannot be done. Rerouting of key phone numbers

49 Synchronization Its Running and all is Well!! 49 49

50 Data Synchronization Review mission critical objects & libraries to ensure what you need is what you get Good Morning Reports Verify object counts between Source & Target libraries to ensure required objects are not missing. 50

51 Data Synchronization Check file synchronization between Source & Target libraries for any missing data. Review IFS/DLO folders and directories to ensure critical application objects are not missing. 51

52 Testing Verify what is in place works!!! Testing Makes Perfect 52 52

53 It s not all bad news when your plan fails during a test frequent testing identifies gaps in your recovery process 53

54 Testing Active Testing Fail Over Test because your business depends on it! 54

55 Active Testing Validates the Recovery Plan in terms of: 1. Recovery Capability 2. Source system configuration 3. Network Recovery 4. Data Integrity 5. Identify Weakness in the Plan 6. Provides Training This all equals success during a Disaster 55

56 Active Testing Roll Swap It s a Validation Process Test until you get the first one at 100% Management expects a working solution Yesterday Roll Swap several times a year to ensure all is well 56

57 Spot Testing Automated process to verify the status of files & the state of the mirrored system. ( 7 x 24 ). Critical to ensure Data and Systems integrity. Definition of things to check must be defined followed by implementation. 57

58 Sample Test Objectives Application interface tested for mid-batch recovery. All communications interfaces tested Communication file retransmissions with vendors Mid-Day failure recovery for Online systems Validation of data recovery Draft of DR plan presented to Team for review 58

59 Bridging the Gaps 59

60 High Availability Must Layer to Business Needs Business Resiliency High Availability Disaster Recovery Transaction Integrity Risk Management Disk Protection Backup 60

61 Planned vs. Unplanned Downtime Planned Events: Account for 90% of downtime Routine backups, software upgrades, application maintenance, hardware and software updates Unplanned Events: Account for 10% of downtime Disaster Related events Hardware failure, Site loss, Power etc. Resiliency Approach Purchase & Implement Solution based on a 10% ROI or aim for 100% 61

62 Disaster Recovery How much money would your company lose if a major outage occurred? THE GOAL No Business tolerance for DOWNTIME Critical systems must be are continuously available Business interruption measured in hours & minutes rather than DAYS. 62

63 RTO, RPO and ROI RTO: Recovery Time Objectives How long can your system be down? RPO RTO RPO: Recovery Point Objectives How much data can you lose? ROI ROI: Return on Investment Goals Planned verses unplanned HA verses conventional hot site Minutes Hours Days 63

64 Planned Downtime Backup Window Daily & Full System Planned IBM & 3 rd party Software Upgrades IBM & 3 rd Party PTF/Fixes Application Maintenance ( Reorgs ) Hardware Upgrades Full Backups Wkly Daily Backups 6 times per week Software Installs + Housekeeping = Planned Outages Per/ Week 3 hrs 6 hrs Per/Year 156 hrs 312 hrs 20 hrs 24 hrs hrs or days/year 64

65 Workload Balancing Production System CPU Backup System CPU running the business spare capacity you can use The backup server has spare capacity The backup database is locked for update Run query and reports on backup server Run test environments on the backup server Run development environments on the backup server 65

66 Business Risks Defines # Layers Single System Single Location Medium Risk No Dependencies Single Layer: Local Redundant Systems 66

67 Business Risks Defines # Layers Double Layer: Local Redundant Systems Plus Offsite DR Two Locations Low Risk No Dependencies 67

68 Systems Availability Benefits Summary!! 1. Recovery from disaster related outage events 2. Backup window reduction 3. Availability for planned outage events 4. High Availability for unplanned outage events 5. Workload balancing 68

69 Bridging the Gaps H/A needs to monitored to ensure integrity Special considerations must be given to designing a messaging model that is responsive Operations support requirements must be defined and a response system implemented. Operations education will be required. 69

70 Bridging the Gaps Printing Special considerations need to be given to report distribution. Report confirmations must be generated to ensure NO duplicated delivery of Spool files to end users or dependent business process. ( FTP ) 70

71 Its become Mission Critical. It represents the way you communicate with customers, partners, employees and the world. is an H/A Application 71

72 Bridging the H/A Gaps Change Control is mandatory!! A process must be developed to ensure changes are performed. 72

73 A Poor Availability Solution Guarantees: Lost Data Business Disruption Conflict Lost Customers 73

74 Best Practices Audit 74 74

75 High Availability High Availability is a living process You must care and feed the solution It is not just a product. It s a discipline Assign qualified staff to manage and operate! Consider the entire application service when protecting Verify what is in place works!!! 75

76 For more information: 76