Documenting a Communications Recovery Plan Leo A. Wrobel

Transcription

1 Documenting a Communications Recovery Plan Leo A. Wrobel Payoff The growing complexity of communications systems means that almost every organization today has a local area network (LAN) administrator whose possible absence during a disaster mandates the development of a well-documented communications recovery plan that can be executed by people unfamiliar with an organization's systems. This article reviews the basic components of such a plan, particularly the need for accurate and up-to-date hardware and software inventory forms. Problems Addressed Technical services personnel often respond tersely when asked by management to produce a disaster recovery plan for a communications system. They view such a request as a noconfidence vote and the plan as a test of their ability to perform a recovery in the event of a disaster. Competency, however, is almost never the issue. In most large organizations, the technical services and communications staffs are capable of recovering from a disaster under virtually any circumstances. Often, the very people controlling the recovery process are those who actually designed and built the communications system in the first place. They know where every wire in the organization runs and have memorized the telephone number of every major equipment vendor and service supplier. In the event of a disaster, these employees would undoubtedly pull together in an almost superhuman effort. Problems with disaster recovery occur when an organization's key personnel, such as the local area network (LAN) administrator, are unavailable for any reason during a disaster. Because of the growing complexity of LAN technology, one of the most effective tests of a disaster recovery plan is to assemble the disaster response team and remove the LAN administrator from the exercise. Loss of a LAN administrator who knows everything about a system can be devastating. The need for a communications recovery plan should therefore be presented to technical services staff from the perspective that someone unfamiliar with the system may have to execute the plan. The goal of every organization should be to document its communications recovery plan in a systematic format that can be followed by any reasonably trained technical services personnel, whether they be from a vendor, a rental company, a major supplier, or a carrier company. Developing an Inventory of Communications Hardware Disaster recovery personnel need to know several details about the organization's hardware, including: Type and model number. Software packages residing on the hardware. Software revision numbers.

2 Date of purchase and cost. Criticality to operations. Power requirements. Other helpful items include: The name, address, and telephone number of the manufacturer. The local distributor or depot for the equipment. The location of secondary-market hardware suppliers, who can be instrumental in providing equipment during a disaster. Exhibit 1 provides an example of a hardware inventory form that can aid recovery personnel in making quick command decisions. For example, if a four-year-old piece of equipment is depreciated over a five-year period, recovery personnel may decide to replace rather than to repair it. Such information is extremely useful when many decisions must be made rapidly and under tense circumstances. Hardware Inventory Form for Emergency Replacement and Restoration Component: Purpose: Manufacturer: - Serial Number: - Associated Software: Criticality Rating (1,2,3,4): -- Date of Purchase: Vendor Name and Telephone Number: -- Remarks: -- A separate inventory should also be kept of all hardware maintained by an organization at a disaster recovery center. The inventory should contain some fairly minute detail, such

3 as whether the hardware is on a movable rack or requires dollies, and the type of power plug the equipment requires. Maintaining Accurate Inventories The best method for keeping track of hardware necessary for restoring communications is through a process called data importing. Automated Data Importing Importing is essentially a means of finding data bases and repositories of information within the organization that are reasonably up to date. For example, when a piece of hardware is purchased, a document or file for the equipment is archived somewhere within the organization. Sometimes, the contract and the documentation for the equipment goes to the accounting department for amortization purposes. In other cases, the files become part of the personal file of the communications manager or analyst. The optimal situation is for them to be stored and accessible on a LAN. In any case, recovery planners should locate and identify these repositories of inventory data so they can be automatically imported into the recovery plan. There are good reasons for taking this approach. The price of some of today's hardware and the necessity for any mission-critical equipment to be protected by a disaster recovery plan means that the savvy technical services manager often includes disaster recovery in the selection criteria for major hardware purchases. Such equipment includes automated call distribution units; Private Branch exchange; major bridges, routers, and gateways; LAN networks; and mainframe computers. It is much more cost-effective to negotiate roll-in replacement guarantees when a vendor is vying for business than to try to add these services later. Yet, given today's levels of staff turnover, failure to import information on such guarantees and on maintenance contracts into the recovery plan could result in a future LAN manager needlessly paying for a disaster recovery plan for equipment already protected. Organizations that make heavy use of internetworked LANs have an advantage because they can automatically transfer files between interconnected departments in several ways without human intervention. For example, when object linking Microsoft Word files, a technical services manager can key in on a specific file name in the accounting department to ensure that updates to a hardware repository file are transferred to the appropriate file in the recovery plan. Manual Updating Importing can also be accomplished through a sneaker net assigning a key person or division to regularly go to the department containing files on equipment, make a floppy disk copy of the appropriate file, and update the recovery plan. But under the pressures of work, busy technologists can easily overlook this task, causing the recovery plan to be dangerously outdated. Updating this way therefore requires that staff realize the importance of the recovery planning process and that the process be enforced, to the point of withholding raises when the task is not performed.

4 Additional Components of an Up-to-Date Plan Importing Information on Personnel and Vendors Up-to-date telephone numbers for personnel and critical equipment vendors are essential to the successful implementation of a disaster recovery plan. Once again, this means importing data from reliable sources regularly and, preferably, without human intervention. Consider employees' home telephone numbers, which are found in several places, such as human resources and the company telephone directory. Care must be taken to ensure that the data contained in these sources is up to date. Here again, importing is best done by object linking files together, but it can also be accomplished through use of a sneaker net and floppy disk. Telephone numbers for key hardware vendors and suppliers can often be found in the network control center, help desk, or other operational environment whose personnel have day-to-day contact with vendors and are often the first to know about changed telephone numbers. Because operations staff are usually regularly involved in escalation procedures, their departments generally document information on second- and third-level management within the vendor community. Developing an Inventory of Software Communications recovery planners must also develop an inventory of all software required for operation of mission-critical communications equipment. This inventory should include: Acquisition date. Original cost. License number. Version number. Exhibit 2 provides an example of a software inventory form. Software Inventory Form for Emergency Replacement or Restoration

5 Software Package: Purpose: Supplier Name and Telephone Number: - License Number: Version Number: - Criticality Rating (1,2,3,4): - Location Where Software Can Be Purchased or Replaced: Remarks: Equipment Room Diagrams Equipment room diagrams should show all installed communications hardware and delineate any special environmental specifications such as air flow, temperature, and power needs. The diagrams should also outline equipment footprints, clearances, and any other information useful to a network installer. They form part of the appendices that should accompany a thorough recovery plan. See Exhibit 3 for a list of the information that should be contained in these appendices. Useful Appendices to a Disaster Recovery Plan - Emergency Phone Lists of Management and Recovery Teams - Vendor Callout and Escalation Lists - Inventory and Report forms - Carrier Callout and Escalation Lists - Maintenance Forms - Hardware Lists and Serial Numbers - Software Lists and License Numbers _ Team Member Responsibilities - Network Schematic Diagrams - Equipment Room Diagrams - Contract and Maintenance Agreements - Special Operating Instructions for Sensitive Equipment - Cellular Telephone Inventory and Agreements

6 Importing Components of the Corporatewide Plan Technical recovery planners should consider importing components of the corporatewide recovery plan dealing with global policy issues. It makes little sense for a technical recovery planner to write procedures for such companywide concerns as loss of a building, physical security, fire and bomb-threat procedures, purchasing, and media affairs. In these cases, the technical recovery planner should direct the reader of the communications recovery plan to the relevant section of the corporate plan. In most cases, the LAN or network recovery plan itself will probably end up being imported into a corporatewide recovery plan for execution by an emergency management team. The process is reciprocal. Assigning Technical Teams It is advisable to split a technical recovery plan into numerous subsections managed by technical teams. Such sections include LAN management, voice communications, data communications, and the emergency network control center. Each of these functions has assigned day-to-day responsibilities, and the team assigned to each will have specific responsibilities during a disaster. A network control center or help desk, for example, could take on a very different function in a disaster by helping to maintain command and control. The Real Reasons for Disaster Planning All companies use automated systems to conduct business and all suffer when these systems fail. Whether they are LANs or other communications systems, automated systems should not increase the risk to a company merely because they are convenient. A communications recovery plan should therefore accurately focus on restoring a business' core operations, or the items most crucial to its profitability, in the event of a disaster. Exhibit 4 presents the seven R's of a successful disaster recovery plan. The Seven R's of a Successful Disaster Recovery Plan

7 1. Recognition of Need for Planning -Protect Human Life. -Recover Critical Operations. -Protect Competitive Position. -Preserve Customer Confidence and Good Will. -Protect against Litigation. 2. Response -Reacting to Initial Report of a Disaster. -Notifying the Police, Fire, and Medical Personnel. -Notifying Management. -Establishing the Executive Management Team (EMT). -Filing Initial Damage Assessment Reports to the EMT. 3. Recovery -Assisting the EMT in Preparation of Statements. -Opening a Critical Events Log for Auditing Purposes. -Using Modified Signing Authority for Equipment Purchases. -Obtaining Necessary Cash. -Maintaining Physical Security. -Arranging Security at the Damaged Site, the Recovery Center, and Emergency Funds Disbursement Centers. 4. Restoration -Coordinating Restoration of the Original Site. -Restoring Hardware Systems. -Restoring Software Systems. -Restoring Power, the Uninterruptible Power Supply, and Common Building Systems. -Replacing Fire Suppression Systems. -Securing the Building. -Rewiring the Building. -Restoring the LAN. -Restoring the WAN. 5. Return to Normal Operations -Testing New Hardware. -Training Operations Personnel. -Training Employees. -Scheduling Migration Back to the Original Site. -Coordinating Return to the Original Site. 6. Rest and Relaxation -Scheduling Compensatory Timeoff. 7. Reevaluate and Reassess -Reviewing the Critical Events Log. -Evaluating Vendor Performance. -Recognizing Extraordinary Achievements. -Preparing Final Review and Activity Report. -Aiding in Liability Assessments. Recommended Course of Action A disaster recovery plan is a complex road map of how to rebuild an organization after a disaster. It should be written for execution by a reasonably well-educated technical person in the event key personnel are unavailable. A thorough recovery plan includes input from all major vendors, suppliers, and departments and must import data from accurate sources. It should delineate recovery tasks

8 systematically and clearly and strike a balance between cumbersome detail that discourages reading and a cursory explanation understandable only to people familiar with a system. It can take two years or more to complete a successful recovery plan, but the effort is well worth the protection the plan affords an organization. Author Biographies Leo A. Wrobel Leo A. Wrobel is president of Premier Network Services Inc., in Dallas.