Preparing for
Preparing for We will use the term disaster very loosely during this session. Immediate, short-term displacement Building evacuation. On-Site event. Immediate, long-term displacement Building fire, Train derailment Upcoming, long-term displacement Flood, Zombie Infestation, Building Construction Each category has different expectations, different preparations, different responses.
Preparing for Planning rarely is a plan that is only put in place by executive or leadership. All levels of employees will be involved in a disaster, so they should know what to expect. Think about the forest, not the trees Don t put extremely specific procedures or items in your plans. Many of these items often change and won t be updated before your disaster happens.
Preparing for Preparedness Business Continuity Avoidance Recovery
Define the stage Preparedness Business Continuity Avoidance Recovery Preparedness is where you help define all the things you need to know to avoid or recover from a disaster. Who knows what Who has access to what Who can make decisions Who can execute decisions or procedures How do you prioritize How do you go back to business as usual?
Define the stage Preparedness Business Continuity Avoidance Recovery Business Continuity is where you keep your business open while a disaster is or recently has happened Communication to customers Communication to employees Communication to other constituents Internal processes and activities
Define the stage Preparedness Business Continuity Avoidance Recovery Avoidance is the plan to avoid a disaster to begin with. Not always possible, but there are ways to help mitigate the effects of a disaster.
Define the stage Preparedness Business Continuity Avoidance Recovery Recovery is the process of picking up the pieces and re-building after a disaster has happened. THIS is where you would think of server logins, passwords, connections, and $$ to recover.
Preparedness The initial step of any DR/BC Plan is to build a catalog of who knows what, and who can do what in your department. May seem obvious, but may not always be. You cannot base this on assumptions. Bob does all of our Microsoft stuff, so he must know how to re-configure our Exchange server! Involve the employees on all levels. May help to schedule a short meeting with them to dig into the process.
Preparedness Things to think about : How servers are setup. How the networking works. How the security of the network/servers/desktops is setup. How to access specific applications. How to access critical documents and reports (remotely?) How to access policies and procedures (like your DRP!) How to access a list of vendors or other employees that have knowledge of your systems.
Preparedness Access. Who has it, and who needs it? Limiting access to servers and services to a single person may make sense for PCI-DSS, but it can be a big failure in a DRP. In a SECURED document, you should document who has access to each server and service. Think of everything that requires a login. If only one person has access, it may be good to assign an auditable backup person incase that person becomes unavailable. Do these people have the resources to access the servers or services remotely, or must they always be on site? What happens if on-site is not available?
Preparedness Who can make decisions? Do you delegate decision (and/or) purchasing decisions to the employees doing the work? Do you require a two-key approach for everything? If so, can you could on two-keys being available? Documenting and making it clear to everybody who has decision making abilities during a disaster is key. Belongs in the DRP. It needs to be made clear that this person is made available to make these decisions. They would be an essential employee.
Preparedness Executing decisions and procedures is often left to the employees who are most familiar with the servers or services they usually maintain. This could also fall to a vendor. Make sure the vendor is aware that they are listed in your DRP. Make sure you don t assign too many services/servers to a single person (or group of people). Some employees are more than happy to jump to the task, but may perform poorly if they have too much on their plate. Document who is responsible for the work.
Priorities Knowing how to prioritize is key in a disaster. Having a clear priority list documented without being under duress will make transitions smoother. During a disaster you will often run into situations where one group is screaming louder than others for service to be restored. Don t fall into this trap of only serving those squeaky wheels. Have a technical review of priorities. Sure, email is most important, but it may rely on the network to be up
Transition to Normal You will need to document how to transition back to business as usual from your disaster mode. Don t wait for everything to be back to normal having a hit list of things that need to be taken care of is ok. ALWAYS have a debrief and a lessons learned. Larger disasters should include not only internal staff, but external constituents as well. Ask the question what can we do better. Modify your DRP with the responses.
Business Continuity
Business Continuity After the initial assessment of the disaster, questions should be asked: What and how do I communicate with my employees? What and how do I communicate with my customers? What and how do I communicate with my constituents? How do we continue to do the work that is expected of us? How are these groups USED to communicating with you?
Business Continuity Employee Communication Every supervisor should have a way to communicate with their employees. Every employee should have a way to communicate with their supervisor. Email may not be enough (is it up?!). Know home and cell phones. Ask during the yearly review process to update this information. Be honest and precise to what is going on. No PR needed in internal communications. Early and often. Employees should be the first to know of a disaster. It s ok to tell them to hold the info from others.
Business Continuity How do customers normally communicate with you? Email? Make sure your email is hosted in a reliable place and you have access to it. Backup mail server hosted in a secondary location? Does it work for outgoing email? Phone calls? Do you know how to answer calls from a remote location? Call forwarding, EC-500, remote agent logins, etc. are all things to think about. In Person? Do you have a way to communicate that appointments can t be meet, class canceled, or anything else? Phone / Email / Texting is key here.
Business Continuity When dealing with customers, don t assume that they always communicate with you (or your employees) the same way. Actually survey your customers and find out how they want to know about disrupted services. Every group deals with customers in a different way. This can t be a universal answer. Ask for help for pre-canned messages from CABS that you can modify on the fly.
Business Continuity Internal processes and activities Do all of your employees have access to the data they need remotely? VPN, ALG, Phones, Remote Desktop? Do you provide laptops, or is the assumption that employees use their own? Do all essential employees have internet access? Access to services remotely? Have you practiced accessing services remotely? Good idea to designate a day where employees work from home (telework). Do this on a regular basis will work out the kinks.
Business Continuity Do your employees know what they would work on while at home? Add this to your communications. CAN they accomplish this from home? What are goals and expectations when you can t see them work? How do you classify a day as productive? Have employee write a short report of what they did that day. Maybe their work speaks for itself Real-time communication is key. Phone, IM, (IRC), Email should be available to employee.
Business Continuity Telecom Business Continuity Offerings: EC-500 (Link cell phone with desk phone) One-X Communicator / Avaya Communicator (Answer personal phone remotely) One-X Agent (Answer agent calls remotely) $$ Remote Call Forwarding (Answer incoming calls to a single phone line) VDN Variables (setup a variable to do different call flows in a disaster) See these options in the booths during lunch.
Avoidance
Avoidance Won t go into this in much detail Plenty of resources and common sense Multi-data centers Don t have your data-centers below grade. Double up on power-supplies, going to multiple UPSs, going to multiple breakers RAID your drives. Use a MAS that can locate in multiple DCs Use Hyper-visors like VMWare, Zen or Hyper-V to make hardware issues less important.
Recovery So, you ve had a disaster. BC didn t work, or you need to rebuild Take a look at section 1 that should have started the template to recover. BUT you will need additional details. Take a look at NIST-800-34. Requires that you know system well and all the moving parts. Need to get into details like version numbers, port numbers, user assignments, etc. System Diagrams! Don t store these on the file-system you plan on restoring
Recovery Plan of attack. Again, set your priorities ahead of time, and execute For the managers, depend on your employees to give updates to you. Don t ask every 10 minutes. They will tell you when services are back. As your plan starts, prepare your reports for funding. Purchasing has an avenue to get emergency funding and POs released quickly to replace hardware. Keep track of hours related to recovery. Know the emergency contacts for your venders.
Recovery People to know : Your purchasing agent. Key people at IT Services (for networking related recovery). If you have an SLA, know how to execute it. Get to know Scott Bryan. He s the DRP guy at IT Services C-Store contacts for servers / desktops / licensing Contacts within CABS for communications
Scared yet?
Summary Plan, Plan, Plan Test, try, and document. Get feedback. Plan for the next incident.
Thanks! Nick Kwiatkowski nk@msu.edu Call/Text: 517-432-2528