Major incidents start to impact businesses within just minutes, often

Transcription

1

2 Major incidents start to impact businesses within just minutes, often before incident resolution teams can even be engaged. With this immediate impact to business, meeting your major incident SLAs is no longer enough. You are under a microscope from customers with expectations that all systems are always on, or you ll inform them immediately and communicate to them along the way. What s more, given the sensitive nature of incidents and the variety of audiences requiring communication, knowing how to adequately manage communication plans can be tricky. Handling them poorly can be even worse than not handling them at all. When it comes to communicating during major incidents, business customers are like Goldilocks: They want information that s just right. They don t want too little, they don t want too much, they want it relevant to them, and they want it right on time. This paper outlines best practices to empower and engage your key stakeholders during major incidents. Align Your Communications with Your Process During the Planning Stage Having processes and systems in place to resolve major incidents means you re already ahead of many large enterprises. Monitoring, contact, triage, resolution, post mortem all constitute areas where pre-planning can bring value. Major incidents happen. The processes and systems in place will determine how successfully you resolve them. The right technology, proper planning and configuration can produce THE REAL COST OF POOR COMMUNICATION Sending notifications too late, to the wrong stakeholders, or with the wrong message can damage your organization (and your job security). Extend your communication beyond resolvers: Customers will contact you by the thousands. Customer satisfaction drops by 15 percent every time a customer has to call back about the same issue. 1 Executives kept in the dark can take your time with inquiries, damage your internal reputation, and cost you your job. Third-party service providers who lack information cannot help with resolution. A service outage starts to affect most businesses within just 15 minutes

3 a turnkey, repeatable process that does not require manual communications and produces faster resolution times. Each of these steps produces a unique set of data. It would be easy to use that data to send one communication to incident resolvers and all stakeholders. Instead, parse the data to produce multiple messages for unique audiences. For instance, in the Triage step, messages might look like this: To incident resolvers: Servers SRDC-RHL09 through RHL16 in the third bank in Server Room 1 at Headquarters are failing intermittently. Please test the servers and all the connections to determine the exact cause so we can take corrective action ASAP. To the IT department head: Servers SRDC-RHL09 through RHL16 in the third bank in Server Room 1 at Headquarters are failing intermittently. We are beginning tests. We will let you know when we have determined the exact cause and can give you an estimated completion time. To executives: We re having an issue with some servers at Headquarters. We are testing to determine the cause, and we will let you know the estimated completion time as soon as possible. To customers: We are experience some service issues. We are working on a resolution diligently, and will restore service as soon as possible. Automate Messages with Templates You need at least four distinct messages. There could also be messages to partners, utilities, PR, corporate communications, and more. Creating all these messages on the fly in the midst of trying to resolve a major incident would cause delay and increase the risk of error. Instead, create a series of templates to automate the tailoring of messages at each stage of the process. Choose a template, update recipients if necessary, update messaging if necessary, add relevant data, and send. Even better, tight integration between monitoring, ticketing, and other systems with communication systems could allow for messages to auto-populate based on the type of incident. Right-Sizing Communications: How to Get it Just Right As an IT professional, you likely didn t sign up for one of the trickiest communication jobs in the business, but getting the communications right is now officially part of your job description. A good rule of thumb on tackling this new task: communicate as broadly as the impact of your incident, as detailed as required (but no more) for any given audience, as far ahead of impact as possible. BREADTH Major incidents are no longer isolated to back-end batch processing nor even live systems that impact a small set of constituents. With so many applications and services interconnected, a single incident could have a domino effect impacting multiple stakeholders. Consider the following audiences when building communication plans: Direct Stakeholders Communications should be sent to anyone whose business processes may be impacted by the incident. Depending on the type of incident, this could be owners of a process dependent on an impacted service, an internal customer reliant on the interrupted service or even end-customers (business or consumer) who will feel the impact in a significant manner. For example, an incident that might simply slow the services of an end consumer might not require communications to consumers, but a business customer reliant on a service that experiences an outage will be more significantly impacted and should be considered in your communications plan. 3

4 Help Desks and Call Centers When services or applications are slow or disrupted, your typical 2015 worker seems to have trigger finger when it comes to calling your help desk. Even a 20 minute outage can result in a flood of hundreds of tickets and phone calls in the help desk of a typical large enterprise. Alerting the staff manning the lines not only prevents them from opening duplicate tickets, but it s also an opportunity to arm them with outbound communication to alleviate the concerns of inbound callers. Management While extra inbound calls have hard dollar costs associated with them, an out-of-the-loop executive has an immeasurable reputation cost. Your highest value customers most likely have direct access to company executives and when they feel a disruption, chances are they re not calling into a call center. Executives should be kept in the loop in real time and never caught off-guard by in inbound client. Customer-Facing Staff In addition to call centers and executives, any customer-facing staff should be kept informed of incidents that might impact their customers. This might include field staff, account executives, sales staff, etc. Downstream Dominoes In today s inter-connected enterprise, a single incident may trigger a multitude of additional incidents. Understanding interdependencies could help prevent or mitigate damage downstream. For example, a power outage might not have an immediate impact outside of restoration teams if a back-up generator kicks in. But if a generator gets below a certain threshold and restoration is not imminent, giving application owners ample time to shut down apps properly could save hours in restart time once the power is back up and running. TIMING The modern business customer - be they internal or outside - is forgiving yet impatient. Intermingled with their reliance on digital systems is an acknowledgement that they will break, but the forgiveness comes packaged with expectations that communications will be transparent and timely. As 451 Research Analyst Donnie Berkholz stated in a blog post We all understand that sometimes things break, because clouds are incredibly complex systems. We re only really looking for two things out of it: (1) don t have the same problem twice, and (2) keep us informed. Optimal timing would allow you to reach stakeholders and clients before they feel the impact of an incident. Let the call center know before an influx of calls, let the client know before their own systems are impacted, let your executives know before someone else tells them. CONTENT As mentioned earlier in the paper, altering the content of your communication to specific audiences is key. Resolution teams need to have specific details as well as calls to action to engage and inform them rapidly. External clients don t need or want details. They simply want to know that the problem is being addressed and, when possible, what to expect next. Don t confuse transparency with overcommunication. Too much information can result in lost confidence. The following guidelines can help with crafting incident messaging for a variety of audiences: 1: Is the information of value to the recipient? 2: Is the information actionable by the recipient? 3: Does the information help to reduce dissatisfaction or confusion? For example, updates at timed intervals can be beneficial in assuring constituents that teams are working on the issue, even if there is no known additional information. 4: Does the information use vernacular familiar to the audience? For example, don t bog down executives with fancy tech acronyms and internal process abbreviations. Use plain language that they can relay as necessary. 4

5 Tactics for Communicating During an Incident When a major incident occurs, time is rarely your friend. The following tactics help you to use communication techniques to actually save time versus wasting precious minutes. Utilize Message Templates During the heat of an incident, the last thing your teams need to be doing is crafting prose. Message templates can save time by providing messaging and response options. Templates can be rapidly altered during incidents for specifics. Integrate When Possible Even better, when communications templates are integrated with monitoring systems, details can be auto-populated into message templates. Automate When Possible Tight integration between your communication and ticketing & monitoring systems allows for messages to automatically be sent without taking your resolution and ops teams off the tasks of restoring services. Enterprise-grade systems will allow for multiple messages to be triggered for different audiences with different messages. Synchronicity Synchronize contact information via data sync from systems of record. Augment basic contact information with contextual information like on-call schedules, groups and skill sets. During incidents, you are rarely looking for a specific person, but for a role or skill set. For example, you don t want Joan, you want the call center manager on duty. Give Stakeholders Control We ve already covered the requirement of communicating to stakeholders beyond resolution teams. We ve also let you know they can be fickle with specific needs. Allowing stakeholders to subscribe to communications lets them control frequency of alerts, mode of communication and subject matter on which they want to be informed. Use Communication to Trigger Collaboration Resolution teams still need to take action quickly. In today s distributed, heterogeneous environments that often requires collaboration. Something as simple as a conference call for a team representing a dozen skill sets can take well over an hour to assemble. Automating notification to conference bridge recipients (with escalation processes for alternates) can assemble the same team in 3-5 minutes, drastically reducing resolution times. Partner with Your Communication Team Once communications leave the walls of your enterprise, it s fair game. While transparency and honesty are important aspects of communicating to stakeholders, you may want a professional to craft the exact wording. Proactively set up meetings with your client or corporate communications team to create the templates and messages to keep in your arsenal for when a major incident hits. The processes and systems in place will determine how successfully you resolve them. The right technology, proper planning and configuration can produce a turnkey, repeatable process that does not require manual communications. 5

6 WHITEPAPER Communication Goes Both Ways Individuals impacted by a major incident can be some of your best field agents. Use proactive communications as a chance to gather data points if you think it might help with troubleshooting or post mortem. Your customers want to know that you ll learn from the experience and feel confident that the same issue won t happen again. Including their input as part of that process gives them a sense of partnership in improving the services you provide. From resolution teams and IT management to executives, from service desks and call centers to partners and clients, a major incident sends ripples of impact throughout your extended organization. Learning how to trigger and tier proactive communication during times of disruption allows for resolution teams to focus on resolution, mitigates impact to client-facing teams, and can drastically increase reputation and stakeholder satisfaction. After an Incident: Post Mortem During an incident, your company communicates as events change, which is crucial. But it s only after an incident is over without the distraction of firefighting - that one can gain true perspective and start to sort out all of the facts. With this 20/20 hindsight, it s beneficial to give stakeholders the confidence that you have identified root cause, that any processes are now in place to mitigate similar incidents and reduce risk First-call resolution: how important is it, really?, atlassian.com/help-desk/first-call-resolution, Cloud Outages, Transparency, and Trust, dberkholz/2015/01/12/cloud-outages-transparency-and-trust, Business Impact of IT Incident Communications, com/rs/alarmpoint/images/xmatters-2015-survey-report.pdf, 2015 CONTACTS ABOUT US MORE ONLINE RESOURCES INTERNATIONAL and press 2 US/CAN TOLL FREE XMATTRS ( ) EMEA +44 (0) AUSTRALIA/APJ SUPPORT opt 2 xmatters cloud-based solutions enable any business process or application to trigger two-way communications (text, voice, , etc.) throughout the extended enterprise during time-sensitive events. With over a decade of experience in rapid communication, xmatters serves more than 1,000 leading global firms to ensure business operations run smoothly and effectively during incidents such as IT failures, product recalls, natural disasters, dynamic staffing, service outages, medical emergencies and supply-chain disruption. xmatters is headquartered in San Ramon, CA with additional offices in London and Sydney. Copyright 2015 xmatters. All rights reserved. All other products and brand names are trademarks or registered trademarks of their respective holders.