MAJOR INCIDENT MANAGEMENT TRENDS 5 Survey Report
Introduction Reliance on digital infrastructures has dramatically increased the impact and frequency of major incidents. In fact, more than 9 of large businesses report major incidents occur at least several times a year and nearly 6 report major incidents occur at least monthly. A survey of more than 400 IT professionals reveals that IT and business leaders within individual companies are mostly aligned on what constitutes major incidents and how to resolve them. However, standard definitions and processes are lacking between companies and across industries. Without these standards, IT departments lack benchmarks and best practices to help drive improvements A more complete picture reveals itself when we combine our recent survey results with our customers experience and our own expertise. In this report we will attempt to put the results of the latest survey in context for better analysis. Key Findings Only 52 % of companies have a major incident team Only 44 % of companies staff major incident teams with dedicated personnel In the absence of a major incident team, various roles lead the major incident response 76 % of business stakeholders are tolerant that major incidents are an unavoidable fact of business 87 % say executives must be informed during major incidents 64 % have target resolution times 76 % frequently miss those target resolution times www.xmatters.com 2
Industry Standards Businesses do recognize the importance of rapid and effective response to major incidents. However, they differ in how they define a major incident and organize response teams. A number of factors determine whether an event rises to the level of a major incident. Most respondents still choose traditional indicators like length of an outage and customer impact. However, some of our customers recognize even degraded service or specific system outages as major incidents, and these views are reflected in the results as well. Numerous Factors Determine When an Incident Becomes a Major Incident For your company what factors determine whether an incident is classified as a Major Incident? Length of outage Customer impact 77% 81% Employee impact Specific services or apps affected Executive management becomes involved Number of services or apps affected 61% 55% 53% 5 SLA is exceeded Performance threshold surpassed Number of calls to the call center 32% 29% 38% 1 2 3 4 5 6 7 9 www.xmatters.com 3
Barely half of companies have a major incident team, and our survey respondents bear out that a major incident team is still not universally the norm. Aligning with what customers and other businesses tell us, larger companies are more likely to have major incident teams. Only 52% of Companies Have a Major Incident Team Does your company have a Major Incident Management team? No 48% Yes 52% Company Size vs. Major Incident Team Y/N All 52% 48% 500-1,000 41% 59% 1,000-5,000 52% 48% Yes No 5,000-10,000 67% 33% More than 10,000 67% 33% 1 2 3 4 5 6 7 9 www.xmatters.com 4
For companies without a major incident team, leadership falls to any number of roles, including IT operations and IT executives. Interestingly, all responses were in the IT department but only a quarter indicated the IT service desk. The size of major incident teams varied as well, but of teams in our survey have 10 employees or fewer. However, fewer than half the businesses in our survey have dedicated personnel on their major incident teams. In keeping with our customers, though, the result varies by company size and bigger companies are more likely to have employees dedicated to major incident management. In the Absence of Dedicated Team Various Roles Lead the Major Incident Response For those members of the Major Incident Management team, how many have this as a full time role? IT operations IT executive It varies by the incident type depends It who depends is available who is available at the time at the time 29% 35% 44% 56% IT service desk 24% 1 2 3 4 5 6 Only 44% of Companies Staff Major Incident Teams with Dedicated Personnel For those members of the Major Incident Management team, how many have this as a full time role? 6 5 4 3 2 1 56% 2% 9% None, they have other 1 2 3 4 More than 5 responsibilities beyond incident response 7% 19% 7% www.xmatters.com 5
Company Size vs. MIM Team Being a Full-Time Role 9 88% 7 6 56% All 5 4 3 2 1 48% 2 4 None, they have other responsibilities beyond major incident response 4 3 26% 2 2 19% 9% 9% 1 6% 7% 9% 1 1 6% 4% 4% 2% 1 2 3 4 More than 5 500-1,000 1,000-5,000 5,000-10,000 More than 10,000 Major Incident Response Many IT executives, managers, and employees are assessed based on their ability to limit the number, duration, and severity of incidents. Most companies in our survey have SLAs in place, but many report that they breach their SLAs regularly. Nearly two-thirds of respondents have SLAs, but the time frames target resolution times vary. However, more than of respondents say they exceed their SLAs. 64% Have Target Resolution Times Are there specified target times to resolve a Major Incident? No 36% Yes 64% www.xmatters.com 6
Wide Range of Resolution Target Times for Major Incidents 35% How fast are the target times to resolve a Major Incident? 3 2 17% 29% 29% 19% 1 7% 5% Less than 15 minutes 15 30 minutes 30 60 minutes 60 90 minutes Longer than 90 minutes 76% Frequently Miss Target Resolution Times How often are resolution targets missed? Most of the time 13% 63% 23% 1% Some of the time Rarely Never 1 2 3 4 5 6 7 9 www.xmatters.com 7
Most companies report several major incidents a year, but results vary by company size. For instance, 51% of companies with 5,000 to 10,000 employees and 3 of companies with more than 10,000 employees report a major incident at least monthly. Some of our larger clients report several major incidents per week, and the discrepancy could be a result of different definitions of a major incident. Company Size vs. # of Major Incidents 9 7 72% 6 62% 5 4 3 34% 42% 4 35% All 500-5,000 More than - 5,000 24% 2 1 18% 13% 14% 12% 13% 12% 13% 3% 1% 1% Daily Weekly Monthly Several times a year Yearly Fewer than once a year Never Automated notification platforms are fairly popular, but we were surprised that more than half use manual communication methods. Regardless of the system, 4 of respondents send mass notifications to everyone in IT instead of to specific resolution team members. Again, though, larger companies are more likely to use automated notifications. Nearly two-thirds of companies with at least 5,000 employees report using an automated notification platform. It s no wonder that companies are automating their notifications, especially since more than 82% of respondents say business application downtime significantly affects revenue. To be honest, this was kind of a no-brainer response because many of our customers consider revenue risk a major factor in declaring a major incident. www.xmatters.com 8
Less Than Half Use an Automated Communications Platform For Notification Other 2% We use an automated communications solution 49% How are IT team members notified in the case of a major incident? Team members are manually contacted 49% Almost Half of Companies Use Mass Notifications for Major Incidents Which best describes your company s notification process for a major incident? Specific individuals re contacted are contacted A mass notification is sent to everyone in IT 4 An issue specific notification is sent only to select teams 45% ecific s sent teams 82% Confirmed Downtime Equals Revenue Risk For your company, does business application downtime have a significant impact on revenue? No 18% No 18% Yes 82% Yes 82% www.xmatters.com 9
Companies Do Have Areas of Agreement Definitions and opinions vary widely across industries and business areas, but they seem pretty well aligned within individual companies. This is a promising trend because aligned lines of business can only help companies resolve major incidents faster. Over of respondents say IT and the business are aligned on what constitutes a major incident. There is even stronger agreement on the need for IT to keep the business informed during a major incident. In fact, executives accept that major incidents are a fact of doing business, but they have no tolerance for being left in the dark while incidents are happening. In relating to the question about what constitutes a major incident, nearly of respondents said service outage or security breach. But more than half of respondents also included intermittent service outages, and a large minority pointed to service degradation, a sign of lower tolerance for anything below optimal service. More than of business stakeholders accept that major incidents occur. In fact, they voice more frustration over lack of communication during a major incident than they do for the occurrence of a major incident. 82% Say IT and Business are Aligned No 18% No Yes 82% 18% In your opinion are the IT teams and the business aligned on what constitutes a Major Incident? Yes 82% www.xmatters.com 10
Outages and Security Breaches Are Major Incidents, as is Basic Service Interruptions For your company which of the following are considered Major Incidents? Service outage Security breach 95% 98% Intermittent service availability 52% Performance degradation 42% 2 4 6 12 87% Stated The Business Must Be Informed During Major Incidents Are business stakeholders and management OK with being kept completely in the dark until Major Incident is fully resolved? No, the business demands regular status updates 87% Yes, the business is willing to patiently wait 13% www.xmatters.com 11
Businesses are More Frustrated about Lack of Information Than the Actual Incident In your experience which of the following is more frustrating to the business stakeholders? Lack of timely communication during the Major Incident 56% The occurrence of a Major Incident 44% Conclusion IT departments have been concerned with major incident management for many years, but it s a new and maturing area of focus for business leaders. As companies find their own way and create their own definitions and processes, it s only natural that they won t always agree. By peeking behind the curtain, we learn that executives have come to grips with the occurrence of major incidents, but they insist on effective communication. Over the next few years, we expect more definitions to become standardized. Right now there is no one right way to organize teams or resolve major incidents. Hopefully these findings give you a look where you stand in relation to your peers. www.xmatters.com 12
68% Methodology and Participants Companies Represented Size More than 10,000 24% 5,000-10,000 14% 1,000-5,000 36% 500-1,000 26% Industry Financial Services 16% Manufacturing 16% Technology 16% Healthcare 13% Education 9% Services 7% Energy and Utilities 5% Transportation 4% Retail 3% Telecommunications 2% Non-profit 2% Hospitality 1% Other 7% 5% 1 2 Location Mexico, Central America, or South America Australia or New Zealand 3% 2% Europe 1 Asia 4% Middle East or Africa 1% United States or Canada www.xmatters.com 13
Participant Demographics Role with IT I manage a team of IT or technology professionals 36% I am a front-line IT or technology professional 64% About Dimensional Research Dimensional Research provides practical marketing research to help technology companies make their customers more successful. Our researchers are experts in the people, processes, and technology of corporate IT and understand how IT organizations operate. We partner with our clients to deliver actionable information that reduces risks, increases customer satisfaction, and grows the business. For more information visit www.dimensionalresearch.com. Contacts About Us CORPORATE HEADQUARTERS 12647 Alcosta Blvd., Suite 425 San Ramon, CA 94583 USA +1 877.XMATTRS (962.8877) EMEA HEADQUARTERS 20 Little Britain London, EC1A 7DH UK +44 (0) 203 427 6326 APJ HEADQUARTERS Level 29 Chifley Tower, 2 Chifley Square, Sydney, NSW 2000 AU +61 2 9238 8023 Copyright 2016 xmatters. All rights reserved. All other products and brand names are trademarks or registered trademarks of their respective holders. xmatters cloud-based solutions enable any business process or application to trigger two-way communications (text, voice, email, etc.) throughout the extended enterprise during time-sensitive events. With over a decade of experience in rapid communication, xmatters serves more than 1,000 leading global firms to ensure business operations run smoothly and effectively during incidents such as IT failures, product recalls, natural disasters, dynamic staffing, service outages, medical emergencies and supply-chain disruption. xmatters is headquartered in San Ramon, CA with additional offices in London and Sydney.