SESSION 408 Tuesday, November 3, 10:00am - 11:00am Track: Service Support and Operations

Transcription

1 SESSION 408 Tuesday, November 3, 10:00am - 11:00am Track: Service Support and Operations Preventative Problem Management: What ITIL Didn't Teach You Gabriel Soreanu Solutions Architect, Cisco Systems gsoreanu@cisco.com Session Description What exactly is preventative problem management, and why are organizations electing to use it? Many say it the most important process for reducing the number of IT outages. In this session, attendees will learn about practical techniques and a governance model for implementing strong preventative problem management. Attendees will explore best practices and discover how to implement, measure, and improve the process while avoiding the roadblocks often associated with preventative problem management. (Experience Level: Intermediate) Speaker Background Gabriel Soreanu is the lead consultant of the ITSM practice at Cisco. He has more than thirty years of experience in IT and communications, specializing in ITSM and process re-engineering. Gabriel has worked with global Fortune 500 customers in the service provider, financial, and public sectors. A frequent speaker at IT conferences, Gabriel has published many industry papers and is a professional engineer. He received his MBA from the City University of Seattle, and he holds several professional certifications, including ITIL Expert, COBIT, TOGAF, and PMP.

2 Preventive Problem Management Gabriel Soreanu Solutions Architect, Cisco ITSM Practice Preventive Problem Management What ITIL Didn t Teach You

3 Introduction Implementation and Governance Incident Analysis Other Preventive Techniques Other Considerations Introduction

4 Cisco ITSM Practice Providing ITSM/technology assessments and consulting assistance Sample Customers Over 300 customers in the past 15 years

5 About Myself Gabriel Soreanu Sr. Consultant, Cisco ITSM Practice Working on: IT Management Consulting Process re-engineering Areas of expertise and certifications: ITIL Expert, TOGAF, COBIT, PMP, P. Eng., MBA Paper published in the Oct. issue of SupportWorld magazine Implementation and Governance

6 Cisco Confidential 9 Goal: restore normal service operation as quickly as possible and minimize impact of incidents Tickets generated by events (manually or automatically) and user calls Resolved timely May not identify underlying cause Goal: identify underlying causes and proactively prevent recurrence of incidents Tickets manually generated by incidents and proactive activities Time is not of the essence Need separate teams and ticketing systems 2013 Cisco and/or its affiliates. All rights reserved. (Truly Reactive) Fixing underlying or unidentified issues Root cause analysis On all high priority and some lower priority incidents (Reactive Proactive) Follow-up activities to root cause analysis Tasks are assigned to various groups for resolution Requires workflow automation (Truly Proactive) Analysis of various data sources and corrective activities Performed by more than one group Coordinated through a virtual team

7 It eliminates defects in the infrastructure before they can trigger outages It is one of the few activities that can reduce the number of potential outages, not only their impact Problem Management reduces the number of recurring incidents, but Preventive Problem Management reduces the number of unique incidents Consequently: It improves infrastructure availability and service quality It reduces reactive work and support effort However, Problem Management and especially Preventive Problem Management is not widely implemented Reactive v. Proactive activities: Replacing a flat tire v. changing the engine oil Not very widely implemented It fixes potential outages not actual ones Benefits are difficult to quantify Usually adopted when service quality cannot be further improved by other support processes Measuring Problem Management success: CSF: reduce the number of incidents by 10% per year KPI: incident rate (number of incidents / number of devices in the infrastructure) KPI for Preventive Problem Management: unique incidents

8 Tier 0 Tier 1 Tier 2 Tier 3 Tier 4 Call Center Help Desk NOC Operations Engineering No monitoring No troubleshooting No monitoring Limited troubleshooting Full monitoring Most troubleshooting Problem Implementation Design Application Development All groups: Can be segmented by technology, geography, or customer Can be internally tiered Take escalations from the lower tiers Engineering and Application Development segmented by technology Separate from Incident Management groups For small organizations it can be part of Incident Management, but it must be segregated Can be centralized or distributed When distributed, it is important to create a virtual team Can include technology SMEs or only analysts If it doesn t include technology expertise it must draw on other technical resources It must communicate with other groups performing proactive work, e.g. Performance and Capacity Management, Security Uses automated workflow to assign corrective tasks and monitor completion

9 Performed by analysts or technical SMEs depending on activity Usually performed by various groups: Problem Management Operations Engineering Coordinated by Problem Management May employ a virtual Problem Management team Many activities can be automated, but will always require manual analysis Continuous activity, tasks repeated daily or weekly Incident Analysis

10 The most important Preventive Problem activity Should be the first one to implement Requires a solid incident ticketing system: Searchable database complex queries Adequate fields, including platform, device, software version, root cause category, etc. Complete information in the records (troubleshooting log) Does not require technical expertise analyst job Can be partially automated, but usually requires manual analysis May use Pareto analysis to focus on critical issues May require collaboration with equipment vendor Identify underlying causes that were not resolved by closing past incidents Look for repeated incidents in the last 30 days on the same: Device Platform Module Software version Review recurring incidents to see if they had the same symptom A recurring incident may be flagged in real time during incident resolution

11 Identify underlying causes that triggered multiple incidents Build Top N lists with highest associated incident counts: Devices Platforms Modules Software versions Analyze the lists for common causes (e.g. software bugs or hardware defects) Send potential issues to specialist groups for further analysis and resolution Discover underlying causes triggering seemingly unrelated incidents Group incidents by root cause category: Access Location Environment Administration Tools and instrumentation Change related issues User or staff procedural errors Contention issues Database errors Look for common causes on categories with high incidents counts

12 Other Preventive Techniques Performed by technical SMEs, usually outside of Problem Management Require specialized tools Consist of manual analysis, automated audits and scripts Generate problems to be fixed by specialist groups in Operations or Engineering Most techniques fall into one of the following categories: Compliance audits Analysis of non-actionable alerts May require collaboration with equipment vendors

13 Alerts (SNMP traps, Syslogs) can be: Actionable high priority, must be addressed in real time Non-actionable low priority, should be reviewed periodically Define a set of non-actionable Syslogs that are good indicators of potential failures Scan your Syslog database periodically for the defined set Start with a small set and gradually increase it based on: Equipment vendor recommendations Past failures and root cause analysis Another set of techniques is similar to Top N, performed on Syslog data Requires a Syslog parser Define standards (templates) for network device configurations (routers, switches, servers, Windows, Linux, Unix, etc.) Templates developed and owned by Engineering Periodically run compliance audits that compare deployed configurations to the standards and generate non-compliance reports Requires network configuration management tools, depending on platform There are tools that verify compliance to industry or proprietary best practices Develop action items from the non-compliance report and assign them to relevant groups Requires workflow automation tool The network configuration management tool may correct non-compliance item automatically For network devices, audit the device configuration repository, not the network devices

14 Both hardware and software have a limited lifespan, usually determined by End of Support time Define a strategy and roadmap for each hardware platform and software version: Triggers for upgrading the hardware or software (bugs, security issues, new features) Upgrade paths for all hardware and software platforms, i.e. replacement candidates Mitigation plans for EoL hardware / software May have different plans for the same hardware model or software version, depending on supported applications and features Review all vendor security and field notices and develop plans accordingly There are many other preventive activities that may not be part of Problem Management. It is important that these activities are monitored and reported. Here is a potential list: Regular maintenance activities Regulatory audits Performance monitoring Capacity plans Disaster recovery testing Strong Validation and Testing Strong Change and Release Management policies

15 Other Considerations If you are just starting implementing Preventive Problem Management: Start small, with a limited number of activities Create a virtual team for communication and building expertise Rely on your equipment vendors for leading practices Strive to adopt a new preventive technique every month or quarter As reactive workload decreases, move more resources into proactive work Ensure management support by communicating progress and success As with any ITIL process, metrics and reporting are critical Remember that Preventive Problem is like detective work: lots of fun!

16 Root cause analysis is very effective in reducing recurring incidents. Preventive Problem Management reduces non-recurring incidents as well, hence it is complementary in reducing the total number of incidents Preventive Problem Management does not require incidents for trigger Successful Problem Management will reduce reactive effort making support resources available for proactive work Successful Problem Management will improve infrastructure availability, service quality and customer satisfaction Thank you for attending this session. Please don t forget to complete an evaluation form!