Assuring Service Quality Despite Limited Resources

Transcription

1 Assuring Service Quality Despite Limited Resources North Carolina Digital Government Summit Sept. 2, 2009 David Hayward Sr. Principal Service Assurance Management

2 Every User Matters. Every Transaction Counts. A quality user experience is your government s lifeline. IT is critical to delivering a quality user experience to fulfill your organization s mission and mandates. 2 Copyright 2009 CA

3 The Government IT Service Challenge Constituents Why can t I complete my transaction? Stakeholders Why do constituents see the problem before IT? IT Operations App Why is this taking so long? Agency & Department Executives Service Owners What s happening in my domain? How is it affecting services? Users are What complaining should I prioritize on? Where is the problem? What caused it? Not my issue System DB Are we meeting expectations or causing dissatisfaction? Network It s difficult to get a handle on each of these problems and even harder to understand how these problems affect each other.

4 Understanding Service Assurance Challenges

5 IT Service Assurance Paradox Managing Technologies vs. Services 05:00 06:00 07:00 08:00 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 % available Web Server App Server Mainframe Database Network Application And who is measuring the real user experience and managing the whole service 99% 99% 99.9% 99.9% % 99.9% End-User Service Unavailable or Slow Available, Performing as Expected? Managing Infrastructure Top-Down in Context of Services is Required 5 Copyright 2009 CA

6 Technical Challenge: Managing Services Service #1 Service #3 Service #5 Service #2 Service #4 Service #6 Client Systems Applications Databases Servers Storage Networks Intelligent Service Modeling and Analytics is Required 6 Copyright 2009 CA

7 Managerial Challenge: People & Processes Service Desk Service #1 Service #3 Service #5 Service #2 Service #4 Service #6 Network Engineering Client Systems Application Engineering Applications Databases Servers Storage Networks System Engineering DB Administration Prioritized & Shared Understanding of Services, Their Condition & Root Cause is Required 7

8 What IT Professionals Say About Service Assurance Challenges

9 IT is Reactive When application-related problems occur, what are the primary ways in which IT most often finds out about the problem? When application-related problems occur, what are the primary ways in which IT most often finds out about the problem? Calls from Users NOC or Monitoring Center Transaction Management products n= 186 Feb % 10% 20% 30% 40% 50% 60% 54% of problems first detected by end users Enterprise Management Associates : 2008 Survey Slide 9

10 Why Is This Number Important? 89% Enterprise Management Associates : 2008 Survey

11 Slow Root Cause Determination/MTTR 89% IT organizations use triage teams to solve service performance problems Adapted from Enterprise Management Associates :2008 Survey

12 Impact on Productivity For these cross-domain teams, what types of personnel are most often included? Sample Size = 180, Valid Cases = 148; Responses not shown received 0% Slide 12 Enterprise Management Associates : 2008 Survey

13 IT App/Service Management Roadblocks In your opinion, which reasons hamper your organization's effectiveness in terms of managing applications and services? 1 Changes to applications and infrastructure not well documented or controlled 2 Poor coordination between support teams 3 Troubleshooting/root cause analysis takes too long Sample Size = 232, Valid Cases = 232 Enterprise Management Associates : 2008 Survey Slide 13

14 Wish List: Top Three Most Wanted Tools What are the top three tools and/or functionalities that your organization currently lacks that would most benefit your current support requirements? 1 Consolidated event correlation 2 Change tracking, verification, audit 3 Transaction management, problem identification, root cause analysis Slide 14 Enterprise Management Associates : 2008 Survey

15 Top Three Pressures Facing IT Operations Management Gartner: 2008 Survey

16 Trends Affecting IT Operations Management Global Trends Increasing IT service delivery orientation Need to improve (and prove) IT operations efficiencies Accelerating adoption of ITIL and process frameworks Improve or establish IT operations accountability Focus on automation Green IT (power & cooling) Data Center Consolidation (cost reductions) Technology Trends Priority shift from management of infrastructure to management of services and processes Increasing technical complexity, which involves new management buyers IT virtualization Increasing commoditization of platform agentry Movement back towards vendor frameworks Reducing TCO of management via deployment of agentless solutions New organizational roles and collaborative responsibilities CMDB and workflow (RBA) as new age framework Adapted from Gartner: 2009 Report

17 Summary: Barriers to Overcome > Too many tools > Silo tools not well integrated > Technology silos prevent end-to-end management > Determining root cause is difficult > Reactive: little or no predictive visibility > Performance becoming most significant service issue > Little time to optimize > Flat or down budgets CA: 2008 Focus Groups 17 Copyright 2009 CA

18 People, Processes and Technology for IT in Lean Economic Times

19 Application Performance Mgmt Service Management Project & Portfolio Mgmt Infrastructure Management Security Management Mainframe Self-Service 2.0 IT Approach for Lean Economic Times External Constituents Citizens Businesses Suppliers Internal Constituents Executives/Officers Staff Compliance IT Mngt & Operations CIO PMO CISO Support Engineering/Development Infrastructure IT Hardware & Software Assets VALUE Focus on what s most important to meet your mission & mandates delivering value to constituents Aim resources at high value deliverables Improve user experience Reduce waste Increase productivity COST 19 Copyright 2009 CA

20 Technology Building Blocks for Transforming IT in Lean Times

21 3 Building Blocks Service Assurance Application Management Infrastructure Management 21 Copyright 2009 CA

22 3 Building Blocks Pivotal Insight Service Assurance Application Management Fault, Performance, Configuration & Capacity Infrastructure Management Networks Physical & virtual systems Databases 22 Copyright 2009 CA

23 #1: Infrastructure Management End-to-End View of Infrastructure Status Discover networks, systems and databases; map to IT services Monitor performance trends/violations and outages Correlate alarms; diagnose network, system and database root cause BACK END USER NETWORK FRONT END MIDDLEWARE Mainframe September WAN/ 5, 2009 WWW Router Firewall Switch Load Web Balancer Servers Portal App Server Database End User Databases 23

24 #1: Infrastructure Management End-to-End View of Infrastructure Status Discover networks, systems and databases; map to IT services Monitor performance trends/violations and outages Correlate alarms; diagnose network, system and database root cause BACK END USER NETWORK FRONT END MIDDLEWARE Mainframe September WAN/ 5, 2009 WWW Router Firewall Switch Load Web Balancer Servers Portal App Server Database End User Databases 24

25 Best Practice: Automated Discovery > Inductive Modeling Discover assets Learn relationships Create model Monitor the model for conditions CA, Inc.

26 Best Practice: Automated Root Cause Analysis > Active monitoring does not just rely on event streams > Distinguish between a flood of events and meaningful alarms Symptomatic event Causal event CA, Inc.

27 Best Practice: Automated Root Cause Analysis > Correlate symptoms > Suppress unnecessary, symptomatic alarms > Pinpoint root cause Must be automated Manual methods overwhelming and slow CA, Inc.

28 Best Practice: Configuration-Aware Root Cause > Manage to standards Gold standard Change Authorization > Know infrastructure impact of change Fault-Awareness Degradation-Awareness Automated detection and roll-back Misconfiguration Original Configuration Copyright 2009 CA 28

29 Best Practice: Proactive Perf. & Capacity Management Time Identify over when something Threshold is too wrong for too long Identify Deviation when unusual from behavior Normal Is happening Automatically reduce noise Time Time Over Threshold = 15 min. Time Window = 1 hour Time Time is Unacceptable = 15 min. Time Window = 1 hour Upper Threshold Basel ine Lower Threshold Automatically determine baseline values Networks Systems Databases Energy

30 3 Building Blocks Pivotal Insight IT Benefits Service Assurance Application Management Fault, Performance, Configuration & Capacity Infrastructure Management Networks Physical & virtual systems Databases Improve Infra MTTR 50% Reduce Infra DT 70% Boost Efficiency 43% 30 Copyright 2009 CA

31 Pop Quiz!!

32 What Does This Mean? M T T I

33 An Important Concept Mean Time To Innocence

34 3 Building Blocks Pivotal Insight IT Benefits Service Assurance Customer Experience & Transaction Behavior Application Management Web portals & servers Application servers J2EE,.NET & client/server apps Fault, Performance, Configuration & Capacity Infrastructure Management Networks Physical & virtual systems Databases Improve Infra MTTR 50% Reduce Infra DT 30% Reduce Help Desk Calls 40% MTTR = Mean-Time-to-Repair DT = Downtime 34 Copyright 2009 CA

35 #2: Application Performance Management End-to-End View of Transaction Behavior Discover applications; understand user experience; establish SLAs Monitor all business transactions through infrastructure; measure response and SLAs Conduct triage across infrastructure; diagnose application root cause BACK END SAP PSFT USER NETWORK FRONT END Siebel MIDDLEWARE Mainframe WAN/ WWW Router Firewall Switch Load Web Balancer Servers Portal App Server Database End User Web Services 3 rd Party Applications Databases 35

36 #2: Application Performance Management End-to-End View of Transaction Behavior Discover applications; understand user experience; establish SLAs Monitor all business transactions through infrastructure; measure response and SLAs Conduct triage across infrastructure; diagnose application root cause BACK END SAP PSFT USER NETWORK FRONT END Siebel MIDDLEWARE Mainframe September WAN/ 5, 2009 WWW Router Firewall Switch Load Web Balancer Servers Portal App Server Database End User Web Services 3 rd Party Applications Databases 36

37 #2: Application Performance Management End-to-End View of Transaction Behavior Discover applications; understand user experience; establish SLAs Monitor all business transactions through infrastructure; measure response and SLAs Conduct triage across infrastructure; diagnose application root cause BACK END SAP PSFT USER NETWORK FRONT END Siebel MIDDLEWARE Mainframe September WAN/ 5, 2009 WWW Router Firewall Switch Load Web Balancer Servers Portal App Server Database End User Web Services 3 rd Party Applications Databases 37

38 Best Practice: Understand Real User Experience > Monitor all transactions all the time 24x7 in production From browser to back-end > Understand your users experience Monitor end-user browser response times Identify users by name and priority See transaction behavior s impact to the service September 5, 2009 CA Wily Technology Copyright 2008 CA 38

39 Best Practice: Set SLAs DMV H.P. Driving Manual Driving Test Online Reg. Unemployment H.P. Unemployment Reg Prof. Skills > Set and measure SLAs on processes > Understand the impact of Application Performance > Communicate in the language of the stakeholders Job Postings September 5, 2009 CA Wily Technology Copyright 2008 CA 39

40 Best Practice: Incident Prioritization > Assign value to successful and unsuccessful transactions > Prioritize incidents based on the impact Criticality of the transaction Priority of the users Severity of the error > Work to resolve most service s critical issues first 40 September 5, 2009 CA Wily Technology Copyright 2008 CA

41 Best Practice: Triage and Root Cause Analysis > Triage Identify the problem source s domain Aid remediation > Diagnosis and root cause analysis Monitor the transaction calls through the application Identify application components or back end calls behind failures 41 September 5, 2009 CA Wily Technology Copyright 2008 CA

42 3 Building Blocks Pivotal Insight IT Benefits Service Assurance Customer Experience & Transaction Behavior Application Management Web portals & servers Application servers J2EE,.NET & client/server apps Improve App MTTR 50% Reduce App DT 70% Reduce App Perf Delays 71% Fault, Performance, Configuration & Capacity Infrastructure Management Networks Physical & virtual systems Databases Improve Infra MTTR 50% Reduce Infra DT 30% Reduce Help Desk Calls 40% MTTR = Mean-Time-to-Repair DT = Downtime 42 Copyright 2009 CA

43 3 Building Blocks Pivotal Insight IT Benefits Infrastructure Impact on Quality & Risk Service Assurance End-to-end infrastructure Applications Customer Experience & Transaction Behavior Application Management Web portals & servers Application servers J2EE,.NET & client/server apps Improve App MTTR 50% Reduce App DT 70% Reduce App Perf Delays 71% Fault, Performance, Configuration & Capacity Infrastructure Management Networks Physical & virtual systems Databases Improve Infra MTTR 50% Reduce Infra DT 70% Reduce Help Desk Calls 40% MTTR = Mean-Time-to-Repair DT = Downtime 43 Copyright 2009 CA

44 #3: Service Assurance Management! Complete View of the IT Service Transaction behavior insight Infrastructure status insight Service quality, risk to service delivery & root cause insight BACK END SAP PSFT USER NETWORK FRONT END Siebel MIDDLEWARE Mainframe September WAN/ 5, 2009 WWW Router Firewall Switch Load Web Balancer Servers Portal App Server Database End User Databases 44

45 #3: Service Assurance Management! Complete View of the IT Service Transaction behavior insight Infrastructure status insight Service quality, risk to service delivery & root cause insight BACK END SAP PSFT USER NETWORK FRONT END Siebel MIDDLEWARE Mainframe September WAN/ 5, 2009 WWW Router Firewall Switch Load Web Balancer Servers Portal App Server Database End User Databases 45

46 Best Practice: Service Modeling & Impact Analysis Services & Components: based infrastructure, application and customer experience Switches 1 1 Transactions Service Service Topology: real-time model, component relationships & impact Databases Servers Component Detail: status, severity, impact & root cause 46 Copyright 2009 CA

47 Best Practice: Service Dashboards & Reports Services listed according to: Business Importance Quality level Risk to quality DMV H.P. Driver s Manual Health Quality Risk Availability Real-time Service Status Indicators Tax Filing Medicare Fish & Game Historical Service Status Details 47 Copyright 2009 CA

48 3 Building Blocks Pivotal Insight Infrastructure Impact on Quality & Risk Service Assurance End-to-end infrastructure Applications IT Benefits Focus staff on your mission Improve quality/predictability Lower IT cost Customer Experience & Transaction Behavior Application Management Web portals & servers Application servers J2EE,.NET & client/server apps Improve App MTTR 50% Reduce App DT 70% Reduce App Perf Delays 71% Fault, Performance, Configuration & Capacity Infrastructure Management Networks Physical & virtual systems Databases Improve Infra MTTR 50% Reduce Infra DT 30% Reduce Help Desk Calls 40% MTTR = Mean-Time-to-Repair DT = Downtime 48 Copyright 2009 CA

49 What if You Could Solve the Paradox? Managing Technologies vs. Services 05:00 06:00 07:00 08:00 09:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 % available Web Server App Server Mainframe Database Network Application And who is measuring the real user experience and managing the whole service 99% 99% 99.9% 99.9% % 99.9% End-User Service Unavailable or Slow Available, Performing as Expected? Managing Infrastructure Top-Down in Context of Services is Required 49 Copyright 2009 CA

50 What if You Could Define & Manage Services? Service #1 Service #3 Service #5 Service #2 Service #4 Service #6 Client Systems Applications Databases Servers Storage Networks Intelligent Service Modeling and Analytics is Required 50 Copyright 2009 CA

51 What if You Could Focus People & Processes? Service Desk Service #1 Service #3 Service #5 Service #2 Service #4 Service #6 Network Engineering Client Systems Application Engineering Applications Databases Servers Storage Networks System Engineering DB Administration Prioritized & Shared Understanding of Services, Their Condition & Root Cause is Required 51

52 Analyst Recommendations

53 Have a Common Framework for IT Ops Tools 60% of the IT operations management market share is in the hands of BMC, CA, HP and IBM Gartner Tivoli Dataquest: IT Operations Management Software, Worldwide, 2007

54 Framework Should Support IT Ops Processes > Example: Fault to Remediation Process > Reducing the MTTR by eliminating manual intervention and automatically passing event data between products Gartner Dataquest: IT Operations Management Software, Worldwide, 2007

55 Settle on a Management Strategy > Focus staff on highest value deliverables > Improve user experience > Reduce waste > Improve productivity CA Leaders in Lean IT: 2009

56 Settle on a Management Process > Set and measure SLAs on key business processes > Monitor 100% of all end user transactions > Employ predictive and proactive monitoring > Prioritize incidents based on service impact > Implement the capability for rapid triage and root cause analysis > Report results and implement continuous improvement processes Adapted from Ashton-Metzler Associates: 2009

57 Getting Started Pick three top services Evaluate staff focus, understanding & approach to monitoring Focus on high value Reduce Waste Evaluate: Triage process Manual versus automated methods Building block (tools) capability & integrations (enable best practices?) Evaluate: How staff determines & weigh risks Percent reactive vs. proactive Continuous improvement methods & success Improve Service Improve Productivity Evaluate: Time/cost to determine root cause Resources to manage problems vs. add value

58 Final Quiz

59 Why Are These Numbers Important?

60 Application Performance Mgmt Service Management Project & Portfolio Mgmt Infrastructure Management Security Management Mainframe 2.0 Analyst Data on Best Practices > Lower overall cost of service delivery by up to 30% Reduce service desk calls up to 40% Improve MTTR up to 50% Reduce downtime up to 70% Reduce performance delays up to 71% Improve staff productivity up to 25% Reduce number of IT tools up to 50% > New Management Technologies Average payback 9 months or less September 5, 2009 Average 433% ROI over 3 years VALUE COST Sources: Achieving Business Value & Gaining ROI with CA s EITM Software, IDC, 2007, and IT Economics, Gaining Business Value with CA s Enterprise IT Management Software: An ROI Study, IDC, 2008, and direct customer data 60 Copyright 2009 CA

61 Assuring Service Quality Despite Limited Resources Thank You