A Single Pane of Glass Will Never Exist. Session 232, February 23, 2017 Ryan J. Klein, IT - St. Joseph Health Kelly Nunez, IT - St.

Transcription

1 A Single Pane of Glass Will Never Exist Session 232, February 23, 2017 Ryan J. Klein, IT - St. Joseph Health Kelly Nunez, IT - St. Joseph Health 1

2 Speaker Introductions Ryan J. Klein St. Joseph Health Anaheim, CA

3 Speaker Introductions Kelly Nunez St. Joseph Health Anaheim, CA

4 Conflict of Interest Ryan J. Klein Kelly Nunez Has no real or apparent conflicts of interest to report. 4

5 Agenda Overview of St. Joseph Health Evolving infrastructure and IT operations and St. Joseph Health Enterprise Monitoring Team The Monitoring Opportunity at SJH Building our Monitoring Strategy People, Process, Technology Our Results & Recommendations 5

6 Learning Objectives 1. Assess the theory of the "single-pane-of-glass" monitoring strategy for infrastructure and applications 2. Analyze the process improvement steps taken by the SJH Enterprise Monitoring team 3. Propose a cohesive idea for a infrastructure and application monitoring solution 4. Create a Monitoring as a Service (MaaS) that is repeatable and expandable 6

7 Benefits Realized for the Value of Health IT Savings and Satisfaction Automated deployment of new systems Integrated alert and notification systems Reduce cost by 27.3% Reduced alert notification time by 73.8% Improved incident engagement and decreased resolution times resulting in higher application uptime and decrease clinical impact 7

8 Overview of St. Joseph Health 8

9 Providence St. Joseph Health at a glance 9

10 St Joseph Health IT Infrastructure & Operations Strategy Vision: Information Technology drives competitive advantage, enhances the healthcare experience, and provides an operationally streamlined service. Mission: We provide solutions and an environment that: Meet and exceed the needs of our health system Are agile and high-performing Promote collaborative innovation Provide proactive and measurable value 10 Strategic Principles

11 Evolving Infrastructure and IT Operations and St. Joseph Health 11

12 Two Types of IT Managing two separate, coherent modes of IT delivery, one focused on stability and the other on agility. Traditional IT DevOps Safety Accuracy Silo Teams Waterfall Methodologies High Touch Traditional Governance Safety Speed Cross Functional Teams Agile Methodologies Low Touch (Automation) New Governance 12

13 A Service Line approach 13

14 SJH Service Inventory Overview 14

15 Enterprise Monitoring Team Vision: Immediately identify and alert on issues to be quickly resolved before patient care is impacted and further researched on how to be prevented from reoccurring. Mission: Provide a strong monitoring solution that proactively engages technical teams to predict and prevent issues. 15

16 Monitoring Strategy Review Previous Method Team focused/no engagement Reactive approach Tool overlap with capability gap Lack of visibility to new deployments No end to end (E2E) monitoring Lack of vendor engagement No governance Limited capabilities for Cloud and SaaS Not conducive to DevOps requirements Strategy Focus: Deployment Approach w/component Monitoring New Method Application focused/fully engaged Proactive approach Consolidate tools and fill gap Implement during provision E2E monitoring Engage vendors Implement governance Incorporate Cloud and SaaS architectures Enable support for DevOps Strategy Focus: Applications 16 Approach w/e2e Infrastructure Monitoring

17 Key Areas of Monitoring Focus Event Correlation End to End View Hybrid Public SaaS 17

18 Enterprise Monitoring Processes People Monitoring, IT, Project Governance and Security Proactive Response & Reporting Process Technology Current and new tools 18

19 Change Monitoring Team Focus Tool focused -> Application focused Increase application accountability within the team Build monitoring knowledge experts by application Tier 1 applications Primary and Secondary Champions Tier 2 and 3 applications Primary Champion Knowledge transfer throughout team People 19

20 Implement Monitoring Checkpoints Governance Teams Integration IT Project Governance Team: Technical Assessment IT Architects Team: IT Architecture Review Security Team: Security Assessment Support Teams Awareness IT Support Teams Application Teams Incident Response Teams Service Desk NOC People 20

21 Overcoming Challenges Application teams lacked knowledge of application architecture Worked directly with vendor Engaged teams often and frequently Teams functioned in silos Became liaison between application and technical teams Marketing campaign to explain MaaS program People 21

22 Implement E2E Monitoring Model Implement Proactive Response Integration with ticketing system (standard) APM and synthetic monitoring with immediate notifications to application teams Auto-baselining infrastructure performance with immediate notification to application/technical teams Additional notification levels options ( , text) Strengthen Reactive Response Integration with Mission Control and Incident Response Mandatory monitoring team participation on all calls IT Liaison for Service Desk and NOC Transparency of all alerts and dashboards Process 22

23 Implement Monitoring Checkpoints Prevent going around the process Governance Processes: Technical Assessment IT Architecture Review Security Assessment IT Processes: Installation/decommission infrastructure New site baseline testing Authorized requesters Proactive response Increase transparency Application Performance: Alerts, baseline health Infrastructure Performance: Alerts, service/process health, device state, baseline health, monitor state Published Dashboards/Reports: Mission Control (Incident Response) Executive summary Operational details Process 23

24 Overcoming Challenges Support Teams didn t want to change their process Marketing campaign to explain MaaS program Worked tightly with teams to update processes IT Liaison for Service Desk and NOC didn t exist Created process Provided training and access Process 24

25 Tool Comparison Identify components Identify availability/performance monitoring Map to current tools Identify the gap Missing because not implemented yet? Missing because not purchased yet? Technology 25

26 Identified Gaps DevOps integration Integrate with vrealize Automation and Puppet Automate deployment of agents (servers) NextGen Network project engagement (automate deployment network) Limited ESX host monitoring vrealize Operations engagement Reclaim funds by replacing current ESX host monitoring solution Limited storage monitoring Support teams and vendor engagement Tool review and implementation Limited knowledge and training vrealize Operations training Tableau training 26 Technology

27 Overcoming Challenges Limited budget for training Sent half team for training Knowledge transfer and documentation Lack of budget for new tools Contract negotiations current tools How close to 100% could we get with current tools? 80/20 Management accepts remaining risk Reclaim funds from overlapping tools Technology 27

28 Results: Monitoring as a Service (MaaS) Program Created a repeatable and standardized service for our Monitoring Framework. Incorporates automatic deployments, notifications, SJH standards, vendor recommendations and industry best practices. Documents application topology and interdependencies of each application. Finds best fit monitoring for each tier application. Provides full service reporting capabilities to all users with total transparency. Aligns with Incident Management and integrates with DevOps and CMDB. Ensure all aspects of an application are monitored and alerting. 28

29 MaaS Options Menu Watch Analyze Integrate Verification of Application Workflow All Upstream and Downstream Applications Documented Infrastructure List verification Complete topology including storage and physical servers Alert notification and Ticketing Default Monitoring thresholds All Service, Processes, Logs, File or Folders Monitored Vendor Recommendations Custom Monitoring threshold Normalization Synthetic Script Other Monitoring Tools Mandatory for all Mandatory for Tier 1 Available for all 29

30 30

31 Benefits Realized for the Value of Health IT Savings and Satisfaction Automated deployment of new systems Integrated alert and notification systems Reduce cost by 27.3% Reduced alert notification time by 73.8% Improved incident engagement and decreased resolution times resulting in higher application uptime and decrease clinical impact 31

32 Recommendations Begin with the consumer and not the technology. Become user and application centric. Obtain a tool set that works for you. Unlike one ring, there is no onetool to rule them all. Make sure tools are flexible, sustainable and supports both the traditional and DevOps environments. Be proactive. Collaborate with hardware and software support teams and vendors. 32

33 Questions Ryan J. Klein St. Joseph Health Kelly Nunez St. Joseph Health *Please complete the online session evaluation 33