Making Data-Driven Decisions for Better DevOps Outcomes

Transcription

1 Making Data-Driven Decisions for Better DevOps Outcomes Andi Mann, Andi Mann Chief Technology amann.com

2 The Dev Lifecycle is Complex Plan Code Build Test/QA Stage Release Config Monitor SDKs API UI Escalation/ Collaboration Metrics and Reporting Common Data Fabric Other Tools No rigid schemas add in data from any other source.

3 The Ops Environment is Complex Server, Storage, Network Server Virtualization Operating Systems Infrastructure Applications Mobile Applications Cloud Services Custom Applications API Services SDKs API UI Ticketing/Help Desk Metrics and Reporting Common Data Fabric Other Tools

4 DevOps is Exponentially More Complex Plan Code Build Test/QA Stage Release Config Monitor Biz PMO Dev QA Sec Build Stage Ops Biz Server, Storage, Network Server Virtualization Operating Systems Infrastructure Applications Mobile Applications Cloud Services Custom Applications API Services

5 One Constant - Data From every tool, process, or component in Dev. On-premises, in the cloud, or with 3 rd party Ops. Across diverse teams, activities, and services.

6 BUT WHAT DATA DRIVES GOOD DECISIONS?

7 I m working super hard!! That s my stapler!

8 Yeah, but what are you achieving? I m gonna need you to come in Sunday. 9

9 Sales? Downloads? Installs? Users?

10 Some DevOps Data that Might Matter Culture Process Quality Systems Activity Impact e.g. e.g. e.g. e.g. e.g. e.g. Retention Idea-to-cash Tests passed Throughput Commits Signups Satisfaction MTTR Tests failed Uptime Tests run Checkouts Callouts Deliver time Best/worst Build times Releases Revenue

11 Specific Data For Each Stakeholder process times team efficiency unplanned work test volume code coverage exception counts build speed failure rates manual builds response time uptime/availability resource usage Biz PMO Dev QA Sec Build Stage Ops Biz time to deliver idea to cash ROI code volume commit volume release speed remediation time code quality access rates performance latency scalability revenue signups satisfaction

12 Shared Data for Multiple Stakeholders process times team efficiency unplanned work test volume code coverage exception counts build speed failure rates manual builds response time uptime/availability resource usage Biz PMO Dev QA Sec Build Stage Ops Biz time to deliver idea to cash ROI code volume commit volume release speed remediation time code quality access rates performance latency scalability revenue signups satisfaction

13 Shared Data for Multiple Stakeholders process times team efficiency unplanned work test volume code coverage exception counts build speed failure rates manual builds response time uptime/availability resource usage Biz PMO Dev QA Sec Build Stage Ops Biz time to deliver idea to cash ROI code volume commit volume release speed remediation time code quality access rates performance latency scalability revenue signups satisfaction

14 Shared Data for Multiple Stakeholders process times team efficiency unplanned work test volume code coverage exception counts build speed failure rates manual builds response time uptime/availability resource usage Biz PMO Dev QA Sec Build Stage Ops Biz time to deliver idea to cash ROI code volume commit volume release speed remediation time code quality access rates performance latency scalability revenue signups satisfaction

15 Computing UK s Metrics that Matter Source: Computing Research UK, DevOps Review 2016: Accelerating Innovation, July 2016

16 Computing UK s Metrics that Matter Source: Computing Research UK, DevOps Review 2016: Accelerating Innovation, July 2016

17 Increase App Delivery Velocity Product Managers identify new opportunities Code continuously delivered to market Auditors have visibility DevOps Teams iterate with continuous insights Customers are happy

18 Tesco uses Machine Data to Accelerate Development and Understand Customers Cut Investigation & Resolution time 95% Reduce Escalations 50%, Accelerate Dev Cycles 30% Operational Analytics with Live Transaction Tracing and End-to-end Infrastructure Insight Activity Tracking Dashboards with Improved Customer Experience and Reduced Lost Revenue

19 Computing UK s Metrics that Matter Source: Computing Research UK, DevOps Review 2016: Accelerating Innovation, July 2016

20 Computing UK s Metrics that Matter Source: Computing Research UK, DevOps Review 2016: Accelerating Innovation, July 2016

21 Improve Code Quality Developers check in code Code Quality Scans White Box Static Security Scans Test Fail: Return Pattern library used for test and QA QA Pattern Library Test Pass: Promote Automated Acceptance Tests Test Pass: Promote to Production Black Box Dynamic Security Scans Production Chaos Monkey Tests Test Fail: Return QA Prod Pattern

22 MEDIA & ENTERTAINMENT APPLICATION DELIVERY Improved DevOps Agility It s like we were working without peripheral vision before and now we have it. Robert Gonsalves, Web Operations Key Customer Benefits Increased success rate of deployments Ability to detect issues before they affect broad production Monitoring deployment process several times per day

23 Computing UK s Metrics that Matter Source: Computing Research UK, DevOps Review 2016: Accelerating Innovation, July 2016

24 Computing UK s Metrics that Matter Source: Computing Research UK, DevOps Review 2016: Accelerating Innovation, July 2016

25 Data-Driven Feedback Shows Business Impact

26 Amaya Gaming Uses Machine Data for Digital Customer Insight Allows DevOps to ensure quality of releases & avoid negative impact on service performance. Analyze which new website features are being adopted, and how, by end users. Insight fed back into the development cycle to improve customer engagement.

27 But Good Data Is Not Enough

28 Find The Value In The Data Planning Development Build Verification Deployment Post-Deploy 100 (0%) 100 (0%) (-5.3%) 100 (0%) 100 (0%) 100 (0%) 160 stories 0 in progress 100% success 100% success 364 deploys 0 CFDs 100 stories 95 complete 8.8 MTTB 3.95 MTTT 0.54 success 1 ticket MTTR 30 points/dev 94 (-6%) days

29 Find The Value In The Visualization

30 Find The Failure in The Data Planning Development Build Verification Deployment Post-Deploy 100 (0%) 100 (0%) 0 (-100%) 100 (0%) 100 (0%) 100 (0%) 160 stories 0 in progress 35% success 100% success 364 deploys 0 CFDs 100 stories 95 complete 8.8 MTTB 3.95 MTTT 0.54 success 1 ticket MTTR 30 points/dev 94 (-6%) days

31 Find The Failure in the Visualization

32 How About Now?

33 How About Now?

34 How About Now?

35 Apply Machine Learning to Your Data

36 ML Lets You Predict (and Prevent) Failure

37 Summary

38 Data-Driven Decisions For Better DevOps Outcomes Improve Velocity Reduce the time it takes to get code through dev/test to market through faster issue resolution and reduced cycle time Improve Quality Real-time visibility into processes like code check-in, build, test, QA to support continuous integration and continuous delivery Improve Impact Instrument customer engagement and application usage to capture critical business events, outcomes, and user behavior Our devs are now able to find and fix issues 5-10 times faster. We can monitor all the automation and handoffs it takes to deploy 5-10 times a day. My code isn t ready until it s Splunk-ready.

39 Sources/Additional Reading splunk.com/devops Resources on Splunk for DevOps incl. case studies, customer stories, partners, products, videos, etc. dev.splunk.com Resources for developing with or on the Splunk platform, incl. SDKs, API Docs, guides, etc. blogs.splunk.com Check the DevOps tag for specifics, including how to deploy Spunk w/ CI/CD tools splunkbase.splunk.com Splunk add-ons, applications, and TAs for AWS, Jenkins, Ansible, Jira, Puppet, Docker, and more Vertu Calls On Splunk Enterprise For Smarter DevOps, Splunk Press Release, 2016, DevOps Review 2016: Accelerating Innovation, Computing Research UK, July State of DevOps Report, DevOps Research and Assessment The DevOps Cookbook, John Allspaw, Patrick Debois, Damon Edwards, Jez Humble, Gene Kim, Mike Orzen, and John Willis The Phoenix Project, Gene Kim, Kevin Behr, George Spafford Data-Driven DevOps: Use Metrics to Help Guide Your Journey, Gartner Inc. 2014, Cameron Haight and Tapati Bandopadhyay Metrics that Matter, Mark Michaelis, IntelliTect DevOps and the Cost of Downtime: Fortune 1000, IDC DevOps Best Practice Metrics: Fortune 1000 Survey, IDC, 2014

40 Thank You amann.com Andi Mann Chief Technology amann.com