SESSION 205 Wednesday, November 1, 11:30am - 12:30pm Track: DevOps and Agile

Transcription

1 SESSION 205 Wednesday, November 1, 11:30am - 12:30pm Track: DevOps and Agile Measuring DevOps: the Key Metrics that Matter Anders Wallgren Chief Technology Officer, Electric Cloud anders@electric-cloud.com Session Description Having the right goals, asking the right questions, and learning by doing are paramount to achieving success with DevOps. Specific milestones and shared KPIs play a critical role in guiding your DevOps adoption and lead to continuous improvement: true agility, improved quality, and faster time to market. This session will share common use cases and realworld examples to walk you through a practical framework for measuring and tracking of your DevOps efforts and software delivery performance in a way that will provide you with data you can act on! Speaker Background Anders Wallgren is chief technology officer at Electric Cloud. Anders has over 25 years; experience designing and building commercial software. Prior to joining Electric Cloud, he held executive positions at Aceva, Archistra, and Impresse and management positions at Macromedia (MACR), Common Ground Software, and Verity (VRTY), where he played critical technical leadership roles in delivering award winning technologies such as Macromedia;s Director 7. Anders holds a B.SC from MIT.

2 Measuring DevOps The Key Metrics That Matter Anders Wallgren CTO, Electric

3 We are here to talk metrics

4 Every business is a software business Embedded/IoT Mobile Enterprise, Web/IT Software is eating the world Marc Andreessen, Aug 2011

5 Software is now the primary driver of innovation & disruption.

6 Can we deliver? FINSERV building software to deliver better service RETAIL building platforms for online sales and support AUTOMOTIVE building services for the connected car FEDERAL delivering on time and with higher quality HEALTHCARE building applications to improve care TELECOM building embedded and online services 100% of those surveyed 58% want fast software delivery (one of top 3 goals) 12% can do it Survey completed by Forrester and 155 F1000 senior IT execs

7 What Is The One Question That Predicts Software Team Performance With Startling Accuracy?

8 To what degree do we fear doing deployments? Source: Puppet Labs 2015 State Of DevOps:

9 High-Performing IT Organizations Do It More Often 2017: 5x lower change failure rate, 96x faster recovery from failures. From: IT Revolution and Puppet Labs 2016/2017 State of DevOps

10 WHY METRICS?

11 The scientific method Observation is fundamental to the scientific method You can't improve what you can't measure

12 Key principles of measuring software Software s most important quality is its adaptiveness and ease of change. Efficiency Effectiveness Rate and cost per release Ability to add more value

13 Types of metrics Internal: inside-out measurements of efficiency (cost/time/materials) Tech progress, product pipeline trends and resource utilization Measures of efficiency and resource consumption External: outside-in measurements of effectiveness (value delivered) Quality, usefulness, performance, and business outcomes Effectiveness and value delivered Culture: (objective and subjective trends in team dynamics) Process overhead, trustworthiness, shared objectives, morale, motivation, team/product/enterprise identity

14 Best Practices OBJECTIVITY SUBJECTIVITY FOCUS ON OUTCOMES IDENTIFY PREDICTORS SIGNAL TO NOISE RATIO

15 Best Practices VALUE TEAM PERFORMANCE AVOID METRICS THAT CAN BE GAMED SHAMING ENCOURAGES GAMING

16 Automatic Unobtrusive Not only production One pane of glass Measure-ability Technology HOW

17 Actionable Accessible Auditable Tickets closed Customer Sat Uptime MTTR VANITY METRICS?

18 Pipeline KPIs

19 What to measure in the pipeline DEV/CI QA Deploy Release Operate Development lead time Idle time Deployment lead time Release frequency MTTR Rework required by defects, build breakage, downtime Defects discovered/ escaped, impact of defects Deployment frequency, duration Time/cost per release Cost/frequency of outages Idle time MTTD Change success rate Predictability On-call after business hours Work-in-progress and technical debt MTTR Performance / utilization Cycle time -- Cycle Time Visibility Scale --

20 What to measure in the pipeline DEV/CI QA Deploy Release Operate Development lead time Idle time Deployment lead time Release frequency MTTR Rework required by defects, build breakage, downtime Defects discovered/ escaped, impact of defects Deployment frequency, duration Time/cost per release Cost/frequency of outages Idle time MTTD Change success rate Predictability On-call after business hours Work-in-progress and technical debt MTTR Performance / utilization Cycle time -- Cycle Time Visibility Scale --

21 Dev/CI: Where to Start? Trunk-based development (or very-short-lived branches) Self-service automation for environment provisioning *-as-code Build quality in (less unplanned work downstream) Build security in (less unplanned work downstream)

22 What to measure in the pipeline DEV/CI QA Deploy Release Operate Development lead time Idle time Deployment lead time Release frequency MTTR Rework required by defects, build breakage, downtime Defects discovered/ escaped, impact of defects Deployment frequency, duration Time/cost per release Cost/frequency of outages Idle time MTTD Change success rate Predictability On-call after business hours Work-in-progress and technical debt MTTR Performance / utilization Cycle time -- Cycle Time Visibility Scale --

23 QA: Where to Start? Automated testing Fidelity of environments (vs. Prod) Self-service automation for environment provisioning Continuous Delivery

24 What to measure in the pipeline DEV/CI QA Deploy Release Operate Development lead time Idle time Deployment lead time Release frequency MTTR Rework required by defects, build breakage, downtime Defects discovered/ escaped, impact of defects Deployment frequency, duration Time/cost per release Cost/frequency of outages Idle time MTTD Change success rate Predictability On-call after business hours Work-in-progress and technical debt MTTR Performance / utilization Cycle time -- Cycle Time Visibility Scale --

25 Deploy: Where to Start? Automate all the things The first deployment shouldn t be to PROD or even PRE-PROD Continuous Delivery Improves deployment frequency, reliability Artifact version control *-as-code

26 What to measure in the pipeline DEV/CI QA Deploy Release Operate Development lead time Idle time Deployment lead time Release frequency MTTR Rework required by defects, build breakage, downtime Defects discovered/ escaped, impact of defects Deployment frequency, duration Time/cost per release Cost/frequency of outages Idle time MTTD Change success rate Predictability On-call after business hours Work-in-progress and technical debt MTTR Performance / utilization Cycle time -- Cycle Time Visibility Scale --

27 Release: Where to Start? Model the software delivery pipeline Ensures reuse, predictability, visibility Fidelity of everything artifacts, tools, processes, environments Visibility

28 What to measure in the pipeline DEV/CI QA Deploy Release Operate Development lead time Idle time Deployment lead time Release frequency MTTR Rework required by defects, build breakage, downtime Defects discovered/ escaped, impact of defects Deployment frequency, duration Time/cost per release Cost/frequency of outages Idle time MTTD Change success rate Predictability On-call after business hours Work-in-progress and technical debt MTTR Performance / utilization Cycle time -- Cycle Time Visibility Scale --

29 Operate: Where to Start? Version control all artifacts (rollback, governance, visibility) Monitor health of systems and applications Self-service provisioning of environments Shared infrastructure to drive down opex/capex

30 What to measure in the pipeline DEV/CI QA Deploy Release Operate Cycle Time -- Visibility Scale --

31 10K+ Releases/year 1M+ System Integrations/yearc 100K Builds/day 30M Lines of Code 100M Test cases run/day 480K Code reviews/year

32 100+ Applications 2 min For Test System Provisioning $2.5B Online sales, >8 People per release (before->after)

33 Gap and Huawei: Two tales of DevOps at scale Developer Build Production Build Regression Test Full Test Feature Delivery Time Huawei Before Huawei After 10 minutes 300 minute 240 minutes 24 hours 30 days 1 minute 10 minutes 60 minutes 6 hours 7 days Gap Before Gap After 20 minutes 150 minutes 300 minutes 24 hours 15 days 20 minutes 120 minutes 150 minutes 6 hours 1 day

34 What to measure in the pipeline DEV/CI QA Deploy Release Operate -- Cycle Time -- Visibility -- Scale --

35 Release and Deploy Dashboards & Reports

36 Business Metrics from DevOps Toolchain

37 What to measure in the pipeline DEV/CI QA Deploy Release Operate -- Cycle Time Visibility -- Scale Doing it like a unicorn!

38 Business KPIs

39 What to measure in the Business LEAD TIME STORIES DELIVERED CUSTOMER SATISFACTION ACQUISITION, RETENTION COST

40 Culture KPIs

41 What to measure in Culture Satisfaction Retention

42 How to create a generative culture Pathological Bureaucratic Generative Low cooperation Modest cooperation High cooperation Messengers shot Messengers neglected Messengers trained Responsibilities shirked Narrow responsibilities Risk are shared Bridging discouraged Bridging tolerated Bridging encouraged Failure -> scapegoating Failure -> justice Failure -> inquiry Novelty crushed Novelty -> problems Novelty implements A typology of organizational cultures R. Westrum

43 The Seven Measures of Culture 1. Organizational investment in DevOps 2. The experience and effectiveness of team leaders 3. Continuous delivery practices 4. Achieving win-win outcomes for dev, ops, and infosec teams 5. Organizational performance 6. Deployment pain 7. Lean management practices

44 Predictors of strong performance 1. Peer-reviewed change approval process 2. Version control for all production artifacts 3. Proactive monitoring 4. High-trust organizational culture 5. Win-win relationship between dev and ops From: IT Revolution and Puppet Labs 2014 State of DevOps

45 Transformative Benefits 10 min FASTER DEVELOP TO DEPLOY 90 days 99% improvement TIME 10 min FASTER DEVELOP TO DEPLOY 120+ min 12X improvement TIME 6 hours FASTER DEVELOP TO DEPLOY 24 hours 75% improvement TIME minutes FASTER AUDITABILITY who, what, when, how 20 days 90% improvement TIME

46 Resources

47 Questions?

48