GE Power: Our DevOps Experience

Transcription

1 GE Power: Our DevOps Experience Eric Haynes, Principal Engineer Gary Kern, Sr. Program Manager September 22, 2016 Imagination at work

2 GE Power DevOps Starting the Journey Problem: Team s planned work halted to handle unplanned work. Slow processes to release software with handoffs between team disjointed and painful.

3 GE Power DevOps Starting the Journey Continuous Delivery

4 DevOps Release Pipeline automation XLR Release Start/End Dates Scrum Master XLR: DEV Integration task state Rally Config XLR ScrumMaster SM DevOps UI SM Sustaining approval, phase complete XLR XLR XLR SM Sustaining approval, phase complete (Calendar function) Dev Dictionary- DEV Integration Dictionary-QA Dictionary - STAGING Dictionary - PROD SQL XLD1 XLD1 XLD2 XLD2 commit Code repos Legacy CI Jenkins CI TRIGGERED + NIGHTLY: Compile Unit Test SCA Build pkg DEV DEV DEV DEPLOY Completed user story QA QA QA DEPLOY Production Ready STAGING DEPLOY Stable Ops validated Business continuity PROD DEPLOY release reports Dictionary- DEV Smoke SMOKE TEST XLD1 CI Build Server VMs (pre-defined) VMs (pre-defined)

5 GE Power DevOps Business Case Costs Benefits Investment Consultation engineering engagements (on-site project: Automate all GE Power software/analytic package releases) vsql database production instance: Go IT hosting and support XebiaLabs licensing for 2 products: XL Deploy, XL Release (xxx licenses XLD + xx licenses $xxxx each) CONSIDERATIONS: - Auditable savings increase credibility - Holistic DevOps should reduce defect counts we have not yet accounted for this effect - Historical data is powerful collect it from as-is process or run trial. Avoid paper estimates. Category Description Productivity TT labor reductions Rev Growth Solution 1 deployments (input by xxxx): 25 hrs saved /deployment * xxx deployments * $xx/hr labor rate Solution 2 Quarterly package deployments (inputs by xxx /xxx /xxx): 25 hrs saved /deployment * xx deployments * $xx Solution 3 software deployments (inputs by xxx, xxx): 25 hrs saved/box deployment * 5+12 boxes * x deployments Solution 4 deployments (inputs by xxx, xxx): 25 hrs saved/deployment * xxx/xxx deployments (2016/2017) Solution 5 package deployments (input by xxx): 25 hrs saved/deployment * xxx deployments Increased capacity to Innovate (by eliminating Hidden Rework Factory ) 2017 Benefit/Cost ratio: 5.0 Successful pitch to executives triggered funding (and project start)

6 GE Power DevOps Project System of Record: use CA Agile Central (Rally). Manage as Scrum project, just like all deliverable software start with 1-week sprints, evolving Product Backlog DevOps project staffing Initial core team co-located. Responsible for initial implementation and production roll-out/training to ~8 global Scrum teams Long-term staffing model scrum teams become self-sufficient at DevOps. SMEs at each site

7 GE Power DevOps Project (cont ) Infrastructure: Core DevOps team built a test pipeline for automating the release packages, then rolled out to real pipeline through scrum team collaboration Collaboration model between Core DevOps team and Scrum teams: Understand current (manual) release process Re-use test automation (smoke & regression) if present Validate automated deployments Training and production rollout (team ownership) Core DevOps team Scrum team

8 GE Power DevOps Project (cont ) LESSONS LEARNED: 1. Iron out funding logistics early 2. Executive (vocal) sponsorship of project is crucial to success 3. Consider staffing with consultants to kick-start the project 4. Co-locate team members 5. Having a foundation of CI helps a DevOps initiative 6. Manual regression testing is a major drag on release cycle time. Automate everything! 7. Re-deployments and roll-backs should be tracked within a single release 8. Having a test pipeline allows for risk-free development and learning 9. Cross-train to allow for dedicated resources and collaboration 10. Maximize delivery velocity by leveraging Scrum elements like commitments, user story estimates, retrospectives 11. To avoid bottlenecks, ensure that teams have end-to-end DevOps ownership. Do not outsource to a central team 12. Assume that 3 rd -party tool plug-in integrations will require significant customizations 13. Collaborate early with internal IT teams for items like production database hosting and LDAP 14. High-visibility metrics and dashboards are needed to drive continuous improvement

9 GE Power DevOps Metrics Category Metric Level 1 Level 2 DevOps User Story Cycle Time (team commitment to customer availability) X DevOps CI Build Success rate X DevOps Daily CI build counts per scrum team X DevOps Broken build/deployment MTTR X DevOps Automated release pipeline used? X DevOps Release Pipeline - holistic Touch Time X DevOps STAGING/PROD Release Pipeline First-Pass Yield X DevOps STAGING/PROD Release Pipeline re-deploy/fix count X

10 GE Power DevOps Roadmap Achieve CD capability for business Invest in automated regression testing Real-time DevOps metrics Invest in Infrastructure as code Predix Cloud DevOps Current : DevOps production roll-out XLR/XLD automation World-wide staffing model