Case Study: United Airlines Transforms Release Management for Its Modern Software Factory

Transcription

1 DEVOPS CONTINUOUS DELIVERY Case Study: United Airlines Transforms Release Management for Its Modern Software Factory Suzan Liu Principal Engineer eqrm Strategy & Planning United Airlines Rob Karlin Sr. Principal Consultant CA Technologies DO3T013S

2 For Informational Purposes Only Terms of this Presentation 2017 CA. All rights reserved. All trademarks referenced herein belong to their respective companies. The content provided in this CA World 2017 presentation is intended for informational purposes only and does not form any type of warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA. 2 #CAWORLD #NOBARRIERS

3 Abstract Suzan Liu United Airlines Principal Engineer eqrm Strategy & Planning Rob Karlin CA Sr. Principal Consultant Release management plays a critical function at United, but it was hindered by manual processes and spreadsheets. As a part of an IT-wide reliability effort, the company has transformed the role that IT release management plays in the organization. To begin, United performed a governance model and maturity assessment to formally define the service offering, standards and processes. They launched a pilot to establish an onshore/offshore release model, shifting the role of the release team in the process to ensure the right type of work was fulfilled. United moved from multiple spreadsheets to a centralized dashboard to track and manage releases and engagement levels. As a part of the overall transformation, many decisions were needed to ensure success and data was heavily leveraged across all main decision points. In this session, you will learn how transparency around metrics and a strong governance program is transforming United s IT release team. You will also see how CA Continuous Delivery Director is helping United s transformation. 3 #CAWORLD #NOBARRIERS

4 Agenda WHO WE ARE RE-EVALUATE RELEASE GOVERNANCE RELEASE TRANSFORMATION LEVERAGING DATA MOVING TO THE FUTURE: CA CONTINUOUS DELIVERY DIRECTOR 4 #CAWORLD #NOBARRIERS

5 Who We Are 5 #CAWORLD #NOBARRIERS

6 United Airlines At a Glance 2016 AT A GLANCE 339 Destinations (airports served) 216 Domestic destinations (airports served) 122 International destinations (airports served) 50 Countries Served 4,500 Daily Departures We are a 88,000 member team serving 143 Million passengers! 6 #CAWORLD #NOBARRIERS

7 eqrm at United Release Management at United eqrm Facts Internal IT service provider in IT Combines Quality Assurance (300+ people) and Release Management (23 people). Formed two years ago and has rapidly expanded, requiring maximizing each role to keep up with expansion pace. Now supports releases from Dev, Stage, QA, UAT, Prod Perf and Pre- Prod to Production. 7 #CAWORLD #NOBARRIERS

8 Re-evaluate Governance 8 #CAWORLD #NOBARRIERS

9 eqrm Governance Model Reevaluation of eqrm Service Offerings 1 eqrm Operating Model (Organization, Roles & Responsibilities & Governance) Added Release Planning capability Broke out Release Automation from Environment Management 2 eqrm Intake, Estimate and Revision Release Management Release Planning Release Coordination Release Execution Quality Assurance 7 8 Test Planning 9 Test Execution Environment Management 6 Release Automation 10 Test Data Management 11 Metrics & Reporting 9 #CAWORLD #NOBARRIERS

10 CURRENT MATURITY Release Governance Created The Release Management Capabilities And Self-assessed Maturity Governance, Policies and Stakeholder Engagement Mechanism for Managing Release Requests Release Models Risk Factor Checklist Methods for Coordinating and Prioritizing Release Schedule, Models and Teams Methods for Building Testing Strategies Bundling and other Streamlining Methods Method for Validating and Monitoring Closure Process Reference: How IT Operations Can Set up an Effective, Centralized Release Management Process. 03 June #CAWORLD #NOBARRIERS

11 Release Transformation 11 #CAWORLD #NOBARRIERS

12 Defined Release Model Created The Release Management Capabilities And Self-assessed Maturity STEP 1 STEP 2 Understand needs of application, portfolio and stakeholders. Ensure future releases are set up for success RELEASE PLANNING RELEASE COORDINATION For each release, ensure all deployment planning is completed and all prerelease activities and identified and completed STEP 3 STEP 4 Deploy the release according to the plan and manage escalation scenarios when releases do not go according to plan RELEASE EXECUTION RELEASE AUTOMATION Manage the release pipeline from development to production leveraging automation tools Reference: How IT Operations Can Set up an Effective, Centralized Release Management Process. 03 June #CAWORLD #NOBARRIERS

13 Transformation Objectives How It Affects People, Process And Technology People Introduced offshore team Created portfolio release manager role Ensure skillset alignment Raise brand value Process Alignment of process to release areas Portfolio Release Managers -> Release Planning Onshore RMs -> Release Coordination Offshore RMs -> Release Execution Technology Standardized toolset Tool training 13 #CAWORLD #NOBARRIERS

14 Redefined Release Scope Support releases that are higher risk and require strong planning. Characteristics Changes over one hour duration Changes with outages Changes that spans across multiple teams and departments Changes that requires multiple rounds of communication and clear communication The introduction of new functionality and applications High-risk area (top priority incidents) Other Guidelines Capital projects will always be supported Core support work may be supported depending on scope, complexity, and risk Changes that do not allow for strong planning are out of scope (e.g., emergency or incident mitigation) Changes that utilizes less than 75% of release management responsibilities will not be supported 14 #CAWORLD #NOBARRIERS

15 Refined Release Management Responsibilities Release lifecycle oversight / governance Demand management Strategic bundling Release coordination / communication Release calendars and reporting Test acceptance Change management coordination Release automation Test environment management Release execution Post release reviews 15 #CAWORLD #NOBARRIERS

16 Leveraging Data 16 #CAWORLD #NOBARRIERS

17 The Role Of Data In The Transformation WORK ON THE RIGHT THINGS, MANAGE RESOURCES AND CONTROL DEMAND RAISE THE BRAND OF RELEASE MANAGEMENT AT UNITED STANDARDIZE APPS AND HELP THEM MOVE TO AUTOMATION RM involved in forecasting Ability to track on-shore/ off-shore RM hours Measure release output Measure level of engagement and reduce low-value work Track consistent metrics for all the apps Know what a good release looks like Help drive release discipline 17 #CAWORLD #NOBARRIERS

18 State Of Data WHAT WORKED High accuracy in reports information in the reports are trustworthy Reports are automatically refreshed latest information is readily available CHALLENGES Data for release management was available in multiple tools and owned by various departments Change Management Reports PMO reports (resources) Internal team excel reports Intake reports Cross-referencing data from multiple resources was time-consuming and cumbersome and therefore rarely done. WHY SPOTFIRE? United was already leveraging the platform and we had a centralized group of Business Intelligence experts Training to get started only took two days! Rollout United is migrating from HP Service Manager to ServiceNow so creating a reporting database to pull information out of is out of scope for now. Dashboards are managed from standardized Excel reports and manually refreshed daily Adoption of the dashboards was quick Dashboards were fine-tuned over time and end users saw new ways to use the dashboards Future-state: automatically feed data from source systems directly into dashboard 18 #CAWORLD #NOBARRIERS

19 Intake Demand Forecast Created centralized dashboard for the intake process, which included release managers in estimation process CHALLENGE Release estimations were often left out of the process leading to weak release resource forecasts SOLUTION Intake process includes release estimations on all efforts, with more detailed forecasting planned for future Estimates for release managers are broken down by onshore and offshore hours Release Estimates 19 #CAWORLD #NOBARRIERS

20 Release Management Dashboard Release Services Leveraged TO ENSURE WE ARE PROVIDING MEANINGFUL SUPPORT TO OUR ORGANIZATION, LEVEL OF ENGAGEMENT IS MEASURED Opening RFC Cutover Script Comm. Script Review Go/no Go meeting Running deployment Low-value work has been cut out over time, freeing up resources for meaningful engagements 20 #CAWORLD #NOBARRIERS

21 Release Management Dashboard Overall Trends Month-by-month total releases Number of releases by release manager Final release status Reduction in # of releases after narrowing release scope 21 #CAWORLD #NOBARRIERS

22 Release Management Dashboard Overall Trends When releases occur by day of week When releases occur by hour of day How long releases last How long outages last 22 #CAWORLD #NOBARRIERS

23 Release Management Dashboard Specific Trends Expedites vs. Normal releases Risk level of releases If our QA team was involved in the testing Final status of changes Outages associated with releases 23 #CAWORLD #NOBARRIERS

24 Release Transformation Next Steps Move from manual RM process to automated and orchestrated platform that can be leveraged by everyone Automate risk identification and the ability to assess release health for better go/no-go decision making Enable automated, more comprehensive visibility and feedback on testing Leverage continuous delivery pipeline data to improve/mature 24 #CAWORLD #NOBARRIERS

25 Moving To The Future: CA Continuous Delivery Director 25 #CAWORLD #NOBARRIERS

26 Release Automation CA Continuous Delivery Director Drive Continuous Delivery Orchestration OPTIMIZE PIPELINE Continuous Delivery Dashboard PLAN & MANAGE RELEASES AUTOMATED DEPLOYMENT DEV QA/TEST PRE-PROD PRODUCTION 26 #CAWORLD #NOBARRIERS

27 Assess Application Release Risk Improve release governance with the change management organization CA CONTINUOUS DELIVERY DIRECTOR Application Tracking Business Value Productivity Metrics Automate the visibility/feedback of work items tested per release Automate delivery risk identification based on scoring: Application release complexity Interdependencies between releases Code changes Release Execution Conflict Avoidance Process Improvement 27 #CAWORLD #NOBARRIERS

28 Gain Release Health Visibility 28 #CAWORLD #NOBARRIERS

29 Improve Release Management Maturity Provide a consistent release platform for onshore and offshore release teams Implement automated release management workflow and approval Mature processes between release teams and QA 29 #CAWORLD #NOBARRIERS

30 Release Governance Improvements with CA Continuous Delivery Director Eliminating manual and scripted processes Release Planning Identify release size, milestones, and dependencies and assign release manager, manage relationships with CA Continuous Delivery Director Managing Release Requests Create integration in CA Continuous Delivery Director Building Releases Leverage CA Continuous Delivery Director for building release templates and sync with ServiceNow Closure Process Formalize closure process for automated releases: feedback to quality team when release fails using CA Continuous Delivery Director Validating and Monitoring Automated validation of release success through CA Continuous Delivery Director 30 #CAWORLD #NOBARRIERS

31 Summary A Few Words to Review Honest Self- Assessment We had to take our own measure of release management and release maturity leveraging industry standards before we could know how to move forward. Measured Transformation We leveraged data to help track and standardize the work, level set for automation and raise the release management brand. From Manual to Automated CA Continuous Delivery Director plays a big role in moving release management from manual/scripted processes to an automated, orchestrated and measured future. 31 #CAWORLD #NOBARRIERS

32 Must See Demos Automate Your DevOps Ecosystem End-to-End Theater Continuous Delivery for Developers Theater Modern Software Factory in the Cloud Theater #CAWORLD #NOBARRIERS

33 Questions? 33 #CAWORLD #NOBARRIERS

34 Thank You. Stay connected at communities.ca.com 34 #CAWORLD #NOBARRIERS

35 Appendix Release Management Tasks High Level Portfolio and Application Planning FOR EACH RELEASE Release Planning Release Coordination Release Execution Identify technical and business owners and stakeholders Understand if release management services is best applied to the portfolio Ensure stakeholder understanding of release Understand frequency of releases and drivers that affect release frequency Work with leadership of the team to establish frequency of communication and communication format Develop release plan and migration schedule Know tier level of applications Identify critical release tasks and activities Create Master SRPs if needed Understand bundling needs and necessity Manage schedule of releases for each application or portfolio Update centralize release management documentation High level risk mitigation planning Dependencies on other RFCs Determine release scope and impact Track release status, risks and issues Develop deployment script for each SRP Prepare stakeholder and validator list Collaborate with stakeholders to coordinate releaserelated activities Ensure all steps and tasks of the release are well defined Understand and define dependencies between tasks to deploy, go/no go tasks, fallback steps and instruction Identifying verification steps Identify project team members Identifying prep tasks Maintain and distribute status Verify agreed upon Q&A testing signoff Develop release-specific communication Add RFC to Change/Release Calendar Conduct Tier 0 &1 impact analysis Define all must-validate applications and application owners Add incident management plan to script Facilitate go/ no go meetings Follow up on approvals Attend CAB to represent release Conduct deployment-day pre-deployment activities Ensures all prep tasks are complete Communicate to escalation manager release is ready to start (if applicable) Conducts deployment day/night activities according to the developed cutover script Confirms and communicates release completion Track status of release participants Track application validation for all applications status with deployment script Manages release failure or fallback situation Escalate releases that are deviating from the plan cutover script Closes the RFC(s) and document issues within in the RFC while closing out RFC Sends final release communication to all stakeholders identified in the cutover script Supplies improvement suggestions to Enterprise Release Manager Supplies KPI metrics updates for the Release to the Release Management Communicates any deviation from the release plan to Release Manager at the close of the release Work with Change and Problem for any P1/P2 incident review Hold Lessons Learned meetings Upload all archivable information to centralized release repository 35 #CAWORLD #NOBARRIERS

36 DevOps: Continuous Delivery For more information on DevOps: Continuous Delivery, please visit: 36 #CAWORLD #NOBARRIERS