DevOps and the Engineering-Led World of the Digital Enterprise

Transcription

1 DAITAN WHITE PAPER DevOps and the Engineering-Led World of the Digital Enterprise HOW DEVOPS IS ENABLING ENTERPRISES UNDERGOING DIGITIZATION TO INNOVATE, COLLABORATE AND SPEED TIME-TO-MARKET White Paper Contents Enterprise Adoption of DevOps The Five Core Principles of DevOps Getting Started with DevOps DevOps in Practice: Lesson Learned The digitization of enterprises of all sizes, and in all industry verticals, requires entirely new approaches to coding and releasing software enhancements to the customer. Market pressures over the past ten years have forced companies into a faster, nimbler software release mindset. It used to be the norm that companies delivered software on 9-month delivery cycles, with 3-month integration and regression test cycles, only to find out that what was delivered at the end of that cycle was not what the customers wanted, or needed. Today the challenging, fast-moving and competitive marketplace requires that enterprises embrace innovation, collaboration and experimentation. Or they will fail. Today s successful digital enterprises like Amazon and Netflix deploy thousands of times a day. Others, like Etsy, deploy up to 80 times a day. It takes them less than an hour to deploy code to production, whereas companies at the other end of the scale may take one to six months to deploy. While few enterprises need the kind of scale, speed and frequency of deployment demanded by Amazon or Netflix, there are nonetheless lessons to be learned. These companies have embraced agility and DevOps methods into their engineering processes to allow them to benefit from experimentation without increasing risk. They understand, and embrace, the fact that some experiments will fail, and are able to limit the number of consumers exposed to those failures. Because they understand that the experiments that succeed bring benefits that far outweigh the impact of those that fail. There is much to be learned from this. 2 Daitan Group 2016 Accelerating. TM

2 High performers (companies using DevOps practices) deployed code 30 times more frequently, and the time required to go from code committed to successfully running in production was 200 times faster high performers had lead times measured in minutes or hours, while low performers (non- DevOps) had lead times measured in weeks, months or even quarters. Furthermore, high performers were twice as likely to exceed profitability, market share and productivity goals. THE DEVOPS HANDBOOK HOW TO CREATE WORLD-CLASS AGILITY, RELIABILITY, & SECURITY IN TECHNOLOGY ORGANIZATIONS. CO-AUTHORED BY DEVOPS INDUSTRY THOUGHT LEADERS --- GENE KIM, JEZ HUMBLE, PATRICK DEBOIS, AND JOHN WILLIS. THE BOOK IS PUBLISHED BY IT REVOLUTION Daitan Group 2016 Accelerating. TM

3 DevOps Not Just for Unicorns DevOps is a change in IT culture, focusing on rapid IT service delivery through the adoption of Agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective. GARTNER GROUP, DEFINITION OF DEVOPS DevOps isn t just for Unicorns. And it isn t just for technology companies. Because every company is effectively becoming a technology company. DevOps is for any enterprise embracing their digital future. Because they all have common demand: a faster time-to-market, reduced costs, and better quality and stability of output. Done right, DevOps can deliver on this. DevOps is the natural extension of the Agile software development revolution which completely changed how software was developed and delivered. But the market has seen that it wasn t enough to embrace Agile in software development. Today s enterprise needs a holistic approach to engineering one that embraces Agile from idea right through to continuous delivery to the customer whether that customer is inside, or outside, the enterprise. DevOps doesn t replace Agile. DevOps methods embrace Agile software development processes, and extends them to encompass the building, validation, delivery and deployment stages of software and brings the Agile mindset to IT processes and systems. DevOps methods encourage IT teams to collaborate, communicate and embrace automation methodologies. They allow IT teams to deploy enhanced products to the consumer continually, rather than just once every few months. DevOps product teams don t separate QA and deployment processes from the development process. DevOps teams integrate them all together. We re going to need to think more like technology companies and maybe a little less like banks. RICHARD FAIRBANKS, CEO, CAPITAL ONE, APRIL Daitan Group 2016 Accelerating. TM

4 Enterprise Adoption of DevOps DevOps adoption is growing. A recent Gartner survey of enterprise IT organizations 1 indicated that 29% of respondents are either using, or piloting, DevOps approaches to IT. Of those that are embracing DevOps, 70% are using it for mission-critical services. The same survey found that 32% of respondents put time to market for internal customers as their top driver for adoption of a DevOps approach to IT, and 20% of respondents put time to market for release to external customers as their top driver. Reports consistently show that DevOps reduces mean time to value (MTTV). Other drivers for adoption included: enterprise impact and value, and IT cost savings. 1 Gartner, DevOps Adoption, September 29, Survey of research circle and DevOps user group members. When DevOps methods are integrated into IT engineering projects, three main benefits are seen: 1. Improved and enhanced communication with enterprise leaders, 2. Enterprises are able to explore and experiment with new ideas that have positive enterprise value, without introducing excessive risk, and 3. Enterprises see faster release cycles, increased stability, and reduced time to market But adoption isn t without challenges. The Gartner survey indicates that an attitude of resistance to change is the leading reason for not expanding further with embracing DevOps, followed by skills gaps. Interestingly, these are similar reasons for resistance to adoption of Agile methodologies, as VersionOne reports in its latest State of Agile Report 2 below. 2 VersionOne, 10th Annual State of Agile Report, While Agile adoption is increasing, there are still obstacles to overcome. The key barriers to further adoption usually hinge around culture, including the ability to change, general resistance to change, and management support. Interestingly, the majority of respondents pointed toward company culture as the reason for failed Agile projects as well. Once these barriers are overcome, the limiting factor most often cited has been availability of personnel with the necessary Agile experience. VERSIONONE, 10TH ANNUAL STATE OF AGILE REPORT, 5 Daitan Group 2016 Accelerating. TM

5 DevOps and the Path to BiModal IT BiModalIT Involves the development of capabilities to enable two types of enterprise. Mode 1 has to do with capabilities relating to known requirements or otherwise operating in a known state. Mode 2 is for supporting innovative exploratory work where requirements are largely unknown and rapidly evolving. BIMODAL IT DEFINITION The market is seeing that enterprises cannot wait to embrace their digital future, and in order to do that they must embrace innovation. Innovation means experimentation. Experimentation means testing with customers (internal and/or external), iterating, learning and trying again. None of this can happen with old waterfall development methods where software releases happen over a period of months. It also cannot happen with IT methods that push new code slowly, with a high rate of failure. This has led to what Gartner has termed BiModal IT. Gartner considers a BiModal approach to IT to be mandatory for today s drive towards enterprise digitization. Mode 1 projects are increasingly being separated from the innovative Mode 2 projects. The two modes are separate, but within themselves completely coherent. Mode 1 projects may focus on the management of decades-old legacy systems which cannot be easily or quickly transformed. They include big deployments of ERP systems, payment systems, HIPAA-compliant electronic health records, or financial systems that include data upon which the whole business relies. Mode 1 development pathways are more predictable and build on clearly understood product requirements and service delivery mandates. Mode 2 projects, on the other hand, are more exploratory. They embrace uncertainty and champion experimental development. Agile, DevOps, containers, open source platforms can be found at the heart of Mode 2 IT projects. Mode 2 IT projects are where enterprise transformation is delivered. 6 Daitan Group 2016 Accelerating. TM

6 The Five Core DevOps Principles Just like discussions of Agile methodology (the Agile Manifesto), it is as important to understand the principles of DevOps as it is to understand what DevOps means in practice. There could be many examples of DevOps practices practices that embrace a systematic approach, with iterative design processes, continuous deployment, using automated tools, by collaborative teams. But all those practices draw upon five generally recognized principles in the DevOps model: Iterative: using Agile principles to iterate towards a goal; this works well for development projects where the final outcome isn t necessarily fully defined as opposed to in waterfall methods, where the outcome is required to be defined at the start. Iterative principles allow projects to change just enough based on feedback. Continuous: in an ideal world, a DevOps project allows both continuous delivery into a staging environment, and continuous deployment into a production environment. This is reflected in the reality of today s technology world. Products and services never stand still. They always change and improve, and that should be expected, and planned for. Collaborative: DevOps teams rely upon one another and trust that each member is pulling their own weight towards the end goal they collaborate and communicate frequently and transparently. Systematic: in a well deployed DevOps environment, teams address processes systematically to maximize efficiency. Automated: the automation of technology underpins and supports all other DevOps principles. Allowing teams to rely upon automated testing and deployment enables products to be delivered reliably, at scale. Without automation, continuous development and deployment is impossible. 7 Daitan Group 2016 Accelerating. TM

7 The ROI and Reward of DevOps High-performance DevOps-enabled organizations outperform their peers 200x the number of releases, and 24x faster recovery time The reward of deploying DevOps methods to reduce time from idea to delivery is immense. But not without challenges. The 2016 State of DevOps Report 3 survey has been running since 2012, and has followed more than 4,600 technology professionals. In 2016, the survey year focused on ROI, measuring performance and business impacts that include: Deployment frequency how often does the organization deploy code. Lead time for changes how long it takes from code commit to code running in production. Mean time to recover how long does it take to restore service after an unplanned outage or service issue. Change failure rate the percentage of changes that result in degraded service or issues that require fixing. The results were notable. The survey quotes the following results. High-performing IT organizations deploy 200 times more frequently than low performers, with 2,555 times faster lead times. They have 24 times faster recovery times and three times lower change failure rates. High-performing IT teams spend 50 percent less time remediating security issues. And they spend 22 percent less time on unplanned work and rework. Employees in high-performing teams were 2.2 times more likely to recommend their organization as a great place to work. Taking a lean approach to product development (for example, splitting work into small batches and implementing customer feedback) predicts higher IT performance and less deployment pain State of DevOps Report, puppet + DORA: 8 Daitan Group 2016 Accelerating. TM

8 DevOps in Practice: Lessons Learned People, Technology, Process, Culture Canary Rollouts an approach to software deployment that leverages the canary in the coal mine metaphor. The goal is to identify a problem as early as possible, and impact the fewest number of customers. In addition, this practice potentially enables an easier recovery, should the need occur. CANARY ROLLOUT DEFINITION, GARTNER Enterprises embracing their digital enterprise present and future are organizing their DevOps initiatives around four key pillars: People/Teams: reflected in new types of functions, such as Site Reliability Engineers, full stack teams, autonomous two pizza teams, feature teams and more. Technology: embracing open source platforms as well as Agile-focused tools such as ChatOps, to enable fast time to market. Systematic Processes: reflected in an approach that uses continuous delivery, a complete embrace of automated build, test and deploy, Fail Forward and Canary Rollouts where everything is instrumented, tested and monitored. Culture: that puts a premium on collaboration and trust, centered on an engineering, rather than traditional IT culture. Fail Forward an approach to software deployment whereby a production environment is provisioned alongside the existing, working production system. If a problem occurs using a canary rollout as described above, the new environment is systematically removed from the rotation no code is removed from the working production environment because backing out code from production is viewed as risk prone. Begin with the end in mind. STEPHEN COVEY, THE 7 HABITS OF HIGHLY EFFECTIVE PEOPLE FAIL FORWARD DEFINITION, GARTNER At Daitan, our teams have real-world, hands on experience with DevOps approaches to the Mode 2 IT projects described in this white paper 9 Daitan Group 2016 Accelerating. TM

9 Getting Started with DevOps STEP 1: REVIEW & SELF-ASSESSMENT At Daitan, we have real-world experience with the challenges, and rewards, of DevOps approaches Mode 2 IT projects. We have seen the reality of IT engineering teams working to embrace engineering culture, and we understand and champion their need to embrace robust programming skills. Because our enterprise clients are demanding increased deployment speed, reduced failure rate, less bugs, and improved team motivation and productivity they see that DevOps methods are delivering on those demands. In all instances, we counsel our clients to always start with the end in mind. Embracing DevOps can be a long and complex journey, full of change and uncertainty. But the rewards are great. This section of the White Paper outlines some of the specific advice our clients have found useful on this journey. At Daitan, we start by working with our clients on a series of selfassessment questions that allows us to review needs. This self-assessment typically begins with the following three key questions. From there proceed to a more detailed analysis. The three questions we start with are: 1. What current Agile practices and tools do you already have in place? 2. What are your release goals and how fit is your architecture for DevOps? 3. Are you building a new team, or evolving an existing team s practices? These questions are relevant whether we are renovating and upgrading existing systems to be DevOps feasible, or initiating brand new systems. 10 Daitan Group 2016 Accelerating. TM

10 Review question 1: Current Agile Practices Self Assessment: Current Agile Practices It is important to start by understanding what exactly is in place already, and how much needs to change in order to meet the software release goals. That estimate should be as accurate as possible, with an inventory of all the tools currently being used that inventory will help with the selection of the group s DevOps Toolchain the combination of tools that enable the development, delivery and deployment of applications throughout the software development lifecycle. Stages in a DevOps Toolchain Image courtesy of: 11 Daitan Group 2016 Accelerating. TM

11 Review question 2: Release Goals and Architecture Requirements Self Assessment: Release Goals and Architecture Requirements The infrastructure and software architecture used influences how far an enterprise can go with DevOps. In discussing this topic with clients, we review: The impact of a modular, if possible microservices-oriented architecture to allow more frequent delivery and deployment; this is often a better approach than employing a monolithic architecture. The comparison of physical versus virtualized machines. Physical (bare metal) machines make it harder for teams to provision and configure environments. Where possible we recommend virtualization and containers, unless there are very specialized and specific hardware performance requirements that make this impossible. Of course, we are also faced with special requirements that mean different approaches are needed. For example, there may be special code review requirements or manual testing procedures that have to be complied with which means that systems have to be less automated. But generally we try to approach software architecture recommendations that include a modularized and virtualized approach. Review question 3: Team Experience & Culture Self Assessment: Team Experience and Culture Our experience has shown over and again that there is frequently a resistance to change mindset when moving even experienced Agile teams to embrace DevOps methods. This issue cannot be overlooked. Our advice generally includes: Anticipate pain especially the pain that will be experienced by the Ops teams clear and understood by everyone. Pain will occur. Expecting and planning for it helps. Make developers accountable everyone should understand the enterprise impact of service downtime, and should be on the same side of the table in helping the Ops teams address and recover from issues. Educate and evangelize resistance can come simply from fear of the unknown. Embrace education. Evangelize new tools. Help every member of the team step up and raise their level of coding skills. Share and model good programming and architecture patterns and practices, particularly when those patterns and habits impact Ops teams. Once the self-assessment is complete, we move onto: 1. Technology and process requirements, and 2. People and culture assessments 12 Daitan Group 2016 Accelerating. TM

12 Getting Started with DevOps STEP 2: REVIEW TECHNOLOGY, PROCESS, PEOPLE AND CULTURE Technology and Process Experience has shown us that technology and process must evolve together towards DevOps. We have seen over and again the significant team benefits that result when routine and error-prone tasks are automated. We see that many software organizations have already implemented processes and tools for the Coding and Building ( Create phase) but less in the subsequent phases of delivery. Automation Advice Our clients also typically though not always have test automation already in place, which is part of the Verify phase. And when that is the case, we recommend addressing automation of the Package and Release process as the next step. This is a common pain point for enterprises trying to get to far more frequent customer releases. Toolchain Advice Our next area of advice is around the right Toolchain for the team and the processes needed by those teams. We often see that teams want to use known tools because there is a higher degree of comfort with something that is known and understood. There is natural resistance to abandoning a tool, with all the builtup knowledge and experience of working with that tool. Our advice is usually to establish a team to decide tools, rather than relying on one person s experience and knowledge. It helps with buy-in, and expands the knowledge base to a wider group. However, the choice of tools can be overwhelming. Blogs, analysts, and forums can be scoured for the latest insights. But those online resources can potentially list dozens of tools that may appear to perform overlapping functions. Our recommendation is usually to pick three that appear to fit the requirements and to get the entire team to assess pros and cons. Those pros and cons should include an assessment of fit with the current architectural environment, the use or migration of existing scripts, and a look at how large and active the support community appears to be. More than once we have experience of clients selecting great tools, but finding that there are few, if any, engineers experienced in using those tools, making ramp up time challenging. For example, we frequently work with our clients to compare Ansible to Chef, SaltStack and Puppet for Continuous Configuration Management. All these tools are excellent, but specific environments may benefit from using one or another. For example, where clients may already have a large library of existing Python scripts in place, or at least some engineers fluent in Python, Ansible usually comes out as a better choice for saving ramp up time, while Chef, considered a mature and very scalable tool, appeal more to Ruby practitioners. SaltStack and Puppet also work well in other specific situations, according to our experience 13 Daitan Group 2016 Accelerating. TM

13 People and Culture Our experience has shown that it is highly desirable, in some circumstances, to bring on an engineer with specific skills in automation and orchestration because this will immediately help relieve that burden from the rest of the team. But this can result in a short term gain only. The final goal is to have everyone on the product gain DevOps skills. This takes time. There s a learning curve. The automation engineer(s) should not be the only members of the team able to perform those tasks indefinitely. The goal is to have everyone on a team gain DevOps skills. But this takes time. Whereas a skilled automation engineer may take 2 hours to create and configure the code and server for an Ansible-based automation, an engineer who is less familiar may take a full day. This is not wasted time. Investing this time makes the team overall stronger and more reliable. In some situations, making engineers accountable for rolling out their features to production, end-to-end, is a good way to motivate them to be more proficient in DevOps tools. Teams should be encouraged to embrace these new skills rather than automatically relying on the automation engineer or the DevOps engineer to solve the task, just because it s easier and quicker. Without doing this, the automation engineer will be overloaded with tasks big and small, and it will reduce motivation as well as efficiency in the whole team. To avoid that, reinforce the team s goal by providing live feedback to the Dev engineers on the automated build-test-deploy pipeline status and the production environment status to make it clear that the service is the end product (not the code or the infrastructure). One way we work with our clients to achieve that is adopting ChatOps - an emerging trend in which a chat room can be used to automate tasks with special commands and also receive alerts raised by bots, which can detect anomalies in the infrastructure or in the application behaviour and report it directly to the chat room, where both Dev and Ops teams may discuss the issue and how to resolve it, together. Many of our customers also implement application performance monitoring tools to provide analytical insights around the customers experience to help the business as-a-whole, not only the Dev and Ops team, understand how their products perform. 14 Daitan Group 2016 Accelerating. TM

14 Conclusion Enterprises embracing the digitization of their enterprise are looking for partners with an engineering mindset who know how to bring in collaborative Agile teams with the very latest knowledge of open source platforms and DevOps Toolchains. Our teams at Daitan understand application architecture, development, infrastructure and networks and can share case studies of rapid deployments of large-scale systems that have accelerated our clients time-to-market. We welcome you to contact us if you have more detailed questions, or if you want to talk over how best we can help you with your Mode 2 DevOps projects. About Daitan Group Daitan Group provides high quality software development services to significantly accelerate time to market for global technology companies. The company s expert Agile teams deliver full lifecycle software product development, maintenance and quality assurance services across today s leading technologies, including: cloud computing and virtualization; communications, collaboration and messaging; and big data/analytics. For more information: DAITAN GROUP HEADQUARTERS 2410 CAMINO RAMON, SUITE 285 SAN RAMON, CALIFORNIA, USA PHONE: +1 (925) FAX: +1 (925) Daitan Group 2016 Accelerating. TM