From Continuous to Autonomous Testing with AI

Transcription

1 COGNIZANT INSIGHTS From Continuous to Autonomous Testing with AI Continuous testing, or DevOps embedded with QA, helps organizations keep pace with market dynamics. Artificial intelligence can augment testing to be autonomous and zero touch. Executive Summary Success in the dynamic digital age often depends on how well businesses can keep their applications up and updated. This creates pressure on IT teams to be nimble and agile faster than ever before to accommodate changes in business requirements. Hence DevOps, the Agile practice that builds in speed by fostering collaboration, is fast-becoming the de facto model of software delivery. In fact, 57% of organizations, surveyed by the Everest Group for its report, Quality Orchestration: QA for the Digital Era, said they have or plan to initiate a DevOps project; moreover, 34% claim to be scaling up their DevOps 1 programs. Embedding quality assurance (QA) in DevOps gives rise to continuous testing which swiftly ensures code changes are integrated effectively and efficiently. Moreover, continuous testing democratizes quality by iterating QA across the software development lifecycle, driving quality at speed. Cognizant Insights November 2018

2 However, continuous testing is often riddled with bottlenecks, such as siloed automation, a lack of end-to-end visibility of requirements, a high volume of tests, etc. This white paper recommends ways of dealing with these challenges. We advocate an artificial intelligence (AI)-based approach that builds on machine learning (ML) to orchestrate quality across the continuous testing pipeline, enabling an autonomous and zero touch QA. AI FOR CONTINUOUS TESTING As the embodiment of human judgment, AI can smoothen the continuous testing process by eliminating manual intervention. With AI, QA teams can trigger unattended test cycles, where defects are identified and remedial measures are triggered in run time, based on insights gleaned from historical data sets and past events. This way, the AI engine will ensure that only a robust code progresses from one stage to the next, orchestrating quality across the software development lifecycle. Orchestration of QA Processes Though most of the QA activities are automated in continuous testing, the code still needs to be manually signed-off from one quality gate to another, based on test results. This siloed automation, or automation lakes, gives way to the accordion effect, or a disrupted flow of elements due to the stagnation in the pipeline. AI-Driven QA Checkpoint Coding Continuous Development GitHub AI/ML Decision- Making Code Quality Continuous Inspection Sonar, Fortify AI/ML Decision- Making User Stories Commit Trigger Compile & Build AI/ML Decision- Making Jenkins Maven Automated Deployment Continuous Verification Deploy Infra, App & Monitors Functional Validation Performance Validation Security Validation Build Artifact Build Repository AWS Azure Google Deployment Success Quality Gate Quality Gate AI Driven KPI Measurement, Reporting and Dashboard ML-Based User Analytics ML-Based Dashboard AI Continuous Monitoring Figure 1 From Continuous to Autonomous Testing with AI 2

3 As ML algorithms determine if the code is a go or no go based on historical data, QA teams can entrust the AI engine to promote the code or shut down features with high probability of causing application outage or production defects. An AI engine assumes the task of checking-off code at quality gates, making this process autonomous. By analyzing the results of these automated tests, an ML algorithm can pass or fail code progression, creating a fully automated workflow. By orchestrating QA processes with AI, QA teams can: Automate quality gates: As ML algorithms determine if the code is a go or no go based on historical data, QA teams can entrust the AI engine to promote the code or shut down features with high probability of causing application outage or production defects. Predict root causes: Triaging, or identifying the root cause of a defect, is one of the reasons for delays in releasing new features. With patterns and correlations, ML algorithms can trace defects to root causes, with the AI triggering remedial tests before the code progresses. As AI takes these judgment calls, the margin of error is significantly reduced. Leverage precognitive monitoring: ML algorithms scout for symptoms in coding errors that were previously overlooked. The algorithm can then flag these symptoms, such as a high memory usage, as a potential threat that could result in an outage. As corrective steps, the AI engine can automatically spin-up a parallel process to optimize the server-resource consumption. Orchestration of QA Tool Chain Each digital journey is unique, and thus all IT and business requirements must be contextualized. This has a significant bearing on the tool estate, especially for QA, a tool-intensive activity. For instance, every time the developer commits a change in the code, the QA teams leverage different tools to validate the code for various aspects such as performance, security and functionality. Often, open-source and commercial-off-the-shelf products are preferred for flexibility and scalability. For instance, for scalability of pipelines at an enterprise level, an orchestration tool such as Concourse is a better fit than Jenkins, since it handles pipeline definition more easily. At an enterprise level, when various tools come together to execute several QA activities, the tool estate expands beyond a point where it could be integrated for ease of use. Timely commission of the right tools, however, becomes a challenge. Moreover, in our experience, 80% of tools that an organization leases are used only 20% of the time. From Continuous to Autonomous Testing with AI 3

4 This necessitates the presence of a central authority to orchestrate tools based on project requirements, domain, automation coverage, etc. The process can be eased by feeding an AI engine with data of deployment history in order to orchestrate tools. This approach can be helpful by: Making QA seamless: As the AI chooses the best-fit tools, based on historical data, it manages to line up tools in advance based on an upcoming requirement and unclogs the delivery pipeline. This enables uninterrupted testing. Ensuring cost optimization: As AI takes over the commissioning of tools, it can abstract basic functions and help QA organizations shift from proprietary to open-source tools, optimizing licensing or acquisition costs. Toolchain Orchestration GitHub Updates on Automation Scripts Appium Apache JMeter HP UFT CX Optimize Se CRAFT Protractor Postman Cucumber Dev Pushes Code Code Quality Analysis Unit Test/ Code Coverage Build Artifacts & Deploy Provisioning Data/Environment BVT/ Smoke Test Inspected with Sonarqube TEMS One TDM Functional Test System Test System Integration Test Regression Test Nonfunctional Test Performance Test Security Test Accessibility Test Jenkins Concourse Figure 2 From Continuous to Autonomous Testing with AI 4

5 Quick Take Supervised-Learning Algorithms Help a U.S.-Based Media & Communication Services Provider Assure Cloud Performance To account for sudden upticks in user load, our client wanted to assure performance of its customer-facing applications hosted on a third-party cloud. The QA team created a smart monitoring system, powered by supervised learning algorithms, which was fed with a series of inputs and desired outputs for performance validation tests. Based on this data, the smart monitoring system ran unattended sanity checks in the background, commissioning tools and tests, to validate performance across varying user loads, central processing unit utilization, etc. The system was then able to independently flag potential threats to performance, and trigger remedial steps to avert a dip in performance. This helped the client save 73% in QA costs, as well as assure component-level performance to meet defined service-level agreements. From Continuous to Autonomous Testing with AI 5 1

6 The new-age QA team engineers quality from the start, rather than testing it in. This requires a working knowledge of coding and user expectations. Augmenting Natural Intelligence with AI In the aforementioned Everest report, the consultant notes the topmost hindrance in driving quality at speed is the lack of industry-specific, technologically sound QA expertise. To assure quality in digital, QA professionals need to upskill with: Domain knowledge: A deep dive into the domain will help in contextualizing clients needs. Business knowledge helps QA professionals assure business requirements, regulatory compliance and business-critical areas that go beyond application functionality. Technological know-how: QA needs next-generation technologies, such as AI, to deliver zero-defect code. QA professionals should be armed with the knowledge to use these new technologies to deliver maximum value for clients. Full-stack quality engineers: The role of QA professionals now shifts from custodian of quality to orchestrator of quality. The new-age QA team engineers quality from the start, rather than testing it in. This requires a working knowledge of coding and user expectations. Together with AI and the natural intelligence of QA teams, organizations can deliver defect-free codes to production. THE WAY FORWARD: TOWARD ORCHESTRATION-AS-A-SERVICE As the QA goalposts shift, the focus of quality assurance 2 is elevated to brand assurance. This is possible with AI where automation is made intelligent and pervasive, and human judgment is augmented to address QA complexities. With AI, QA teams can ensure businesses stay relevant by stitching together insights gleaned from customer inputs and business acumen. Before leveraging AI to drive QA in digital, however, organizations need to invest in technology and training. Human testers are required to encode the AI with business process flows and critical scenarios. In addition, the knowledge of application development lifecycles is crucial to orchestrate quality. This proposition, when abstracted to a commercial model, allows enterprises to lease AI engines that have been trained and tested to deliver orchestration-as-a-service. QA teams must evaluate the enterprise s business challenges and technology landscape to identify the right AI technology (e.g., ML, conversational AI or natural language processing). A second layer of evaluation would require assessing the enterprise s data logs for relevance and richness, and the enterprise s automation maturity. Unless the enterprise maintains data from various processes or has significantly high levels of automation, AI engines will not take off. Once these check-boxes are ticked, the enterprise can look to onboard an AI engine to orchestrate processes and toolchains. From Continuous to Autonomous Testing with AI 6

7 ENDNOTES ABOUT THE AUTHOR Anant Hariharan Senior Director & Technology Leader, Cognizant s Quality Engineering & Assurance Practice Anant Hariharan is a Senior Director and Technology Leader within Cognizant s Quality Engineering & Assurance Practice. As a seasoned technology leader with proven enterprise quality engineering delivery expertise, Anant has experience implementing end-to-end technology stacks across DevOps, QA automation, continuous integration, big data testing, test data management and service virtualization. His background spans consulting, technology solutions, client relationship management, delivery management, large program management and business development. Anant is a renowned speaker in industry conferences including STAR and Quest, and has guided research publications by technology architects from his team. He can be reached at Anant.Hariharan@ cognizant.com. From Continuous to Autonomous Testing with AI 7

8 ABOUT COGNIZANT QE&A Cognizant Quality Engineering & Assurance (QE&A) helps businesses succeed in digital with an industry-aligned digital assurance proposition. With 800-plus clients across industry verticals and a global footprint, Cognizant is a recognized market leader in Quality Assurance. Cognizant s deep business and technology expertise helps our clients drive quality at speed with Zero-Touch QA. Cognizant s QA Hub ecosystem accelerates innovation by bringing together partners and communities to get quality right, the first time. Learn more at ABOUT COGNIZANT Cognizant (Nasdaq-100: CTSH) is one of the world s leading professional services companies, transforming clients business, operating and technology models for the digital era. Our unique industry-based, consultative approach helps clients envision, build and run more innovative and efficient businesses. Headquartered in the U.S., Cognizant is ranked 195 on the Fortune 500 and is consistently listed among the most admired companies in the world. Learn how Cognizant helps clients lead with digital at or follow World Headquarters 500 Frank W. Burr Blvd. Teaneck, NJ USA Phone: Fax: Toll Free: European Headquarters 1 Kingdom Street Paddington Central London W2 6BD England Phone: +44 (0) Fax: +44 (0) India Operations Headquarters #5/535 Old Mahabalipuram Road Okkiyam Pettai, Thoraipakkam Chennai, India Phone: +91 (0) Fax: +91 (0) Copyright 2018, Cognizant. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any means,electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Cognizant. The information contained herein is subject to change without notice. All other trademarks mentioned herein are the property of their respective owners. TL Codex 3910