In-Process Automation

Transcription

1 In-Process Automation business assurance & testing Author: Sajitha Radhakrishnan SL CORE APPS-TESTING-TESTING-FT-BFSI HCL Technologies Ltd., Bangalore, India WHITEPAPER December 2014

2 TABLE OF CONTENTS INTRODUCTION 3 IN-PROCESS AUTOMATION THE CONCEPT 4 STANDARD REGRESSION AUTOMATION APPROACH IN-PROCESS AUTOMATION APPROACH IN-PROCESS AUTOMATION LIFE CYCLE IN-PROCESS AUTOMATION ADVANTAGES CHALLENGES FACED IN-PROCESS AUTOMATION CRITICAL SUCCESS FACTORS HCL S IN-PROCESS AUTOMATION TOOL 7 IN-PROCESS AUTOMATION CASE STUDY 8 CONCLUSION 9 ACKNOWLEDGEMENT 9 BIOGRAPHY 9 ABOUT HCL , HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 2

3 INTRODUCTION The World Quality Report, published yearly by Capgemini, Sogeti and HP, is recognized as the IT industry s largest research study. This report surveys the current quality and testing practices used across the globe as well as the emerging trends that may affect the future of testing. Some of the key findings of the World Quality Report are: y Quality Assurance functions are becoming structurally more mature number of organizations with a fully-functional Testing Center Of Excellence increased from 4% in 2011 to 26% in 2014 y Organizations continue to increase the proportion of their IT budgets for testing from 18% in 2012 to 26% in This is expected to rise to 29% by 2017 y The share of testing budget for transformational projects was 41% in 2012, which increased to 46% in 2013 y QA teams are still engaged too late in the application development life cycle, which contributes to the increase of testing s share within IT budgets to manage operational and quality inefficiencies All the findings from the report point to the fact that while testing as a function is growing, the IT industry is concentrating on improving the time to market and reducing the QA budget. The market is increased test automation and automating early in the application development process by using testing services. The report also mentions that 40% of the global QA budget was estimated to be spent on functional testing, where only 20% was spent on regression testing and the remaining 20% on other testing activities. In traditional automation projects, the testing team has focused on automating only the regression tests. With the IT industry focusing on cost optimization, it has become imperative to focus on functional test automation. Even if we succeed to achieve 20% reduction in functional test execution effort, we could save millions of dollars. The In-Process Automation approach proposes a shift left paradigm to the Software Development Lifecycle by automating the functional test cases instead of regression tests. This helps target 60% of our testing spend, rather than the traditional 20%. The automation development process runs parallel to the application development phase and the scripts get executed during the first execution cycle in the System Integration Testing (SIT) phase to reduce the testing cycle time. 2014, HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 3

4 IN-PROCESS AUTOMATION THE CONCEPT To understand the In-Process Automation approach better, let us look at the standard regression automation approach. STANDARD REGRESSION AUTOMATION APPROACH Previous Release Current Release UAT Release & Maintenance Analysis Design Coding CIT SIT UAT WEEK Test case preparation from manual team 3 Scripting of regression Scenarios for previous release 4 Execution of Automation Regression pack 2 Identification of test cases for Regression testing from the manual test cases Tweaking should be done for the scripts before execution if there are any changes from the previous build to the current build Cycle should repeat for CIT, SIT and UAT The above diagram describes the traditional automation approach of automating the regression test cases. Once the functional testing of an application is completed and signed off for a release, the manual team identifies the test cases to be added to the regression suite from the functional test suite. These test cases are then handed over to the automation team that conducts the feasibility and automates the regression suite during the development phase of the next release. This automation regression pack is then used for regression testing in the subsequent testing phases of the release. The automation is termed Lagging Release Automation. Now let us see how the In-Process Automation approach is implemented. IN-PROCESS AUTOMATION APPROACH Analysis Design Coding CIT SIT UAT Release & Maintenance WEEK Entry criteria Suspension criteria Exit criteria 1 2 Identification of functionalities for automation Framework definition 3 4 Feasibility study on Application Scripting of 1 st module 4 5 Scripting for n th module Dry run of (n-1) th module Scripts 6 7 Execution of entire Automation pack Cycle should repeat for all the modules Execution of entire Automation pack 8 9 Execution of entire Automation pack Execution of Automation Regression pack In this approach, the automation development life cycle runs parallel to the application development life cycle. During the analysis and the design phase, the functionalities that need to be automated are identified and the automation framework is defined 2014, HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 4

5 and developed. If an existing framework is available, any necessary modifications are also completed. Once the application goes into the coding phase, the feasibility study of the application is conducted and the scripting begins. It continues throughout the coding phase and once the application reaches the Component Integration Testing phase, the first time execution of the automation pack is conducted. The necessary modifications are done with respect to the application and then the automation pack utilized for the subsequent phases of the life cycle. The same automation pack can then be utilized as the regression pack, providing maximum testing during every application release. IN-PROCESS AUTOMATION LIFE CYCLE To make this approach a success, a set of activities recommended to be performed during the course of application development life cycle. These are: 1. Understanding the business flow during the requirements phase a. Static testing and analysis of the Business Requirements Document (BRD), Functional Specifications Document (FSD), and technical specifications b. Understanding the functionality and deciding on the modules to be automated c. Creating a common test template to be used for automation Pre-conditions for the above mentioned activities to be conducted during this phase are: a. Signed-off technical and functional specification documents 2. Develop/ modify the automation framework during the design phase a. Analyzing the system design b. Creating the test scenarios c. Designing the framework d. Identifying the functional flow and object details Pre-conditions: a. Module prioritization information b. Design documents and the application flow with the object names c. An application prototype if available 3. Develop the automation scripts during the application development phase a. Use of programming approaches like descriptive programming to complete the development of automation framework b. Automation scripting with the object details and design flow available from the earlier phase Pre-conditions: a. Stable test environment b. Software for the agreed modules c. Application queue and database details, if applicable 2014, HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 5

6 4. Perform a dry run of the automated scripts during the unit testing/ Component Integration Testing (CIT) phase. a. Executing the script on developed modules, as and when the automation scripting completes b. Preparing the test data for the modules under development Pre-conditions: a. Stable test environment 5. Perform automated functional testing during the SIT phase a. Reworking on the framework and the test scripts for any gaps identified in correspondence to the application under test b. Complete the functional automation testing and share the results c. Analyze and decide on the automation scripts that can be used by User Acceptance Testing (UAT) team Pre-conditions: a. Stable test environment b. Faster defect turnaround 6. Maintenance and execution of the automation scripts in the subsequent phases and across releases a. Select the tests to be reused for regression b. Perform the regression tests for subsequent releases Pre-conditions: a. Stable test environment b. Faster defect turnaround IN-PROCESS AUTOMATION ADVANTAGES 1. Achieve early test automation, resulting in reduced manual testing effort 2. Increased test coverage caused by high volume of testing through automation 3. Reduced time to market/ cycle time caused by automation 4. Reusability of automated test scripts across CIT, SIT, UAT, and regression 5. Early defect detection during functional testing 6. Increased regression scope with functional test automation CHALLENGES FACED 1. High script maintenance effort for In-Process test automation 2. Frequent design changes impact the automation scripts 3. Automation development based on design document and scripts will be validated only once the application development is completed 2014, HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 6

7 IN-PROCESS AUTOMATION CRITICAL SUCCESS FACTORS y Stable application environment during the coding and subsequent phases With the automation team working on development environments, stable application environment and scheduled downtimes are important y Historically low volume of defects during CIT Helps in less rework and maintenance y Stability index to remain high There should be minimal or zero changes to the functional specifications document during the coding phase to ensure less maintenance of the developed scripts y The automation team should coordinate with the development team to prioritize the modules coding based on mutually agreed automation plan y The development team should be able to provide faster resolution of all defects identified, even during the development window. This ensures faster coding of automation scripts with very less downtime y A common test template should be used across testing teams for test scenario preparation for consistency. This test template should be designed taking into consideration the requirements and expectation of the automation team y All specification documents should be delivered to the automation team as per plan indicating the object, interfaces, and navigation flow information HCL S IN-PROCESS AUTOMATION TOOL HCL has developed a ready-made tool capable for evaluating the In-Process Automation feasibility and applicability for the application under test based on a comprehensive set of inputs. Some of the inputs include: 1. Application and project methodology 2. Program release information 3. Historic defect and turnaround information 4. Documentation for the application and release 5. Environment details 6. Test scenario/ case information 7. Execution and productivity information HCL s tool provides the following outputs: 1. A feasibility score of the application indicating whether the solution is technically viable 2. An applicability score which indicates the Return on Investment (ROI) on the efforts spent 2014, HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 7

8 3. Recommendations to improve the feasibility and applicability scores An Applicability Index less than -20% (translates to 20% additional effort compared to traditional approach) is typically Not Recommended for In-Process Automation. The In-Process Automation is applicable even if there is a 20% additional effort - as there could still be a reduction in execution cycle time. An applicability index less than 10% is considered as Feasible. An applicability index greater than 10% is Strongly Recommended for In-Process Automation. A Feasibility Index less than 30% is typically Not Recommended for In-Process Automation. A Feasibility Index between 31% and 70% is categorized as Feasible for In-Process Automation, though some level of process tweaking would be required (provided in Recommendation section). A Feasibility Index greater than 70% is a strong contender for High Feasibility for In-Process Automation. IN-PROCESS AUTOMATION CASE STUDY The In-Process Automation approach was implemented for one of HCL s investment banking clients for a platinum project. HCL had proposed to automate at least 50% of the functional test cases during the development phase. The grid below indicates the expected effort saved for In-Process Automation (without manual execution) against the effort spent on automating the test cases, which would have been executed at least once manually. Activities Case 1 Case 2 Number of test scenarios (Automatable) Number of test cases (Automatable) Test scenario preparation effort (Person days - Assuming productivity of 20 TS per day) Test cases preparation effort (Person days - Assuming productivity of 40 TC per day) Test data preparation effort (Person days - Assuming 20% of test case creation effort) Test cases planned to be automated Automation scripting effort (Person days - Assuming productivity of 3 TS per day) Total preparation effort - Manual (Person days) Total preparation effort - Automation (Person days) Total preparation effort (Person days) Manual execution effort (Person days - Assuming productivity of 25 TC per day and 4 cycles) Automation execution effort (Person days - Assuming productivity of 200 TC per day and 4 cycles) Total execution effort (Person days) Total effort (Person days) Percentage Effort save between case 1 and case 2 24% 2014, HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 8

9 CONCLUSION The In-Process Automation approach helps to start the automation process early and get the benefits of automation throughout the testing life cycle. A transparent development life cycle and coordination between the testing and development teams helps this approach succeed. As the automation test execution is faster, defects can be detected early in the testing cycle, ensuring more defect fixing time for the development teams. The development and testing teams need to deploy a disciplined approach since the automation cycle runs parallel with the development life cycle. The manual testing team also gets an opportunity to conduct random/ ad hoc testing to discover niche defects. In today s industry where decreasing the time to market is the motto for any application, an approach like In-Process Automation should be implemented to facilitate the testing process and the team. ACKNOWLEDGEMENT The author gratefully acknowledges the contributions of Raam Kumar, Neha Kotak, and the members of her project team for their work on the original version of this document. BIOGRAPHY Sajitha Radhakrishnan is an ISTQB Certified Test Manager with 10 years of experience in Manual and Automation Testing in the Finance Domain. She has expertise in different types of testing and testing methodologies for multi-tier applications and strong hands-on knowledge of several process analysis approaches/ tools. She has extensive experience in test planning, estimation, status tracking and reporting, metrics collection, and preparation. Sajitha currently manages implementation of the In-Process Automation approach for one of the platinum projects of an HCL client. This is a messaging platform migration project where the application under test interfaces with 200+ back end applications. The automation testing includes SWIFT Network Validation testing for around 216 message types, message comparison using service virtualization, and functional testing using Quick Test Professional and Green Hat. 2014, HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 9

10 ABOUT HCL About HCL Technologies HCL Technologies is a leading global IT services company working with clients in the areas that impact and redefine the core of their businesses. Since its emergence on the global landscape, and after its IPO in 1999, HCL has focused on transformational outsourcing, underlined by innovation and value creation, offering an integrated portfolio of services including software-led IT solutions, remote infrastructure management, engineering and R&D services and business services. HCL leverages its extensive global offshore infrastructure and network of offices in 31 countries to provide holistic, multi-service delivery in key industry verticals including Financial Services, Manufacturing, Consumer Services, Public Services and Healthcare & Life sciences. HCL takes pride in its philosophy of Employees First, Customers Second which empowers its 95,000 transformers to create real value for customers. HCL Technologies, along with its subsidiaries, had consolidated revenues of US$ 5.5 billion, for the Financial Year ended as on 30th September 2014 (on LTM basis). For more information, please visit About HCL Enterprise HCL is a $6.7 billion leading global technology and IT enterprise comprising two companies listed in India HCL Technologies and HCL Infosystems. Founded in 1976, HCL is one of India s original IT garage start-ups. A pioneer of modern computing, HCL is a global transformational enterprise today. Its range of offerings includes product engineering, custom & package applications, BPO, IT infrastructure services, IT hardware, systems integration, and distribution of information and communications technology (ICT) products across a wide range of focused industry verticals. The HCL team consists of over 100,000 professionals of diverse nationalities, who operate from 31 countries including over 505 points of presence in India. HCL has partnerships with several leading global 1000 firms, including leading IT and technology firms. For more information, please visit Hello there! I am an Ideapreneur. I believe that sustainable business outcomes are driven by relationships nurtured through values like trust, transparency and flexibility. I respect the contract, but believe in going beyond through collaboration, applied innovation and new generation partnership models that put your interest above everything else. Right now 100,000 Ideapreneurs are in a Relationship Beyond the Contract with 500 customers in 31 countries. How can I help you?