Automation Test Framework Assessment Step ahead for successful test automation

Transcription

1 Automation Test Framework Assessment Step ahead for successful test automation 13th Annual International Software Testing Conference 2013 QAI STC th September 2013 Author Rahul Shirudkar, Specialist Senior Fairmont, Level 2, High Street, Hiranandani Business Park Powai, Mumbai - 76, Maharashtra, India

2 P a g e 1 Abstract Organizations, these days, understand value of automated software testing and implement automation in different projects. It has been observed that many a time automation fails and if we analyze failures most of them are due to lack of budget, time constraints, tool incompatibility or lack of expertise. In automated testing, framework plays a key role in automation success and so does in failure. Some of the common framework failure reasons are too many efforts to develop and maintain framework, framework scalability issues, framework not utilizing capabilities of underlying tool or not so easy to use and requires special skills. The objective of this paper is to outline a systematic approach of automation test framework assessment (ATFA), by which developed framework will be assessed against predefined assessment criteria by an independent assessment team. The outcome of this assessment will help automation projects on Go-No Go decision for using assessed framework before investing high efforts in automation and thus provide better opportunity for successful automation. This paper provides list of different framework criteria which can be used for assessment and also defines roles and responsibilities for the assessment team. Target Audience: Area of Application: Test Managers, Test Leads, Test Professionals and Technical Leads working on/supporting automation projects Automation Testing 1. Introduction Advances in testing over the recent years have made us understand value of test automation, but often automation is not used or it fails. As per the IDT (Innovative Defense Technologies) automated software testing survey [3] [4] reasons given for failure of automation were: Lack of time, Lack of budget, Tool incompatibility, Lack of expertise and others [For e.g. environmental issues, application not stable or frequent changes to functional requirements].

3 P a g e 2 Test framework is required in test automation as one automation tool cannot do everything [1] ; it is a group of tool working together that provides automation solution. So investments need to be done in framework development to supplement automation efforts [2]. If we analyze all of the above failure reasons, we will realize that test framework is linked to most of these reasons. For e.g. If developed framework is not easy to use, understand and maintain then we may end up spending most of efforts (and hence budget) in using and maintaining framework rather than actual automation. Another example would be if framework designed is so complex that automation testers cannot use it and requires special skills to use effectively. Then they may avoid using it. For most of the test automation project, we follow standard process as shown in Figure 2. We start with planning where we find out scope for automation. Then we select the tool for automating the scripts. With selected tool; we tend to develop automation framework. Once test framework development completes, we start script development and the process continues with executing and maintaining automaton scripts. In this standard process, we invest efforts in developing the framework and then spend large amount of efforts in automation using this framework. What if the framework fails to fulfill the automation need? Our invested efforts will be wasted and eventually may result in automation failure. So it is worth to validate the developed framework against automation requirements by independent resources (other than the one who developed it). This will ensure that framework will work as expected and minimizes the risk of failure. The next section of this paper describes the detailed process of Automation Test Framework Assessment to validate the framework against the automation needs.

4 P a g e 3 2. Automation Test Framework Assessment (ATFA) Process Automation Test Framework Assessment (ATFA) is the process of assessing the developed test automation framework against pre-defined assessment criteria, analyzing the outcome of assessment and providing the recommendation that will help automation project team to decide whether to use developed framework for further automation or not. Figure 3 shows process with test framework assessment as a formal phase in test automation. This assessment is done when test framework is developed. If framework fulfills business needs as per assessment criteria defined, then automation team can start developing automated script. If not, they may have to enhance automation framework or sometimes look another automation tool depending upon the assessment results. This assessment process starts with identifying the team who can do the assessment, after which assessment criteria is defined. When test framework is developed, team assesses and creates results and finally provides recommendations. Figure 4 shows step by step process of test framework assessment.

5 P a g e 4 Each step in assessment process is explained below in detail Identify Assessment Team First step is to form assessment team who will conduct framework assessment. Team should have knowledge of test frameworks and underlying automation tool. The Assessment Lead understands the automation requirements and defines assessment criteria. These criteria are reviewed and approved by test manager. Then the Assessor evaluates test framework against criteria and create detailed assessment results. At the end assessment lead also generates assessment summary and provides recommendations. Same resource can play both roles if needed. The roles and responsibilities of the assessment team are defined below. Assessment Lead Define assessment criteria and get it reviewed and approved by the Test Manager. Create Test Framework Assessment plan Review assessment results Create Assessment Summary and provide recommendations Assessor Evaluate test framework against predefined assessment criteria Record results and create detailed assessment results 2.2. Define Assessment Assessment lead works with test manager to understand the project needs in terms of automation and application technology, and then start defining the assessment criteria. The different assessment criteria are listed below Test Framework Setup Easy to Setup Easiness of setting up framework on new system. For e.g. Is it an installable with some custom settings in system or simple copy of folders? Easy to Understand Easiness for understanding different components of the framework. Easy to Use Ability to use features of underlying tools Easiness in using the framework components. Ability to use the features of underlying tools. For e.g. If framework has its own custom reporting functions, can

6 P a g e 5 Easy to Maintain Scalability Supported application the framework still access the reporting feature of the automation tool? Easiness in modifying the existing components of the framework. Ability to support new applications, new technology, new tools. Currently supported application type and technology. Test Script Development Easy to Create/Maintain Reusability Support to programming logics Easy to unit test Synchronization Easiness in creating and maintaining test script. For e.g. Can a functional expert/ business analyst create and update test script or it will need knowledge of tool, programming language? Ability to call existing test script. Ability to implement decision [if else conditions], iterate through certain steps in test script [loops]. Easiness in testing the currently added steps in test script. For e.g. Can we test the newly added steps independently or we need to execute complete test script? Ability to add synchronization points in test script flows. Test Data Management Parameterization Iterations Recording Output values Ability to replace actual data values in test script with variables for which values can be passed from external source. Ability to run the test script with multiple data sets in same execution cycle. Ability to store the outcome in external files. Test Execution Easy to Run Batch Execution Scheduling Exception Handling Easiness in executing the test script. For e.g. If execution can be started with one click or some additional configuration is required before executing the script. Ability to run multiple test scripts in one run. Ability to schedule the execution at specified time. Ability to manage run-time script and application error. This also includes the ability to log error information.

7 P a g e 6 Defect logging Ability to create defect in Defect Management system when failure is detected. Test Reports Readable Report Format Summary Report Detailed Execution Report Easy to read and understand. Formats in which reports can be generated For e.g. Excel, HTML, PDF, text file. Ability to generate summary report for the execution. Ability to generate step by step details for each and every test script. Integration with other tools Test Management System Defect Management System Build Management System Can framework read test script stored in test management system and send results back to it? Is framework supports integration with defect management system? Can framework be integrated with build management system to start execution of script when new build is deployed? Transition Easy to Handover Supporting Documentation Easiness of handing over scripts/framework to client or other vendor. Documentation available with framework. For e.g. User guide, technical design documents, coding standards etc. Assessment lead may add additional criteria as per project need and then works closely with test manager to set expected level for each of these criteria. The expected levels are defined as below Must Have Important (To Have) Nice To Have Not Required The framework must support the feature/criteria before starting the script development The framework should support the criteria but scripting can be started without meeting the criteria The framework may or may not support the criteria The framework don t need to support the criteria

8 P a g e 7 Figure 5 shows the sample assessment sheet with expected levels set Assess Test Framework When test framework is developed, assessor starts assessment of framework based on the criteria defined by assessment lead. The assessor can use static analysis techniques such as code review, inspection and walkthrough, or can evaluate framework by setting up on machine, then develop and execute sample scripts, or can use both the techniques. Findings are recorded in assessment sheet. The assessment results are defined as below Framework fulfills the criteria. For e.g. Framework allows data to be read from external files. Meet [Expected Level] Framework fulfills the criteria with workaround. For e.g. Framework cannot directly upload test results in test management tool. But can generate Meet with Workaround results in a format which can be uploaded. Meet Partially Does not Meet Framework fulfills the criteria partially. For e.g. says that framework should support HTML and Excel reports but framework only support excel Framework does not fulfill the criteria. For e.g. Framework does not allow calling one test from another. Figure 6 shows sample assessment sheet after assessment results recorded with comments.

9 P a g e Assessment Results and Recommendations Once assessment is complete, assessment lead reviews the assessment results and provides one of the following recommendations to test manager. Framework is ready to use for test automation Framework can be used for test automation with suggested improvements Framework cannot be used for test automation; either framework needs to be redesigned or other tools needs to be considered for automation 3. Conclusion Framework plays a key role in test automation and hence is considered as one of the critical success factors for automation. If it is not satisfying the automation requirements, it is likely to happen that automation testers avoid using it and this could increase probability of automation failure. It is important to validate framework and confirm that it is reliable, before actually using it in script development. Automation Test Framework Assessment process is precautionary steps that will help us to validate developed framework, confirming that it is fulfilling all our automation needs. This process starts with identifying assessment team, Assessment lead defines the assessment criteria and then assessor evaluates developed framework against it. The outcome of this process is recommendation for using the framework or suggesting improvements or consideration for other tools or approach if framework cannot be used. Framework assessment efforts are less as compare to actual test automation efforts, so it is worth to spend these little efforts at initial stage as a step towards successful automation.

10 P a g e 9 4. References [1] Sharon Robson, Five reasons why Test Automation fails Knowledge Engineer and Software Testing Practice Lead, Software Education, May 2012 [2] Shrini Kulkarni, "Top Ten Classic Reasons why Test Automation projects Fail", igate Global Solutions Hyderabad, India [3] Elfriede Dustin, Thom Garrett, Bernie Gauf, Implementing Automated Software Testing, 2009 [4] 5. Author s Biography Rahul has eight years of experience in software testing which includes both manual testing as well as test automation. His core skills are test automation and test tool administration. He is highly experienced in developing automation frameworks for different applications like web based, native clients and packages (SAP). He has strong expertise in tools like HP Quick Test Professional, Selenium and HP Quality Center. He is a bachelor s degree holder in Computer Engineering and ISTQB (Foundation Level) certified professional. He has also passed HP Quick Test Professional 9.2 and HP AIS Application Lifecycle Management V11 examinations.