MERCURY CUSTOMER PERSPECTIVE WHITE PAPER: USING MERCURY TESTDIRECTOR TO DEVELOP A SOFTWARE DEFECT REPORTING AND RESOLUTION PROCESS

Transcription

1 MERCURY CUSTOMER PERSPECTIVE WHITE PAPER: USING MERCURY TESTDIRECTOR TO DEVELOP A SOFTWARE DEFECT REPORTING AND RESOLUTION PROCESS

2 ABOUT THE AUTHOR Punky McLemore is a quality assurance (QA) testing manager at Hewlett- Packard. Prior to working at HP, McLemore was a QA testing manager at a large U.S. financial institution, where she was responsible for testing all new applications for this multi-billion-dollar enterprise. The institution s QA staff employed up to 80 testing specialists and the number of Mercury TestDirector users reached 120. McLemore has 16 years of experience in QA testing and has been a Mercury customer/user for seven years. TABLE OF CONTENTS About the Author 2 Defect Fix 8 Introduction 3 Retest Defects 9 Learning from Experience 3 Customized Software Quality Defect Reporting and Resolution Process 5 Defect Reporting 6 Defect Status Definitions 9 Defect Status Modifications Status Modifications by User 10 Configuring New Processes Into Test Automation Tools 11 Defect Resolution Meeting 7

3 Introduction In this whitepaper, McLemore shares her unique viewpoint and wealth of experience gained from using Mercury TestDirector to support a variety of large-scale projects, the setup and management of the processes, and how easily the processes could be changed and maintained. This paper will present some of the best practices she accumulated as the project lead for testing the corporation s new, automated foreign exchange system a three-year project, extensively utilizing Mercury TestDirector. Learning from Experience Mercury TestDirector is an extremely versatile product in all areas especially for tracking issues and discrepancies, as well as managing defect resolution processes. To help make these processes more effective and achieve better results, this paper will present a set of best practices that other QA test organizations can use to design or streamline their own defect tracking and resolution processes. The information in this paper is based on McLemore s experience as a testing manager at a multibillion-dollar U.S. financial institution, where she was responsible for testing all new applications. One of her projects included creating and running a complex defect tracking and resolution process for the company s newest application an automated foreign currency exchange system. Previously, the company s financial managers had to conduct currency exchanges by phone. With the new system, brokers were able to trade currency electronically, in real-time. As a result of having to mimic a manual process electronically, the group assumed that the majority of requirements could be gathered from studying the manual system. Interviews with brokers, videos of traders in action, and many months of requirements gathering meetings took place to hammer out the needs of the business. The system had to be almost perfect in its accuracy because of the possibility of misaligned deals and the potential for losing hundreds of millions of dollars in a few keystrokes. Quality targets were vague, as the team had no current electronic system on which to base its expectations. Communication among the members of the project group was key to the success of this project. The project was complex involving people from all over the world and testing was expected to be executed 24x7. As fate would have it, the IT group was also looking to automate the manual processes used in the development cycle to meet the challenges posed by the project. The team needed a way to communicate issues, defects, resolutions, scheduling, etc., as well as track what was going on. The project team needed a system that would be highly reliable and easily customizable, with performance that would support users all over the world. The system would have to be easy to use as the users would be coming from manual processes and would not be used to an automated solution. (This was also the problem that was faced with the system they were trying to build, since the brokers who would be using the system were not used to completing transactions via a computer.) 3

4 After a few false starts with other automation tools, the team discovered Mercury s solutions. The financial institution s QA department decided to try Mercury TestDirector as the foundation for all of its testing processes. They found that TestDirector had all of the ingredients needed: High availability: Mercury TestDirector did not crash. Good performance: It was fast enough for long distance users from Asia, Europe, and Australia. Usability: It was easy to customize and use. After the team s favorable experience with Mercury WinRunner, the decision to adopt Mercury TestDirector for functional and regression testing came quickly and naturally. The company used TestDirector to model all of its processes for defect resolution and scheduling releases. They defined everything in TestDirector along application lines all of the enterprise s applications were referred to as projects. The currency exchange project took more than three years to complete and delivered outstanding results. All discrepancies, deficiencies, or other abnormal system behavior noted during the test phases of the software development lifecycle (SDLC) were recorded, tracked, and resolved in accordance with the company s newly defined problem reporting procedures. In Mercury TestDirector, a discrepancy is defined as an unexpected event that occurs while testing or when the expected result does not equal the actual result. A discrepancy could be a defect, a request for an enhancement or new functionality, or any issue requiring corrective action. The TestDirector Defects tab was the tool used for defect reporting. The defect resolution system tracked every single issue and discrepancy in the project not just system, hardware, and software issues related to the application but all process issues as well, like who was responsible for each step, how to obtain equipment, etc. The remainder of this paper will describe the steps McLemore defined for the company s new defect reporting and resolution process. 4

5 Customized Software Quality Defect Reporting and Resolution Process DEFECT REPORTING START DISCREPANCY IS IDENTIFIED DISCREPANCY IS LOGGED IN TESTDIRECTOR STATUS = NEW USE STANDARD FOR DEFECT DOCUMENTATION DEFECT RESOLUTION MEETING RENEW NEW OPEN & REOPEN DEFECTS NEED CLARIFICATION? ADJUST SEVERITY IF NEEDED STATUS = OPEN ASSIGN DEFECT DEFECT FIX RETEST DEFECTS BUSINESS REPRESENTATIVE ASSIGN TO? DEVELOPMENT GROUP TEST GROUP BEHAVIOR ACCEPTED? ENVIRONMENT GROUP RECEIVE BUILD PACKAGE AND EXECUTE SMOKE TEST SMOKE TEST PASSED? LOG DEFECT CLOSE DEFECT NO CHANGE REQUIREMENT ASSIGN DEFECT TO DEVELOPER FIX CODE & UNIT TEST ASSIGN DEFECTS AND EXECUTE DEFECT RETEST NO RETEST FAIL? NO STATUS = REOPEN ASSIGN TO DEVELOPMENT FIX IN THIS RELEASE? NO SCHEDULE FOR LATER RELEASE STATUS = FIXED PERFORM REGRESSION TEST NEW DEFECTS? LOG DEFECT NO ASSIGN TO DEVELOPMENT RENEW NEW OPEN & REOPEN DEFECTS DEPLOY BUILD WITH FIXED DEFECTS CLOSE DEFECT SET STATUS = READY TO RETEST SET STATUS = READY TO RETEST EXECUTE SMOKE TEST ENVIRONMENT FIXED? NO STATUS = REOPEN CLOSE DEFECT Software Quality Assessment (SQA) defect process flowchart 5

6 It is extremely important to dedicate sufficient time and resources to defining all processes upfront. Only by conducting thorough planning can a defect reporting and resolution process provide valuable information and streamline the processes for IT. The following sections will describe steps that can be taken for creating reporting processes, defining defect security levels, recording defect status, and tracking all modifications. Defect Reporting All discrepancies and issues relating to a project that were uncovered during each phase of the SDLC were reported in Mercury TestDirector. When a defect was opened in TestDirector the following process was followed: 1. A release team member would log a defect report in Mercury TestDirector as soon as enough information was gathered in order to recreate or explain the situation. The default Status in TestDirector was set to New. Everyone on the team was encouraged to log defects as soon as possible to allow optimum time for investigation and repair 2. The person who logged the defect assigned an initial severity to the defect based on how it affected the application and/or the test process itself. Severity was defined as: a. Show Stopper: Prevented work from proceeding in an area that would affect deployment schedules. No work-around was available. A fix was required immediately. b. High: Same as Show Stopper, except that a work-around existed and the fix could wait until the next scheduled build. c. Medium: A defect that needed to be fixed as soon as possible prior to deployment. d. Low: A defect whose fix could come in a subsequent release (a later project). e. Enhancement: A suggestion for improvement that could be fixed anytime. 3. A defect writing standard was used to ensure that the defect contained all pertinent information. The specifics of the discrepancy were incorporated, including print screens, associated test cases, selection criteria, and any other critical information to help resolve the issue. 4. The Assign To field was used to indicate who would likely conduct an initial investigation. At this point, defects were assigned to the appropriate group leader representing development, environment support, business lead, or product manager. This fast-track assignment was done so that defects could be reviewed and possibly resolved prior to the defect resolution meeting described below. If the resolving group could not be determined, the defect was assigned to the project manager. 5. If a discrepancy was identified by someone other than a release team member, the specifics were forwarded to the test lead to be logged in Mercury TestDirector. 6

7 Defect Resolution Meeting A sure time-waster in the testing phase was when defects were not acted upon in an efficient manner; some even fell through the cracks. The defect resolution meeting ensured that all defects were reviewed and resolved efficiently. Defects were therefore discussed in a periodic meeting with representatives from the development, project management, product management (user acceptance), and business groups involved in the project together with the test lead. The highest number of defects in the system approached 5,000 and the team handled 25 to 50 defects per defect meeting. The test lead updated Mercury TestDirector during the meeting, making for an efficient use of meeting time. The meeting centered on a review of the defect report. This report was prepared in TestDirector and contained all information needed in order to determine the clarity, severity, and assignee. The defect report was sorted by severity and status so that high profile and new bugs would be addressed first. New defects were reviewed first for: Summary and Description: If a defect lacked the clarity needed to adequately investigate the discrepancy, the defect was assigned back to the defect originator to provide more information. The defect was kept in New status. The defect originator provided the necessary information by updating the defect in Mercury TestDirector. Defects were then assigned back to the project managers via the test lead and would follow the process for New defects. Severity: Reviewed for accuracy and required consensus of group. Assign To: Assigned to the appropriate team (environment, developer, or business analyst) for investigation and resolution. Note: Whenever an assignment was changed, Mercury TestDirector was setup to send out an to the assignee. This is a very powerful tool, as the assignee immediately knows if they need to work on a defect. Status: Changed from New to Open. Mercury TestDirector was set up to send to the assignee each time a defect was assigned and the test lead received an each time a defect changed status. The project manager and test lead received an each time a new defect was entered. The test manager received an each time status was changed to User Error or Works as designed. The test manager would then use this information for mentoring. The development manager received an for each status changed to Reopen. TestDirector is so easily customizable that these changes were made by a non-programmer and the changes were made often to support process improvement initiatives. One tester used 50 percent of his time to administer TestDirector and make all of the experimental and real changes. This is one of the most powerful reasons to use TestDirector. 7

8 Next, the team would review any Open defects. Here is where the Comments came in handy. At each stage of defect resolution from New to Closed, comments were entered to document the defect s journey. The Comments revealed the why, how, and when a defect would be resolved. Appropriate assignments were made to move the defect s resolution along. If the system s behavior was acceptable and required no fixes (systems functioned as expected), the defect was updated with the acceptance information and assigned to the test analyst who set the defect status to Closed. The team would review what was entered in a user-defined field called Cause. This enabled the team to do root-cause analysis with consensus from all the representatives as well as pull reports at a later time (for process improvement purposes). There would also be an optional review of any disputed Closed defects. Defect Fix As a result of the defect resolution meeting, defects were assigned to the appropriate lead from one of the resolving groups: development; environment support; or the business group. 1. The defect was assigned to the business group if there were any questions regarding the requirement related to a defect. Requirement changes were made at this point as needed. The defect was assigned to the development lead if any code changes were needed, or identified as an enhancement for a later release of the application. 2. The defect was assigned to the development lead if a bug in the code was suspected. The development lead assigned the defect to a developer. When the developer fixed the code, the developer changed the defect status to Fixed. The fixed defects made their way into build packages and were handed off to the test group. A report from Mercury TestDirector was produced that reflected all defects fixed in a build. When a build was handed off, the test lead set the status of the Fixed defects to Ready to Retest and assigned the defect to a tester for re-testing. Note: The strength of TestDirector is the ability to define attributes to defects that support various processes. One of these is the Code Fix process. When configuring TestDirector, the team took into account the attributes needed to track the defects as they went through the fixing process, setting developer priorities, keeping fix-time statistics, etc. The same procedure applied to all the other groups. The team controlled the quality of the information for each defect by a system of attribute ownership. 3. The defect was assigned to the environment support group when configuration and setup errors were noted. These discrepancies usually surfaced when the team ran a smoke test. (Smoke testing is non-exhaustive software testing that determines whether the most crucial functions of a program work, but not bothering with the finer details of the program. The term was derived from hardware testing, in which the device passed the test if it didn t catch fire the first time it was turned on.) The environment support group fixed the problem, set the status of the defect to Ready to Retest, then assigned the defect back to the test lead for retest and subsequent closure. 8

9 Retest Defects Defects fixed without changing code as in environment defects were retested and closed immediately. Defect fixes requiring code changes were included in a build package and were retested and either closed or reopened depending on the outcome of the test. This information was handled very efficiently in Mercury TestDirector. 1. The test analyst then reviewed any setup information and the Ready for Retest defects for retesting in the build. Any identified Test Pending defects were also reviewed for retest consideration in the cycle. 2. The test analyst updated the status to Ready for UAT (User Acceptance Testing) when the defect detected in UAT/production and was fixed and tested by integration test analysts. 3. The test analyst then retested the defect and updated its status appropriately. 4. If the defect was not resolved, the test analyst updated the status to Reopen and the process resumed at Step Test leads provided status on all unresolved defects on a daily basis. 6. At the conclusion of each release, during the post-release phase, the test lead scanned the defect statuses for defects that were not resolved during for the current release. Defect Status Definitions Each defect had a status, which was updated throughout the test cycle. Every time the defect status was updated, comments were added into the defect R&D comments field: DEFECT STATUS New Open Fixed Ready for Retest On Hold Closed Reopen GOALS OF THE ITIL PROCESS By default, the status of the new defect was New when added by a test analyst. The test lead changed the status from New to Open and updated the R&D comments field within an acceptable timeframe (dependant upon severity) and the defects were assigned to the appropriate teams. Developers updated the status from Open to Fixed when the defect was fixed and unit tested. The turnaround time for defect fixes was driven by the defect severity. Project/product/release managers updated status to Ready to Retest when the defect had been fixed, the system tested and identified as being included in a build. Test analysts updated the status from Ready to Retest to On Hold when the defect could not be tested in the build due to another defect that prevented the retest execution. Test analysts updated the status from Ready to Retest to Closed after re-testing on the new build of the application. Test analysts updated the status from Ready to Retest to Reopen when the defect retest failed. Test analysts updated the status from Closed to Reopen when the previously closed defect reappeared in a course of testing. Defect status codes 9

10 Defect Status Modifications Status Modifications by User The table below is a simplified version of the status changes that Mercury TestDirector allows, providing for an easier learning curve and enforcing a preferred workflow. Status changes include: Any changes to the system that were made according to appropriate change control procedures. As necessary, the failed test or sections of the test were re-executed following corrective action. At any time during the process, the project managers, system analysts, developers, and test staff could use Mercury TestDirector to retrieve the status of all defects stored in the TestDirector Repository. Mercury TestDirector s reporting capabilities provided a way to monitor the overall progress of defect resolution. USER Test Analyst Project/Product/Release Managers User Acceptance Analyst Developer CHANGE FROM: Initial Entry New New Open Ready to Retest Ready to Retest Ready to Retest Ready to UAT New Open Fixed Initial Entry Ready to UAT Ready to UAT Open Reopen STATUS TO: New Open Closed Closed Test Pending Reopen Closed Ready to UAT Open CR Created Ready to Retest New Reopen Closed Fixed Fixed Defect status modifications 10

11 Configuring New Processes Into Test Automation Tools All processes evolved in time. It took over a year to go from a chaotic process to a high-performing team that had documentation; repeatable, measurable results; and a program for improvement. Mercury TestDirector was pivotal in the project s success because it enabled the team to model and remodel iteratively, as the team defined and redefined its processes. When success with using Mercury TestDirector was apparent in building the defect resolution process, the team decided to tackle automation. In preparation for automation, the team decided to use the Test Plan in TestDirector as a repository for manual tests, as well as the tool for managing and tracking all efforts. The team was able to use the concepts learned in TestDirector to build robust test plans, as well as build test sets that included manual tests and automated scripts that were originally built in Mercury WinRunner. 11

12 Mercury is the global leader in business technology optimization (BTO). We are committed to helping customers optimize the business value of IT Mercury Interactive Corporation. Patents pending. All rights reserved. Mercury Interactive, the Mercury logo, Mercury TestDirector, and Mercury WinRunner are trademarks or registered trademarks of Mercury Interactive Corporation in the United States and/or other foreign countries. All other company, brand, and product names are marks of their respective holders. WP