1 TEST PROCESS IMPROVEMENT SUKANYA RAGHAVAN suraghavan@inautix.co.in INautix Technologies Pvt Ltd 10 th Floor Tidel Park, No.4. Rajiv Gandhi Salai Taramani, Chennai 600113
2 ABSTRACT This paper brings in various KPA s in which Organization can focus for test process improvement. The idea behind when and why an organization should think of test process improvement is vital. The comparison of various models and which of them proved successful for our organization is explained. TMM and TPI are two of the models that are discussed in detail and how TMM could be compared and applied to CMM levels and leverage the benefits of them to an Organization s testing Practice. We have discussed two process areas one in the test definition/preparation phase and another in test execution phase which has brought in a greater degree of success in terms of ROI. TOPICS DISCUSSED Definition of Test Process Improvement and ways History on Various Models Assessment Process and Assessment Steps Comparison on few Testing Models Success story with two test process improvement initiatives out of TMM assesment Test Process Improvement and Ways "The continuous improvement of all processes." is the primary focus for any process improvement. The two ways to make process changes: o o Random changes This approach can be applied when we need immediate change to be applied and can be applied to only selected few areas for a immediate test process change Planned incremental changes This approach can be applied when we need to have a structured and planned changes needed for the testing practice as a whole or for a team, for which time and effort would be comparatively more. For aforesaid ways the primary input or entry criteria would be to know where the current testing process stands for our team/organization. There is no way we can improve until we know where we are and we have a scorecard of what we perform. The immediate result of knowing where we stand in our testing process is to compare and benchmark ourselves with various industry specific standards and models and baseline like TMM, TPI, CMM and CMMi etc
3 HISTORY ON VARIOUS TESTING MODLES Test Models Origins Acronym Details Author/Organization BTM Beizer s Testing Model Authored by Beizer in 1990 TOM Test Organization Maturity Model Authored by Gerrad Consulting in the U.K. in the late 1990s TMAP Test Management Approach Authored by Martin Pol, Rudd Teunissen and Erik Van Veenendaal in 1995 and now owned by Sogeti which is part of Capgemini. TSM Testability Support Model Authored by Dr. David Gelperin in 1996 TPI 2 Test Process Improvement Model Authored by Tim Koomen and Martin Pol in 1996 TMM SM3 Testing Maturity Model Authored by Dr. Ilene Burnstein, at the Illinois Institute of Technology in 1996 TCMM Technical Capability Maturity Model Authored by Torry Harris in 2001 TIM Test Improvement Model Authored by Thomas Ericson in 2002 TMAP Next Test Management Approach for Next generation Authored by Tim Koomen, Michiel Vroon, Leo van der Aalst & Bart Broekman in 2005 TMMi TM Test Maturity Model integrated By TMMi Foundation in 2007. In 2010 level 4 and 5 framework was released. IMG 1 (List of Various Test models with their history) ASSESMENT PROCESS Any assessment process as discussed earlier would need to know the current status and a thorough assessment is not a quick and simple process; therefore, it must have management s full support. An organization has two options o o Performing it themselves Hiring a consultant.
4 The industry recommendation is that any organization performs the initial assessment themselves with independent validation by an outside source. There will be a higher acceptance of the findings if it is performed internally. If an outside consultant writes the assessment report, the first thing the organization will try to do is disprove the negative findings. If it is performed internally, and if the team is viewed as being unbiased, the report will be more readily accepted. IMG 2 (Assessment Process)
5 ASSESSMENT STEPS The below mentioned steps defines an assessment process and each step internally has about 5 or 6 stages which exhaustively completes the test process assessment. At the end of the assessment a list of areas of improvement is identified which needs to be improved for the entire process to be in par with the industry standard or a baseline model against which the testing process was benchmarked. IMG 3 (Steps involved in the Assessment Process)
6 Comparison on few Maturity Models Capability Maturity Model (CMM) The CMM is used to assess the maturity level of an organization to develop software products. Five levels of maturity Level 1: Initial Level 2: Repeatable Level 3: Defined Level 4: Managed Level 5: Optimized Each level is characterized by a set of key process areas. NO SPECIFIC FOCUS ON TESTING AS A KEY PROCESS Test Process Improvement (TPI) A test process is evaluated w.r.t. 20 key areas. The maturity level of a test process is assessed on a scale of 1 13. The 13 levels of maturity are partitioned into three groups: controlled, efficient, and optimizing. GENERIC PARTIONING which exposes high level areas of improvement Testing Maturity Model (TMM) The TMM model gives guidelines concerning how to improve a test process. The maturity of a testing process is evaluated on a scale of 1 5. Level 1: Initial Level 2: Phase Definition Level 3: Integration Level 4: Management and Measurement Level 5: Optimization, Defect Prevention, and Quality Control IS READILY USABLE FOR ANY ORGANIZATION OF ANY SIZE HOW TMM Assessment model worked for us IMG 4 (TMM Model for our Organization)
7 After a thorough assessment there were few findings which the team had to implement for their process improvement. The team was assessed at L2 for test planning and tracking and test execution. After analyzing the current gap in these two the team implemented two process improvements out of TMM assessment which lead to tremendous process stability and increased satisfaction index from clients Success Story with Business Process Testing (BPT) Primarily implemented as a process improvement initiative out of the assessment for test planning and tracking. HP ALM Business Process Testing software is a web-based test design solution that allows creating manual test cases that are reusable. It provides a script-free mechanism for business analysts and SMEs to design and align tests with business processes for accurate functionality coverage. It more suits for projects that are challenged by frequent releases and demands for shorter test cycles Business Components: The Business Components module enables you to create and manage business components in HP Application Lifecycle Management (ALM). These components provide the basis for Business Process Testing. Business components are testing units that perform specific tasks in a business process. Workflow in BPT Framework Identify the reusable operations or functions in the system Create a business component with manual test steps Add Parameters to business component if the reusable function needs to be tested with different input values After creating the components, create the test case scenarios in test plan using following test case type Associate the business components to the test cases based on system work flow Create test set in Test Lab and associate the business process test cases for manual test execution
8 IMG 5 (Manual BPT Flow ) Advantages of BPT Framework BPT Framework allows non-technical subject matter experts to quickly create test cases using the business process components. When creating manual tests, Business Process Testing dramatically reduces the need for repeating steps and facilitates text reusability. Business process test creation is accelerated by the ability to add components across multiple test cases Business Components supports to test data parameterization which enables to validate the scenario with various combination of inputs Business Components helps to quickly identify the good candidates for automation. Changes to a component will propagate to all the linked test cases and reduce maintenance Disadvantages of BPT Framework BPT Framework can be used only if you have QC access along with the subject matter expertise in the system. Need to purchase additional license for BPT Framework if you want to use this for your test scripts.
9 Success Story - II The second take away from our assessment lead to the invention of a Requirement Volatility template which enabled us track the health of our project from requirement analysis till test execution which made us matured in our way of test planning and test tracking. REQUIREMENT VOLATILITY Requirements volatility (RV) refers to additions, deletions and modifications of requirements during the systems development life cycle. RV creates rework in design and code that increases the system development cost and time and compromises the system quality. Ignoring requests for requirement changes can cause system failure due to user rejection, and failure to manage RV can increase the development time and cost. IMG 6 (Requirement Volatility Tracker) Requirements Volatility Formula = {(No. of requirements added + No. of requirements deleted + No. of requirements modified) / No. of initial approved requirements} * 100 Unit Of measure = Percentage
10 Risk Factors for RV Large systems with a large number of requirements have scope for many changes. Frequent turnover of developers and/or users in the development team will affect continuity and lead to changes in requirements. Projects that are forced to be completed earlier than the required time may lead to an incomplete and incorrect requirements definition and subsequent changes. Finally, users may add, modify or delete requirements due to lack of planning on their part or due to indecisiveness. IMG 7 (Requirement Volatility Risk) RV Risk Management Methods Reject RV - The easiest suggestion to manage RV is to freeze the requirements and reject the volatility. Eliminate Avoidable Causes of Volatility - Verification and validation of requirements using inspections, reviews and walk-through exercises can ensure that requirements are correct, consistent and complete, and they can help to avoid changes to requirements during later stages. Use Methods to Work With Volatility - Prototypes and pilot projects are useful in resolving requirement uncertainties in strategic systems and in systems with new technologies Manage Volatility and Its Effects - Structured requirement definition methods can help to trace a requirement to the system s objectives and to determine the impact of a requirement change on system objectives.
11 Conclusion: The TMM assessment leads us the way for improving our test process and paved way for 2 new process improvement initiatives and helped us realize the high satisfaction index from our stakeholders. REFERENCES http://www.astqb.org/ www.isaca.org http://onlinelibrary.wiley.com www8.hp.com BIOGRAPHY: I have about 12 years of Testing Industry experience at various Organizations which includes the Cognizant Technology Solutions and the Tata Consultancy Services and at present the inautix Technologies India Private Limited. Most of my testing career was based overseas at the United Sates wherein Organizations like BNYMellon, AIG and Solbright were key in nurturing me with abundant testing exposure and knowledge. I completed my Bachelors of Science with Physics as my Specialization and graduated with My Masters In Computer Applications from SSN College Of Engineering, Chennai. Having to come back to the place where my industry experience commenced here in India makes me feel nostalgic and wish could be a source of inspiration to the team, project and Organization and to the testing and Quality Assurance fraternity with my contributions.