Keywords - Software Testing, Risk-based Testing, Metrics, Control and Measurement

Transcription

1 Measurement and Control for Risk-based Test Cases and Activities Ellen Souza, Cristine Gusmão, Keldjan Alves, Júlio Venâncio and Renata Melo Department of Systems and Computing at Pernambuco University Rua Benfica, 455, Madalena, , Recife PE, Brasil {eprs, cristine, kao, jvmj, Abstract Risk-based testing is an approach that consists of a set of activities regarding risk factors identification related to software requirements. Once identified, the risks are prioritized according to its likelihood and impact, and the test cases are projected based on the strategies for treatment of the identified risk factors. Then, test efforts are continuously adjusted according the risk monitoring. Most risk-based testing approaches focuses on activities related to risk identification, analysis and prioritizing. However, metrics are fundamental as they quantify characteristics of a process or product and support software project management activities. In this light, this paper proposes and discusses risk-based testing metrics to measure and control test cases and test activities progress, efforts and costs. Keywords - Software Testing, Risk-based Testing, Metrics, Control and Measurement 1. Introduction Testing has been fundamental for improving and ensuring software quality. However, this set of activities may take so much time, cost and resources. The domain of inputs and outputs of a system is very large, as well as, there are many possible paths to testing [1]. Therefore, a common practice of software testing is to be performed by an independent group of testers. This practice often results in a testing phase being used as project time buffer to compensate for project delays, thereby compromising the time devoted to testing [2]. Regarding software risks, problems motivated by unexpected, non-designed or even ignored risks are very common. And then, the need for using a risk management methodology increases as software complexity and size of the systems grow [3]. Risk-based Testing (RBT) is an approach that aims to minimize some of these software risks and testing problems. RBT consists of a set of activities regarding risk factors identification associated to software requirements. Once identified, the risks are prioritized according to its likelihood and impact and the test cases are projected based on the strategies for treatment of the identified risk factors. The test efforts are continuously adjusted according to the risk monitoring. Thus, RBT justifies test efforts finding the most important defects as early as possible against lowest cost [4]. Therefore, measurement process is needed at least for assessing the status of projects, products, processes and resources [5]. According to Amland (1999), in all test projects it is important the use of metrics to monitor progress and quality, even more when using risk-based testing where more time is spent identifying and analyzing software important areas which enables to have an idea about the needed resources to design and execute the test cases [6]. In addition, metrics are admittedly an excellent way to evaluate a software product as they can give us consistent numeric values which enable assert product quality and justify several kinds of decisions. In this paper, we introduce the idea and importance of metrics to control and measure risk-based test cases and activities. In order to evaluate and explain the proposed metrics, the activities of a risk-based software test process [7,8] were simulated by three volunteers. Section 2 presents the background to clarify the main concepts involved in this work. Section 3 explains the proposed metrics for risk-based test cases and activities, while section 4 explores the approach in further detail through a case study. Section 5 concludes the paper and describes some future work. 2. Background Risk is the possibility of an organization to suffer offense or loss on at least one objective of the project, like: time, cost, quality and others. The PMBOK Guide [9] defines two kinds of risks: positive and negative. The positive risk associates an idea that there are earnings instead of losses for the organization. These losses are denominated negative ones. The SEI (Software Engineering Institute) elaborated a taxonomy risk [10] which groups some definitions in order to classify many different risks as: project, business and technical or product risks. The product risks are the ones that can be mitigated through software testing. James Bach introduced first the RBT concept. He described his idea in the article: The Challenge of Good Enough Software in 1995 [11]. We may think of RBT as an approach that tackles with quantitative assessment to risks in order to fulfill quality assurance for software through most important test /09/$ IEEE

2 Commonly, a simple risk model is used in RBT. It is based on the likelihood of a fault along with impact caused by it. Afterwards, the risks are sorted according to its exposure. Furthermore, there are several RBT approaches regarding the existing literature [12,13,14,15,16]. Bach (1995) defines an approach based on heuristics which uses two distinctive treatments for risks identification activity. However, the assessment and control for risk factors are not trivial and requires knowledge and experience about the business [17]. Amland (1999) developed a set of metrics for risk analysis aiming support test activities. The big challenge to him was to define a cost and quality indicator for faults. However, in accordance to the author a suitable simple approach was used. Chen (2002) defines an execution strategy for regression testing based on specification. She defines regression test for components that were updated and regression test for components that seemingly were not modified. Chen s approach is intended to select test cases rather than requirements and it is based on system documentation which must be up to date. Jørgensen (2004) provides the requirements and prototype to develop a support tool for RBT risk analysis and test plan creation activities. Besson (2007) proposes an approach based on use where any test activity implies on risk reduction. The author controls test effort grounded in Pareto s Principle which states that 20% of the functionalities in the system allows the user to accomplish 80% of his tasks. 3. Metrics for Risk-based Testing approach Control and Measurement A metric quantifies a characteristic of a process or product and support software project management activities like planning, organizing, controlling and improving [18]. The software testing metrics can serve as key indicators of the testing activities and how these metrics can be used to improve your testing practices. In this section a set of metrics is proposed to indicate risk-based testing progress, effort, cost, stability, quality and others categories. The Goal Question Metric (GQM) approach, developed by Victor Basili 1, was used to define the proposed metrics. We have observed three classes of metrics for riskbased testing control and measurement. The first one is related to test cases control and is explained in Section 3.1. The second evaluates the RBT activities by controlling the needed efforts and resources and is presented in Section 3.2. The third is to control and 1 Goal Question Metric Approach (GQM), available at measure the impact and advantages of RBT adoption in an organization Metrics to Control and Measure the Riskbased Test Cases Several studies have been conducted in order to identify the main risks of software projects. Here we discuss a significant set of metrics used to control and measure the risk-based test cases. For each requirement, risks are identified and analyzed. After that, test cases are designed to assess the risk factors events. When those tests are executed and a fail can be observed, testers report the defect so it can be fixed and the risk, related to the test case, mitigated. Amland already suggests some metrics for risk-based testing like, number of tests planned, executed and completed, number of faults per function, number of hours used in testing per fault found, the number of hours used in fixing per fault (to correct the defect and return the function to re-test) and others [6]. In addition, Konda (2008) presents a set of metrics in order to measure defect removal accurately that is also useful for risk-based testing [19]. Table 1 to 10 enumerate the recommended metrics for RBT test cases control and measurement. The Gn line contains the measure objective which is the information we want to acquire. The Qn.m line is the questions that help us plan and manage the test progress. Finally, the Mn.m is the metric that we have to collect to answer the questions. Table 1. Metrics for Risk-based Test Cases G1. To verify if an acceptable level of mitigated risks has been reached. Considers risk status. Q1.1. How many risks have been mitigated? Q1.2. How many risks have been mitigated per requirement? M1.1 Number of planned risk-based test cases / Number of executed riskbased test cases, by status M1.2 Number of planned risk-based test cases per requirement / Number of executed risk-based test cases per requirement, by status Even though executed risk-based test case does not always imply in mitigated risk, this metric is important to verify if an acceptable level of mitigated risk factors per requirement/feature have been reached. Table 2. Metric for Prioritized Risks G2. To know the average of most important and prior requirements. Q2.1. How many requirements have a high risk exposure? M2.1 Number of requirement with high exposure / Number of requirements After the risk identification, the risks should be analyzed and prioritized using the risk exposure technique

3 (RE). The risk exposure is the product of the probability of a non-satisfactory result to occur, and the loss associated to this non-satisfactory result. An interval using the risk exposure value can be defined to identify high, medium and low risk priorities for example. Table 3. Metric for Identify Risky Categories G3. To know the risk categories that presents more risk factors. The risks can be classified according to some categories or taxonomies. Q3.1. How many risks have been identified per categories? M3.1 Number of risks per categories / Total number of risks The SEI risk taxonomy for Product Engineering contains five elements and each element is characterized by its attributes. The Requirements element, for example, contains attributes related to Stability, Completeness, etc. [10]. In this light, we can find which source of risk is more relevant in the project. Table 4. Metric for Identify Treated Risks G4. To verify how much the risks are decreasing in each iteration or test cycle. Q4.1. How many risks are found per meeting? M4.1 Number of identified risks / Number of meetings Table 5. Metric for Verify Risk Reduction [20] G5. To verify risk reduction leverage. Q5.1. Is the cost of the reduction technique in accordance with the leverage value? M5.1 (Risk exposure before reduction - Risk exposure after reduction) / Cost of risk reduction If the leverage value is not high enough to justify the action, then we can look for other, less costly or more effective reduction technique. Table 6. Metrics for Effort Identification G6. To support planning by providing effort estimations. Q6.1. How much time is M6.1 (Number of spent on meetings to identified risks / Number identify relevant risks per of requirements) / Meeting requirement? duration Q6.2. Is the time spent on M6.2. Compare the value brainstorm meetings of M4.1 with a threshold. worthwhile? Q6.3. How much time is spent on meetings to analyze relevant risks per requirement? M6.3. Time spent to analyze all requirements / Number of requirements The threshold, mentioned at M6.2, could be based on the average of past meetings or another kind of statistical measurement. These metrics are useful since the employees are paid by the hour. Table 7. Metric for Defect Identification G7. Indicates the quality of the risk-based test. Q7.1. How many defects M7.1 Number of defects of were found per severity? a certain severity / Total number of defects Table 8. Metrics for Identify the Effectiveness of RBT G8. Effectiveness of risk-based test. Q8.1. How effective the risk based test cases are? M8.1 Number of test cases that fails (i.e., a defect was found ) / Number of Q8.2. How many change requests are made per requirement / risks? executed test cases M8.2. Number of raised change requests per requirement / risks Table 9. Metric for Non-identified Defect G9. Escaped defect with risk-based test. Q9.1. How many defects M9.1 Number of change are found by the client (end requests raised by client per user) using risk based tests? severity. This metric is related to the one in Table 7. Table 10. Metric for Required Effort G10. The effort required to find a defect with risk-based test cases. Q10.1. Amount of time to M10.1 Total hours spent on find a defect with risk test execution / number of based test cases? defects detected during the same period. This metric is to identify the possibility of loss associated with the occurrence of a defect. It ranges from a loss of comfort to the loss of many human lives Metrics to Control and Measure RBT- Activities This Section presents metrics that can be used to rank and track RBT activities. They are used basically to find problems and improve the activities and are shown from Table 11 to 15. Table 11. Metric for Time Identification G11. To know the average time spent to analyze a requirement with a certain number of lines. Q11.1. What is the mean time to analyze a line of a requirement? M11.1 (Sum (analysis duration of a requirement / Lines of a requirement)) / Number of requirements

4 Table 12. Metrics for Identify the Productive of RBT Risk Identification G12. To verify how productive the risk identification meetings are, in a quantitative manner. Q12.1. How many risks are M12.1 Number of risks / found per meeting? Number of meetings Q12.2. How many risks are found per meeting and per employee that joined the meeting? M12.2 (Number of risks / Number of meetings) / Number of employees Table 13. Metric for Identify the Productive of RBT Risk Identification G13. To identify the number of low risks factors identified. Q13.1. How many risks were discarded during the meeting? M13.1 Number of discarded risks / Total risks If the number of discarded risks is high, it means the elicitation is too broad. It might be wise to focus on the project scenario so that the relevant risks are determined faster. Table 14. Metric for Identify the same Risk Exposure G14. Same Risk Exposure Identification Q14.1. How many test cases shared the same risk exposure? M14.1 Number of test cases with the same risk exposure / Total number o test cases If many test cases share the same risk exposure, the risk based approach is meaningless because they can't be prioritized. In this case, one of the metrics used to compute the risk exposure can be used to prioritize the requirements. Table 15. Metric for Evaluate the Risk Identification Activity G15. Evaluate that the identified risks are useful/meaningful to design the test cases. Q15.1.How many test cases were designed making use of the identified risks? M15.1 Number of risks used to design test cases / Number of identified risks 4. Case Study and Results The next step was to use the set of metrics presented in section 3 at RBTProcess [7,8] - a Risk-based Software Testing Process Model. The process activities are shown in Figure 1 together with its roles and phases. The Health-Watcher system requirements document [21] was used to identify and analyze the risk related to its requirements. The document contains nine functional requirements and nine non-functional requirements. For this article, we have considered only the functional requirements. Three undergraduate students, with testing basic knowledge, took part as volunteers executing the RBTProcess activities and collecting the time spent to realize each one. Figure 1. RBTProcess: Risk-based Software Testing Process Model 4.1. Identifying Risks The first step used for the risk identification was the Taxonomy Based Questionnaire (TQB). The TBQ consists of questions under each taxonomic attribute designed to elicit the range of risks and concerns potentially affecting the software product [10]. In this case study, we choose eight questions related to four risk categories (stability, completeness, clarity and validity). After that, a brainstorm meeting was carried out in order to validate the identified risk and may discover others. Table 16. Identifying Risks metrics - RBT Test Cases M3.1 Stability: 11 / 24 = 45% Completeness: 7 / 24 = 29% Clarity: 3 / 24 = 12% Validity: 3 / 24 = 12% Table 16 shows the values for RBT test cases metric M3.1, which enable us to identify the risk category that presents more risk factors and indicate that requirements aren t stable. Table 17 shows useful metrics for planning and estimate test activities. Table 17. Identifying Risks metrics - RBT Activities M minutes per requirement with 40 lines. M risks per meeting M / 3 (volunteers) = 8 risks

5 4.2. Analyzing Risks Analysis is the conversion of risk data into risk decision-making information. The volunteers compute the risk exposure for each requirement/feature considering the identified risk in previous activity and using the risk exposure formula proposed by Amland [1]. Table 18. Analyze Risks metrics for RBT Test Cases M6.3 < 2.5 minutes per requirement Table 18 shows the needed time to analyze, that is, to prioritize a requirement while Table 19 indicates the number (average) of requirements with the same risk exposure (RE). The Table 19, line M4.1b and Table 20 are examples of inefficient RE values and indicate that another metric has to be used to prioritize the requirements. Table 19. Analyze Risks metrics for RBT Activities M4.1a 2 / 9 = 22 % M4.1b 4 / 9 = 44 % Table 20. Calculated Risk Exposure (RE) for Health- Watcher system requirements Requirement/Feature Risk Exposure FR16. Change logged employee 1.19 FR01. Query information 1.38 FR13. Register new employee 1.38 FR14. Update employee 1.38 FR12. Update complaint 1.38 FR10. Login 1.56 FR15. Update health unit 1.69 FR11. Register tables 1.75 FR02. Specify complaint Planning Tests Adopting RBT approach, the Test Manager has further information like requirement stability, clarity, importance, preference and so on. Table 21 provides a priority classification that helps test managers to decide when, what and how to test each feature explained as follows: Table 21. Priority classification for Health-Watcher system requirement Requirement/Feature RE Priority FR14. Update employee 0.75 Low FR13. Register new employee 0.83 Low FR10. Login 1.00 Medium FR16. Change logged employee 1.00 Medium FR11. Register tables 1.07 Medium FR02. Specify complaint 1.13 Medium FR12. Update complaint 1.13 Medium FR15. Update health unit 1.19 High FR01. Query information 1.40 High 1. When: supposing that three test cycles are defined in the project scheduler, the test manager can decide to test first, the requirements with High priority; second, the requirements with Medium priority and third, the ones with Low priority. 2. What: considering the principle of diverse halfmeasures explained by James Bach (1999), where he emphasizes to do not let risk-based testing be the only kind of testing you do due to the possibility of a poor risk analysis, the test manager can decide, for example, to perform also functional test for the requirements with High priority [18]. 3. How: the test manager, using risk information, can decide, for example, to perform white-box or structural testing for requirements with High priority. He also can decide to automate test cases for a risk category Designing Tests An important difference between testing and risk-based testing is that the test cases are design to asses the identified risk factors. Assuming that: i. we designed a functional test case for each requirement flow, ii. we designed a risk-based test case for each identified risk and iii. for requirements with medium and low priority, was executed just RBT, Table 22 presents the number of tests needed to test all the requirements flow, also the number of risk-based test cases and the number of test cases resulted from a combination of functional and RBT explained in Section 4.3. Table 22. RBT X Non RBT Test Case Design Feat. Prior. No. Flows No. RBT No. Functional and RBT designed FR01 High = 6 FR15 High = 7 FR02 Medium 6 5 No. Risks = 5 FR12 Medium 6 4 No. Risks = 4 FR11 Medium 1 4 No. Risks = 4 FR10 Medium 5 4 No. Risks = 4 FR16 Medium 1 3 No. Risks = 3 FR13 Low 7 2 No. Risks = 2 FR14 Low 1 0 No. Risks = 0 Total Executing and Evaluating Tests As we did not test the Health-Watcher system, the values in Tables 23 and 24 are examples to explain the proposed metrics in Section 3. The metrics presented in Tables 23 and 24 exists in any software test process, except that the test cases are based on risks. So, they also verify if an acceptable level of mitigated risks has been reached through test execution.

6 Table 23. RBT Execution Results Feat. No. RBT Planned No. RBT Executed No. RBT Pass No. Defects FR FR FR FR FR FR FR FR Total Table 24. Test Execution Metrics M / 24 = 45 % Executed M1.2 FR % Executed FR % Executed M7.1 Blocker. 40%, Critical. 20%, Major. 10% Minor. 25%, Trivial. 5 % M / 11 = 54 % M8.2 FR01. 1 CR, FR15. 1 CR, FR02. 2 CR FR12. 2 CR, where CR = Change Request 4.6. Controlling Tests The test execution results are the main input for the risk control. When a risk-based test case is executed and it pass, that means the identified risk does not exist but, when it fails, and the defect is fixed, the identified risk is mitigated. Table 24 shows that 54% of the RBT test cases found a defect. 5. Conclusion and Future Work This paper presented a set of Metrics to measure RBT activities. RBTProcess, an approach of Risk-based Testing was used to evaluate this set of metrics. The main goal of RBT is to identify the most common risks that occur in projects developed within a specific application domain, organizing this information and allowing its reuse in risk management processes for specific software projects. This work is an ongoing project. There are a number of limitations that have to be overcome, as risk metrics, metrics collection, adaptation mechanisms for the RBTProcess in use by an organization, evaluation of recommended practices for given risks, among other points. The authors are working on a tool (RBTTool) for collecting and analyzing data and using it for evaluating and improving the suggested test cases based on identifying risk and to plot the tendency of evolution of the risks. 6. References [1] Kaner, C., J. Falk and H.Q. Nguyen, Testing computer software, Wiley; 2nd edition, [2] Myers, G.J., The Art of Software Testing, John Willey and Sons, 1979 [3] Rios, E., Análise de Riscos em Projetos de Teste de Software, Alta books, 2005 [4] L. van der Aalst, Risk Based Test Strategy Judged. In 7th ICSTEST, Düsseldorf-Germany, [5] Fenton, N.E. and S.L. Pfleeger, Software Metrics: A Rigorous Approach, PWS Publishing Company, 1997 [6] S. Amland, Risk Based Testing and Metrics: Risk analysis fundamentals and metrics for software testing including a financial application case study. In 5th International Conference EuroSTAR 99, Barcelona-Spain, [7] E. Souza and C. Gusmão, RBTProcess - Modelo de Processo de Teste de Software baseado em Riscos. In 13º WTES - Workshop de Teses e Dissertações em Engenharia de Software, [8] E. Souza, C. Gusmão and H. Rocha, RBTProcess - Proposta de Modelo de Processo de Teste de Software baseado em Riscos. In III EBTS Encontro Brasileiro de Teste de Software, 2008 [9] PMBOK Guide, Project Management Institute; 3rd edition, [10] Carr M.J., S.L. Konda, I. Monarch, F.C. Ulrich and C.F. Walker, Taxonomy-Based Risk Identification, Software Engineering Institute Technical Report, Carnegie Mellon University, [11] J. Bach, The Challenge of Good Enough Software, presetend at American Programmer, [12] J. Bach. James Bach on Risk-Based Testing: How to conduct heuristic risk analysis. In Software Testing & Quality Engineering Magazine, 1999, pp [13] S. Besson, A Strategy for Risk-Based Testing, available at [14] Y. Chen, Specification-based Regression Testing Measurement with Risk Analysis, Master Thesis, University of Ottawa-Canada, [15] L.K.U. Jørgesen, A software tool for risk-based testing. Graduation Work, Norwegian University of Science and Technology-Norway, [16] L.H. Rosenberg, R. Stapko, A. Gallo, A. Risk-based object oriented testing. In Twenty-Fourth Annual Software Engineering Workshop, NASA SEW24, [17] J. Bach, Troubleshooting risk-based testing. In at Software Testing & Quality Engineering Magazine, [18] Software Metrics Guide, available at etrics.html#p31, 2008 [19] K.R. Konda, "Measuring Defect Removal Accurately", available at _testmetrics.htm, [20] S.L. Pfleeger, Risk Business: what we have yet to learn about software risk management. In Journal of Systems and Software, 53, 2000, pp [21] S. Soares, E. Laureano and P. Borba, Implementing Distribution and Persistence Aspects with AspectJ. In Proceedings of the 17th ACM conference on OOPSLA 02, ACM Press, 2002, pp