Automated Statistical Testing Suite for Software Validation

Transcription

1 Automated Statistical Testing Suite for Software Validation Thomas Thelin Dept. of Communication Systems, Lund University Box 118, SE LUND, Sweden Abstract Software validation is an important activity in order test whether the correct software has been developed. Several testing techniques have been developed, and one of these is statistical usage testing (SUT). The main purpose of SUT is to test a software product from a user's point of view. Hence, usage models are designed and then test cases are developed from the models. The development of test cases from the usage model can be made automatically by using a tool. In order to increase the efficiency of the test phase, software tools are important. This paper presents a new tool for statistical usage testing called MaTeLo. The tool is developed in a joint European project with six industrial partners and two university partners. The purpose of the tool is to automatically produce test cases based on usage models. The usage model can either be specified as message sequence charts, UML sequence diagrams or state charts. Furthermore, besides producing test cases, the tool calculates important metrics, for example, the reliability of the software and the coverage of the usage model. Keywords: Statistical usage testing, usage-based testing, testing tool, Markov model, software validation. 1. Introduction One main purpose of the testing phase is to evaluate the quality of a software product. An important quality attribute is reliability, which is defined as failure-free operation during a specified time period (ISO/IEC 9126:1, 2001). SUT is a testing technique that provides testers with a reliability measurement (Adams, 1984; Musa, 1998). SUT is a similar technique as usage-based testing and operational profile testing (Musa, 1993), and was introduced in the cleanroom principle (Cobb and Mills, 1998; Linger, 1994). Although not many companies use the cleanroom approach, some concepts of cleanroom are still in use (Kelly and Oshana, 2000). The purpose of SUT is to validate the developed software from the intended usage of the software (Runeson and Wohlin, 1995; Runeson and Regnell, 1998). The assumption is that not all test cases can be executed, and hence some test cases have to be chosen. In this case, the selection is based on one or several usage profiles, and then the test cases are selected from these profiles. Consequently, the functions that will be used the most will be tested before the software is released, and hence the software will be of a certain quality. In order to support the model building and the test case generation, automation is needed. This paper describes a tool (named MaTeLo), which is under development (MaTeLo, 2003). A prototype of the tool is ready, and the final version will be finished in June The purpose of the tool is to automatically derive test cases based on usage models. Furthermore, the tool also calculates quality metrics, e.g. reliability of the software. MaTeLo is developed in a joint European project, named Markov Test Logic (MaTeLo) Automated Statistical Testing Suite for Software Validation (IST ). The outline of the paper is as follows. In Section 2, the objective of SUT is presented. The European project is described in Section 3, and in the Section 4 the different parts of the MaTeLo tool are pre- 1

2 sented. In Section 5, the value of the tool is described, and the paper ends with a summary and a discussion of the future plans in Section Statistical Usage Testing The purpose of statistical usage testing (SUT) is to test the developed software from the intended usage (Cobb and Mills, 1990). The idea is to use statistics and reliability metrics to achieve highquality software. Hence, the main objective is not to cover all requirements (although this is a secondary objective), but to cover the intended usage of the software. In addition, the goal is not necessarily to detect all the faults, but the ones that are most critical from the intended usage of the software (Whittaker and Thomason, 1994). The origin of SUT is often referred to the paper by Adams (1984). However, SUT was first used as a tool in the concept of cleanroom (Cobb and Mills, 1990; Runeson and Wohlin, 1995). As SUT has evolved, some related techniques have been developed. Usage-based testing (Runeson and Regnell, 1999) and operational profile testing (Musa, 1993) are related methods that use the concept of SUT with small changes. Furthermore, inspection techniques based on the ideas of SUT have been developed (Thelin et al., 2003a; Thelin et al., 2003b). Today, cleanroom is not used by many companies (although exceptions exist), but the usage as well as research of SUT increase (Poore, 2000). SUT is based on reliability theory and estimation techniques (Le Guen and Thelin, 2003); state charts are the main representation used. In order to perform SUT, one or several usage profiles are developed, which describes the users way of using the software (Runeson and Regnell, 1998). The testing is thus focused on detecting the faults that are most likely going to be stimulated by the users, and hence leading to a failure. Faults that will not affect the user are not that interesting to find, and because these are not leading to a failure. The first step of SUT is to develop a usage model of the intended usage. This is often represented as a state chart, but also other inputs can be choose, for example, UML sequence diagrams. Each state represents a condition in the software and each transition represents an event. A probability value is connected with each transition. The probability values show the likelihood that an event will happen. When the state chart is completed with states, transitions and probabilities, the state chart is used to generate test cases, and by using statistical theory, the reliability of the software can be calculated. The testers of the software are provided with the reliability estimate as well as with a coverage measure of the usage mode. These are used as stopping criteria for the testing. Examples of SUT used in real applications are presented by Kelly and Oshana (2000) and Kouchakdjian and Fietkiewicz (2000). 3. MaTeLo A European Project MaTeLo is the name of the tool as well as of the European project, which includes 8 partners. The project started in January 2002 and will end in June The main objectives of the project are to: Investigate formal specification languages for various industries, specifications simulation methods and associated tools. Develop an automatic generator of test cases based on statistical methods. The test cases should be derived from a Markov usage model. Provide test execution in heterogeneous environments, test results analysis and quality reports generation for regular follow-up during the development. Provide metrics to accurately evaluate software reliability and performance throughout the development process. Integrate the testing tools in real development environments for industrial validation. 2

3 The industrial partners involved in the project represent automobile, avionics, aerospace, and telecommunications. In addition, two universities are involved in the project. The technical partners of the project are: Alitec, France Lund University, Sweden Fredrich-Alexander-University, Germany Danet, Germany The end users involved in the project are: Magneti Marelli, France Israel Aircraft Industry, Israel Alenia Spazio, Italy NEC, Franc 4. MaTeLo Principles MaTeLo is based on different logical parts, see Figure 1. Users of the tool can specify the input to the tool in three different notations. They can either use MSC:s or UML sequence diagrams, which specify the user interaction with the system that will be tested. Users can choose either representation and then it is translated into a state chart, or users of the tool can directly specify the usage in a state chart, which is the third option. All these models describe the usage of the software. Hence, the technical details are hidden from the users. An example of a state chart is shown in Figure 2. Scenarios (MSC, UML-SD) Converter Usage knowledge MM skeleton (XML) MU editor Markov Model TC generator TC converter Test Cases (XML) Test Cases (TTCN-3) Metrics (e.g. Reliability, Coverage) Figure 1. Overview of the logical components in the MaTeLo tool. 3

4 Figure 2. A screen shot of the usage model editor. When the state chart has been derived, probabilities in the transitions between states are either automatically calculated using a pre-defined distribution or they can be specified by the user. This is carried out in the Markov Usage Editor (MU). In the MU, one or several usage profiles can be used. This is valuable since different intended users will utilize the software differently. When the usage models have been designed, the tool checks whether the model has been correctly designed and then it converts the usage models into test cases. The test language used is TTCN-3. In Figure 3 a screen shot of the test interface of MaTeLo is shown. After the test cases have been derived, they are run on the system under test and the outcome is analysed. Several important metrics are automatically calculated in the tool, for example, the reliability of the system and the coverage of the usage models. This provides useful information to the testers of Figure 3. A screen shot of the test case generator. 4

5 whether the software is good-enough to be released. The output of the testing also shows what failures have been detected. The reliability calculation is based on theory of Markov models. 5. The Value of MaTeLo The main objective of MaTeLo is to utilize SUT to derive test cases from usage models. Hence, this means that the testers will develop a model of the usage of the software. Since different users will use the software in different ways, several usage profiles can be developed. For example, the usage profiles can be divided into novice users and experienced users. Since the usage of the software will differ between these groups, the states, transitions and in particular the probabilities will differ. Another benefit of MaTeLo is that if someone is more familiar with Message Sequence Charts (MSC) or UML sequence diagrams, the user can specify the basic input to the tool in these notations. After the communication between the user and the software under test has been specified, MaTeLo automatically converts the input to a state chart. Then the user can provide probabilities to the transitions in the chart, or even this can be made automatically by using a certain distribution. When all usage profiles have been specified in the state chart, test cases are automatically produced. The format for these is either TTCN-3 or ASCII. Hence, user can choose either between a specific test description language or plain text as output representation. Another main benefit of MaTeLo is the automatic calculation of quality metrics, such as reliability and coverage of the usage models. This provides valuable input to the testers, so they know when to stop testing. The reliability value can be decided beforehand; then the tool will provide answers of whether it is necessary to continue testing or not. In summary, the main benefits of MaTeLo are: Tests software from the users point of view by using different usage profiles. Automatically converts usage models into test cases. Calculates important metrics (for example, reliability) that can be used as stopping criterion of the testing. Provides options for several input formats, i.e. UML sequence diagrams, MSC:s and state charts. 6. Summary and Future Plans This paper has described a tool, which is based on statistical usage testing. At the front-end of the tool, testers provide the tool with models the intended usage of the software. This means the most critical failures will be detected, since the test cases cover the usage and not the technical specification. The output of the tool is test cases, which can be run on the system under test. Furthermore, the tool also provides testers and managers with important quality metrics, for example, reliability. The tool is currently under development and will be finished in June Currently, a prototype is ready, and this will be empirically evaluated against the current test process used by the end users. The empirical evaluation will be conducted in the end users environment and compared to their currently used testing method. Some of the questions that will be answered by the evaluation are: Is the MaTeLo tool easily integrated into the development process? Is the MaTeLo tool easy to use for the testers? Which of SUT (using MaTeLo) and the current test process is most efficient and effective? Which of SUT (using MaTeLo) and the current test process is easiest to use for a developer and a tester? 5

6 Furthermore, together with the evaluation, validation of the tool will be made. The purpose of the validation is to investigate whether the requirements specified in the beginning of the project comply the implementation of the tool. The evaluation and validation will be monitored and summarized by researchers at Lund University. References Adams, E. N., Optimizing Preventive Service of Software Products, IBM Journal of Research and Development, Cobb R. H. and Mills, H. D., Engineering Software Under Statistical Quality Control, IEEE Software, 7(6):44-54, Fowler, M. and Scott, K., UML Distilled A Brief Guide to the Standard Object Modeling Language, Addison-Wesley, USA, Grabowski, J., TTCN-3 - A New Test Specification Language for Black-Box Testing of Distributed Systems, Proc. 17th International Conference and Exposition on Testing Computer Software, ISO/IEC :2001, Software Product Evaluation General Guide, International Standard 9126, ITU-T Z.120 Message Sequence Charts, MSC, ITU-T Recommendation Z.120, Kelly D.P. and Oshana, R.S., Improving software quality using statistical testing techniques, Information and Software Technology, 42(12): , Kouchakdjian, A. and Fietkiewicz, R. Improving a product with usage-based testing, Information and Software Technology 42(12): , LeGuen, H. and Thelin, T., Practical Experieces about Statistical Usage Testing, Proceedings Software Technology and Engineering Practice (STEP 03) Workshop on Software Test and Reliability Estimation Process, Linger, R., Cleanroom Process Model, IEEE Software, pp , MaTeLo project website, http;// Musa, J., Operational Profiles in Software-Reliability Engineering, IEEE Software, 10:(2):14-32, Musa, J., Software Reliability Engineered Testing, McGraw-Hill, USA, Poore, J., Introduction to the special issue on: model-based statistical testing of software intensive systems, Information and Software Technology 42(12): , Runeson, P. and Wohlin, C., Statistical Usage Testing for Software Reliability Control Informatica, 19(2): , Runeson P. and Regnell, B., Derivation of an Integrated Operational Profile and Use Case Model, Proc. 9th International Symposium on Software Reliability Engineering, pp , Thelin, T., Runeson, P. and Wohlin C., Prioritized Use Cases as a Vehicle for Software Inspections, IEEE Software, 20(4):30-33, July/August Thelin, T., Runeson, P. and Wohlin C., An Experimental Comparison of Usage-Based and Checklist-Based Reading, IEEE Transactions on Software Engineering, 29(8): , August Whittaker J. A and Thomason, M. G., A Markov Chain Model for Statistical Software Testing, IEEE Transactions on Software Engineering, 20(10): ,