Project description: Specification coverage and traces distances

Transcription

1 Project description: Specification coverage and traces distances M. Willemsen 8th January 2004

2 Contents 1 Project description Introduction Context Problem description Project phasing Project organisation

3 Chapter 1 Project description 1.1 Introduction This document describes the project Specification coverage and traces distances at the University of Nijmegen, which should result in a tool implementing a new theory concerning specification coverage and traces distances in automatic formal testing. The graduate student is Mathieu Willemsen at the University of Nijmegen, the Netherlands. The general outline for this document is first some context information, followed by the problem description, the project phasing and some general project information at the end. 1.2 Context Software testing is a necessary part of software development. Software testing today is mostly done by having people examine the specification. Then,based on the specification, they devise test-cases, execute the test-cases and compare the results against the specification of the software. These tests are mostly made using established heuristics and their own insight. This process is costly and time-consuming. Further adding to the cost is that there needs to be regression testing every time there is a change in the software. There are many different kinds of tests that can be performed: stress testing, security testing, acceptance testing, performance testing, robustness testing, etc. We ll be dealing with black box functional testing, also known as conformance testing, in this project. Black box means that the internal working of the implementation is unknown to the test team. Functional that the test team is only interested in determining whether the behavior of the implementation conforms with the specification. Today most specifications are still written in natural language. Even though this makes them easy to read, it also causes ambiguity. As this project will be dealing with automated testing any ambiguity in the specification will make automatisation extremely difficult. So a suitable formal language in which to 2

4 write the specification is needed. The formal way used to specify the behavior of a program used within Tor- X to is in the form of a Labeled Transition System. specification as a Labeled Transition System in a appropriate formal language. A LTS is a formal description describing the states in the program and the valid input and output transitions between the different states. Briefly, a LTS is a four-tuple (S,L,T,s i ) with L the set of labels and S the set of states. s i is the begin state, T is a partial function of S x L x S giving the transitions in the system. A small LTS can be visually represented as a network of in- and outgoing labels,drawn as arrows, between states, which are drawn as circles. Larger ones can be better expressed in a process algebraic language. Through a test derivation algorithm a test tool can generate sound sequences of allowed inputs and outputs out of the formal specification, which together form a test case. The test case can then be used to test the implementation by inputting the valid inputs and observing the outputs of the implementation. If the implementation does not accept valid input, accept invalid input or gives the wrong output then a difference between the specification and the implementation has been found which indicates an issue. The generated sequence of observable inputs and outputs is called a trace. Goga has come up with an interesting theory concerning specification coverage. The basic idea of the theory is that if two traces are very similar, then they are probably both dealing with the same part of the specification and if two traces are very different, then they are probably generated from quite different parts of the specification. In the theory traces are converted to a special representation called a marked trace, the reason being that these marked traces posses certain useful properties. The level of difference between marked traces is converted into a distance. What is exactly meant with difference is left blank for the moment. The distance calculation algorithm is based on edit distance theory, but adapted to work for marked traces. By going one level higher and looking at the level of difference between two sets of traces a level of coverage between these two sets can be derived. Again what is precisely meant with the level of difference is left blank for the moment. By using certain algorithms to generate a finite set of marked traces from the specification and determining the level of difference between this derived finite set and the (infinite) set of all possible traces from the specification, a specification coverage can be determined. Advantages of such a theory are that feedback can be generated for formal testing tools. This feedback could help formal testing tools to better decide when to stop testing. This theory could also be a basis for further work in the field of test generation and test selection. Where this theory could work as a test purpose and help decide how to generate and select tests with the highest specification coverage. 3

5 1.2.1 Problem description Goga s theory is a new idea and hasn t been tried or tested in practice. The current version of his theory contains some ambiguity, which needs to be removed. The scope of this project will limit itself to finite deterministic LTS s as specification. This is to keep the complexity of the project at an reasonable level for this kind of project. One would like to be able to express how confident one is that the implementation conforms with the specification. The main goals of the graduation project are a valid theory concerning specification coverage based on the concept of trace distances and a working implementation of that theory. Possibly also determine whether something can be said about the amount of specification coverage that a single trace will add to an already existing test set. This would be useful for test selection and test generation. These main goals can be subdivided into smaller goals. 1. Validating whether the marked trace distance derivation algorithm has the required distance properties and whether the algorithm is sound in the context of current intuition concerning the difference between traces. Improve and/or adapt if problems are found. 2. Validating the coverage concept. Improve if possible or if the current theory is wrong. 3. Create a working implementation based on the latest theories. The number one priority in this project is getting a valid theory. Getting a good implementation comes second Project phasing In this section a phasing of the project is worked out. Because the subject of the project is new and untested, it s hard to plan exactly how long the different phases of the project will take. The relative newness of the subject also makes it hard to anticipate the possible problems that might arise during the course of the project. The approach taken to bring this project to a good ending is by first familiarising myself with the theory. After getting enough knowledge of the theory, I need to create my own non-ambigu version of the theory. Possibly already making large changes to it if needed. The next step is implementing the theory. The language choosen for this is Clean. The reason for choosing Clean as the implementation language is that most of the theory was described in a functional way, so that implementation in Clean would be relatively easy. Clean is also a high level programming language meaning that the programmer doesn t need to worry about a lot of low level programming problems. After finishing the implementation, I will use it to test whether my current version of the theory satisfies all requirements. When issues are encountered, I will improve my theory, make the nesecary changes to the implementation and 4

6 recommence the testing process. Once a theory has been found that satisfies the requirements. This theory can be implemented as a tool for TorX. Providing feedback and laying a basis for further work. In all phases there needs to be a good documentation of the work done and decisions taken. A table listing a rough estimate of time for the different phases in the project. Phase First familiarise myself with the theory. Possibly adjust the theory if needed. Make a prototype implementation of the theory for use in further testing. Perform tests to discover any faults in the theory Draw conclusions from test results. Rework theory based the results of the previous phase Implement a integrated tool for TorX. Dealing with unforseen problems Duration 4 weeks 8 weeks Project organisation The graduation project will be carried out at the Software Technology group of the Computer Science Faculty at the University of Nijmegen(Radboud Universiteit Nijmegen). The duration should be around a half year. The main supervisor is Jan Tretmans. He is assisted by Rene de Vries and Peter Achten. Progress meetings between Mathieu Willemsen and supervisors will be held in one to two week intervals to evaluate project status, provide Mathieu Willemsen with feedback to improve his work and discuss the subjects of interest at that time. 5