Introducing Structured Testing

Transcription

1 Introducing Structured Testing Master s Thesis Lund Author: Hanna Färnstrand Supervisors: Henrik Andersson, Sogeti Lars Hall, Apptus Technologies Patrik Wåhlin, Apptus Technologies Examiner: Per Runeson, Department of Computer Science, Lund University

2 Abstract Software development often runs on a tight schedule. When facing a fast approaching deadline, immature companies tend not to want to spend time on formal test processes, but would rather focus their effort on adding the required functionality into the system. Testing is often viewed as a necessary evil. However, with testing costs often reaching half of the project costs, more and more software companies become aware of the advantages of structuring the test process to save money and at the same time lessen the risks of releasing faulty products on the market. This Master s thesis strives to introduce a structured testing process in a company where the testing is currently performed ad-hoc. A comparison of five existing test process framework models was done. Then, a summary of test process elements viewed as basic in all of the framework models was drawn up. These elements were considered necessary to introduce in the start-up phase of the process improvement. Action research was then performed through interviews, document studies, observations and questionnaires in order to assess the strong and the weak parts of the current test process, followed by improvement suggestions to the weak parts. The improvement actions were influenced by the start-up test process elements while considering the development process. The result is a set of templates, diagrams and instructions guiding the user in how to introduce some first important steps of structured software testing. Some of the improvement actions were then implemented in a chosen project and later evaluated. The introductory test process elements in the improved test process model include recognizing testing as its own process, early access to test basis, review of requirements for testability, early test planning, use of test techniques, test case specification and scripting, defect management and test reporting. The difficulties in the studied test process are thought to be common in immature test processes, and the improved test process elements suggested in this thesis could thus be generalized for use in other organizations. The thesis report provides the reader with some basic understanding of the concepts of software testing. Also, existing models on test process improvement are described. 2

3 Acknowledgements The author would like to thank the following people for their valuable support during the making of this Master s thesis: Henrik Andersson at Sogeti, for coming up with the subject, giving me advice on test process assessment and improvement, and giving me comments on my work. Per Runeson at SERG, for helping me decide which way to go, commenting my ideas and thoughts and by reviewing my written work. Jörgen Porath at Apptus, for providing me with insight into and information about the organization and the test process, and for letting me try my ideas. All the other people at Apptus; Patrik Wåhlin, Lars Hall, Louise Nygren, the developers in the Web Crawler project, and everybody else who in any way helped me in assessing and improving the test process. 3

4 Table of contents ABSTRACT... 2 ACKNOWLEDGEMENTS INTRODUCTION BACKGROUND PURPOSE A PRESENTATION OF THE COMPANIES INVOLVED Sogeti Apptus Technologies OUTLINE TERMINOLOGY SOFTWARE TESTING WHAT IS SOFTWARE TESTING? SOFTWARE QUALITY LEVELS OF TESTING Unit test Integration test System test Acceptance test Regression test THE RELATIONSHIP BETWEEN THE TEST PROCESS AND THE DEVELOPMENT PROCESS PROBLEM AREAS CONCERNING TESTING THE NEED FOR A STRUCTURED TESTING APPROACH RISKS IN INTRODUCING STRUCTURED TESTING BASIC TESTING TECHNIQUES Static techniques Dynamic techniques TEST DOCUMENTATION The test plan The test specification The test report WHO SHOULD DO THE TESTING? WHEN SHOULD TESTING START? WHEN SHOULD TESTING STOP? TEST TOOLS TEST PROCESS IMPROVEMENT FRAMEWORK MODELS The Test Process Improvement (TPI) model The Testing Maturity Model (TMM) The Testability Maturity Model The Test Improvement Model (TIM) The Minimal Test Practice Framework (MTPF) METHODOLOGY RESEARCH METHODS Survey Case Study Experiment Action research THE APPLICABLE METHOD DATA COLLECTION Available sources Data collection methods Drawing conclusions from collected data Validity

5 5 THE ACTION RESEARCH WORK OUTLINE ASSESSING THE TEST PROCESS IMPROVING THE TEST PROCESS THE ASSESSMENT INTERVIEW Collected data from the assessment interview Analysis of data from the assessment interview ARCHIVAL RECORDS Collection of data from archival records Analysis of data from archival records OBSERVATIONS Collection of data from observations Analysis of data from observations MEETING WITH THE TEST MANAGER EXISTING TERMS, CONCEPTS AND PRACTICES PRE-IMPROVEMENT QUESTIONNAIRE Collection of data from the pre-improvement questionnaire Analysis of data from the pre-improvement questionnaire POST-IMPROVEMENT QUESTIONNAIRES Collection of data from the post-improvement questionnaires Analysis of data from the post-improvement questionnaire RESULTS RESULTS FROM THE RESEARCH IMPROVEMENT MEASURES Specific improvements in the chosen project Results from the specific improvements in the chosen project CONCLUSIONS CONCLUSIONS REFERENCES BOOKS JOURNAL ARTICLES CONFERENCE PAPERS APPENDIX I THE ASSESSMENT INTERVIEW APPENDIX II THE PRE-IMPROVEMENT QUESTIONNAIRE APPENDIX III THE CURRENT TEST PROCESS MODEL APPENDIX IV THE IMPROVED TEST PROCESS MODEL APPENDIX V A GUIDE TO TEST PLANNING APPENDIX VI A REQUIREMENTS REVIEW CHECKLIST APPENDIX VII A GUIDE TO TEST TECHNIQUES APPENDIX VIII A GUIDE TO TEST EVALUATION APPENDIX IX THE IMPROVED TEST PROCESS ASSESSMENT QUESTIONNAIRE

6 1 Introduction 1.1 Background This report is the description of the work behind, and the conclusions drawn in, a Master s Thesis in the field of Information and Communication Engineering. The thesis work was performed by the author in collaboration with Sogeti, Apptus Technologies, and the Software Engineering Research Group (SERG) at the department of Computer Science, Lund University. The subject of the thesis was proposed by Sogeti, and the research was performed on-site at Apptus Technologies. The target group intended in this report is persons with some basic knowledge of software engineering, including software development and/or software testing, but with no further expertise in the software testing area. A table explaining some words and expressions used in the report is provided at the end of chapter Purpose The problem stated by Apptus Technologies is that a structured testing process is lacking. Testing is currently performed in an ad-hoc manner, and the results depend to a large extent on the individual tester. The company has not yet come to a point where faulty software on the market constitutes a problem, but attributes this to being successful in finding very competent developers, and not having projects large enough to make structured testing necessary. As the company grows and the magnitude of the projects increases, the need for a more structured process however becomes evident as costs for and risks related to releasing products not thoroughly tested most certainly will rise. The purpose of this Master s Thesis is to fulfil the following goals: Evaluate the current test process at Apptus Technologies. The assessment includes finding the strong and the weak parts of the test process, i.e. the elements that are in need of improvement. Create an improved test process, adapted to the needs, wishes and possibilities stated by Apptus Technologies. Implement the improved test process in a chosen project at Apptus Technologies. Note that in this report, the current test process refers to the test process as it was at the start of the research period, whereas the improved test process refers to the new, changed process suggested as a result of the research, and implemented at the end of the thesis work. 1.3 A Presentation of the Companies Involved The thesis was performed by the author in cooperation with the companies Sogeti and Apptus Technologies Sogeti Sogeti is Cap Gemini s local professional services division. The Swedish offices of the Sogeti Center of Test Excellence (SCTE) are located in Lund and Stockholm. SCTE focuses on consulting companies in software testing-related activities such as test process improvement, test automation and test execution. Sogeti is involved in the development of the test process improvement model TPI and the renowned test 6

7 method TMap (Koomen et al. 2006). The company s goal with this thesis is to find a generic method for test process improvement. Sogeti s contribution to the thesis work was to provide material, assistance and support regarding test-related questions and activities. For more information about Sogeti, the reader is referred to < Apptus Technologies Apptus Technologies, headquartered in Lund, Sweden, is a software company that provides database solutions for e-commerce and e-directory applications at some of the world s most heavily trafficked sites. Customers include Eniro, CDON and the Swedish Public Radio. At the time of writing, Apptus had about 60 employees, where most of them developers. The developers worked in small project groups of up to ten people each. The company s aim with the thesis is to find and introduce a more structured test process. Apptus contribution to the thesis work was to create the possibility for a study and assessment of their current test process and the chance to create and implement an improved test process in one of their projects. For further information regarding Apptus Technologies, the reader is referred to < 1.4 Outline The outline of this report is as described: Chapter 1 Introduction In the introductory chapter, the background and the purpose of the Master s Thesis are explained. Also, the companies involved are presented. Last, some testing related terms are explained to the reader. Chapter 2 Software Testing In chapter 2, some fundamentals of software testing are described. The theory explained in this chapter strives to provide the reader with enough background information to understand the concepts and terms in the following chapters. Chapter 3 Test Process Improvement Chapter 3 explains existing models on test process improvement. Elements from these models are used in the test process assessment in chapter 5. Chapter 4 Methodology Chapter 4 addresses research methodologies and explains the methodology used in this thesis. Data collection methods are also presented. Chapter 5 The Action Research Work Outline The assessment process and the data collected in it are presented in chapter 5. An analysis of the collected data, resulting in a definition of the problem areas in the current test process, is also found in this chapter. Chapter 6 Results The results from the assessment of the current test process and the implementation of the new test process are described in chapter 6. 7

8 Chapter 7 Conclusions In chapter 7, the conclusions drawn from the thesis work are presented. Chapter 8 References The references used and referred to in this report are presented in chapter 8. Appendix I The Assessment Interview The questions asked in the assessment interview can be found in Appendix I Appendix II The Pre-Improvement Questionnaire The questionnaire used to assess the testing in a specific project is presented in Appendix II. Appendix III The Current Test Process Model A flowchart of the current test process model in relation to the development process, can be found in Appendix III. Appendix IV The Improved Test Process Model The improvements suggested to the test process are presented in a flowchart in Appendix IV Appendix V A Guide to Test Planning In Appendix V, a guide to help testers plan for tests and complete the test plan template is found. Appendix VI A Requirements Review Checklist A checklist that can be used for reviewing requirements for testability is presented in Appendix VI. Appendix VII A Guide to Test Techniques A guide to choosing the appropriate test technique can be found in appendix VII. Appendix VIII A Guide to Test Evaluation In Appendix VIII, a guide on how to evaluate test results is presented. Appendix IX The Improved Test Process Assessment In Appendix IX, the assessment of the implemented improved test process is presented. 8

9 1.5 Terminology An explanation of terms used in this report, in alphabetical order: Black-box Testing without considering the inner structures of the code; testing viewing the item to be tested as a black box. Metrics Quantified observations of the characteristics of a product or process. Milestone A tangible event that is expected to occur at a certain time in the project s lifetime, used by managers to determine project status (Burnstein 2003). MTPF Minimal Test Practice Framework (Karlström et al. 2005) Risk The chance that a failure occurs in relation to the expected damage when the failure really happens (Koomen and Baarda 2005). Test A group of related test cases and test procedures (Burnstein 2003). Testability The ability to perform cost-effective testing on a system (Koomen and Pol 1999). Test basis The documentation that the testing is based on, for example the requirements, functional or technical specification. Test case The items that make up the basis for the execution of the test. Test cases contain, at the least, the test inputs, the execution conditions and the expected outputs. Test The components making up the environment where the test takes environment place; the hardware, software, procedures, means of communication, etc. Tester The person planning and/or executing the test cases. Test object The (part of the) software or information system to be tested (Koomen and Baarda 2005). Test phase The phase in a development project that comprises test activities. Test plan A framework based on the planning of the test process; defining what to test, when to test, and how to test. Test process The set of activities involved in testing, made up of at least three phases; test planning, test setup and preparation, and test execution (Yutao et al. 2000). Test process Optimizing the quality, costs, and lead time of the test process, in improvement relation to the total information services (Koomen and Pol 1999). Test technique A structured way of defining test cases from the test basis Test tools Automated aids for the test process. Test unit A collection of processes, transactions and/or functions that are tested collectively (Koomen and Baarda 2005). Testware The products of testing, such as specifications, test plans, files, etc. TIM Test Improvement Model (Ericson et al. 1997). TMM Testing Maturity Model (Burnstein 2003). TPI Test Process Improvement (Koomen and Pol 1999). White-box testing Table 1 - Terminology Testing based on the code or on the technical design documents, requiring knowledge of the internal structure of the system. 9

10 2 Software Testing 2.1 What is software testing? In the software development process, testing is used to show that the products work as intended or as required. Burnstein (2003, p. 7) gives two alternate descriptions of the definition of testing : Testing is generally described as a group of procedures carried out to evaluate some aspect of a piece of software. Testing can be described as a process used for revealing defects in software, and for establishing that the software has attained a specified degree of quality with respect to selected attributes. Koomen and Pol (1999, p. 5) uses the following definition for testing: Testing is a process of planning, preparation, and measuring aimed at establishing the characteristics of an information system and demonstrating the difference between the actual and the required status. Software testing involves comparison between a test object and a frame of reference with which that object should comply (Koomen and Pol 1999, p. 5). Testing gives an indication of the difference between the actual status of an object or output of an action, and the desired or required status or output. If the test case shows that the actual status and the desired status are one and the same, it is considered passed. On the other hand if the actual and the desired status are not the same, the differences provide the tester with useful information about the system. The process of software testing in turn encompasses two processes. Burnstein (2003, p. 6) defines these as validation the process of evaluating a software system or component during, or at the end of, the development cycle in order to determine whether it satisfies specified requirements; and verification the process of evaluating a software system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase. Validation includes inspecting the final product, and is done by executing software (Koomen and Pol 1999, p. 10). In short, the validation process answers the question Have we built the right product? The verification process means examining intermediate products of the development phases. This is sometime referred to as evaluation, and is mostly done by inspecting or reviewing documents (Koomen and Pol 1999, p. 10). Verification answers the question Have we built the product right? 10

11 2.2 Software quality If the purpose of testing is to establish that the software has sufficient quality, the need for a definition of quality is evident. What constitutes good software quality depends on what is considered important for a specific system. Criteria to base software quality on could be: The product meets the user s expectations. This definition of software quality is very subjective and depends on who the user is and the way he or she is going to use the product. All users will probably not have the same expectations on the product. Also, the user will only judge the product on what he or she can see and experience. The product meets all of its specified requirements. Meeting all the requirements is not on its own a sufficient measurement of software quality. To a large extent, this depends on the quality of the requirements. If requirements are wrong or missing, the quality of the software might be severely reduced. The product meets all of its specified quality metrics. Burnstein (2003, p. 24) describes a quality metric as a quantitative measurement of the degree to which an item possesses a given quality attribute. Examples of quality attributes are the so called -ilities, such as reliability, portability, usability and maintainability (Sibisi and van Waveren 2007). This is a good basis for establishing software quality, but quality attributes are often difficult to measure. Different quality attributes could also influence and counteract each other. If, for example, extra security is added to the product through a log-in feature, usability might be decreased because it will take longer and be more difficult to access the product. 2.3 Levels of testing The process of testing is divided into different levels. These levels reach from low (unit tests) to high (system and acceptance tests), as shown in figure 1 below. Unit (e.g. a class) System or sub-system Unit test Integration test System test Integrated units Acceptance test Finished system Figure 1- Levels of testing 11

12 2.3.1 Unit test A unit test is a test performed on a single unit in the system. A unit is the smallest possible testable software component (Burnstein 2003, p. 137). What is viewed as a unit depends on, for example, the development method, but is often a single method or a class. Since the component is relatively small in size and often can be isolated, unit testing is often the easiest level to test on. In many organizations, the person performing unit tests is also the developer of the unit, since unit testing focuses on evaluating the correctness in the code. Wohlin (2005, p. 155) states that the purpose of unit testing is to ensure that the unit works as intended. Mainly the functionality, meaning that a set of input data generates the expected output data, is tested Integration test Integration testing strives to ensure that the single units work together. Interfaces between units or systems are thus tested for defects. Experienced testers recognize that many defects occur at module interfaces (Burnstein 2003, p. 153). Integration tests are often iterative. They first take place between single units and then between sub-systems (or clusters in object-oriented programming), eventually building up a complete system. This can be achieved either by a top-down or a bottom-up strategy. The purpose of integration testing is to validate that the system and its interfaces work according to the specified design or system architecture (Wohlin 2005, p. 155) System test A system test tests the system as a whole. The test cases in a system test are often based on the requirements in a black-box manner (see section 2.4.1). System tests do not only test functional behaviour, but also non-functional such as performance and usability. Also, system tests can detect defects stemming from hardware/software interfaces. When the software product is intended for use in systems with heavy traffic loads such as database access systems with many simultaneous users, it is important to perform load tests. A load is a series of inputs that simulates a group of transactions (Burnstein 2003, p. 165). Load tests can be either stress tests or volume tests. A stress test tests a system with a load of transactions and data that causes it to allocate its resources in maximum amounts for a short period of time (Burnstein 2003, p. 169). The goal of stress tests is to find the circumstances under which the system will crash. They detect defects that could occur in the software s actual working environment, and which would not turn up in other tests under normal testing conditions. Volume tests aim at measuring the maximum load of data that the software can handle by feeding the system large volumes of data. The purpose of a system test is to ensure that the system performs according to its requirements (Burnstein 2003, p. 163) Acceptance test The acceptance test resembles the system test in many ways, with the exception that it is usually performed by the customer, allowing them to validate that the product meets their goals and expectations (Burnstein 2003, p. 177). The acceptance test is 12

13 also based on requirements, and system test cases may be reused. When the acceptance test is approved, the system is installed at the customer s site. The approval of the acceptance test often marks the end of the software development project. If there are no customers, but the product is intended for the market, the acceptance test often takes place in the form of a beta test (Wohlin 2005, p. 156). In a beta test, a limited number of potential users use the software for a certain time, and then evaluate it. If the beta users are satisfied, the software is released to the market Regression test Regression test is not a level of testing, but rather the retesting process that is needed on any level after a change in the software or in the requirements. If a change has been made to the code, regression testing is performed by running the old test cases again to ensure that the new version works and that no new faults have been introduced because of the change. 2.4 The relationship between the test process and the development process A software process is a set of activities and associated results which produce a software product (Sommerville 2001, p. 8). Fundamental elements of all software processes are software specification (definition of functionality, requirements), software development (production of design and code), software validation (testing), and software evolution (further development to meet changing needs). Different processes organize these elements differently according to their needs. To describe a software process, a software process model is used. Examples of common process models are the waterfall model and evolutionary development. For further information about software process models, the reader is referred to Sommerville (2001). The development process follows a life-cycle model from the initial idea to the finished and delivered product. A common presentation of the relationship between the development process and the test process is the V-model, as shown in figure 2. In this model, the activities, divided into phases, are sequential. When one phase is finished, the next phase starts. 13

14 Initial idea Expectations Operation Requirements, functional design Acceptance test Technical design System test Coding Unit and integration test Figure 2 - The V-model (Koomen and Pol 1999, p. 15) The left side of the model represents the phases of building the system; from the initial idea to the implementation (coding) of the system. The right side of the model shows the test execution phases, or levels. The V-shaped line shows the sequence of the phases. The arrows represent the paths from the test basis to the test execution. The dotted line roughly defines the formal responsibilities for the phases; above the line, the customer is responsible, while the developers are responsible for the phases below the line. Another common process methodology is Agile (Highsmith 2002). Agile is a collection of different process models including extreme Programming (XP), Scrum, and Lean Software Development (Ryber 2006, p. 49). Common to all agile models is putting the person in the centre instead of tools and processes, and to support software development and respond to quick changes while avoiding unnecessary work and comprehensive documentation (Highsmith 2002, p. 29). Developers should be allowed to work fast and at the same time handle changes by using a flexible process. In agile methods, test cases are defined before programming starts. Unit tests determine if the product is ready for release and regression tests are constantly run since deliveries come at a fast pace. In agile programming, the tester can be viewed as an internal customer, who writes test cases like requirements, and accepts the software after a successful test. A graphical representation of an agile process model is shown in figure 3. 14

15 Coding Project definition Design and adjustment Evaluation Production Figure 3 - Agile development (Koomen and Baarda 2004, p. 133) The relationship between the test process model and the development process model depends on the development process and the current test level. However, according to Koomen et al. (2006, p. 64), there are two fixed relationships that can be determined; the start of the preparation phase (requirements, functional design) directs when the test basis becomes available, and the execution phase (realization) determines when the test object becomes available. 2.5 Problem areas concerning testing Common test process problem areas are described below (Koomen and Pol 1999). Insufficient requirements It is difficult, if not impossible, to write a perfect requirements specification. Since the requirements specification is one of the most important bases for testing, it is important for testers to engage in and provide input into the specification phase. The requirements specification should be (Lauesen 2002, pp ): Correct each requirement is correct and reflects a customer need or expectation. Complete all necessary requirements are included. Unambiguous all parts know and agree upon what each requirement means. Consistent requirements match and do not conflict. Ranked for importance and stability requirements state a priority and the expected frequency of changes. Modifiable requirements are easy to change without losing consistency. Verifiable there exists an economically feasible way to check that the product meets a requirement. Traceable it is possible to see where requirements come from and where they are used in code and design. The tester s task in the specification of requirements is first and foremost to make sure that the requirements are testable. Testability includes unambiguousness, consistency, completeness, and verifiability (Ryber 2006, p. 29). Testers should be included in the formal inspection of requirements documents, as to provide a tester s view. Use case models, diagrams, or flowcharts of the system idea could also be used as a mean of communication when checking requirements. 15

16 An infeasible amount of test cases Even for a small system, there can be an infinite number of possible test cases. This is especially true for a system that can take a large number of different combinations of input data, and where many actions can take place at the same time. The tester needs to find out what input data and actions are most important to test. Since it is impossible to cover every single combination of input data, the issue is to find the subset of test cases that has the largest possibility of detecting the largest number of serious defects (Ryber 2006, p. 31). Costs of testing In many projects, testing-related costs count for up to 50% of the total software development cost. Boehm (1976) argues that these large costs stem from introducing testing too late in the process, and that these costs could be lessened if a defect could be detected and fixed at the same stage as it is introduced. This praxis needs good planning and a structured test plan. 2.6 The need for a structured testing approach The characterization of an unstructured software testing process (ad hoc testing) is that the situation is somewhat chaotic; it is nearly impossible to predict the test effort, to execute tests feasibly or to measure results effectively. Koomen et al. (2006, p. 52) reports the following characteristics of unstructured testing, based on findings from studies of both structured and unstructured testing: Time pressures Time pressures that could be avoided mainly stem from the absence of a good test plan and budgeting method, the absence of an approach stating which test activities should be carried out when and by whom, and the absence of solid agreements on terms and procedures for delivery and reworking of the applications. No insight in the quality of the produced system There is no insight in or ability to supply advice on the quality of the system due to the absence of both a risk strategy and a test strategy. Also, both quality and quantity of the test cases are inadequate since no test design techniques are being used. Inefficiency and ineffectiveness of the test process Lack of coordination between the various parties involved in the testing could result in objects being tested more than once or not at all. Inefficiency is also caused by lack of configuration and change management, incorrect use or non-use of testing tools, and the lack of prioritization and risk analysis in the test planning. A structured test process is, according to Koomen et al. (2006) a process that lacks the disadvantages mentioned above. In other words, a structured testing approach is one that: can be adapted to any situation, regardless of the system being developed and irrespective of who the client is, delivers insight into the qualities of the system, finds defects at an early stage, prevents defects, puts testing on the critical path for as short time as possible, generates reusable test products (e.g. test cases), and is comprehensible and manageable. 16

17 2.7 Risks in introducing structured testing Risks in introducing structured testing include: Introducing a test process seems too expensive The solution to this problem is to put the cost into perspective. Give managers tools to compare the cost of introducing structured testing to the cost of not finding an adequate number of faults and problem areas in time. Employees are reluctant to work in new ways Improve the process gradually in small steps. Koomen and Pol (1999, p. 25) states that it is of utmost importance that employees recognize the need for improvement and agree to it. If not, the chances of failure are much greater. And if the change process has failed once, the next attempt at a process change will face even greater reluctance. There is not enough time to perform testing Many testing activities can be performed in parallel with development. Though staff needs to be appointed to testing, it does not have to be awarded much more time. The number of possible test cases is too large to handle There are techniques that can reduce the number of test cases needed. See section Basic testing techniques A test technique is a structured approach to deriving test cases from the test basis (Koomen and Pol 1999, p. 11). Different techniques are aimed at finding different kinds of defects. Using a structured test technique most often results in more effective detection of defects than ad-hoc identification of test cases. According to Koomen et al. (2006, p. 71), some advantages of using test techniques are: the tests are reproducible since the content of the test execution is described in detail, the test process is independent of the individual who specifies and executes the test cases, and the test specifications are transferable and manageable. There are two kinds of testing; static and dynamic, which are explained in sections and When to use the respective techniques depends on what item is under test. In figure 4 below, the division between static techniques (reviews) and dynamic techniques (execution) is shown. 17

18 Requirements Static (reviews) Specifications Design Code Test plans User manuals Dynamic (Execution) Figure 4 - The roles of static and dynamic testing techniques (Burnstein 2003, p. 305) Dynamic testing techniques are further divided into three basic categories; black-box, white-box, and grey-box testing techniques. What testing technique to use depends on the level the testing is carried out on. Figure 5 below shows the hierarchy of these categories (Ryber 2006, p. 89): Testing techniques Static techniques Dynamic techniques Black-box techniques Grey-box techniques White-box techniques Functional testing Non-functional testing Figure 5 - Testing techniques Static techniques A static testing technique is a verification activity and involves reviews, i.e. evaluating documents, design, or code. The code is however not executed. Burnstein (2003, p. 304) defines a review as a group meeting whose purpose is to evaluate a software artefact or software artefacts. This could be done with varying levels of formality and either manually or with the help of tools. Two common types of reviews are inspections and walkthroughs. 18

19 Inspections are usually of a more formal nature and participants are provided with the material to be inspected before the meeting. They follow a prepared checklist when inspecting the document. Walkthroughs, on the other hand, are less formal. The producer of the material guides the participants through the material, often in a line-by-line manner as if he or she would manually execute the code (Burnstein 2003, p. 310). The participants can ask questions during the walkthrough. The main advantage of using reviews is that defects can be found at a very early stage, before programming has started, and are therefore prevented from being propagated into the code. Thus, the defects are cheaper to repair, and the rework time is reduced. Also, some types of faults which can not be found during execution of the code can often be found in a review (Ryber 2006, p.91). An additional advantage is that those persons involved in the review acquire an understanding of the material under review, and an understanding of the role of their own work Dynamic techniques Burnstein (2003, p.304) describes dynamic testing techniques as techniques where the software code is tested by executing it. Input data is fed into the system, and the actual output data is compared to the expected output data. Dynamic techniques can only be applied to the code. Black-box techniques Black-box techniques are used to perform so-called behavioural testing (Ryber 2006, p.94). Input is fed into the system, and the output is compared to the expected output, verifying that the system has performed the right action from what the tester can see. The tester does not care what happens to the data within the system; he views the system as a black box. Black-box techniques can be used for testing both functional and non-functional features. Functional tests are used for testing the functionality of a system, i.e. the functions the systems should perform. Functional features of a system are related to data; what it is used for, how it is recorded, computed, transformed, updated, transmitted, and so on (Lauesen 2002, p. 71). Functional tests focus on the inputs, valid as well as invalid, and outputs for each function. Non-functional tests are supposed to test the non-functional features, sometimes referred to as quality features, of the system. Examples of non-functional features are performance, security, robustness, and maintenance. Non-functional tests do not focus only on the output, but also on measurements such as the speed of a transaction. White-box techniques White-box techniques are also known as structural testing, or glass-box testing. The purpose of white-box testing is to verify the internal logic or structure of the system. Thus, knowledge of the internal structure is necessary. The tests include data-flow or control-flow tests. Validating only the output from a test is often not enough. Tests often have an impact on the internal state of the system. The impact does not show up in the black-box tests, but may affect subsequent test execution and cause test cases to fail. Guptha and Singha (1994) argue that observing the internal state of the system is essential to 19

20 assure that tests execute successfully. Therefore, a combination of black-box and white-box techniques is preferable. Grey-box techniques A grey-box technique is merely a combination between white-box and black-box techniques. This technique can be used for e.g. looking into the internal structure of the system in order to get an understanding of how to write better black-box test cases for the system. Examples of dynamic testing techniques are described below (Ryber 2006, p.92): Data Data testing is the most common form of black-box testing. It involves categorizing data into groups. Data test techniques include: Equivalence partitioning (EP) The input data domain is partitioned into equivalence classes of both valid and invalid data, where all instances of data within an equivalence class is assumed to be processed in the same way by the software. An example is a system that can take the numbers 1 to 10 as input. In this example, three equivalence classes are needed. One equivalence class is the range of valid numbers; Another class would contain the invalid numbers below the range; < 1. The last class would contain the invalid numbers above the range; > 10. The tester can then select a chosen number of test input values from each equivalence class. In the example above, input values could be -7, 5, and 14. Boundary value analysis (BVA) Experienced testers know that many defects occur on, above or below the edge of an equivalence class (Burnstein 2003, p. 72). In BVA, the tester chooses input values close to the edges. In the previous example, possible input values could be 0 (invalid, below lower edge), 1 (valid, lower edge), 10 (valid, upper edge), and 11 (invalid, above upper edge). Domain testing When several variables work together and thus need to be tested together, domain testing can be used. Domain testing works in the same way as EP and BVA, but with more variables at the same time, creating domains from combining equivalence classes. One example is an insurance premium which is calculated from combining a person s age, gender, and profession. If the number of variables is three or less, the domains can be displayed graphically. Then, BVA is used by choosing input values on the edges between domains (Ryber 2006, p. 130). Flow Flow tests involve testing flow of information within the system. Examples of flow test techniques are: Use case model testing A use case is a description of a workflow where an actor (a person, the system itself, or an external system) interacts with the system and performs a task. The use case describes all the steps taken in order to accomplish the task along with alternative actions which can be taken. Input parameters which ensure that the entire workflow is run through are then chosen as test cases. Coverage and control flow graphing Coverage and control flow graphing requires knowledge of the internal structure of the code. Coverage graphs show 20

21 how large a portion of a specific program part or prime is covered by test cases. For example, the coverage of program statements, branches, conditions or paths can be measured. Combinations of primes are used to derive test cases. When a certain percentage of primes have been tested, the system is said to have reached that percentage of coverage. If, for example, all possible paths have been covered by the test, the system has achieved 100% path coverage. State transition graph testing State transition testing is often used in embedded systems. All possible system states are first defined. Graphs show the system states as nodes and transitions between states as arcs. The actions required to change states are then defined and connected to the arcs. Test cases can then be derived from different paths of state transitions, e.g. risk-based, probability-based or coverage-based. Logic Logic testing is primarily used when testing complicated combinations of variables. Examples of logic testing techniques are: Decision trees Rules for the system, based on the requirements, are created. The rules cover all combinations of input data. The rules are then combined in a decision table. Then, the rules are checked for consistency using a decision tree. The branches of the decision tree represent the different groups of data on each level, further divided into other levels of branches. The test cases are derived from the leaves, i.e. the ends of each branch. Cause-and-effect graphing Cause-and-effect graphing is used when the tester needs to test combinations of conditions. Causes (distinct input conditions) and effects (an output condition or a system state change) are displayed as nodes. Arcs between the causes and the effects show what effects come from what causes. If two or more causes need to be combined in order to reach an effect, Boolean logic (AND, OR, and NOT) is used to combine the arcs. The graph is converted into a decision table. From the table, test cases can be derived. Risk-based testing Risk-based testing includes techniques that derive test cases in an ad-hoc manner. Much experience is needed on the part of the tester. Examples of techniques are: Error guessing Experienced testers usually know where in a system defects tend to occur. On those grounds, he or she can make an educated guess about where defects will occur in a similar system, and can thus design test cases to address those problem areas. Risk-based testing Ryber (2006, p. 219) defines a risk as the probability of a defect occurring multiplied with the influence the defect will have on the system. In risk-based testing, risks are identified and test cases addressing risk areas are prioritized. Exploratory testing Testers explore the system by testing without using predesigned test cases. While the tester learns and experiences the system, he or she comes up with new test cases. The benefits include utilizing the tester s creativity while at the same time lessen overhead documentation. Exploratory testing is not always viewed as its own testing technique, but rather as a testing approach that can be applied to other testing techniques (Itkonen et al. 2007). 21

22 2.9 Test documentation Software test documentation includes documentation produced before, during and after test execution. The IEEE Standard for Software Test Documentation (IEEE , 1998) describes a set of basic software test documents and specifies the form and contents of these, but does not specify what documents are required. The purpose of the standard is to provide a common frame of reference, a checklist for test documentation contents, or a baseline for evaluating the current test process. The standard includes a test plan, a test specification, and a test report. The documents included in these are also deliverables. When implementing test documentation into an organization, the IEEE standard recommends introducing the different documents in phases. In the initial phase, the planning and reporting documents should be introduced. The process should begin at the system level, since the need for control during system testing is critical. In subsequent phases, the needs of the organization and the results of the prior phases should dictate what sequence of introduction of the documents to use (IEEE , 1998). The IEEE standard might not be appropriate for immature organizations. Some of these might not be able to produce the information and the metrics needed for answering the questions proposed by the IEEE test documentation. The standard could however be seen as a target to aim at when the organization accomplishes the appropriate maturity level The test plan Burnstein (2003, p. 197) defines a plan as a document that provides a framework or approach for achieving a set of goals According to Burnstein, test planning is an essential practice for any organization that wishes to develop a test process that is repeatable, manageable and consistent. All of the studied test process improvement models (see chapter 3) mention test planning and the production of a test plan as vital elements of the test process. When properly planned, resources can be optimally distributed and thus more efficiently used, which in turn leads to higher productivity (Samson 1993). Since test planning should include the tools, test base documents, and such elements needed for testing, these can be produced and made available in time. Thus, no time on the critical path will be wasted on the production of them. Also, since test planning can provide earlier access to the test basis and thus enable earlier test specification, test activities can run in parallel with development activities. This will shorten the time that testing is solely on the critical path. Test planning should begin early in the software life-cycle, preferably already in the requirements definition phase. The test plan is the result of the test planning, and contains milestones to help determine the project status. These milestones can make for good points in time to follow up and evaluate the process, so that the project leader can make sure that the project runs as expected (Burnstein 2003, p. 197). Also, the test plan serves as a means of communication between different parties and 22

23 enables transfer of learning between them, and can also serve as a method of communicating the quality of the test process to the customer (Koomen et al. 2006). Depending on the organizational policy, test plans can be organized in several ways. The structure may be hierarchical, with separate test plans for each level of testing (see section 2.3), or with just one master test plan. Koomen et al. (2006, p. 87) claim that the hierarchical approach is better suited for larger companies, and projects which adopt the waterfall development method, while agile development methods usually integrate test planning into the project plan or use a single test plan. In the IEEE standard (IEEE , 1998), the test plan prescribes the scope, approach, resources, and schedule of the testing activities. It identifies the items to be tested, the features to be tested, the testing tasks to be performed, the personnel responsible for each task, and the risks associated with the plan. The standard further specifies structuring the test plan with, but not limited to, the following elements: a) Test plan identifier a unique identifier to each test plan b) Introduction a summary of the software items and features to be tested, and references to relevant documents such as the project plan c) Test items the test items and their versions, and references to relevant documents such as the requirements specification d) Features to be tested the features and combinations of features to be tested, and a reference to their design specifications e) Features not to be tested the features that are not to be tested and the reasons why f) Approach the activities, tools and techniques that will be used in testing each group of features, the completion criteria and constraints on testing g) Item pass/fail criteria the criteria on how to determine whether a test passes or fails h) Suspension criteria and resumption requirements the criteria to determine whether to suspend all or just a portion of the testing, and what needs to be retested when testing is resumed i) Test deliverables the deliverable documents, such as the test plan and the test design specification j) Testing tasks the tasks necessary to prepare for and perform testing k) Environmental needs the necessary and desired properties of the test environment, including hardware, security settings and test tools l) Responsibilities the appointed responsibilities, such as test management, design and execution m) Staffing and training needs the skills needed by the staff, or the training needed to acquire skills n) Schedule the identified milestones, time needed and deadlines o) Risks and contingencies the high-risk parts of the project, and contingency plans for each p) Approvals the names of each person who must approve of the test plan The test specification In the IEEE standard, the test specification is covered by three document types; 23