MSc Software Testing and Maintenance MSc Prófun og viðhald hugbúnaðar Fyrirlestrar 25 & 26 The SWEBOK Chapter on Software Testing IEEE http://www.swebok.org/ 19/10/2005 Dr Andy Brooks 1
Repeat after Andy: software contains faults, software contains faults,... 19/10/2005 Dr Andy Brooks 2
Software testing consists of the dynamic verification of the behavior of a program on a finite set of test cases, suitably selected from the usually infinite executions domain, against the expected behavior. Static techniques (no execution) are beginning to mature. Exhaustive testing on even simple programs can take months or years. Selecting test techniques often relies on risk management and engineering expertise. Observed behavior can be checked against user expectations (validation), against a specification (verification), or against implicit requirements or reasonable expectations. 19/10/2005 Dr Andy Brooks 3
Software Testing 1. Software Testing Fundamentals 2. Test Levels 3. Test Techniques 4. Test Related Measures Testing-Related Terminology Key Issues Relationships of Testing to Other Activities The Target of the Test Objectives of Testing Intuition and Experience Specificationbased Code-based Fault-based Evaluation of the Program Under Test Evaluation of the Tests Performed 5. Test Process Usage-based 19/10/2005 Dr Andy Brooks 4
1. Software Testing Fundamentals 1.1.2. Faults vs. Failures A fault (defect) is the cause of a malfunction. A failure is an undesired effect observed during system operation. Testing can reveal failures, but it is the faults that can and must be removed... the cause of a failure cannot always be unequivocally identified. 19/10/2005 Dr Andy Brooks 5
1. Software Testing Fundamentals 1.2 Key Issues 1.2.1. Test selection criteria/test adequacy criteria What is a suitable set of test cases? Is a set of test cases adequate? Can the testing be stopped? 1.2.3. Testing for defect identification A succesful test case is one which causes a failure. 19/10/2005 Dr Andy Brooks 6
1. Software Testing Fundamentals 1.2 Key Issues goðsvar 1.2.4. The oracle problem A human or mechanical agent is needed to calculate the correct outcomes of tests. Can you rely on: i. the values of constants in books, ii. your pocket calculator, or iii hand calculations? 19/10/2005 Dr Andy Brooks 7
1. Software Testing Fundamentals 1.2 Key Issues 1.2.5. Theoretical and practical limitations of testing Complete testing for real programs is not possible. Dijkstra: program testing can be used to show the presence of bugs, but never to show their absence 1.2.6. The problem of infeasible paths There may be control flow paths that cannot be exercised by any input data. 19/10/2005 Dr Andy Brooks 8
2. Test Levels 2.1 The target of the test 2.1.1. Unit testing Unit testing verifies the functioning in isolation of software pieces which are separately testable. Involves executing code. Testing can be performed by the developers themselves. 19/10/2005 Dr Andy Brooks 9
2. Test Levels 2.1 The target of the test 2.1.2. Integration testing Integration testing is the process of verifying the interaction between software components. Traditional top-down or bottom-up strategies are used for traditional, hierarchically structured software. Modern integration testing involves identifying functional threads. The user enters a search term in the GUI dialogue box, which is transformed to an SQL query in the middle layer, and then the database in the bottom layer is queried... Mock objects may be needed if functionality is not currently implemented. program stubs 19/10/2005 Dr Andy Brooks 10
a b c l d e f g h i m j Procedures b and c could be stubbed in a top-down approach. Procedures e and g could be stubbed in a bottom-up approach. l, m, or n could be mock objects. n 19/10/2005 Dr Andy Brooks 11
2. Test Levels 2.1 The target of the test 2.1.3. System testing System testing is concerned with the behavior of a whole system. Most faults should have already been identified during unit and integration testing. System testing often involves testing the non-functional requirements: Security Speed Accuracy Reliability System testing often also involves testing external interfaces to other software and hardware. 19/10/2005 Dr Andy Brooks 12
2. Test Levels 2.2 Objectives of Testing 2.2.1. Acceptance/qualification testing Acceptance testing checks the system behavior against the customer s requirements, however these may have been expressed. Customers or developers may undertake typical tasks. 2.2.2. Installation testing Installation testing is much like system testing except the software has now been installed in the target environment. 2.2.3. Alpha and beta testing This testing involves trial usage in-house (alpha) or by external parties (beta). 19/10/2005 Dr Andy Brooks 13
2. Test Levels 2.2 Objectives of Testing 2.2.4. Conformance/Functional/Correctness testing Does the observed behavior of the software conform to the specification? 2.2.5. Reliability achievement and evaluation Statistical measures of reliability can be achieved by generating random test cases according to the operational profile. 1 failure every 5000 logical tasks. 1 failure every 50000 logical tasks. 1 failure every 500000 logical tasks.? Frequency of logical tasks in internet banking Login and password...33% Display account information...33% Pay bill...25% Transfer money between accounts...5% O.s.f. operational profile 19/10/2005 Dr Andy Brooks 14
2. Test Levels 2.2 Objectives of Testing 2.2.6. Regression testing Regression testing is the selective retesting of a system or component to verify that modifications have not caused unintended effects... 2.2.7. Performance testing Does the software meet performance requirements such as capacity and response time? 2.2.8. Stress testing The software is tested beyond the maximum design load. number of simultaneous users, simultaneous applications,... 19/10/2005 Dr Andy Brooks 15
2. Test Levels 2.2 Objectives of Testing 2.2.9. Back-to-back testing The same set of tests is applied to two versions of the system and the results compared. 2.2.10. Recovery testing Recovery testing checks restart capability after a crash. 2.2.11. Configuration testing Configuration testing analyses the software under different memory, network, server, database, browser,... configurations. 19/10/2005 Dr Andy Brooks 16
2. Test Levels 2.2 Objectives of Testing 2.2.12. Usability testing How easy is it for users to learn to use the software? How long do users take to recover from mistakes? ATMs have a walk-up-and-use useability requirement. 2.2.13. Test-driven development (TDD) Tests cases are written before the code is executed. Under TDD, sometimes, less attention is paid to producing software requirement specification documents. 19/10/2005 Dr Andy Brooks 17
3. Test Techniques 3.1 Intuition and experience 3.1.1. Ad hoc testing í sérstöku augnamiði Tests are based on skill, intuition, and experience with similar programs. A widely practiced technique. things that are empty division by 0 overflow characters in a numeric field 3.1.2. Exploratory testing Tests are dynamically designed, executed and modified. Not defined in advanced. 19/10/2005 Dr Andy Brooks 18
3. Test Techniques 3.2. Specification-based world s oldest woman is 125? 3.2.1. Equivalence partition The input domain is divided into subsets of equivalent classes and an input is taken from each subset as being representative of the class. Age<0 ( -5 ), 0<=Age<=135 ( 45 ), Age >135 ( 142 ) Equivalence partitioning may also involve classes such as: is a number and is not a number number of inputs The size of the set of test cases (test suite) can grow very quickly. 19/10/2005 Dr Andy Brooks 19
3. Test Techniques 3.2. Specification-based world s oldest woman is 125? 3.2.2. Boundary-value analysis Test cases are chosen near boundaries of the input domain since faults tend to be found near extreme values. Robustness testing is an extension of this technique and includes values outside the input domain. Age ( -1, 0, 1 ) ( 134, 135, 136) 19/10/2005 Dr Andy Brooks 20
3. Test Techniques 3.2. Specification-based 3.2.3. Decision Table (Cause-Effect Graphing) Decision tables represent relationships between conditions (inputs) and actions (outputs). Test cases are derived by considering all possible combination of conditions and actions. If the customer is billed using a fixed rate method, a fixed monthly charge applies if electricity used is less than 100 kwh. If electricity usage is greater, then the customer is charged according to Schedule A. If the customer is billed using a variable rate method, Schedule A charging applies to the first 100 kwh of electricity consumption and Schedule B charging to additional consumption. example 19/10/2005 Dr Andy Brooks 21
Decision Table example conditions 1 2 3 4 fixed rate account T T F F variable rate account F F T T < 100 kwh T F T F >= 100kwh F T F T actions fixed monthly charge x Schedule A charging x x x Schedule B charging x 4 rules 4 tests 19/10/2005 Dr Andy Brooks 22
3. Test Techniques 3.2. Specification-based 3.2.4. Finite-state machine-based The program is modelled as a finite-state machine and tests derived to cover the states and transitions. 3.2.5 Testing from formal specifications Tests can be derived from specifications written in a formal language. 3.2.6 Random testing Tests are generated at random. The input domain is sampled at random. 19/10/2005 Dr Andy Brooks 23
3. Test Techniques 3.3. Code-based 3.3.1 Control-flow based criteria If The strongest criteria is path testing: all entry-to-exit control flow paths in the flowgraph are excuted. Other criteria are statement testing, branch testing, condition/decision testing. Adequacy is measure by the coverage achieved by the set of tests cases (test suite) expressed as a percentage: e.g. 100% statement coverage 19/10/2005 Dr Andy Brooks 24
// Pay overtime at "time and a half" if (hours > STANDARD) pay = STANDARD*RATE+(hours-STANDARD)*(RATE*1.5); else pay = hours*rate; If A single test may involve executing the if-else statement, so the statement is covered. But a single test will mean the condition only takes one of its two possible values (TRUE,FALSE) and only one of the two possible branches are executed. The condition and branches are not fully covered. 19/10/2005 Dr Andy Brooks 25
3. Test Techniques 3.3. Code-based 3.3.2 Data-flow based criteria double pay = 0.0; How are variables defined, used, and killed (undefined)? The strongest criteria, all definition-use paths, requires that for each variable, every control flow path segment from a definition of that variable to a use of that definition is executed. A weaker criteria is all-uses. double pay = 0.0; System.out.print ("Enter the number of hours worked: "); int hours = scan.nextint(); System.out.println (); // Pay overtime at "time and a half" if (hours > STANDARD) pay = STANDARD*RATE+(hours-STANDARD)*(RATE*1.5); else pay = hours*rate; NumberFormat fmt = NumberFormat.getCurrencyInstance(); System.out.println ("Gross earnings: " + fmt.format(pay)); 19/10/2005 Dr Andy Brooks 26
3. Test Techniques 3.4. Fault-based 3.4.1. Error guessing Test cases are designed with the aim of identifying likely faults in a program given a history of faults in earlier projects, as well as the experience of the software engineer. 3.4.2. Mutation testing A mutant is a slightly modified version of a program ( negate decision or replace operator,...). The test suite is modified so that a test case kills the mutant. 19/10/2005 Dr Andy Brooks 27
3. Test Techniques 3.5. Usage-based 3.5.1. Operational profile The test environment reproduces the operational environment of the software. Future reliability of the software can be estimated from the results. Test inputs are assigned a probability distribution according to their occurrence in actual operation. 19/10/2005 Dr Andy Brooks 28
3. Test Techniques 3.6. Nature of application Specialized testing fields: Object-oriented testing. Component-based testing. Web-based testing. GUI testing. Concurrent programs. Protocol conformance. Real-time systems. Safety-critical systems. 19/10/2005 Dr Andy Brooks 29
3. Test Techniques 3.7. Select and combine 3.7.1. Functional and structural Specification (functional) and code-based (structural) techniques should not be seen as alternative but as complementary. 3.7.2. Deterministic and random Select and combine techniques. 19/10/2005 Dr Andy Brooks 30