Testing Testing is the most important component of software development that must be performed throughout the life cycle Testing must be carried out by developers continuously More methodical testing must be done at the end of each life cycle 1
Testing There are two type of testing Non-Execution-Based testing Review specification Review design Review source code Execution-Based testing Execute source code 2
Verification and Validation Verification (V&V) Determining whether a phase has been done correctly Are we building the product right?[boehm, 1984a] Validation The process of testing before the product is released to the client Are we building the right product?[boehm, 1984a] 3
Quality Issues The term quality DOES NOT refer to excellence or luxury A quality product is a product that satisfies its specification The job of SQA is to ensure the high quality of the product Does the software satisfy its specification? 4
Quality Issues Managerial Independence It is important to have different managers for each development and SQA team Independent managers come into play when an organization has to make decisions: Do we deliver on time with many errors? Do we deliver late with fewer errors? This way if they do not agree they can consult a more senior manager 5
Review Specification document Design document Source code Review must be performed by someone other than the author Preferably by more than one person 6
Walkthrough A walkthrough team must consist of 4 to 6 experienced senior technical staff members They can find the critical errors Groups represented in a walkthrough: Manager responsible for drawing up the specification Client representative Development team representative SQA team representative 7
Walkthrough The material for the walkthrough must be distributed to the members in advance Each member must review the document and prepare the following lists List of items reviewer does not understand or is not clear about List of items reviewer believes to be incorrect 8
Managing Walkthrough Which of the following should chair the walkthrough? Representative for the document being reviewed Representative for design team Representative for SQA Representative for the client 9
Managing Walkthrough The person leading the walkthrough guides other members though the document to: Uncover errors, not to correct errors Reasons: The solution could be poor in quality The solution will cost more (Salary for all team members VS. an individual) Not all items are incorrect; need more analysis Short time to perform the walkthrough 10
Managing Walkthrough Two ways to conduct a walkthrough Participant driven Each participant addresses his or her issues Representative for the specification team must:» Clarify issues» Explain the point if the reviewer is mistaken» Agree with the error 11
Managing Walkthrough Two ways to conduct a walkthrough: Document driven The document representative walks through the document The reviewers interrupt the representative:» By prepared comments» To ask questions This method leads to detecting more faults» Faults are spontaneously detected with this method 12
Managing Walkthrough Verbliizng statements will lead to fault detection Specification walkthrough Design walkthrough Plan walkthrough Code walkthrough 13
Managing Walkthrough The primary task of the walkthrough chair is to invoke questions and stimulate advanced discussions A walkthrough is an interactive, multi-sided process Walkthrough must not be used for evaluating the participants Detecting faults must be the goal of a walkthrough 14
Inspection Five steps are involved during inspection Overview Document is presented by the producer Document is distributed to inspection team Preparation Inspection team members try to understand the document in detail 15
Inspection Five steps are involved during inspection Inspection Walking through the document with inspection team Every item is covered Every branch is taken at least once Rework Responsible person or teams resolve all faults and problems noted during inspection 16
Inspection Five steps are involved during inspection Follow-up The chair has to ensure that every issue is addressed and resolved A re-inspection team is reconvened if more than 5% of the material inspected is reworked Inspection should not be used to evaluate individuals 17
Inspection Case studies [Bush, 1990] At JPL on average, each 2-hours of inspection exposed 4 major faults and 14 minor faults This is a saving of $25,000 per inspection 18
Comparison of Inspection and Walkthrough Walkthrough is a two step process Preparation Team analysis More informal than inspection 19
Comparison of Inspection and Walkthrough Inspection is a five step process Overview Preparation Inspection Rework Follow-up Inspection is formal 20
Strengths of Reviews Effective way of detecting faults Faults are detected in early phases of software process Good for modularized products Object-Oriented Paradigm Consists of mostly independent modules and classes 21
Weaknesses of Reviews If the previous phase documents have not been modified it is very hard to conduct a review Could be used for performance evaluation of individuals involved 22
Metrics for Inspection Fault density Fault per page Fault per 100 line of code Fault categories Major fault per unit of category Minor fault per unit of category 23
Metrics for Inspection Fault detection rate The number of major and minor faults detected in an hour Fault detection efficiency The number of major and minor faults detected in a person hour 24
Execution-Based Testing The process of executing code to find faults Terminology: Bug IEEE standard terminology for a fault Failure Observed incorrect behavior of the product as a result of fault Error Programmer mistake 25
Execution-Based Testing Program testing can be a very effective way to show the presence of bugs but is hopelessly inadequate for showing their absence [Dijkstra, 1972] If a product produces an incorrect output, it means the product definitely contains faults If a product produces correct output, it means the product may contain faults 26
Utility What Should Be Tested Usability of the product User needs are met when the correct product is used under specified conditions 27
Reliability What Should Be Tested Measurement of the frequency and criticality of a product failure It is important to know: How often, on average, the product fails Mean Time Between Failure (MTBF) How long would it take, on average, to repair it Mean Time To Repair (MTTR) 28
Robustness What Should Be Tested A robust product should not fail under heavy usage Performance How well does the product perform? Response time Space requirements 29
Correctness What Should Be Tested Does the product produce correct output when given correct input when operated in permitted condition? The input and output specification may be satisfied with incorrect result 30
Testing Versus Correctness Proofs Correctness proof is a mathematical technique It can show if the product is correct 31
Who Should Perform Executedbased Testing? Testing is the process of executing a product with the intention of finding faults [Myers 1979] Testing is a destructive process If the programmer attitude toward the code is a protective one, then the chance of finding faults (destructive task) is low 32
Who Should Perform Executedbased Testing? Majority of testing should be performed by an independent SQA team Programmers should not test their own modules Notion of destroying their own creation Module testing should be done by programmer and/or an independent SQA team Misunderstanding of the specification or design Integrated testing must be done by an independent SQA team 33
Who Should Perform Executedbased Testing? The tester (programmer or an independent person) cannot input random values and say It looks good by looking at the output The correct output must be calculated by an expert with a given input. This way we can insure the software is performing correctly or it is fault tolerant 34
When Testing Stops Testing stops when a product is irrevocably discarded 35