Introduction To Software Testing. Brian Nielsen. Center of Embedded Software Systems Aalborg University, Denmark CSS

Transcription

1 Introduction To Software Testing Brian Nielsen Center of Embedded Software Systems Aalborg University, Denmark CSS

2 Software development cycle 1. Programmer produces code he believes is bug-free. 2. Product is tested. 20 bugs are found. 3. Programmer fixes 10 of the bugs and explains to the testing department that the other 10 aren't really bugs. 4. Testing department finds that five of the fixes didn't work and discovers 15 new bugs. 5. Repeat three times steps 3 and Due to marketing pressure and an extremely premature product announcement based on overly-optimistic programming schedule, the product is released. 7. Users find 137 new bugs. 8. Original programmer, having cashed his royalty check, is nowhere to be found. 9. Newly-assembled programming team fixes almost all of the 137 bugs, but introduce 456 new ones. 10. Original programmer sends underpaid testing department a postcard from Fiji. Entire testing department quits. 11. Company is bought in a hostile takeover by competitor using profits from their latest release, which had 783 bugs. 12. New CEO is brought in by board of directors. He hires a programmer to redo program from scratch. 13. Programmer produces code he believes is bug-free.

3 Autonomous Systems

4 Extreme environmental conditions

5 Safety Critical Systems Rotterdam Storm Surge Barrier

6 Sensor-networks

7 Consumer Electronics

8 System Characteristics Mass production Limited Hardware (speed, power,memory) Safety critical Autonomous Distributed One-shot development Control of physical environment Mixed signal I/O Extreme environments State-based behavior Real-time constraints (hard and soft) Technical / scientific algorithms

9 Spectacular Software Bugs ARIANE-5 INTEL Pentium II floating-point division 470 Mill US $ Baggage handling system, Denver 1.1 Mill US $/day for 9 months Mars Pathfinder Radiation theraphy, Therac-25

10 Ariane 5 crash An overrun exception stopped both replicated position calculation computers (SRI s) at time H Fail-stop was a wrong design decision Overrun caused by higher velocity in Ariane 5 compared to Ariane 4 Overrun occured in a module which was irrelevant after take-off Simulations were not run with the new velocity (would have caught the failure)

11 Software Qualities Maintainer Good Documentation Readable Code Good Design Functionality Ease of use Ease of learning User Reliability Correctness Efficiency Low Cost Portability Increased productivity Customer

12 Costs of Poor Quality Increased time to find and fix problems Increased cost to distribute modifications Increased customer support Product liability Failure in the market place

13 Software Quality Assurance Quality Control Process definition and standards Formal technical review Test planning and review Quality Assurance Analysis and reporting Measurement

14 Verification vs. Validation: Verification: are we building the product right? The software should conform to its specification Validation: are we building the right product? The software should do what the user really requires

15 Verification vs. Validation: Verification: are we building the product right? The software should conform to its specification Validation: are we building the right product? The software should do what the user really requires Assuring that a software system meets a user s needs

16 What is testing?

17 A Self-Assessment Test [Myers]

18 A Self-Assessment Test [Myers] A program reads three integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.

19 A Self-Assessment Test [Myers] A program reads three integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral. Write a set of test cases to test this program

20 A Self-Assessment Test Test cases for: [Myers]

21 A Self-Assessment Test [Myers] Test cases for: 1. valid scalene triangle?

22 A Self-Assessment Test Test cases for: [Myers] 1. valid scalene triangle? 2. valid equilateral triangle?

23 A Self-Assessment Test Test cases for: [Myers] 1. valid scalene triangle? 2. valid equilateral triangle? 3. valid isosceles triangle?

24 A Self-Assessment Test Test cases for: [Myers] 1. valid scalene triangle? 2. valid equilateral triangle? 3. valid isosceles triangle? 4. 3 permutations of previous?

25 A Self-Assessment Test Test cases for: [Myers] 1. valid scalene triangle? 2. valid equilateral triangle? 3. valid isosceles triangle? 4. 3 permutations of previous? 5. side = 0?

26 A Self-Assessment Test Test cases for: [Myers] 1. valid scalene triangle? 2. valid equilateral triangle? 3. valid isosceles triangle? 4. 3 permutations of previous? 5. side = 0? 6. negative side?

27 A Self-Assessment Test Test cases for: [Myers] 1. valid scalene triangle? 2. valid equilateral triangle? 3. valid isosceles triangle? 4. 3 permutations of previous? 5. side = 0? 6. negative side? 7. one side is sum of others?

28 A Self-Assessment Test Test cases for: [Myers] 1. valid scalene triangle? 2. valid equilateral triangle? 3. valid isosceles triangle? 4. 3 permutations of previous? 5. side = 0? 6. negative side? 7. one side is sum of others? 8. 3 permutations of previous?

29 A Self-Assessment Test Test cases for: [Myers] 1. valid scalene triangle? 2. valid equilateral triangle? 3. valid isosceles triangle? 4. 3 permutations of previous? 5. side = 0? 6. negative side? 7. one side is sum of others? 8. 3 permutations of previous? 9. one side larger than sum of others?

30 A Self-Assessment Test Test cases for: [Myers] 1. valid scalene triangle? 2. valid equilateral triangle? 3. valid isosceles triangle? 4. 3 permutations of previous? 5. side = 0? 6. negative side? 7. one side is sum of others? 8. 3 permutations of previous? 9. one side larger than sum of others? permutations of previous?

31 A Self-Assessment Test Test cases for: [Myers] 1. valid scalene triangle? 2. valid equilateral triangle? 3. valid isosceles triangle? 4. 3 permutations of previous? 5. side = 0? 6. negative side? 7. one side is sum of others? 8. 3 permutations of previous? 9. one side larger than sum of others? permutations of previous? 11. all sides = 0?

32 A Self-Assessment Test Test cases for: [Myers] 1. valid scalene triangle? 2. valid equilateral triangle? 3. valid isosceles triangle? 4. 3 permutations of previous? 5. side = 0? 6. negative side? 7. one side is sum of others? 8. 3 permutations of previous? 9. one side larger than sum of others? permutations of previous? 11. all sides = 0? 12. non-integer input?

33 A Self-Assessment Test Test cases for: [Myers] 1. valid scalene triangle? 2. valid equilateral triangle? 3. valid isosceles triangle? 4. 3 permutations of previous? 5. side = 0? 6. negative side? 7. one side is sum of others? 8. 3 permutations of previous? 9. one side larger than sum of others? permutations of previous? 11. all sides = 0? 12. non-integer input? 13. wrong number of values?

34 A Self-Assessment Test Test cases for: [Myers] 1. valid scalene triangle? 2. valid equilateral triangle? 3. valid isosceles triangle? 4. 3 permutations of previous? 5. side = 0? 6. negative side? 7. one side is sum of others? 8. 3 permutations of previous? 9. one side larger than sum of others? permutations of previous? 11. all sides = 0? 12. non-integer input? 13. wrong number of values? 14. for each test case: is expected output specified?

35 A Self-Assessment Test Test cases for: [Myers] 1. valid scalene triangle? 2. valid equilateral triangle? 3. valid isosceles triangle? 4. 3 permutations of previous? 5. side = 0? 6. negative side? 7. one side is sum of others? 8. 3 permutations of previous? 9. one side larger than sum of others? permutations of previous? 11. all sides = 0? 12. non-integer input? 13. wrong number of values? 14. for each test case: is expected output specified? 15. check behaviour after output was produced?

36 Testing Testing: to check the quality (functionality, reliability, performance, ) of an (software) object -by performing experiments -in a controlled way

37 Testing Testing: to check the quality (functionality, reliability, performance, ) of an (software) object -by performing experiments -in a controlled way

38 Testing Testing: to check the quality (functionality, reliability, performance, ) of an (software) object -by performing experiments -in a controlled way In avg errors per 1000 LOC % of development time and cost in embedded software

39 Testing Testing: to check the quality (functionality, reliability, performance, ) of an (software) object -by performing experiments -in a controlled way In avg errors per 1000 LOC % of development time and cost in embedded software

40 What is a Test? Software under Test

41 What is a Test? Test Data Software under Test

42 What is a Test? Test Data Output Software under Test

43 What is a Test? Test Data Output Software under Test Correct result?

44 What is a Test? Test Data Output Software under Test Oracle Correct result?

45 What is a Test? Test Cases Test Data Output Software under Test Oracle Correct result?

46 Types of Testing system Level integration unit robustness performance white box black box Accessibility usability reliability functional behaviour Aspect

47 Types of Testing system Level integration unit robustness performance white box black box Accessibility usability reliability functional behaviour Aspect

48 Quality-Characteristics (ISO-9126) Functionality Suitability, accuracy, security, compliance, interoperability Reliability maturity, fault tolerance, recoverability Usability understandability, learnability, operability Efficiency time behaviour, resource utilization Maintainability Analysability, changeability, stability, testability Portability Adaptability, intallability, conformance, replaceability

49 Quality-Characteristics (ISO-9126) Functionality functional testing Suitability, accuracy, security, compliance, interoperability Reliability reliability testing maturity, fault tolerance, recoverability Usability usability testing understandability, learnability, operability Efficiency performance testing time behaviour, resource utilization Maintainability maintainability testing?? Analysability, changeability, stability, testability Portability portability testing? Adaptability, intallability, conformance, replaceability

50 Module test test driver Module under test test stub test stub

51 Whitebox Example int invoice (int x, int y) { int d1, d2, s; if (x<=30) d2=100; else d2=90; s=5*x + 10 *y; if (s<=200) d1=100; else if (s<=1000) d1 = 95; else d1 = 80; return (s*d1*d2/10000); } Test Cases

52 Whitebox Example int invoice (int x, int y) { int d1, d2, s; if (x<=30) d2=100; else d2=90; s=5*x + 10 *y; if (s<=200) d1=100; else if (s<=1000) d1 = 95; else d1 = 80; return (s*d1*d2/10000); } d1=100 if d2=100 d2=90 if x>30 s>200 if s>1000 d1=95 d1=80 return Test Cases

53 Whitebox Example int invoice (int x, int y) { int d1, d2, s; if (x<=30) d2=100; else d2=90; s=5*x + 10 *y; if (s<=200) d1=100; else if (s<=1000) d1 = 95; else d1 = 80; return (s*d1*d2/10000); } d1=100 if d2=100 d2=90 if x>30 s>200 if s>1000 d1=95 d1=80 return Test Cases est Data Expected Output =5 Y=5 75 =31 Y= =30 Y=

54 Black box testing requirements output nput events SUT y domain testing x

55 Module Integration calls uses interacts M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12

56 Generic Embedded System Embedded system User interface Specific interface Power supply System environment Application Software Digital I/O A/D+ D/A OS Plant I/O-drivers Processing Unit Actuators Sensors Interfaces to other systems

57 System test 2*CRTG (4 channels) 2 * 200 k

58 Test Equipment Complete Type Approval Test System (3 M )

59 Who Should Test?

60 Who Should Test? Developer Understands the system But, will test gently And, is driven by deadlines

61 Who Should Test? Developer Understands the system But, will test gently And, is driven by deadlines

62 Who Should Test? Developer Understands the system But, will test gently And, is driven by deadlines Independent tester Must learn system But, will attempt to break it And, is driven by quality

63 Who Should Test? Developer Understands the system But, will test gently And, is driven by deadlines Independent tester Must learn system But, will attempt to break it And, is driven by quality

64 Testing Objectives To identify as many errors as possible To bring the software to an acceptable level of quality To determine risk of release Perform testing efficiently, effectively and within budget and time constraints Compile a record of software errors for future error prevention and correction (learning organization).

65 Risk Based Test Strategy Make best possible use of resources by identifying and prioritizing quality aspects and subsystems Higher risk more testing No risk no testing Risk = chance of failure damage Use frequency Chance of error being present Complexity New tools/techniques Inexperienced developers Cost of repair Loss of market share Legal claim

66 Testability = how easy can a program be tested operability - it operates cleanly observability the results are easy to see distinct output is generated for each input incorrect output is easily identified controllability processing can be controlled tests can be automated & reproduced decomposability - software modules can be tested independently simplicity - no complex architecture and logic stability - few changes are requested during testing understandability - program is easy to understand

67 Testing Quality assurance activities by running code examines functionality in the form used by customers Software testing is a formal process carried out by a specialized testing team in which a software unit, several integrated software units or an entire software package are examined by running programs on a computer. All the associated tests are performed according to approved test procedures on approved test cases.

68 Testing Dynamic testing is the process of executing a program or system with the intent of finding error (Glenford Meyers definition) Static testing is any activity that aims at finding defects by inspecting, reviewing, walking through, and analyzing any static component of the software (code, documents, and models) Debugging is an ad hoc activity performed by individual developers to find and remove bugs from a program. Testing is a planned activity

69 There is More than Testing software process quality control requirement management reviewing inspection CMM tatic analysis certificatio debugging walk-through

70 Lifecycles & testing

71 requirements V - Model acceptance test spec acceptance test specification system test spec system test architecture spec integration test spec integration test detailed design module test spec module test implementation code unit test spec unit-test

72 Introducing, Detecting and Repairing Errors Liggesmeyer 98

73 Life Cycle : Testing as a Process T L O completion test generation preparation planning & control test execution I

74 Life Cycle : Testing as a Process T L O completion test generation preparation planning & control test execution I Testing as phase of development process vs.

75 Life Cycle : Testing as a Process T Testing as phase of development process vs. I L O Testing as a process itself with its own phases in parallel with development process planning & control preparation test generation test execution completion

76 Life Cycle : Testing as a Process T Testing as phase of development process vs. I L O Testing as a process itself with its own phases in parallel with development process requirements specification implementation planning & control preparation test generation test execution completion quality report

77 Challenges of Testing

78 Challenges of Testing New embedded systems more functionality increasingly advanced faster time-to-market higher quality

79 Challenges of Testing New embedded systems more functionality increasingly advanced faster time-to-market higher quality Testing more to be tested more complicated in less time more thorough

80 Challenges of Testing New embedded systems more functionality increasingly advanced faster time-to-market higher quality Testing more to be tested more complicated in less time more thorough skilled developers and testers advanced testing tools and techniques well organized using solid development method

81 Summary

82 Some Testing Principles

83 Some Testing Principles Testing starts during the requirements phase

84 Some Testing Principles Testing starts during the requirements phase The programmer shall not be the tester

85 Some Testing Principles Testing starts during the requirements phase The programmer shall not be the tester A test case specifies the test inputs and the expected outputs

86 Some Testing Principles Testing starts during the requirements phase The programmer shall not be the tester A test case specifies the test inputs and the expected outputs Test cases shall also cover invalid and unexpected inputs

87 Some Testing Principles Testing starts during the requirements phase The programmer shall not be the tester A test case specifies the test inputs and the expected outputs Test cases shall also cover invalid and unexpected inputs Test cases shall test that the program does what it should do and that it does not do what it should not do

88 Some Testing Principles Testing starts during the requirements phase The programmer shall not be the tester A test case specifies the test inputs and the expected outputs Test cases shall also cover invalid and unexpected inputs Test cases shall test that the program does what it should do and that it does not do what it should not do Test cases shall be recorded for reuse

89 Some Testing Principles Testing starts during the requirements phase The programmer shall not be the tester A test case specifies the test inputs and the expected outputs Test cases shall also cover invalid and unexpected inputs Test cases shall test that the program does what it should do and that it does not do what it should not do Test cases shall be recorded for reuse A test is successful when it detects an error ( but the project manager thinks differently! )

90 Some Testing Principles Testing starts during the requirements phase The programmer shall not be the tester A test case specifies the test inputs and the expected outputs Test cases shall also cover invalid and unexpected inputs Test cases shall test that the program does what it should do and that it does not do what it should not do Test cases shall be recorded for reuse A test is successful when it detects an error ( but the project manager thinks differently! ) No risk, no test

91 Lessons Testing starts when project starts Have a development approach How to develop (waterfall, cyclic, etc) : roles and activities Clear expectations between testers and developers Clarity about phases and tasks Easier planning Test execution takes 3 steps Test, fix, re-test Let the programmers find bugs 80 % of bugs can be found by programmers (but they don t) Every morning discuss the issues found Testers create test report at end of the day Next morning discuss which issues have to be solved Programmer and tester fix issue together

92 Non-Structured testing Start too late Don t organize and manage your test process No test plan (V&V plan) Unclear responsibilities Conflicting interests No fixed or stable specifications Don t learn from a defect from a test from statistics Use test tools in a chaotic environment No configuration management and change control No standard test procedures Don t educate Do not provide career options

93 Model Based Development

94 Traditional Software Development The Waterfall Model Problem Area Analyse REVIEWS Design Costly in time-to-market and money Errors are detected late or never Application of FM s as early as possible Implementation Testing REVIEWS Running System

95 Models A model is a simplified representation of the real world. Engineers use models to gain confidence in the adequacy and validity of a proposed design. Focus on one or more aspects of interest: Safety Liveness Peak time Performance

96 Modelbased Validation UML FORMAL METHODS Analysis Design Model Specification Verification & Refusal Validation Implementation Testing Monitoring

97 Modelbased Validation UML FORMAL METHODS Design Model Automatic Code generation Analysis Verification & Refusal Validation Specification Implementation Testing Monitoring

98 Modelbased Validation UML FORMAL METHODS Design Model Automatic Code generation Analysis Verification & Refusal Automatic Test generation Validation Specification Implementation Testing Monitoring

99 Modelbased Validation UML FORMAL METHODS Design Model Automatic Code generation Analysis Verification & Refusal Automatic Test generation Validation Specification Automatic Monitoring generation Implementation Testing Monitoring

100 Multiple V-model Model Prototype(s) Development Iteration(s) Final Product design design design test test test build build build Different objects, objectives, techniques in different stages Complexity of ES: functional decomposition and stepwise integration, but tight coordination (e.g independent development of HW and SW) Cost: Detect and correct errors earlier => Test earlier

101 END

102 Control Systems

103 Procedure P; begin while A do begin X; if B then if C then Y else Z else begin V; W end end end

104 Testing: A Definition

105 Testing: A Definition Testing is not: static analysis reviewing or inspection walk-through debugging bug fixing auditing quality control validation of the specification

106 Testing: A Definition Testing is not: static analysis reviewing or inspection walk-through debugging bug fixing auditing quality control validation of the specification But testing: is one of the instruments for measuring quality increases value of product by establishing confidence in its quality helps in assessing risk of putting product into operation

107 Verification & Validation

108 Verification & Validation Verification :

109 Verification & Validation Verification : building the product right concerns development process intermediate products e.g. checking the testability of the specification

110 Verification & Validation Verification : building the product right concerns development process intermediate products e.g. checking the testability of the specification Validation :

111 Verification & Validation Verification : building the product right concerns development process intermediate products e.g. checking the testability of the specification Validation : building the right product whether final product meets requirements and needs e.g. by reviewing the specification