Testing. And Software Product Management. Autumn 2017 CSM14104 Software Product Management 1

Transcription

1 Testing And Software Product Management Autumn 2017 CSM14104 Software Product Management 1

2 Lecture Outline and learning objectives What is testing? Definition Limits of testing Test strategy Testing in software product development Test levels Testing in the life cycle Seven Principles of testing Note: individual testing techniques and test case design methods are not covered Autumn 2017 CSM14104 Software Product Management 2

3 What is testing? Autumn 2017 CSM14104 Software Product Management 3

4 Testing What do you think of these definitions for testing: Testing is about trying if the product works Works for whom? Where? When? How do you know if it works or not? Does it matter if it is slow as long as it works (somehow)? Does it matter if it works only most of the time? Does it matter if it is more difficult to use or has less features than a competing product? Does it matter if it has features nobody asked for? Testing is about looking for bugs What s a bug? Does the software have to crash? Produce wrong results? Be unreliable? Do you actually want to find bugs or are you (secretly) wishing that you won t find any? How hard are you trying to find bugs? Do you consider only the happy paths? Are you testing with only the likely (correct) inputs or also with the improbable, illogical, missing or outright toxic inputs? Are you really trying to break it? Do you test only in the development environment or also in realistic user s contexts? Autumn 2017 CSM14104 Software Product Management 4

5 Testing is hard but absolutely necessary test (verb), Oxford Dictionary of English: take measures to check the quality, performance, or reliability of (something), especially before putting it into widespread use or practice In practice, testing is the single most important quality assurance method in a software development project! Autumn 2017 CSM14104 Software Product Management 5

6 Software and System Testing as an activity What do you do when you test: Testing [1] : An activity in which a system or component is executed under specified conditions, the results are observed or recorded, and an evaluation is made of some aspect of the system or component. [1] IEEE Standard for Software and System Test Documentation. IEEE Std , Autumn 2017 CSM14104 Software Product Management 6

7 Software and System Testing as a process Why and when do you do what you do: Testing [2] : The process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects. [2] ISTQB Glossary Autumn 2017 CSM14104 Software Product Management 7

8 Stakeholders, quality, value Weinberg, G.: Quality is value to some person Kaner & Bach: A Bug is an attribute of a software product That decreases its value to a favored stakeholder Or increases its value to a disfavored stakeholder Without a sufficiently large counterveiling benefit. Autumn 2017 CSM14104 Software Product Management 8

9 Stakeholders, quality, value Testing [3] is an empirical technical investigation conducted to provide stakeholders with information about the quality of the product or service under test [3] Cem Kaner & James Bach, Black Box Software Testing Foundations, Autumn 2017 CSM14104 Software Product Management 9

10 Big Picture Autumn 2017 CSM14104 Software Product Management 10

11 Limits of testing When is testing complete? Testing is complete when there are no defects left in the test object (e.g. software product) Is this possible to achieve? Can we prove that the software is correctly designed using some formal method (i.e. logic and mathematics)? Can we test the product/system exhaustively with all possible inputs in all possible contexts? Autumn 2017 CSM14104 Software Product Management 11

12 Limits of testing Unfortunately, the answer is: no There are theoretical limits to what can be proved automatically with algorithms (remember halting problem?) and proving by hand is not practical Besides, the implementation still needs to be verified against the design Autumn 2017 CSM14104 Software Product Management 12

13 Limits of testing Even a simple function can defy exhaustive testing Imagine a Java method that takes two parameters of type Integer and returns a third Integer In order to exhaustively test the function you would need to execute it with all possible pairs of input values and compare the result to the expected value There are ( Integer.MIN_VALUE +Integer.MAX_VALUE) 2 different pairs - a very big number (= 2 64 )! Even if distributed on a number of fast cores for parallel computation, it could take years in calendar time to run all test cases (and from where do you get the expected value for each input?) Or, imagine a legitimate program executing a never ending loop each sequence of n executions of the loop is a distinct path through the program and needs it own test case (path coverage) How many paths are there? Autumn 2017 CSM14104 Software Product Management 13

14 Test strategy Complete, exhaustive testing is not possible, so, what do we do? We need to come up with a test strategy for the project Defines the project s testing objectives and the means to reach them Sets priorities for testing based on risk assessment Distributing test activities to the subsystems/major components depending on expected risk and cost/severity of failures Critical parts and functions should be tested extensively Less risky or infrequently executed parts of the system may be tested only lightly The goal is to find an optimal distribution of the limited time and effort available for testing to the different parts of the system under development As a product manager, you need to understand the product priorities and risks and help the test manager & devs to get their priorities right Autumn 2017 CSM14104 Software Product Management 14

15 The fundamental test process 4 [4] Andreas Spillner, Tilo Linz, Hans Schaefer: Software testing foundations - a study guide for the certified tester exam : foundation level, ISTQB compliant, 4th Edition. Santa Barbara, CA : Rocky Nook, Inc., Autumn 2017 CSM14104 Software Product Management 15

16 Testing for quality characteristics Freedom from defects is one important aspect of quality i.e. implemented features produce correct results and don t crash But there are also other aspects to quality (see ISO & 25012) Test cases can be defined that test also non-functional quality characteristics System functions may be required to return a result within a certain time limit, i.e. the Response Time (sub-characteristic of Performance Efficiency) must be less than a fixed number of milliseconds in 98% of calls We can implement test cases that not only call the function and verify the result but also record the response times to provide the statistics needed for checking that the whole requirement is satisfied Autumn 2017 CSM14104 Software Product Management 16

17 Testing in product development Autumn 2017 CSM14104 Software Product Management 17

18 Different kinds of testing Testing can be characterized by Which part of the product are you testing (the test object)? On what basis do you define the test cases? What aspects of quality are you investigating? What is the overall goal of testing? Is testing scripted with pre-defined test cases? Are test activities automated? Autumn 2017 CSM14104 Software Product Management 18

19 Test levels [4] Andreas Spillner, Tilo Linz, Hans Schaefer: Software testing foundations - a study guide for the certified tester exam : foundation level, ISTQB compliant, 4th Edition. Santa Barbara, CA : Rocky Nook, Inc., Autumn 2017 CSM14104 Software Product Management 19

20 Test levels component test Test Object: a single unit or component (class, module) Goal: verify the external behavior and internal implementation of the smallest self-contained elements of a software system in isolation from other components Test Basis: the internal design and implementation (white-box) and the external specification (i.e. interace definition, black-box) May require stubbing (mocking) of external dependencies The execution of component tests is usually automated as part of a Continuous Integration pipeline Autumn 2017 CSM14104 Software Product Management 20

21 Test bed [4] Andreas Spillner, Tilo Linz, Hans Schaefer: Software testing foundations - a study guide for the certified tester exam : foundation level, ISTQB compliant, 4th Edition. Santa Barbara, CA : Rocky Nook, Inc., Autumn 2017 CSM14104 Software Product Management 21

22 Black-box and White-box [4] Andreas Spillner, Tilo Linz, Hans Schaefer: Software testing foundations - a study guide for the certified tester exam : foundation level, ISTQB compliant, 4th Edition. Santa Barbara, CA : Rocky Nook, Inc., Autumn 2017 CSM14104 Software Product Management 22

23 Test levels integration test Test object: a sub-system or other coherent entity in the architecture of the system that is composed of individual (tested) components Integration testing can also span several sub-systems, i.e. in different system scopes Integration strategy determines the scope Goal: verify the interactions of the components within the sub-system and the interactions between sub-systems Discover problems in interfaces and behavioral assumptions of separately developed components Check interactions with acquired COTS software Test basis: architecture of the system, use-cases or system functions that span several subsystems May require stubbing (mocking) of external dependencies Execution of tests may be automated (CI pipeline); i.e. smoke tests Autumn 2017 CSM14104 Software Product Management 23

24 Test levels system test Test object: the system as a whole Goal: validate that the implemented system meets the needs of the customer and end users Test basis: requirements, risk analyses, mission statements, specifications (e.g. scenarios and use cases, user stories), user manuals, system architecture & design notes, users, competing products, similar products, previous versions of the product Should be done in an environment that matches the production environment (not in the production env.) All external dependencies must be in place Test data must be as realistic as possible Autumn 2017 CSM14104 Software Product Management 24

25 System test - scripting System tests are often scripted A script defines a sequence of interaction steps with the system where the input data and actions and the expected response of the system are exactly defined in each step The tester performs the actions as directed by the script, observes the results and records and reports any anomalies or deviations from the expected result Can be tedious and boring if the same tester repeats the same tests many times over - all kinds of errors are possible in test execution There is a strong tendency to automate system test execution Need tools that can automate the interaction with and observation of system responses in the UI (Robot framework, Selenium etc.) Especially relevant for regression tests Autumn 2017 CSM14104 Software Product Management 25

26 System test exploratory testing 5 Scripted tests can typically only find problems and defects that the test designer had in mind when writing the test cases Remember: complete testing is not possible They loose some of their effectiveness over time as the software changes and new defects are introduced that the existing tests are not designed to find Maintenance of the test cases and automation is a problem in most projects Exploratory testing is a popular and effective way to do manual testing on new functionality There is no predefined script but the tester selects a charter or agenda (scope and objectives) for a test session (functions to test, what to observe) and forms test cases on the fly She interacts with and observes the system recording any problems she finds and the steps needed to replicate it The tester must be able identify with the users and their needs based on her experience and knowledge of the product [5] Autumn 2017 CSM14104 Software Product Management 26

27 Test levels acceptance test Test object: the whole system Goal: validate the suitability and adequacy of the system from users point of view (fitness for use) Test basis: user and system requirements (use cases, user stories etc.), business processes, risk analyses, forms, reports, laws and regulations, maintenance and system administration procedures and rules Done in the customer s actual production environment Individual functions/features can be tested for acceptance as they become ready in incremental development (e.g. agile) Autumn 2017 CSM14104 Software Product Management 27

28 Types of acceptance testing Contract acceptance testing Fulfillment of test criteria as stated in the development contract User acceptance testing Fitness for purpose for all user groups Operation acceptance testing System administration and maintenance (backup/restore, user management, security etc.) Field testing Distributing pre-release versions of the system to selected customers to get a wider coverage of possible operational environments Alpha & beta testing, dogfooding Autumn 2017 CSM14104 Software Product Management 28

29 Testing subsequent versions of the product Software does not wear out like physical systems All faults once injected stay there until discovered and removed (or not..) Software systems undergo many changes during their lifetime The operational environment changes Customers needs change Bugs are corrected The retirement of a system may also incur new development Data migration, new interfaces and dependencies, partial replacement All these events involve testing the changed parts and checking for regression in the rest of the system Need two testing strategies for hot fixes and scheduled releases Autumn 2017 CSM14104 Software Product Management 29

30 General principles of testing Based on 40 years of experience Autumn 2017 CSM14104 Software Product Management 30

31 Principles 4 Principle 1: Testing shows the presence of defects, not their absence Testing can show that a product fails, i.e. there are defects Testing cannot prove the program is defect free If all the tests pass, it only tells you that these tests cannot find defects Adequate testing reduces the probability that hidden defects are present in the test object Principle 2: Exhaustive testing is not possible Like we saw earlier, even simple functions can require an astronomical number of test cases for exhaustive testing [4] Andreas Spillner, Tilo Linz, Hans Schaefer: Software testing foundations - a study guide for the certified tester exam : foundation level, ISTQB compliant, 4th Edition. Santa Barbara, CA : Rocky Nook, Inc., Autumn 2017 CSM14104 Software Product Management 31

32 Principles Principle 3: Testing activities should start as early as possible The goals of testing should be decided early and testing activities should focus on them to find defects early Principle 4: Defect clustering Defects cluster together; most defects are found in a few parts of the test object If many defects are detected in one place, there are usually more nearby Autumn 2017 CSM14104 Software Product Management 32

33 Principles Principle 5: The pesticide paradox Insects and bacteria become resistant to pesticides; similarly, if the same tests are repeated over and over, they loose their effectiveness A test case can typically only find defects that the designer of the test case thought of The software changes during development but do the test cases? Existing test cases do not cover all the software components and possible input combinations; new tests can find new defects Principle 6: Testing is context dependent Testing must be adapted to the risks inherent in the use and in the environment of a software system No two systems should be tested exactly in the same way Safety-critical systems should be tested differently than games Autumn 2017 CSM14104 Software Product Management 33

34 Principles Principle 7: No failures means the system is useful not! Finding failures and repairing defects does not guarantee that the system meets user expectations and needs Early involvement of the users in the development process and the use of prototypes help to avoid falling victim of this fallacy Autumn 2017 CSM14104 Software Product Management 34