INF 3121 Software Testing - Lecture 01 Introduction. Fundamental concepts in testing 1. Why is testing necessary?? 4. Fundamental test process 5. The psychology of testing 1
1. Why is testing necessary? ü LO: Describe with examples the way in which software can cause harm to a person, environment or company ü LO: Distinguish between a defect and its effects ü LO: Give reasons why testing software is necessary ü LO: Explain why testing depends on the level of risk, time and budget 2
Software systems context Software systems are an important part of life: Air traffic control SS Web browsers Content management systems Telecom networks Word processors Most people had experience with software not working as expected. If the SW system doesn t wok correctly, it can lead to problems like: Loss of money Loss of business reputation Injury or death 3
Causes of software defects Human errors Internal Fatigue Lack of training Lack of understanding Lack of interest External Time pressure Complex code Many system interactions Changed technologies Non-controllable events (i.e. environmental conditions) Environmental conditions Radiation Magnetism Pollution 4
Causes of software defects Both causes of errors produce defects ( = faults, bugs) in the code. Defects, if executed, may result in failures of the SW system (the system will fail to do what it should). Failures can affect seriously the users of the SW system, i.e.: Break pedal not working in some cars Miscalculations in financial SW systems 5
Four typical scenarios 6
Cost to repair 7
Role of testing Testing has an important role in all the stages of SW products life cycle: Planning Development Maintenance Operations 8
Role of testing is To reduce the risk of problems occurring during operations To check if the SW system meets: - legal requirements - Industry specific standards To learn more about the SW system 9
Testing and quality Testing: Measures the quality of certain characteristics of the SW in terms of defects found Functional aspects Non-functional aspects (Reliability, Usability, Portability) Gives confidence in the quality of the SW That is, if there are few defects found and properly tested Teaches us lessons to apply in future projects By understanding the root causes of defects, processes can be improved. This can prevent defects from reoccurring. 10
How much testing is enough? It s impossible to test everything! Testing should provide sufficient information for the stakeholders to take informed decisions about: Release of the software Next development steps, etc It depends on: Level of risk: Technical risks Business risks Project constraints Project constraints: Time Budget 11
? ü LO: Recall the common objectives of testing ü LO: Provide examples for focus of testing in different phases of the software life-cycle ü LO: Differentiate testing from debugging 12
Test activities Test planning Test control Test analysis Test design Test implementation Test executing Checking results Evaluating exit criteria Test result reporting Test closure 13
Definition The process consisting of all life cycle activities: - both static and dynamic, concerned with: - planning, preparation and evaluation of : to: - software products and related work products - determine that they satisfy specified requirements - to demonstrate that they are fit for purpose - and to detect defects. 14
Depending on the objectives of the test process, testing can be focused on: Confirming that the SW system meets the requirements Causing as many failures as possible Check that no defects have been introduced during dev. changes Assess the quality of the SW (with no intention of finding bugs) 15
Objectives for testing Finding defects reduce the probability of undiscovered defects Gaining confidence in the level of quality Providing information for decision-making Preventing defects 16
3. Test Principles ü LO: Explain the seven principles in testing 17
The seven testing principles offer some guidelines common for all testing: P1: Testing shows presence of defects Testing can show that defects are present, but cannot prove there are no defects. Testing reduces the probability of undiscovered defects remaining in the software; but even if no defects are found, this is not a proof of correctness. 18
P2: Exhaustive testing is impossible Testing everything (all combinations of input and preconditions) is not feasible except for trivial cases. We use risks and priorities to focus test effort. 19
P3: Early testing Testing should start as soon as possible in the development life-cycle and should be focused on defined objectives. 20
P4: Defect clustering A small number of modules contain most of the defects discovered during prerelease testing. 21
P5: Pesticide paradox If the same set of tests will be repeated over and over, it will no longer find new bugs. 22
P6: Testing is context dependent I.e., safety-critical SW is tested differently from an e-commerce site. 23
P7: Absence-of-errors fallacy Finding and fixing defects does not help if the SW system is un-usable or does not meet user s expectations. 24
ü LO: Recall the five fundamental test activities and respective tasks from planning to closure 25
Plan and control Plan means: what, how, when, by whom? Scope, objectives and risk analysis Test levels and types that will be applied Documentation that will be produced Assign resources for the different test activities Schedule test implementation, execution, evaluation Control and adjust the planning to reflect new information, new challenges of the project. 26
Analysis and design Review test basis: Requirements Product architecture Product design Interfaces Risk analysis report Analysis: general test objectives are transformed into: Design: Test conditions Test cases Test cases Test environments Test data Create tracebility 27
Implementation and execution Implement: Group tests into scripts Prioritize the scripts Prepare test oracles Write automated test scenarios Execute: Run the tests and compare results with oracles Report incidents Repeat test activities for each corrected discrepancy Log the outcome of the test exectuion 28
Evaluate exit criteria and report Evaluate: Assess test execution against the defined objectives Check if: More tests are needed Exit criteria should be changed Report: Write or extract a test summary report for the stakeholders. 29
Test closure activities What deliverables have been delivered? Are there any remaining ones we need to re-plan? Check the closure of incident reports Archive the items delivered: SW code Set of tests & results Documentation created/updated (this is necessary for future reuse) Analyze lessons learned for future releases. 30
5. The psychology of testing ü LO: Recall the psychological factors that influence the success of testing ü LO: Contrast the mindset of a tester and of a programmer 31
Independence test levels A certain degree of independence is often more effective at finding defects and failures. However, the developer can very efficiently find bugs in their own code. Applying a certain level of independence of the testing depends on the objective of testing. Independence levels: Tests designed by the same person that wrote the code Tests designed by another person from the same team (dev. colleague), but same organization Tests designed by a person from a different team (QA colleague), but same organization Tests designed by a person from a different organization / company (outsourcing the testing) 32
Tips and tricks Looking for failures requires: Curiosity Professional pessimism Attention to details Good communication skills Experience on error guessing Communicate failures in a constructive way: fact-focused; give factual reports and review findings Be clear and objective Confirm that: You have understood the requirements The person that has to fix the bug has understood the problem 33