Testing and QA in Incremental Software Development Project

Transcription

1 T Software Project Testing and QA in Incremental Software Development Project SoberIT Contents Testing as a part of incremental development Test Planning From project management viewpoint Test execution Test reporting Test status reports Defect reporting Bugzilla tutorial by Mikko Rusama Designing and managing test cases Equivalence partitioning and boundary value analysis Error guessing Example - how to manage test cases Exploratory testing approach Peer testing using exploratory approach 2 1

2 Testing is an integral part of sw development Often, testing is seen as some separate, last phase of software development process That can be outsourced to separate testing team That only deeds to be done just before the release if there is any time Testing can not be separated from rest of the software development Testing has to be involved from the beginning Testers can, and should, contribute in each phase of the software development life-cycle Testing is much more than the final acceptance testing phase 3 Testing as part of incremental development 4 2

3 Process models from waterfall to XP Functionality Requirements Time Design Implementation Testing Waterfall Incremental e.g. RUP Agile - XP 5 The V-model of testing Requirements Acceptance testing Functional specification System testing Build Architecture design Module design Unit testing Integration testing Test Coding V-model is an extension of the waterfall model You imagine a little V-model inside each iteration However, you might want to be more iterative on iteration level, too. Do not take the V-model as a process for the whole project 6 3

4 Two extremes of organizing testing Waterfall model Agile models (XP) Coders Tester Customer Coder Coder Testers Leading idea: Testing in collaboration 7 Definition of Software Testing Testing is a process of planning, preparation, execution and analysing, aimed at establishing the characteristics of an information system and demonstrating the difference between actual and required status. (Pol & Veenendal 2002) 8 4

5 Execute tests incrementally Each iteration delivers tested software Don t plan test execution as a separate phase after development Unit tests are executed as a part of coding activity Test driven development Functional system tests can be designed and executed simultaneously with implementation Enables fast feedback Remember tracking What was tested What version and environment When it was tested By whom What were the results 9 Involve the customer early Take the customer with you to specify or review the test cases The customer plays the oracle role Give the customer the opportunity to Execute pre-specified or exploratory tests Play around with the system Before the FD phase 10 5

6 Test Planning 11 Great testing challenges Agency: All testers act on behalf of some client. How do testers faithfully represent them? Evaluation: Not all failures are obvious. When a bug occurs, how do testers recognize it? Coverage: There's too much to test. How do testers decide what to test and what to ignore? Logistics: Testing must pace and respond to the project. How do testers apply their efforts at the right time and place to be useful? Reporting: Testing is about information. How do testers report their work in such a way as the information will be received and used? Stopping: Even though there's too much to test, not all testing is worthwhile to perform. How do testers know when enough's enough? Bach Test planning decisions 12 6

7 Purpose of the test planning To identify the items being tested, the features to be tested, the testing tasks to be performed To prescribe the scope, approach, resources, and schedule of testing activities To assign the responsible for each task To identify the risks associated with the plan To make conscious decisions of all these things Communicating, not just recording 13 Checklist for test planning Overall test objectives (why) What will and won t be tested (what) Test approach (how) Test phases Test strategy, methods, techniques, Metrics and statistics Resource requirements (who) Tester assignments and responsibilities Test environments Test tasks and schedule (when) Risks and issues 14 7

8 Overall test objectives (why) Why are we testing? What is to be achieved by the tests, and what are the risks associated with testing this product? The quality goals for each iteration This course Document your objectives in project plan chapter 5.3 Metrics that are used to evaluate the quality of the results in the end of each iteration Document in project plan chapters 6.x 15 What will and won t be tested (scope) Identify components and features of the software under test High-enough abstraction level Prioritize Both functional and non-functional aspects Consider time, resources and risks Everything can t be tested and everything that is tested can t be tested thoroughly Identify separately components and features that are not tested This course Document in project plan chapter 5.3 For each iteration 16 8

9 Test approach (how) How testing is performed in general and each iteration Levels of testing Test techniques Functional, non-functional Methods and techniques Tools and automation exploratory testing Used in peer testing (use also to supplement the planned tests) What other QA activities are used and how document/code reviews or inspections coding standard collecting continuous feedback from the customer Reporting and defect management procedures how the testing results are utilized and the feedback provided to steering the project Scope of test documentation On what level and how test cases are documented What other test documentation is produced This course Document in project plan General approach in chapter Details for each iteration in chapters Resource requirements (who) People How many, what expertise Responsibilities Equipment Computers, test hardware, printers, tools. Office and lab space Where will they be located? How big will they be? How will they be arranged? Tools and documents Word processors, databases, custom tools. What will be purchased, what needs to be written? Miscellaneous supplies Disks, phones, reference books, training material. Whatever else might be needed over the course of the project. Define responsibilities Identify limited / critical resources Location and availability This course Document in the project plan 18 9

10 Test environments Identification of test environments Hardware, software, os, network,... Prioritization and focusing test suites on each Number of combinations can be huge Regression testing in different environments Scheduling implications Test lab Different hardware and software platforms Cleaning the machines Setting up the test data Moving from platform to another People vs. hardware needs 19 Testing tasks and schedule (who) Work Breakdown Structure (WBS) Areas of the software Testable features Assigning responsibilities Mapping testing to overall project schedule Both duration and effort Build schedule Number of test cycles Regression tests Releases External links, i.e. Beta testing Consider using relative dates This course Document in the project plan 20 10

11 Test case organization and tracking Prioritizing tests The most severe failures The most likely faults Priorities of use cases End-user prioritizing the requirements Most faults in the past Most complex or critical Positive / negative Create test suites Test-to-Pass (Positive testing) Test-to-Fail (Negative testing) Smoke test suite Regression test suite Functional suites Different platforms Priorities, different passes 21 Test planning during PP and early I1 Project level Plan test approach (strategy) Document in project plan chapter 5.3 Including what will and won t be tested in each iteration Plan test environments and tools Document in project plan chapter 5.4 Plan test case organization and tracking Organizing tests into test suites Prioritization of test suites Documented as an initial test suite structure for the test cases You have 2 weeks left to do project level and I1 planning! Iteration level Measurable quality goals for the developed functionality What will be tested What won t be tested Details of the test approach How testing is performed Priorities of testing Testing rounds i.e., how many times and when certain tests are executed Tasks and schedule Resources Responsibilities Test deliverables Document in the project plan chapter 6.x 22 11

12 Test Reporting 23 Defect tracking and reporting Document your choices in project plan chapter 5.3 What bugs are reported Why defect tracking You don t forget found defects You get metrics Bug lifecycle When and how bugs are managed When and what bugs are fixed Who decides, when and how Use Bugzilla or some other defect tracking system Bugzilla provided by the course 24 12

13 Bug metrics Reported Closed Open I I I3 DE Total Total open This phase reported Blockers 1 0 Critical 2 1 Major 5 10 Minor Trivial Total Description of blocker and critical bugs found and open other QA metrics unit test coverage code reviews source code metrics 25 Quality assessment 1/2 Functional area File conversions GUI editor Encoder Admin tools Coverage Max functional areas Testers assessment of the current quality status of the system Quality Comments Only few minor defects found, very efficient implementation. Not started 2 critical bugs found during last test round, lot of small problems Nothing serious yet Legend Coverage: 0 = nothing 1 = we looked at it 2 = we checked all functions 3 = it s tested Quality: = quality is good = not sure = quality is bad 26 13

14 Quality assessment 2/2 Evaluate the quality of the different functional areas of the system how much effort has been put on test execution what is the coverage of testing what can you say about the quality of the particular component based on your test results and gut feeling during testing is the number of reported bugs low because of lack of testing or high because of intensive testing Assess the quality status of the system against the goals of the increment 27 Test report and log Test report template provided Summary of testing tasks and results No detailed lists of passed and failed test cases Includes evaluation of the quality Test log Provides a chronological record of relevant details about the execution of tests Who tested, when and what (version, revision, environment, etc.) Lists all executed test cases Results, remarks, bugs and issues of each test case Execution times, used data files, etc. See TestCaseMatrix.xls, for example

15 Test Case Design 29 Well-timed test design Early test design test design finds faults faults found early are cheaper to fix most significant faults found first faults prevented, not built in no additional effort re-schedule test design test design causes requirement changes Not too early test case design Design in implementation order All tests do not have to be designed before implementation starts Start test design from the most completed and probable features Avoiding anticipatory detailed test design and deprecated, incorrect test cases that are not based on the actual features Test planning and design must begin early, test case design not necessarily 30 15

16 A good test case Effective Exemplary Evolvable Economic Finds faults Represents more than one singular task Easy to maintain Cheap to use More about this issue: 31 Do not forget the expected values The expected values explicitly define what is the correct result Important if the correct result is not obvious Subconscious desire for the test to pass less work to do, no incident report to write up It looks plausible, so it must be OK less rigorous than calculating in advance and comparing If expected values are not defined The tester wants and expects to see correct behavior The tester does not know the correct behavior Tester assumes the correct behavior Looks OK to me! 32 16

17 Deriving test cases from use cases If the functional requirements are modelled as use cases it is sensible to utilize them in functional testing Use case!= test case Testing is interested in the uncommon and abnormal scenarios One use case leads to several test cases Prioritize use cases and use this prioritization when prioritizing tests Maintain traceability between use cases and test cases Use cases are not complete specifications Testing only the conditions that are mentioned in use case is usually not enough See Robert V. Binder s Extended Use Case Test Design Pattern 33 Use case example: use case 1. User slides a card through the card-reader 2. Card-reader scans employee ID from card Exception 1: Card can t be read Log event Use case ends 3. System validates employee access Exception 2: Employee ID is invalid Log event Use case ends 4. System unlocks door for configured time period Exception 3: System unable to unlock door Log event Use case ends 5. User opens door Exception 4: Door is not opened System waits for timeout System locks door Use case ends 6. User enters and door shuts Exception 5: Door is not shut System waits for timeout Log event Set alarm condition Use case ends 7. System locks door Exception 6: Door fails to lock System attempts to lock door Log event Set alarm condition Use case ends 34 17

18 Use case example: test cases Test Case 1: Valid employee card is used Slide the card through the reader Verify door is unlocked Enter building Verify door is locked Test Case 2: Card can t be read Swipe a card that is not valid Verify event is logged Test Case 3: Invalid employee ID Swipe card with invalid employee ID Verify door is not unlocked Verify event is logged Test Case 4: System unable to unlock door Swipe card Injected failure of unlocking mechanism Verify event is logged Test Case 5: Door is not opened Swipe card Verify door is unlocked Don t open the door and wait until timeout is exceeded Verify door is locked Test Case 6: Door is not shut after entry Swipe card Enter building Hold door open until timeout is exceeded Verify alarm is sounded Verify event is logged Test Case 7: Door fails to lock Swipe card Enter building Injected failure of locking mechanism Verify alarm is sounded Verify event is logged 35 Error-guessing and ad hoc testing Always worth including After systematic techniques have been used Can find some faults that systematic techniques can miss Supplements systematic techniques Consider Past failures Intuition Experience Brain storming What is the craziest thing we can do? Lists in literature, error catalogs 36 18

19 Is this a good test case? ID: TC TITLE: File dialog test ENVIRONMENT: Notepad 5.0 build 2195, JRE build 10, Windows 2000 v , sp 4 PRIORITY: 1 DESCRIPTION: Testing the file open dialog STEPS: Double click Notepad shortcut on the desktop Select File menu Select open from file menu EXPECTED RESULTS: File menu dialog opens The title of the dialog is Open Open and Cancel buttons are both available Clicking the Cancel button closes the dialog DEPENDENCIES Preceding test case: TC Opening Notepad (passed) 37 Test Case Specification (IEEE Std 829) 1. Test-case-specification identifier: specifies the unique identifier assigned to this test-case specification. 2. Test items: describes the detailed feature, code module and so on to be tested. It should also provide references to product specifications or other design docs from which the test case was based. 3. Input specifications: specifies each input required to execute the test case (by value with tolerances or by name); identifies all appropriate databases, files, terminal messages, memory resident areas, and external values passed by the operating system; specifies all required relationships between inputs (for example, timing). 4. Output specifications: describes the result expected from executing the test case. Results may be outputs and features (for example, response time) required of the test items. Also provides the exact value (with tolerances where appropriate) for each required output or feature. 5. Environmental needs: Environmental needs are the hardware, software, test tools, facilities, staff, and so on to run the test case. 6. Special procedural requirements: describes any special constraints on the test procedures which execute this test case (special set-up, operator intervention, ). 7. Intercase dependencies: lists the identifiers of test cases which must be executed prior to this test case, describes the nature of the dependencies

20 Another simple approach to test cases Test case ID Priority Description Expected result Notes TC Testing the indent functionality It s possible to move indenting of the current line or selected lines to the right, and left. It s possible to align the indentation of the selected lines. Req The common details of a test suite and test procedure (like test environment) are documented elsewhere Avoiding copy paste Test catalogs are utilized to describe common details of test cases test all available ways of performing the function (menu, keyboard, gui buttons, menu short-cut, short-cut keys, ) Test settings or preferences that affect this function This may leave too much space for an inexperienced tester 39 Test Catalogs Test catalog is a list of typical tests for a certain situation Based on experience on typical errors that developer make Textual input/numbers the largest number that is supposed to be accepted the smallest number that is supposed to be accepted one larger than the largest number that is supposed to be accepted one smaller than the smallest number that is supposed to be accepted Machine boundaries (signed byte overflow) (unsigned byte overflow) (32 767) (32 768) (signed 16-bit overflow) (65 535) (65 536) (unsigned 16-bit overflow) ( ) ( ) (signed 32-bit word overflow) ( ) ( ) (unsigned 32-bit word overflow.) 40 20

21 Common pitfalls in test case definition Poor test case organization One big pile of test cases Don t know what a certain set of test cases actually tests or which cases test a certain functionality Don t know what was tested after testing Testing wrong things Prioritize and select the most important tests Consider the test case s probability to reveal an important fault Writing too detailed step-by-step scripts Not enough time for detailed scripting Few detailed, but irrelevant test cases designed and executed -> bad quality of testing, no major defects found Don t program people 41 Example how to manage test cases TestCaseMatrix.xls Test Suite Use case Test Case Priority Description Input values Expected results Notes Result File management UC-1 Open test documents of Test documents of different sizes. Different text documents can Test data files: different sizes. Including empty document and be opened and edited testdata\sizetes document which size exeeds successfully, if document is ts\size*.txt available memory. too big for the system to handle, a proper error message is shown. FM1 1 Pass File management UC-2 FM2 1 Pass File management UC-2 FM3 2 Issue File management UC-2 FM4 2 Fail File management UC-3 FM5 3 Pass Main hierarchy MH1 1 Pass Main hierarchy MH2 2 Pass Main hierarchy MH3 1 Fail Main hierarchy MH4 3 Fail Main hierarchy MH5 3 Fail Formatting FO1 2 Pass Formatting FO2 3 Issue Formatting FO3 3 Pass Formatting FO4 2 Pass Formatting FO5 2 Pass Regression RG1 1 Pass Regression RG2 1 Pass Regression RG3 1 Pass Regression RG4 1 Fail Regression RG5 1 Pass When do you write these test cases? (Hint: not at the end of the project) 42 21

22 Peer testing with exploratory approach 43 Exploratory Testing Manual testing by professional skilled testers Freedom, flexibility and fun for testers Optimized to find bugs Continually adjusting plans, re-focusing on the most promising risk areas Following hunches Minimizing time spent on documentation Focus on manual validation making testing activities agile 44 22

23 What is Exploratory Testing? In scripted testing, tests are first designed and recorded. Then they may be executed at some later time or by a different tester. Tests In exploratory testing, tests are designed and executed at the same time, and they often are not recorded. You build a mental model of the product while you test it. This model includes what the product is and how it behaves, and how it s supposed to behave Tests Product James Bach, Rapid Software Testing, Totally repeatable tests don t clear the minefield mines fixes 46 23

24 Variable tests are more effective mines fixes 47 Exploratory testing defined Exploratory testing is an iterative process of simultaneous learning, test design, and test execution. James Bach, 2003 To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing. James Bach, 2001 Exploratory testing involves simultaneously learning, planning, running tests, troubleshooting and reporting results. Cem Kaner, 2001 Our mission is to test whatever is needed, on short notice, without the benefit, or burden, of pre-defined test procedures. Jonathan Bach, Session Based Test Management,

25 Exploratory testing vs. ad-hoc testing? Exploratory testing is planned, goal driven and managed not an ad-hoc process Testing approach, goal and coverage can be well planned Detailed test log is created during the testing and defects are reported Test report should be reviewable by another person in order to determine what was tested, how and what are the results Testing is not something that anyone masters One important outcome of a test process is a better, smarter tester! 49 Session Based Test Management Charter Time Box Reviewable Result Debriefing 50 25

26 Charter Architecting the Charters is test planning Brief information / guidelines on: What should be tested? Areas, components, features, Why do we test this? goals How to test (approach)? Specific techniques or tactics to be used Test data What problems to look for? Might include guidelines on: Tools to use What risks are involved Documents to examine Desired output from the testing 51 Time Box Short: 60 minutes (+-15) Normal: 90 minutes (+-15) Long: 120 minutes (+-15) Focused test effort of fixed duration Brief enough for accurate reporting Brief enough to allow flexible scheduling Brief enough to allow course correction Long enough to get solid testing done Long enough for efficient debriefings Beware of overly precise timing 52 26

27 Reviewable results Charter Effort Breakdown Duration (hours:minutes) Test design and execution (percent) Bug investigation and reporting (percent) Session setup (percent) Charter / opportunity (percent/percent) Data Files Test Notes Bugs Issues 53 Debriefing The test lead reviews session sheet to assure that he understands it and that it follows the protocol The tester answers any questions Session metrics are checked Charter may be adjusted Session may be extended New sessions may be chartered Coaching / Mentoring happens 54 27

28 Peer Testing in FD iteration Peer group pairs are on course web pages Plan and prepare for peer testing already before DF Delivering and installing the system Meetings (preparation and debriefing) Agreeing on total effort Hand-off the system to the peer group The system under test All relevant documentation User and installation manual Known bugs, bug reporting guidelines Test Charter (at least 2 charters) one general charter, provided by course and at least 1 from the group whose system is tested Peer testing execution Peer testing reports delivered to the other group Agree this with your peer group 55 Peer test reporting Iteration FD peer test deliverables Peer test reports and session logs x 2 (Own and peer group s report) Defect reports directly into bug tracking system Peer group reports on the other groups product Bug summary listing as an appendix in the test report In final report you should assess peer group s testing efforts and results 56 28