Value based Testing Guideline

Transcription

1 Value based Testing Guideline 1. General Instructions 1.1. Overall Plan and Schedule Basically, we encourage you to do the testing concurrently with developers. This could be formal or informal. However, you need to do at least three times formal Value based Testing for your project s grading. Formal means that you need to follow the Value based testing procedure and document your testing procedure and results with the template Test Plan and Cases.doc and Value based Testing Procedure and Results Template.xls we will introduce you later. Table 1. The Overall Plan for Three times Value based Testing Testing # Due Day Scope Goal Before CCD Must finish the test Document a formal Practice on how to use the related before your scheduled testing procedure documents to do Value based CCD and submit it with before CCD Testing and record the testing CCD report procedure and results; Formal test before CCD to minimize the bugs during the CCD With IOC Working Set #1 With Transition + Support Set (TS Set) M, 4/5 Follow the prioritization of the last regression test to do other regression tests after developers fix the bugs. More details see Section 3.3: 4) M, 4/19 After adding new test cases for the newly added functionalities, prioritize again and test Formal Regression test to verify bugs that are found before and during CCD Formal Regression test after adding extra components or functionalities after CCD 1.2. Deliverables: Value based Testing Procedure and Results Template.xls that records the procedure and results of testing and fills all the related information 1

2 Note: What is the difference between the Word version of Testing Procedure and Results.doc and the Excel version of Value based Testing Procedure and Results.xls? Should we submit them both? No. The excel version has already included all parts of the word version, you just need to submit the excel version. But you could refer to the word version template in the ICM EPG to get more information Testing Exit Criteria: Table 2. The Exit Criteria for Three times Value based Testing Testing # Before CCD With IOC Working Set #1 With Transition + Support Set (TS Set) Exit Criteria At this stage, we don t require that you need to Pass all the test cases for this Iteration 1 before CCD. What we require is that you follow the right way to do Value based Testing, document the real situation of your testing procedure and results and do at least two regression tests (just like the example in Section 3.3). Of course, the more regression tests you do before CCD, the better. For example, it is not good that you have some serious bugs found in your CCD, but you Pass every test case before CCD. All test cases for Iteration 1 are executed; All test cases for Iteration 1 are Passed ; TBD 1.4. Grading criteria: 1. Good test cases. Some candidate criteria are: Completeness: the test cases have covered all the requirements for this iteration; Correctness: the test cases correctly reflect the requirements that need to be tested; Clear steps; Explicit inputs and outputs; Includes both invalid and valid if needed; Correct dependencies; 2. Reasonable ratings: Reasonable ratings for each test case s testing priority, cost, etc. 3. Correct Value based testing procedure: Follow the right procedure according to value based testing procedure. 2

3 4. All related information complete, clear and consistent 2. Related Template: Test Plan and Cases.doc (TPC) Value based Testing Procedure and Results Template. xls (Vb_TPR) Value based Testing Procedure and Results Template.xls is a follow up template for Test Plan and Cases.doc. The Purpose of this template is to help testers to record the test results, and help testers to prioritize testing cases from value based perspective. It has 5 sheets, for Sheet 1 and 5, and you could find more instructions in IICMSw_TPR_Template.doc. We will introduce in details about Sheet 2, 3, and 4. Table 3. Overview of Vb_TPR Sheet Content 1. Test Preparation See more information in IICMSw_TPR_Template.doc 2. Regression Testing More details in Section Dependency&Prioritization More details in Section Testing Log More details in Section Testing Incident Report See more information in IICMSw_TPR_Template.doc Please open the template Value based Testing Procedure and Result.xlsx, we will introduce you how to use this spreadsheet for your value based testing step by step: 3

4 2.1. Regression_Testing sheet: Figure 1 Overview of Regression_Testing sheet Basically, this sheet records all the test cases that need to be executed for this iteration. In this sheet, you would find fields you need to fill in: For Test Case Number, Test Item, Pre condition, Post condition, Pass/Fail criteria, Assumptions and Constraints, Traceability and Dependencies, you could directly reuse them from Test Plan and Cases (TPC). Note: One error we find in most of your TPC is that you misunderstand the Dependencies. For this field you need to list all test cases that this test case depends on i.e. the test cases that must be executed prior to the execution of this test case. For example, in this sheet, you will see that TC depends on TC 01 01, which means TC must be executed and passed before you execute TC Also you could see TC has Dependencies filled with TC 01 01, TC 01 04, TC 01 05, TC 01 06, and TC 01 07, which means, before you are able to execute TC 01 09, you need to execute and pass TC 01 01, TC 01 04, TC 01 05, TC 01 06, and TC first. Make sure you list all the test cases that this test case depends on, not only directly, but also indirectly. For example, in the sheet, you will see that TC depends directly on TC 01 04, TC 01 05, TC and TC 01 07, Meanwhile, TC

5 depends on TC 01 01, so for TC 01 09, you need also include TC here, because the dependency has transitive property. Think of it as a way of Testing Path. If this test case doesn t depend on any other test cases, fill it with NA For Input Specifications & Steps Description and Expected Output, they are pretty the same as in Test Plan and Cases (TPC) with a minor change. Here you need to explicitly state Step by Step how you execute the test case and the corresponding Step by Step expected output. Please see the example in the spreadsheet. Note: If one of the steps in one test case fails, the test case should be labeled as Fail. For Test Type, you could use the drop down list to choose one from Valid and Invalid. Basically, to test each requirement item, you need to consider about both valid inputs and invalid inputs. There is a testing case design strategy Equivalence partitioning (See more details at This is also a typical reason that why one requirement could generate multiple test cases. For example, please see the spreadsheet, the requirement item SSRD CR 01 corresponds to multiple test cases TC TC In fact you could generate many invalid test cases based on Assumptions and Constraints. For Testing Priority : You could choose the proper rating using the drop down list. Test Plan and Cases (TPC) has four ratings for Testing Priority: Must have, Should have, Could have and Want to have. Basically, Testing Priority is rated according to the corresponding requirement s business importance from the client s view which you have done in SSRD. However, to better differentiate the ratings, we extend Testing Priority to 5 level ratings. The mapping is in the table below. We know that for the first iteration, all requirements are nearly Must have. To differentiate the ratings into VH and H, the clue is: VH Must have means it is the core business functionality which will bring the direct benefit for the client, H Must have means if you don t implement this functionality, the VH Must have ones can t work, but it doesn t bring direct benefit for the client. For example, for an Online Shopping System, the core functionalities include Shopping Cart and Online Transaction which bring direct benefit to the client. So they should be rated VH. However, for the Login and Account Management functionalities, they are also Must have, because if you 5

6 don t implement these functionalities, Shopping Cart and Online Transaction can t work. However, they don t bring direct benefit to the client. So we should rate them as H. Table 4 Suggestive Rating for Testing Priority Rating VH (Very High) H (High) M (Medium) L (Low) VL (Very Low) Condition Must Have Could Have Should Have Want to Have For Estimated Testing Cost, you could choose the proper rating from the dropdown list. Again, the rating is from VL to VH For Pass?, you could choose the rating from the drop down list, which has 3 values: Y, N, NA Table 5 Status of Test Cases Value Y N NA Condition All steps in the test case generates the expected outputs As long as one of the steps in the test case generates an unexpected outputs The test case is not be able to be executed, there are some candidate reasons: This test case depends on another test case which fails; External factors, such as the testing environment e.g. the precondition could not be satisfied, or there is no required testing data; For Observed Results, if the test case fails, you should record its observed unexpected results, report it to Bugzilla and record the Bug ID in the Bug# field otherwise, you could leave both Observed Results and Bug# blank; For Actual Testing Cost (Hours) please record the time in terms of hours (e.g. 0.3 hour, 1.5 hours ) you spend in executing this test case, no matter whether it fails or pass. If you choose NA for the field Pass? just leave it blank. 6

7 Based on your recorded actual testing cost, you need to compare them and then choose one rating from the drop down list (again from VL to VH) for Actual Testing Cost Rating. You need to compare with your previous estimation, and refine your estimation rating for your next regression testing Dependency&Prioritization sheet: Don t be scared by this table, it is simple for you to fill in. You just need to fill in the Dependency Matrix based on the Dependencies field content in the Regression_Test sheet. All the blue cells are automatically calculated. You could find the example below which tells you how you fill the dependency matrix. Figure 2 Dependency Sheet Note: Please change the test cases numbers according to your reality. If your number of your test cases is more than 30, let me know, I will extend this table for your team. 7

8 Figure 3 The Value (Additive) would give you the testing priority that you should follow, the larger the number, the higher the priority. You don t need to care about Value (Multiplicative) for this time. You just need to follow the Value (Additive) for your prioritization. After you finish this regression testing, the testing results will be summarized automatically at the bottom of this sheet. Figure 4 Regression Testing Summary 2.3. Testing_Log sheet: After you finish one regression test, you need to manually record the results in the Testing_Log sheet. The result would include the Testing Summary which you could get the numbers from Figure 4. You also have to record the testing order you follow according to Value based testing order logic (more details in Section 3.2). Please also Red the test case that fails, Green the ones that passes. 8

9 Figure 5 Testing Log 3. Behind Principles and Calculation: 3.1. Principles: Value based testing principle: Prioritize the testing cases. Three factors should be considered for prioritization: Test Priority: as mentioned above, it serves as a measure for Business Importance from the client s view. The rating is from VL to VH, the value is from 1 to 5. Estimated Testing Cost: the rating is from VL to VH, the value is from 1 to 5. As mentioned above, you could refine this estimation for your next regression testing based on this time s actual testing cost rating. Impact: it means if this test case fails, what impact it will bring. This is measured by the Sum of Test Priority of all other testing cases which depends on this test case. After you choose the Test Priority rating for each test case, and determine the dependency among them, you will automatically get Impact for all test cases. 9

10 Impact (TC 01 01) =TestPriority (TC 01 03) +TestPriority (TC 01 04) +TestPriority (TC 01 07) +TestPriority (TC 01 09) = =14 Figure 6 Impact Caculation For Prioritization, combine the three factors together using both Additive Value Function and Multiplicative Value Function. Additive Value Function: Multiplicative Value Function (you don t need to consider about it for this time): 1, 0, 3.2. Testing order logic: Key Concepts: Before introducing the testing order logic, there are some key concepts you need to know: 10

11 1) Ready to Test: in fact, it is a status of test cases, and its definition is: A test case is Ready to Test only if the test case has no dependency or all the dependencies that it depends on have Passed. 2) Not Tested Yet: it is another status of test cases, and its definition is A test case is Not Tested Yet means this test case has not been tested yet so far. The test case status transition chart is shown as below, the dash line means it doesn t change the status of the failed test case into NA, but change the status of all test cases that depend on the failed test case into NA. <<No dependencies or all test cases in Dependencies Set have been passed>> Failed <<Change the status to NA for all test cases that depends on this failed test case>> Not-Tested-Yet Ready-to-Test NA Passed Figure 7 Testing Cases Status Transition 3) Dependencies Set: A test case s Dependencies Set is the set of the test cases that this test case depends on. It is a set of those test cases that you fill in Dependencies for this test case. Here we emphasize again that the Dependencies Set should include all dependent test cases, either directly or indirectly as mentioned in section 1.1. Of course, the Dependencies Set of one test case is a sub set of the Whole Set for prioritization. Testing Order Logic: The logic behind is simple, and there are some rules of thumb you need to follow: 1) Value First: Test the one with the highest value. 2) Dependency Second: If the test case with the highest value is not Ready to Test, which means at least one of the test cases in its Dependencies Set is Not Tested Yet. In such situation, prioritize the Not Tested Yet test cases according to Value First in this Dependencies Set and start to test until all test cases in the Dependencies Set are Passed. Then the test case with the highest value is Ready to Test. 11

12 Note: If one of the test cases in the Dependencies Set Failed, you need to change the status of the failed test case to Failed and set all test cases (of course including the one with the highest value) that depends on the failed test case to NA. The Failed and NA test cases are left to the next round regression testing after developers fix the related bugs. 3) Shrink the prioritization set ASAP: Exclude the tested one (no matter Failed or Passed ) and the NA out of the prioritization set. Value-based Prioritization for One Regression Testing Round Pick the one with the highest Value Exclude the Passed one for prioritization N <<- -In the Whole Set- ->> <<In the Dependencies Set>> Have dependencies? N Start to test <<Ready-to-Test>> Failed? N Y Y All dependencies passed? Y <<Ready-to-Test>> Exclude the Failed one and the others NA that depends on it for prioritization Multiple Regression Tests Until all Test Cases Passed Figure 8 Value based Testing Procedure Some considerable situations you might encounter in the process: When multiple test cases have the same highest Value: 1) Choose the one which is Ready to Test ; 2) If more than one test case are Ready to Test, then randomly choose one from the Ready to Test and start to test; 3) If no one is Ready to Test, choose the one whose Dependencies Set has the least Not Tested Yet test cases and start to test these Not Tested Yet test cases according to Value First 12

13 3.3. An explicit example: For simplicity, assume we have 9 test cases from TC to TC However, this example has almost included all the situations you might encounter when you do value based testing. Don t be too serious about the ratings in this example, they might be dummy, because we want to include all the situations you might encounter. 1) Before the first round test. All test cases have not yet been tested, so Boolean (Fail)=0. So Value is determined by Test Priority and Estimated Test Cost. Figure 9 Explicit Example (1) 2) An explicit example: From the above sheet prioritization, TC has the highest priority and it doesn t have any dependency. So start to test it, assume that it passes, so choose Y for Pass? in Regression_Testing sheet and you will find that in the corresponding cell s traffic light in Dependency&Prioritization will automatically turn Green. 13

14 Figure 10 Explicit Example (2) Exclude it of the prioritization, and look at the remaining 8 test cases, and pick the one with the highest priority. You could see that there are two test cases TC and TC with the same priority Choosing which one depends on their dependencies. We could see that TC has one dependency TC which has Passed, which means TC is Ready to Test, but TC has the dependencies TC (passed), TC (Not tested yet) and TC (Not tested yet), which means TC is not Ready to Test. According to the rule of thumb, so test TC first. Assume that TC is passed, so choose the Y for Pass? in Regression_Testing sheet and exclude it in the prioritization set. Then look at the remaining 7 test cases, TC has the highest priority; however, TC is not Ready to Test, since it has dependencies TC and TC which have not been tested yet. In order to test TC 01 07, we should determine which to test first for TC and TC (this implies the rule of thumb Value First, Dependency Second ), again, the value determines this. Since TC has higher value (0.00) than TC ( 2.00), also TC s dependency TC has already been passed, so test TC first, assume it marked Passed in the <<Dependencies Set>>, of course at the same time has been excluded in the <<Whole Set>>. And then test TC 01 05, assume it passed, again, marked it Passed in the <<Dependencies Set>>, of course at the same time has been excluded in the <<Whole Set>>. 14

15 Figure 11 Explicit Example (3) Now since the dependencies have all been passed, and TC has the highest value and Ready to Test in the remaining 5 test cases, so start to test it. Assume TC Failed, then choose N for Pass? in Regression_Testing sheet, and check if other test cases depends on it, we could see that TC depends on it, so block TC by choosing NA for Pass? in Regression_Testing sheet. At the same time you need to report the bug for TC to Bugzilla and record the Bug# 1201 here. And the spreadsheet will automatically generate the bug s priority. Then exclude both the Failed TC and NA TC in the prioritization set. Figure 12 Explicit Example (4) 15

16 Look at the remaining 3 test cases, again, choose the one TC with the highest priority value and it is already Ready to Test, assume it passed. Then look at the remaining TC and TC 01 08, and we could see that they both have the same priority and both are Ready to Test, so randomly choose one to test. Assume they are both passed at last. The result is shown below: Figure 13 Explicit Example (5) So until now, all executable test cases have been executed for this regression test round. You need to manually record the testing result summary and the test order in Testing_Log sheet. Red the test case that fails, Green the ones that passes. Figure 14 Explicit Example (6) 3) After this time s regression testing Current failed test cases values also give the relevant bug prioritization. This would provide bug prioritization information for developers to fix bugs. 4) The next round regression testing After developers fix all the bugs, the tester starts to do another regression testing, refine the estimated testing cost first based on last time s actual testing cost rating, then follow the prioritization to test. 16

17 For example, in this case, after the developer fix the bug for TC The tester starts to do another regression test. Based on the current test priority, we could see the TC has the highest value, so test it first. Since it has dependencies, so need to test TC 01 01, TC and TC first, according to the testing order logic, In TC s Dependencies Set, you need to prioritize them too, according to the rules of thumb Value First and Dependency Second, the order should be TC >TC >TC Assume that the developer doesn t introduce the new bugs when he fixes the bug, so they all Passed, so you could test TC now. Figure 15 Explicit Example (7) Assume TC passed this time, although TC is ready for testing now, however its value ( 2.00) is very low. So according to the testing logic, the following order should be TC >TC >TC >TC >TC (Assume they all passed). Figure 16 Explicit Example (8) 17

18 You could see that for this regression testing, all test cases are passed; So until now, all executable test cases have been executed for this regression test round. You need to manually record the testing result summary and the test order in Testing_Log sheet for this round of regression testing. Again, Red the test case that fails, Green the ones that passes. Since all the test cases are passed for this regression test round, so you have finished this time s test. Figure 17 Explicit Example (9) 18