HP ALM: Less Test Cases, More Coverage September 17, 2014

Transcription

1 HP ALM: Less Test Cases, More Coverage September 17, 2014

2 Brought to you by Vivit Testing, Quality and Application Lifecycle Management Special Interest Group (TQA-SIG) Leaders: Damian Versaci, Christopher J. Scharer, Robert Linton, Bernard P. Szymczak Jr., Andreas Birk

3 Hosted by Mihai Grigorescu TCoE Tools Lead at Accenture Vivit Leader of the South Africa Chapter

4 Today s Presenter Huw Price Managing Director Grid-Tools

5 Housekeeping This LIVE session is being recorded Recordings are available to all Vivit members Session Q&A: Please type questions in the Questions Pane

6 Webinar Control Panel Toggle View Window between Full screen/window mode. Questions

7 Challenges Ambiguous (or non-existent) requirements Test cases are currently manually designed by hand Limited ability to gather coverage statistics for test cases Limited re-usability of test cases Poor linking of Test Cases to Test Data Very poor complexity analysis

8 Inspiration from Hardware Testing Think of software as a circuit board: Solid, unambiguous requirements Test cases built using mathematical methods Accurate coverage statistics Ability to test hardware with billions of logic gates

9 What is Coverage? It is not: Code Covered Number of Tests Covered Tests Run Percentage of use cases All Paired Combinations It is: Designing Sufficient Tests To VERIFY That The Design And Code Correctly Implement The Requirements Did you get the right answer for the right reason - Two or more defects may sometimes cancel each other out - Observability

10 What is Coverage? If the customer is a business client or a preferred personal client and they have a checking account, $100,000 or more in deposits, no overdraft protection and fewer than 5 overdrafts in the last 12 months, set up free overdraft protection otherwise do not give overdraft protection. This function has sixty-four possible combinations of input from which to select test cases: So which ones to choose? Copyright 2008 Bender RBT Inc.

11 Different Coverage Techniques Combinatorial All Pairs Constrained All Pairs Does not support Expected Results - You have to work it out for each combination which is very time consuming Combinatorial is something that accidentally increases functional coverage (and only to a point) Combinatorial does not give you any solid metrics on how good your testing actually is you can only infer that x out of y combinations have been covered, but it has no bearing in actual functional logic. You end up with lots of false errors that have to be checked by hand

12 Different Coverage Techniques Assume C and F are not observable events. Assume A is stuck at FALSE. Enter as a test case A(T), B(T), D(T), E(T). T T T T T Results should be C(T), F(T) and G(T). T T Cause-Effect Graphing Observable Events and Path Sensitizing Copyright 2008 Bender RBT Inc.

13 Different Coverage Techniques Optimized Flow Chart Modelling Advanced Graph Optimization Most projects already have a flow chart Similar results to Cause and effect Supports Looping Supports Constraints Not applicable for non sequential based logic

14 Introducing Quality Earlier Complexity Design Use Cases Test Cases Requirements Testing Test Data Development Automation

15 Building Requirements Using flowcharts, design the requirement in terms of individual design steps. The direction of flow between the blocks is unambiguous unlike freeform text. Consistency of logic may be verified programmatically.

16 Building Test Cases

17 Building Test Cases

18 Building Test Cases

19 Building Test Cases From flowcharts, come paths. Paths through a requirement can be: Use Cases Test Cases Paths are generated using mathematical techniques. Ensures that coverage can be measured and optimized.

20 Building Test Cases Test cases, being paths, can be checked for uniqueness. Test cases for different test stages (e.g. SIT, UAT) do not need to be duplicated manually. For example, unit tests can be used as a basis for functional tests.

21 Build Test Cases from Requirements - Example Baseline Agile Designer Test Cases Created Time Taken 5 hours 2 hours Test Coverage - % 16% 100% Agile Designer was also able to shrink 326 possible test cases down to just 17, retaining 100% coverage Based on interviews and analysis, client expected to see an 80-90% reduction in bugs in this case Change requests: Client must manually check and change all of the test cases

22 Requirements-based Testing Model of Test Plan Delivery Full Traceability Business Analyst / Requirements Manager The Requirements are conceived and created in Agile Designer. Project Manager / Development Manager Relevant project managers fill out detail. This can be passed back up the chain to Bas, etc. Test Manager Test Managers ensure implementation issues are tackled. Also can be passed up the chain for verification. Tester Testers receive comprehensive, logical test plans and test cases.

23 The HP Test Case Factory

24 IMPORTING TEST CASES Here are our tests in ALM/QC:

25 Agile Designer Test Cases We import We de-duplicate We optimize.

26 Agile Designer Test Cases When we have imported these into Agile Designer we can see we have 5 paths but as we mentioned before some cover similar paths. A occurs 4 times B occurs 4 times C occurs 4 times R occurs 2 times G occurs 2 times This gives us the perfect opportunity to optimize and improve the test cases

27 Agile Designer Test Cases Use intelligent comparison methods to identify duplicate design steps in your flow. Agile Designer uses this information in order to determine how the flow should be constructed from the test cases.

28 Agile Designer Test Cases We now have our optimized set of test cases which are in a logical, unambiguous flow. The flow chart should fully represent the system under test. The optimized set will reduce the amount of testing. If you make a change to the logic you will automatically know exactly what test to change and what new tests need to be designed.

29 Agile Designer Test Cases A comparison of statistics: Statistic Previous Current Reduction Nodes (Design Steps) Decision Branches % % This shows us a 50% reduction in steps we need to take whilst giving us 100% functional coverage!

30 Re-using Test Cases Unit Tests SIT Tests Regression Tests Functional Tests All Tests

31 Advantages Can be Imported/Exported as required into numerous forms, including ALM and Excel. Coverage Metrics give us fully quantifiable costing amounts for test Case cycle Easy to recognise incorrect flows, missing functions or new functions which require adding. Automatic handling of changes: a change to the flow induces automatic change in the test cases. No manual checking required.

32 What value does Agile Designer provide To Build better Requirements To Estimate Complexity To Create Perfect Test Cases To Improve my Existing Test Cases To link Test Cases to Change To Create and Manage my Test Data Language Use Cases Logic 100% Coverage Used Stand Alone to Improve a Process Risk Path Modelling AND OR OR Embedded in Your Software Lifecycle Each Component Has Value on its own AND can be used in conjunction with other functions Earlier Delivery AND Higher Quality

33 Q & A Q & A

34 Don t Miss Register Today Vivit Mobility Virtual Summit October 8, 2014 Register at: HP Discover 2014 Barcelona December 2 4, 2014, The Fira Barcelona, Gran Via Register at: 754_us/en/large/tsg/pl_ot_aw_homepage_vanity%20url/disc over_mkg/

35 Thank You Makes you more intelligent!

36