QUEST Webinar: Intelligent Defect Sampling An Innovation for Focused Testing WEBINAR SERIES WEBINA SERIES. Featuring

Transcription

1 QUEST Webinar: presents the Intelligent Defect Sampling An Innovation for Focused Testing WEBINAR SERIES WEBINA SERIES Featuring Randy Rice Rice Consulting Services

2 DEFECT SAMPLING AN INNOVATION FOR FOCUSED TESTING RANDALL W. RICE RICE CONSULTING SERVICES , Rice Consulting Services, Inc.

3 THE BIG IDEA As testers, our job is to find defects. However, we often look for only the symptoms of defects failures. All testing is sampling What if We had ways to sample for defects? Defects really do cluster? We could predict where best to look for defects? We looked for defects differently? 3

4 THE BAD NEWS As testers, we suffer from the needle in a haystack problem. Looking for something very small in something very large. However, a metal detector helps! 4

5 WE NEED PEOPLE TO TEST WHO DON T THINK LIKE TESTERS CEO TO TEST MANAGER

6 WHY? When professional testers miss obvious things, business stakeholders rightly look for other solutions. It s common for: Our methods to get old and ineffective Testers to grow complacent People to fail to learn Our perspectives to get too narrow People to have misplaced priorities 6

7 POLL QUESTION #1 Do you feel challenged by the problem of Too much testing, too little time and resources? 7

8 A GOLD MINING ANALOGY Gold mining is an expensive task, even on a small scale. Testing can also be expensive. There are often places that look promising for gold. But there is no way to tell from the surface if gold is really underground. The only way to know for sure is to drill holes and sample the dirt. In testing, we can do something similar as a way to test our intuition and suspicions about where defects may be. Test automation is a creative way to drill for defects. 8

9 9 NO HELPFUL SIGNS!

10 WHAT DO WE SEEK? Failures Externally Visible Through Functional (Black-box) Testing Defects 10 Internally Visible Through Structural (Whitebox) Testing and Static Analysis

11 THE PROBLEM External Too much to cover, too little time Unclear expected behavior Test automation mainly used for confirmatory testing Internal Lack of technical knowledge by testers Lack of motivation from developers Lots of code! Many interactions However, there are some very helpful static test tools 11

12 THE CONCEPT Sample 1 Sample 2 Above the surface we use a lot of guesswork. Sample 3 Sample 4 Sample 6 Sample 5 Failure Failure Visible Feature Sets Defect Below the surface is where the defects are. Defect 12

13 ADJUSTING THE SAMPLES (TESTS) Sample 2 Sample 7 Sample 8 When we find a defect, there may be others nearby. Sample 9 Sample 10 Sample 5 Visible Feature Sets Defect Defect Defect Defect 13

14 LOOKING BELOW THE SURFACE Sample 2 Sample 7 Sample 8 Sample 9 Sample 10 Sample 5 Failure Failure Failure Failure Failure Visible Feature Sets Defect Defect Defect Defect 14

15 PATTERNS TELL US SOMETHING Sights are probably OK, but we need to work on accuracy Other variables could have an impact: Wind, distance, etc. Sights are off, but we are consistent in our aim. Variation Consistency 15

16 THE KEY LOOK FOR PATTERNS Patterns help us find interesting defects. Commonality Concentration Contrast Types Trends Pearson's correlation 16

17 HOW CAN WE SAMPLE? Classes Risk Relationships Statistical Methods Randomly 18

18 CLASSES SAMPLE 1 Input Types Output Types Process Types Input Values Defect Types? States Output Values Transient Data Values 19

19 CLASSES SAMPLE 2 Input Types Output Types Process Types Input Values Defect Types? States Output Values Transient Data Values 20

20 Low Impact High RISK LEVELS SAMPLE 1 Very High High Moderate Low 21 Low Likelihood High

21 Low Impact High RISK LEVELS SAMPLE 2 Very High High Moderate Low 22 Low Likelihood High

22 RELATIONSHIPS Is it possible to withdraw money from a closed account with a positive balance? 23

23 STATISTICAL SAMPLING Population analysis Used to model test data based on production data. 20% of accounts < $1,000 60% of accounts between $1,000 and $1,500, inclusive 20% of accounts > $1,500 20% 60% 20% 0 $1,000 $1,000 - $1,500 > $1, Value of annual customer orders

24 TEST DATA MODELING 20% of test accounts have balances less than $1, % of test accounts have balances between $1,000 and $1,500, inclusive. 20% of test accounts have balances greater than $1, % of accounts < $1,000 60% of accounts between $1,000 and $1,500, inclusive 20% 60% 20% 20% of accounts > $1,500 0 $1,000 $1,000 - $1,500 > $1, Value of annual customer orders

25 RANDOM SAMPLING Not a substitute for other approaches However, it can be helpful at times to help get off the beaten path. Examples: The first n rows of data Every nth occurrence Random number generators 26

26 POLL QUESTION #2 Do you currently use test design techniques such as equivalence classes and/or statistical sampling? 27

27 HOW CAN WE LEVERAGE THIS? Test Automation Frameworks to vary inputs and actions: Randomly Based on defect patterns and clustering 28

28 EXAMPLE There are 10,000,000 possible values for a field variable to be tested. 1. Extract or build test data based on the values to test and the randomizing formula chosen. 2. Create automated tests using the data as input. 3. If a failure is seen, capture the test values and test a lower to higher range. 29

29 EXAMPLE (2) Failure 5,001, ,000,000 Failure Range 4,999,999 5,010,000 5,001,004 30

30 THE MISSING LINK FEEDBACK FROM MISSED DEFECTS 31

31 ENDING THE CYCLE OF DEFECTS Defects = Rework = Lost $$ The cycle continues until broken Same types of defects Same types of defects Create Find Fix 32

32 THE FEEDBACK LOOP New types of defects New types of defects Create Find Fix Identify root causes and fix the process 33

33 USING DEFECT FEEDBACK FOR TEST DESIGN New types of defects New types of defects Create Find Fix New tests Identify root causes and fix the process 34

34 POLL QUESTION #3 Do you perform root cause analysis to prevent defects based on the information from testing? 35

35 Failure Costs (Fix, Rework) Review & Test Costs Prevention Costs Cost of Quality When you consider the cost of a postrelease defect, the value gained in preventing defects is like finding gold! Cost of Production Cost to Build (Development, Delivery) 36

36 Failure Costs (Fix, Rework) Review & Test Costs Prevention Costs Found Value! Cost of Quality Profit!! When you consider the cost of a postrelease defect, the value gained in preventing defects is like finding gold! Cost of Production Cost to Build (Development, Delivery) 37

37 SUMMARY Failures are only symptoms of defects. Not every defect manifests a failure. Not all failures are defects. Before spending valuable time designing and performing non-productive tests, intelligent test sampling is a good way to focus your efforts. No drill, no dig! 38

38 BIO - RANDALL W. RICE Over 35 years experience in building and testing information systems in a variety of industries and technical environments ASTQB Certified Tester Foundation level, Advanced level (Full) Director, American Software Testing Qualification Board (ASTQB) Chairperson, QAI s annual software testing conference Co-author with William E. Perry, Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems Principal Consultant and Trainer, Rice Consulting Services, Inc. 39

39 CONTACT INFORMATION Randall W. Rice, CTAL Rice Consulting Services, Inc. P.O. Box Oklahoma City, OK Ph: Fax: Web site: