A Consumer s Guide to Software Testing. Greg McNelly Progressive Insurance June 2010

Transcription

1 A Consumer s Guide to Software Testing Greg McNelly Progressive Insurance June 2010

2 Introduction About me: Programmer turned tester Last 12+ years in a large insurance IT shop About this talk: Testing is interesting, challenging and valuable, but Innovation is a struggle because of outdated, wrong and entrenched thinking. You can, and should, expect better testing

3 The Cost of Testing How much of your project budgets are typically consumed by testing? My informal observation= 35 to 50% Why is it so expensive? There is no other major field of human endeavor where such levels of waste and delay are tolerated -- and the results are so poor. Ross Collard on software development

4 The Testing Challenge The Wason Exercise Adapted from Rapid Software Testing, a course by Michael Bolton and James Bach

5 The Testing Challenge Did you: Think? Ask questions? Challenge constraints? Make assumptions? Understand the context? Provide information? Assess risks?

6 The Testing Challenge What did I, the customer, do wrong? Waited until the last minute Staffed my test team with warm bodies Provided little support Shared little information Handcuffs and blindfolds Obviously, I need help

7

8 Buyer s Advice Things to consider when buying testing for your project or system: Timing Skills and Tools Risk Planning and Tracking Decision Making

9 Buyer s Advice: Timing Defects tend to be seeded early. Fixes tend to become exponentially more expensive as defects age. Testers need time to acclimate to context. Testing should begin as soon as important questions can be investigated.

10 Buyer s Advice: Skills and Tools Tester skills should match the domain and technical context of the project to minimize strain on other team members. Testers may need a dedicated test environment to maximize confidence in their observations. Testers may need special access to their test environments, and the development environment. Test tools range greatly in scale and purpose don t buy the one-size-fits-all snake oil.

11 Buyer s Advice: Risk Test everything is impossible. A test team should seek to identify and prioritize risks posed by a malfunctioning system. A test team should actively engage the stakeholders in the investigation of risks.

12 Buyer s Advice: Planning and Tracking Good enough process and structure Work breakdown with estimates/budgets are necessary for planning. Establish communication channels and frequency. Leave room for investigation. Prediction of testing, not prediction of results You may need to plan for secondary passes of testing

13 Buyer s Advice: Decision Making Asking testers for sign-off can be dangerous: Can really only sign-off that planned testing has been executed. Focus (and pressure) becomes achieving sign-off, rather than finding important problems. This is false authority. The PM and customer will override if necessary. When will testing be complete? is a rat-hole conversation.

14 Summary Engage your test lead/manager early in your project. Match tester skills, tools and environments to the testing mission. You can t test everything, so expect your testers to identify and prioritize risks and help them. Work processes and structures need to be good enough for planning and tracking without handcuffing the testers. Testers seek and provide information, you make the deployment/ship decision.

15 References / Suggestions Rapid Software Testing course by Michael Bolton and James Bach The Quality Payoff Ross Collard, Collard & Company, June, 2007 in CEO magazine The State of the Practice 2010 Ross Collard (ross@rosscollard.com) Perfect Software: and Other Illusions about Testing book by Gerald M. Weinberg

16 Contact Me