Testing and the Bottom Line

Transcription

1 Testing and the Bottom Line A New Method to Estimate the Value of Good Software Testing Juha-Matti Tirilä Codenomicon Ltd Codenomicon Ltd. 1

2 Outline What are the problems in estimating the cost of testing What happens when code is tested well: the effect on bugfix price of early discovery Our method for estimating the cost impacts of software testing

3 This is why a functional cost model matters

4 Why develop a new cost model Problem: the cost structure of testing is typically misunderstood The difference in nature between testing and development Cost of testing cannot be analyzed with the same tools as the cost of development

5 Where do bugs come from

6 Cost of development Direct cost: we have to pay this to actually produce the desired outcome

7 Cost of testing Indirect cost: if we do not test properly, how much is it going to cost us Direct cost: the price of performing the testing Plus the cost of fixing the issues? It depends.

8 Why does "it depend"? You may have found the bug otherwise just as well, only now you find it earlier, so there s no extra cost Even though it costs you money, testing will have positive consequences beyond the particular bug (That are very difficult to measure)

9 Testing acts as an enabler

10 Cost-benefits of early bug discovery Especially in a security context: vulnerabilities detected by third parties tend to be expensive due to Negative reputation Downtime Increased need for customer support Etc...

11 Cost-benefits of early defect discovery Thinking of security critical bugs: Post-release vs. pre-release Impact on accountability Flexibility in resource allocation; no need to fix straight away

12 Cost-benefits of early defect discovery The person who wrote the code in the first place still working Maybe even still remembers the logic of the code in question Organization-level practices in maintaining good quality, improved performance in the long run

13 Cost-benefits of early defect discovery Especially in security testing, it is a rather limited set of bugs that appear again and again so that testing: Improves your knowhow Accelerates production speed With the best tools: very to-the-point reports on what types of bugs were discovered easy to fix bugs educate developers to avoid similar mistakes in the future

14 Our method, graphically, phase 1

15 The model explained Product release at T = 6 months The price of bugs increases until T, then drops a little, starts to rapidly rise immediately afterwards Some bugs cost nothing The effect of testing: all bugs discovered earlier so that the distribution tends to the left, and down

16 Comparison to traditional incident probability calculations One approach: try to determine the probability and cost of impact, and evaluate your testing budget against this expected loss Our approach: more statistically oriented, more geared towards estimating the average savings irrespective of whether issues ever surface or not

17 But you never have the data before it s too late, right? Remark: the dual presentation done above is something you in reality never get. You have a certain level of testing, and you don't know what the other alternatives would have cost you. How to deal with this problem? Solution: use statistical distributions and models, and tune the parameters of the model to reflect your development environment.

18 A peek at the parameters Agree upon a trend for how the cost of a bug fix will develop towards release, and after release Agree upon the way testing is going to affect discovery times of issues Agree upon "variance", i.e. how widely the cost per delivery time is distributed

19 The example revisited, with statistical distributions

20 The example, continued

21 Benefits of using distributions Very little immediate effect on the estimated price of fixing bugs BUT Provides lots of additional information The mechanism is easy to understand and tune to match a particular development environment In the long run, better estimates to rely on

22 Benefits of using distributions You can readily calculate various statistics, such as: The expected cost of fixing an average bug, with and without testing The expected cost of fixing bugs detected in pre/post -development phases Etc..., to your liking, any statistics computable from a statistical distribution So for example also estimate risks

23 Challenges Of course, the quality of all the estimates depends on how well you tune the parameters But: You can use real-life data to estimate the parameters

24 Thank you!