Software Reliability and Testing: Know When To Say When. SSTC June 2007 Dale Brenneman McCabe Software

Size: px

Start display at page:

Download "Software Reliability and Testing: Know When To Say When. SSTC June 2007 Dale Brenneman McCabe Software"

Georgina Lucas
6 years ago
Views:

1 Software Reliability and Testing: Know When To Say When SSTC June 2007 Dale Brenneman McCabe Software 1

2 SW Components with Higher Reliability Risk, in terms of: Change Status (new or modified in this build/release) Business Criticality (mission-, safety-, etc-critical) Defect History (old defect areas predict future defects) Complexity (complex is likely less thoroughly tested) Should Have Applied Higher Quality Test Completion Criteria, in terms of: Functional Test Thoroughness Defects/Rates Approaching Zero Code Coverage 2

3 Agenda I. SW Reliability Concepts and the Relationship to Complexity II. III. Determining Test Completion Criteria (When To Stop Testing) If High Reliability Is Desired Applying Tools to Generate Objective Information About Complexity And Test Completion Criteria 3

4 I. SW Reliability Concepts & the Relationship to Complexity A. SW Reliability Definitions B. Processes to Promote SW Reliability C. Complexity Metrics Impacting Reliability - Module D. Complexity Metrics Impacting Reliability - Class and Component Quotations without references indicated are from Software Reliability Principles and Practices, by Glenford J. Myers. 4

5 I. SW Reliability Concepts & the Relationship to Complexity Software Reliability Definitions Software Reliability is the probability that the software will execute for a particular period of time without a failure (an occurrence of a defect), weighted by the cost to the user of each failure encountered 1. Focus is on Failures, Not Defects [Reliability] is related to the manner in which the [software] is used 2. Focus is on Impacts To Users, Not To Developers Reliability must be stated as a function of the severity of errors, as well as their frequency 5

6 I. SW Reliability Concepts & the Relationship to Complexity Processes to Promote Software Reliability Reliability is dependent upon at least: - Quality Designed In (Fault Avoidance) The use of Development techniques (structured/oo/etc.) that both - Find defects early - Simplify structure (reduce dependencies between logic/data), which reduces probability of incorrect programming causing failure untested logic/data combinations causing failure - Quality Validated & Verified (Fault Detection) The use of Testing techniques that both - Validate (for user requirements) - Verify (for functional specs.) a significant percentage of critical operational logic/data combinations. The success of, and effort required for, these techniques is impacted by structure. 6

I. SW Reliability Concepts & the Relationship to Complexity Complexity Metrics Impacting Reliability - Module For these two modules, which is more likely to have Reliability problem?

7 I. SW Reliability Concepts & the Relationship to Complexity Complexity Metrics Impacting Reliability - Module For these two modules, which is more likely to have Reliability problem? (Due to likelihood that testing did not cover everything in the code) Cyclomatic Complexity, v(g) [i.e., McCabe metric] measure of logical Complexity (number decisions + 1) also measures number of linearly independent paths, so is minimum # tests needed to qualify a module indicates Size & Reliability 7

8 I. SW Reliability Concepts & the Relationship to Complexity Complexity Metrics Impacting Reliability - Module Essential Complexity, ev(g) measure of logical paths containing unstructured constructs, i.e., Unstructuredness indicates Maintainability Module Design Complexity, iv(g) [integration] measure of logical paths containing calls/invocation of immediate subordinate modules, i.e., Integration Paths Containing Global Data, gdv(g) global data also impacts reliability, because it creates relationships between modules Branches, which is related to v(g) Maximum Nesting Level Others v(g) = 10 ev(g) = 3 iv(g) = 3 8

9 I. SW Reliability Concepts & the Relationship to Complexity Complexity Metrics Impacting Reliability - Module Cyclomatic Complexity & Reliability Risk 1 10 Simple procedure, little risk More Complex, moderate risk Complex, high risk >50 Untestable, VERY HIGH RISK Cyclomatic Complexity & Bad Fix Probability % % > 50 40% Approaching % Essential Complexity (Unstructuredness) & Maintainability (future Reliability) Risk 1 4 Structured, little risk > 4 Unstructured, High Risk 9

10 I. SW Reliability Concepts & the Relationship to Complexity Complexity Metrics Impacting Reliability - Classes & Components OO Metrics for Classes provide insight into object oriented programming practices, and to related testing effort; based on Chidamber and Kemerer s work Polymorphism Encapsulation Inheritance Design Complexity for Components, S 0 = iv(g) measure of decision structure which controls invocation of modules within the design; quantifies testing effort for paths relative to design calls Integration Complexity for Components, S 1 = S 0 - (# modules) + 1 measure of the integration tests that qualify the design tree; the # of paths needed to test all calls to all modules at least once, a quantification of a basis set of integration tests; measures minimum integration testing effort of a design; each subtree test validates integration of several modules Sum/Maximum/Average of v(g), ev(g), gdv(g), branches, etc. 10

11 II. Determining Test Completion Criteria for High Reliability A. When Do We Stop Testing? B. Functional Test Thoroughness C. Defects/Rates Approaching Zero D. Code Coverage Impossibility of Testing All Combinations Levels of Code Coverage - Module Levels of Code Coverage - Component 11

12 II. Determining Test Completion Criteria for High Reliability When do we Stop Testing? When Time or Money Have Been Spent? - Most organizations plan better than this Test organizations normally think (at least subconsciously): 1. When We Have Executed Certain Functional Tests - Includes performance, security, etc. - How many tests to re-run after bug fixes? 2. When Critical Defects & Defect Found Rates Are Reducing - Considering impact/severity 3. When We Have Executed Tests that meet a Certain Level of Code (Logic) Coverage 12

13 II. Determining Test Completion Criteria for High Reliability Functional Test Thoroughness For high reliability, we may consider: execute thorough reqts.-based tests (SW reqts. & user manuals) execute tests based on design information for re-builds w/ fixes, re-execute all, or some significant portion, of above tests Bad Idea Example: Wait til end to run performance tests 100% Number of Functional Tests 0% Build 1 Build 2 Time Initial Tests Rerun Tests 13

14 II. Determining Test Completion Criteria for High Reliability Defects/Rates Approaching Zero Critical Defects Outstanding & Rate of Defects Found Time For high reliability, we may consider: obtain re-builds w/ fixes, and re-execute functional tests, until: there are no outstanding critical defects the rate of new defects found (build-to-build) approaches zero Bad Idea Example: Ignore when many non-critical defects are found in late builds 14

15 II. Determining Test Completion Criteria for High Reliability Code Coverage Tests based on functional requirements are essential - it is your primary mechanism, and might be your only mechanism when you are time constrained and high quality is not critical But... Requirements do not execute, Code does! Tests need to focus on both: is the code doing what it should do? (positive) is the code not doing what it should not do? (negative) Requirements-based tests alone can result in: Use of language-required syntax that can lead to failures Lack of testing of logic/data created during implementation, which can lead to failures; in areas such as implicit (undocumented) functionality (design decisions, exception handling), and functional combinations not identified in requirements 15

16 II. Determining Test Completion Criteria for High Reliability Code Coverage For high reliability, we may consider: from execution of reqts.- based tests, determine whether code coverage indicates: functionality for which reqts. or reqts.-based tests are missing logic for which additional test cases should be created Bad Idea Example: Assume that developers tested all logic Defined % Goal Code Coverage 0% Build 1 Build 2 Time Initial Tests Rerun Tests 16

17 II. Determining Test Completion Criteria for High Reliability Impossibility of Testing All Combinations 18 times It is IMPOSSIBLE to test All Statistical Paths, and this isn t even considering data combinations So, from structural coverage view, When Should We Stop Testing? Not care? (planes do, missiles do) Module entry? All lines? All branches? All boolean outcomes (MC/DC)? A set of cyclomatic (basis) paths? All Statistical Paths = If allow 1 Nanosecond per test, time is : 0 Optimum 10 Number of Tests? 18 T = = 31.7 Years 10 x 3600 x 24 x 365

18 II. Determining Test Completion Criteria for High Reliability Levels of Code Coverage - Module (Units of Code) Module Entry - test entering a module at least once Line - test anything on each line with code; can miss some branch outcomes, dependent on coding style Branch - test each branch outcome; some tools provide statement level testing, which when implemented as testing any part of each statement can miss some branch outcomes Condition or Boolean (MC/DC) - test each condition of a multiple condition decision; required for highest risk airborne sw Cyclomatic Paths (Structured Testing) - test a set of linearly independently paths from entry to exit; based on Tom McCabe s Structured Testing; also called Basis Path Testing, based on graph theory; description in an NIST document shows why this provides improved reliability 18

19 II. Determining Test Completion Criteria for High Reliability Levels of Code Coverage - Module What module coverage level should you attempt? Minimum # tests would be: Module Entry Coverage = 1, Line Cov. = 2, Branch Cov. = 3, Cyclomatic Path Cov. = 4, All Paths Cov. = 6 19

20 II. Determining Test Completion Criteria for High Reliability Levels of Code Coverage - Component Component Entry - test entering each component (subsystem, executable, etc.) at least once Component Integration Calls - test entering each component from each calling component Component Integration Paths - test entering each component from each call occurrence in each calling component Module Entry - test entering each module at least once Module Integration Calls - test entering each module from each calling module (fan-in testing) Module Integration Paths test entering each module from each call occurrence in each calling module; a variation is to include a test of at least one path in calling module that does not call this module; this is Structured Testing recommendation Sums of Other Module Coverage Data - see types for Modules 20

21 II. Determining Test Completion Criteria for High Reliability Levels of Code Coverage - Component Black lines mean not yet tested What coverage level will you attempt during Integration Test? Module entry? - the 5 red boxes (untested) need tested Module integration calls? - the 9 black lines need tested (they include the red boxes) Module integration paths? - the 9 black lines plus all paths in the calling modules that contain calls; Structured Testing recommends this level as minimum Other? 21

22 III. Applying Tools for Complexity & Test Completion Info. A. Structural and Semantic Code Analysis B. Structural Complexity Metrics, Info., & Visualization C. Code Coverage Metrics, Info., & Visualization D. Procedures for Applying Code Coverage Tools 22

23 III. Applying Tools for Complexity & Test Completion Info. Structural and Semantic Code Analysis Two Major Analysis Types - Semantic & Structural Semantic Analyses code content May have sophisticated pattern matching algorithms Often include potential run-time error checking (memory, race, boundary); naming conventions; data mining May include run-time capture of potential problem info. Structural Analyses code structure May have sophisticated logic path determination algorithms Often include checking for poor structure; for poor design May include run-time capture of code coverage info. The two types are complementary; BOTH have value 23

24 III. Applying Tools for Complexity & Test Completion Info. Structural Code Analysis - Visualization And Metrics Structure and code coverage can be expressed in objective terms At Organization/Enterprise Level Multi-level groupings of metrics on structure (size, complexity, maintainability, etc.) and on code coverage At Component/Design Level Structure & class charts, w/ user interaction Summary metrics analysis Overlay of static & code coverage info. At Module (Unit) Level Flowgraphs w/ code, calls, test paths, etc. Detailed metrics analysis Overlay of static and code coverage info.

(unstructuredness) Design complexity Can include any metrics of your

25 III. Applying Tools for Complexity & Test Completion Info. Structural Metrics - Module Cyclomatic complexity Essential complexity (unstructuredness) Design complexity Can include any metrics of your choice OO Metrics for Classes - Polymorphism - Encapsulation - Inheritance - More Available 25

26 III. Applying Tools for Complexity & Test Completion Info. Coloring Based on Coverage - can be based on any metric - interactive drilldown to code details 26

27 III. Applying Tools for Complexity & Test Completion Info. Some of many applications of code coverage data: Coverage Metrics - for Modules - cyclomatic paths (vg, ac) - integration paths (ivg, test ivg) - branches (# br., # test) - can report on other static & coverage metrics - have the most complex modules (reliability risk) been tested? - have the changed modules (reliability risk) been tested? - have other modules known as high reliability risk been tested? 27

28 III. Applying Tools for Complexity & Test Completion Info. Module - Untested Graph & Source Listing - Show untested logic, so can analyze & improve test cases 28

29 III. Applying Tools for Complexity & Test Completion Info. Tested Branches Untested Test Paths Other data from tools may help identify what remains to be tested 29

30 III. Applying Tools for Complexity & Test Completion Info. Procedures for Applying Code Coverage Tools A. Defining Process Improvement For Your Organization - Requiring Measurement and Analysis B. Obtaining/Providing Management Support C. Determining Owners of Processes & Automated Tools D. Considering Test Process Enhancement Alternatives - tool support - available vs. service team - tool usage - ad hoc vs. baselining vs. coverage level reqts. E. Piloting Process Changes & Automated Tool Usage F. Socializing - Presenting Results and Obtaining Feedback G. Implementing Process Changes & Automated Tool Usage Into Your Organization 30

31 Conclusion SW Components with Higher Reliability Risk, in terms of: Change Status (new or modified in this build/release) Business Criticality (mission-, safety-, etc-critical) Defect History (old defect areas predict future defects) Complexity (complex is likely less thoroughly tested) Should Have Applied Higher Quality Test Completion Criteria, in terms of: Functional Test Thoroughness Defects/Rates Approaching Zero Code Coverage 31

32 Thank You! Acronyms - SW - Software - OO - Object Oriented - vg - cyclomatic complexity - ac - actual complexity; coverage of vg paths - evg - essential complexity - ivg - integration (module design) complexity - gdv - global data complexity - S0 - design complexity for component - S1 - integration complexity for component 32

Uncovering Risk in Your ICD-10 Conversion. Key Risk & Effort Metrics for ICD Data Testing

Uncovering Risk in Your ICD-10 Conversion. Key Risk & Effort Metrics for ICD Data Testing Key Risk & Effort Metrics for ICD Data Testing Abstract If you are implementing ICD-10 support in your software applications, many things are important to your management of the process. Good development