Deploying Code Coverage Analysis to Improve Software Quality

Transcription

1 Deploying Code Coverage Analysis to Improve Software Quality Vector Software, Inc.

2 > Vector Software, Inc. About Vector Software

3 Our Company > Vector Software develops embedded software testing products > U.S. Based company founded in 1990 by embedded developers > First product released in 1994 for Lockheed Martin s C-130J Super Hercules > Corporate headquarters in Rhode Island > Sales and support office worldwide > Providence > Boston > Jacksonville > Denver > Phoenix > London > Düsseldorf > Shanghai s olid r efer ence account s in mult iple indus t r ies Aerospace Automotive Medical Railway Industrial DO-178B ED-12B ISO FDA IEC CENELEC EN IEC 61508

4 Our Clients AEL Sistemas Astrium Boeing Bell Helicopter Belcan BAE Systems Cessna Aircraft EADS Electric Boat Goodrich Aerospace General Dynamics Hamilton Sundstrand Honeywell ITT Communications India Space Research Korean Aerospace Lockheed Martin L-3 Communications Northrop Grumman Pratt & Whitney Raytheon Rockwell Collins Rolls Royce Saab Group Thales Turkish Aerospace U.S. Air Force XMobots Robotic Autoliv Autoeversystems AIV Beko Bosch Chang'an Automotive Delphi Dongfeng Automotive ENSO Fujitsu Hyundai Magna Powertrain Magneti Marelli MOTONIC Siemens TRW Valeo Yura Abbott Laboratories Acist Medical Baxter Healthcare Beckman Coulter Boston Scientific Biosafe Cardinal Health Covidien Enteromedics GE Healthcare Hoana Medical Hospira Medical Impact Instrumentation Medtronic Philips Medical Sirona Dental Smiths Medical Terumo Heart Varian Companies who have safety or business critical applications have processes which specify rigorous testing have unit and integration testing requirements need the ability to automate regression testing have requirements to prove code coverage realize that NOT testing is NOT an option ABB Allied Telesis Alcatel Ansaldo Signal Alstom Brunel Bombardier CAP Gemini Elin EBG Electro-Motive Diesels GE Transportation Invensys Rail ŠKODA ELECTRIC Safetran Systems Siemens Triconex Thales Union Switch & Signal Verocel Amazone Analog Devices Arçelik Bechtel Bettis Berthold Technologies British Energy Brunel Caterpillar Changzhou Scale Ltd. Claas Endress+Hauser Flextronics Foster Miller General Electric HAWE Hydraulik ICS Tripex Invensys Ingersoll Rand Itron Krauss-Maffei Mettler Toledo AG NetApp Oak Systems Sanmina Sunways ViaSat Wilo ZIEHL Aerospace Automotive Medical Railway Industrial DO-178B ED-12B ISO FDA IEC CENELEC EN IEC 61508

5 > Vector Software, Inc. Understanding Code Coverage Analysis

6 What is Code Coverage Analysis? > Data gathered during execution of an application > Highlights what lines of code have been executed > More important is which lines were NOT executed > Indicates poorly tested areas of the application > The more unexecuted code, the more risk of bugs > Can be deployed during all phases of testing > Unit testing > Module integration testing > Software integration testing > Functional testing > System testing > Especially important for legacy or fielded code which just works

7 Levels of Code Coverage Statement Decision or Branch (If Statements, loops, case statements)

8 Levels of Code Coverage (Cont.) Modified Condition/Decision Coverage

9 Levels of Code Coverage (Cont.) Basis Path Coverage (unique paths through each function)

10 Why is Code Coverage Analysis Important? > Required by many of the industry standards as a way to prove that the testing is sufficient > Helps projects answer critical questions: > Am I done testing? > Have I identified all the requirements for this project? (if all of the requirements are tested and you still have uncovered code you may need more requirements) > Is there any dead code in my application? (code that cannot be reached) > Helps organizations focus their testing resources on areas that will yield the greatest benefit > Provides critical information on how well legacy code is being tested

11 > Vector Software, Inc. Industry-Specific Software Testing Guidelines

12 Safety Critical Industries Avionics > DO-178B DO-178C DO-278 Aerospace Defense > Military Standards Automotive > MISRA ISO Medical > IEC FDA Guidance Industrial Controls > IEC Railway > CENELEC

13 DO-178B Testing Requirements > Structural Testing > Testing requirements and demonstrating associated code coverage > Satisfied with a combination of system, integration, and unit testing > Level C (Statement) > Prove that every executable statement has been invoked at least once > Level B (Decision) > Above, plus, prove that every point of entry and exit, and each decision point outcome, have been invoked at least once > Level A (MC/DC) > Above, plus prove that each component of a complex conditional independently drives the outcome of the conditional

14 IEC Part 3: Software Requirements > A. Probabilistic Testing > Probability of error reduced by testing with several independently chosen values for tests > B. Dynamic Analysis and Testing > Section covers a variety of tests, many of which can be achieved through a mix of testing on the entire software build (system testing) and unit/module testing Technique / Measurement SIL Test case execution from boundary value analysis R HR HR HR 2. Test case execution from error guessing R R R R 3. Test case execution from error seeding - R R R 4. Test case execution from model-based test case generation R R HR HR 5. Performance modeling R R R HR 6. Equivalence classes and input partition testing R R R HR 7a. Structural coverage (entry points) 100% HR HR HR HR 7b. Structural coverage (statements) 100% R HR HR HR 7c. Structural coverage (branches) 100% R R HR HR 7d. Structural coverage (MC/DC) 100% R R R HR

15 FDA Developer Testing Guidelines A software products testing can be organized into unit, integration, and system testing Testing starts with unit test and concludes with system testing Testing entails running software products under known conditions with defined inputs and documented outcomes Essential element of test is the expected result The amount of structural coverage is commensurate with the level of risk posed Regression analysis and testing are employed to provide assurance that a change has not created a problem

16 IEC Safety Classification Effects Software Documentation Class A Class B Class C Software development plan Software requirements specification Software architecture Must contain contents to sections 5.1 IEC 62304:2006. The plan's content list increases as the class increases, but a plan is required for all classes. Software requirements specification conforming to 5.2 IEC 62304:2006. The content list for the software requirements specification increases as the class increases, but a document is required for all classes. Not required. Software architecture to 5.3 IEC 62304:2006. Refined to software unit level for Class C. Software detailed design Software unit implementation Software unit verification Software integration and integration testing Software system testing Not required. All units are implemented, documented and source controlled (5.5.1). Not required. Define process, tests and acceptance criteria (5.5.2, 5.5.3). Carry out verification (5.5.5) Not required. Integration testing to 5.6 IEC 62304:2006. Not required. System testing to 5.7 IEC 62304:2006. Document detailed design for software units. (5.4). Define additional tests and acceptance criteria (5.5.2, 5.5.3, 5.5.4). Carry out verification (5.5.5). Software release Document the version of the software product that is being released (5.8.4). List of remaining software anomalies, annotated with an explanation of the impact on safety or effectiveness, including operator usage and human factors. Summary of safety classification effects on the code development documentation and process

17 ISO Automotive Software Standards > Adaptation of IEC generic standard > Deals with electronic software elements related to safety > Detailed references to required testing > Risk-management approach > Automotive Safety Integrity Levels (ASIL) > Similar to IEC Safety Integrity Levels (SIL) > Four levels (A-D) to specify necessary safety measures for avoiding risk Methods ASIL A B C D ISO Table 14 Structural Coverage Metrics at the Software Unit Testing Supported by VectorCAST 1a. Statement Coverage HR HR HR HR Yes 1b. Branch Coverage R HR HR HR Yes 1c. MC/DC (Modified Condition / Decision Coverage) R R R HR Yes

18 > Vector Software, Inc. Deploying Code Coverage Analysis on Legacy Applications

19 Testing Challenges of Legacy Code > How do I know where the code might break? > Implicit trust in the code since it has been fielded, but still a risk > Changing code in or around it could cause latent bugs to emerge > It is too expensive to go back and fully unit test > Requires at least 1 Test Line of Code (LOC) per Deliverable LOC > Can Consume 50% of Development Budget > Would eliminate any advantages of code reuse

20 Testing Legacy Code > System / Functional Testing > Execution of entire application to ensure conformity to overall system requirements > System testing cannot guarantee full code coverage > Generally performed by a dedicated test group > 60-70% coverage is common > Regression Testing > Re-runs previously developed tests against changed software > Identifies whether changed software behaves correctly > Compares expected against actual test results > Measures code coverage against previous coverage levels

21 System Testing Facts > Performed by QA Group or a dedicated system test team > Goal is to prove functional requirements > Challenges with system testing > Completeness of testing is difficult to judge > Completeness of requirements is not known > Extra functionality might be deployed > Dead Code might be deployed > Code coverage analysis addresses these challenges!

22 Deploying Code Coverage Analysis During System Testing > Easy to deploy > Able to instrument one file or many files > Takes advantage of the testing you are already doing! > Gives you insight for good decisions based on current testing activities > Depending on the tool, typically easy-to-use and easy-to-deploy > Minimal impact on budget and schedule > Can be used for both legacy code and new development > Still have risk! > Can t achieve 100% code coverage during system testing > Error handling > Defensive programming > Hard to reach code > Need to utilize other testing approaches to further reduce risk

23 > Vector Software, Inc. Integrating Code Coverage Analysis with Other Types of Software Testing

24 Other Levels of Testing > Static Analysis > Compliance with coding standards like MISRA > MISRA focuses on language restrictions to mitigate reliability faults > Finding errors in code statically, by analyzing use of memory for instance > Dynamic Analysis > Unit Testing > Testing smallest piece of testable software > Unit is C/C++ source file or Ada package > Requires generation of test stubs and drivers > Driver simulates calling unit and stub(s) simulates called unit > Enables generation of tests to ensure complete code coverage > Integration Testing > Logical extension of unit testing > Allows testing of all units that make up a functional process > Requires multiple units under test at the same time > Identifies problems when units are combined

25 Unit and Integration Testing Facts > Performed by Developers not QA Groups > Goal is to prove low-level requirements (correctness) > Secondary goal is to prove completeness of testing > Challenges of Unit Testing > Unit testing is expensive (extremely labor intensive if done manually) > Unit testing is tedious for developers > Unit testing is not easily repeatable

26 Unit and Integration Testing Units (.c,.cpp) Test Driver Source Code Unit(s) under test Stubbed dependents Real dependents Unit Test Harness

27 Achieving 100% Structural Coverage System Testing (always) Unit Test (sometimes) Coverage 60% - 70% Achieved Driver Units Stubs Coverage 30% - 40% Achieved Structural Coverage 100% Achieved

28 Testing Methodologies Method 1: New Development Unit Test Integration Test Functional Test Coverage Driver Units Stubs Low-Level Requirements Driver Units Stubs Full Application High-Level Requirements Functional Requirements Coverage 100% Method 2: Legacy Application Functional Test Unit Test Coverage Coverage 60% - 70% Achieved Driver Units Stubs Coverage 30% - 40% Achieved Structural Coverage 100% Achieved

29 How To Determine When to Unit Test? > Newly developed code > Code that has a high complexity > Use static metrics like McCabe Cyclomatic Complexity > Code that historically has been fragile > Goal: Be smart about deploying unit test resources! > Instead of blindly just unit testing everything or nothing! > Result: Building up a suite of unit and integration tests over time > Supports the other key organizational goal of continuous integration and test

30 Why Regression Testing (Repeatability) is Important > Testing multiple source code configurations > Application deployed on different physical targets > PowerPC, ARM, x86... > Application built with multiple tool chains > Green Hills, GNU, VisualStudio > Application built with multiple compile options > Macro values, optimization flags > Testing multiple source code versions > Production release code > Beta release code > R&D code > Testing across multiple machines or architectures > Test suite may take weeks to run on one machine > Application may be deployed on multiple operating systems (Windows, Linux) > Testing what has changed > Interim testing of only those source files that have changed > Based on file checksums, or change dates

31 > Vector Software, Inc. How Automation Improves Software Quality and Compliance to Standards

32 Characteristics of an Ideal Automated Software Test Tool > Allows stand-alone code coverage to support testing legacy code > Fully automates harness generation to support low-level unit testing > Supports and fully automates integration with a wide array of embedded targets > Integrates with your existing requirements-tracking tool to support requirements traceability > Provides both static and dynamic analysis > Integrates with best of breed solutions across development lifecycle

33 Software Development V-Model Requirements Specification System Testing Architecture Integration Testing System Design Integration Module Testing Module Design Unit Testing Coding

34 Regression Testing leads to Better Quality! 79% of all software errors in medical devices arise from the addition of new functionality and regression testing issues. -- FDA An ideal tool should Provide a full command line interface to support test repeatability and integration with existing testing infrastructure Encourages continuous integration and test repeatability

35 > Vector Software, Inc. Gathering Code Coverage Metrics from an Embedded Target

36 Code Coverage Analysis on Embedded Targets > Target needs enough resources to support overhead > Executable size growth due to source code instrumentation > Memory for data capture > Target needs communication mechanism back to host > Coverage data needs to be captured from memory for analysis on host > Streaming in real-time or buffering and dumping memory are common options > Different tools have optimized to varying degrees > Product evaluations are critical to ensure a specific tool will work

37 > Vector Software, Inc. VectorCAST Overview

38 VectorCAST/Cover > Stand-alone Code Coverage Analysis > System / Functional Test Verification > Running your existing tests to see what code is being executed > System tests can be executed different ways: > from scripts, manually through GUI, simulator, > Generally performed by a dedicated test group

39 VectorCAST/C++ and VectorCAST/Ada > Unit Testing > Testing smallest piece of testable software > Unit is C/C++ source file or Ada package > Requires generation of test stubs and drivers > Driver simulates calling unit and stub(s) simulates called unit > Enables generation of tests to ensure complete code coverage > Integration Testing > Logical extension of unit testing > Allows testing of all units that make up a functional process > Requires multiple units under test at the same time > Identifies problems when units are combined

40 VectorCAST/Lint > Static Source Code Analysis > Uses Industry Standard Lint Analysis Engine for C/C++ > Enforce coding standard compliance > Out-of-the-box MISRA support for C and C++ > Automate code inspection process > Custom compliance rules > Highlight language insecurities and dangerous usage > Safety Critical Profile > Deliver defect prevention > Highlight code complexity metrics > Visualize code structure to aid code comprehension

41 BAE Systems BAE Systems Developer of the HAWK Advanced Jet Trainer BAE Systems use VectorCAST for their continuous build and integration process. The off-the-shelf Tool Qualification Documents provided by Vector Software, were used to quickly and easily allow BAE Systems to certify their system to DO-178B.

42 Boston Scientific Corporation Boston Scientific Corporation Developer of the COGNIS Cardiac Resynchronization Therapy Defibrillator Use VectorCAST/C++ with MC/DC for the most safety critical aspects of their application.

43 Bombardier Transportation Bombardier Transportation Developer of high-speed rail control solutions Bombardier development teams uses VectorCAST for testing European Rail Traffic Management System/European Train Control System (ERTMS/ETCS), Automatic Train Protection (ATP) and Specific Transmission Modules (STM).

44 Hyundai Motor Company Hyundai Motor Company Automotive Manufacturer Used VectorCAST/C++ and VectorCAST/RSP to perform low-level requirements testing, and structural coverage analysis.

45 Endress + Hauser Endress + Hauser Industrial measurement and automation VectorCAST automated embedded software testing and code coverage tools allows E+H to identify software quality issues early in the development cycle and to provide the necessary IEC SIL 3 certification evidences.

46 How the VectorCAST Solution is Different Full Embedded Software Testing True Embedded Test Automation > Easiest to Use test tool on the market > Easy continuous integration and test for repeatability > Integration with best of breed static analysis tools > Unit and Integration testing > Code coverage analysis through lifecycle > Automated regression testing > Testing on embedded targets and simulators > Qualification materials for FAA, FDA, and others World-Class Service and Support > Our engineers are our support staff > Local support via phone and > Consulting and Technical Training SEMI-AUTOMATED TOOLS Benefits of using VectorCAST > Reduced Time-To-Market > Easier Certification / Qualification > Verifiable savings of time and money NO AUTOMATION FREE AND OTHER MANUAL TOOLS

47 Connect With Us Proven Solutions for Reliable Software Americas 1351 South County Trail Suite 310 East Greenwich, RI United States of America T: F: E: EMEA 33 Glasshouse Street Suite 3.08 London W1B 5DG United Kingdom T: F: E: Germany St. Töniser Str 2a Kempen Germany T: F: E: Asia Pacific Rm 403, Building 6 No.88 Daerwen Rd Zhangjiang Hi-tech Park Pudong New Area Shanghai China T: F: E: info@apac.vectorcast.com vectorcast.c o m