Applicability of Model-Based Design Quality Metrics to Medical Device Software Dave Hoadley PhD, MathWorks Paul Jones, Office of Science and Engineering Laboratory, CDRH, FDA May 2017 2017 The MathWorks, Inc. 1
Introduction A growing regulatory challenge is how to efficiently assess the safety and effectiveness of a complex device with significant software components Sponsors provide disparate collections of arguments and evidence for each device submission between companies and even within each company Is there another more concrete approach? 2
State of the practice for the Quality System Process Software development Software Requirements Architecture Design Implementation Test & Validation?= QA A people-intensive process, with the largest investment in test and validation 3
Issues Requirements Hard to analyze thousands of natural language requirements Derived (surprise?) requirements from implementation, testing Testing White-box software test coverage is not assessed Quality metrics are elusive Documentation & Process Documentation is non-uniform and verbose Process has opportunity for undetected errors 4
How could software and system modeling help? Analyze requirements Consistency, completeness Ambiguous natural language executable specification Provide insight into design quality Document derived requirements natural language implementation Details emerge early, not during test Assess Test Coverage Instrument model to measure coverage Objectives can be defined for a desired quality level 5
Ex: Requirement model Power button must be pressed for 0.5 to 3 seconds to power on What happens after 3 seconds? How often do I sample the button? Does device go on after we let go or before? 6
Some concrete examples Subjective Statement Objective Goal Software design document is unambiguous, complete, and consistent Natural language requirements and model are mutually traceable Model checker shows lack of inconsistency Software implementation is complete, consistent, and correct Implementation source code is traceable to model Implementation source code is acceptably free of static and runtime defects Device behaves as intended for the context of use Use case scenarios achieved model coverage goals Use case scenarios achieved code coverage goals in intended environment 7
Concrete examples continued Subjective statement Objective goal Verification of risk control measures confirm expected behavior Simulated all inputs to the model for full range of values/conditions causing hazards. No unsafe behavior detected. Model checker demonstrates model safety property assumptions are valid Validation of risk control measures confirm expected behavior Simulated all inputs to the device in use environment for full range of values/conditions causing hazards. No unsafe behavior detected 8
Model-Based Design Process Software development Software Requirements Architecture Design Implementation Test & Validation?= QA More investment in Requirements Design 9
Quality objectives Many standards define a set of objectives and activities per quality level, as fcn(risk) DO-178C Design Assurance Levels E-A ISO 26262 Automotive Safety Integrity Level A-D IEC 61508 Safety Integrity Level 1-4 Example: DAL A (potential for catastrophic failure) requires 100% MC/DC coverage of implemented software, along with dozens of other objectives Should FDA consider such concepts for objective software quality assessment? 10
Examples of Model-Based Design objectives and metrics Evidence (automated artifacts) Model to requirements traceability Simulation test results report Model coverage report Model standards check report Software design documentation Code to model traceability Code to model test results report Code coverage report Code standards report Absence of design errors Property proofs Absence of runtime errors 11
Conclusion Software models are a process improvement over only natural language requirements most report ~40% effort reduction >10^6 USD / device Objective evidence of software quality can be created by tools with a defined modeling language Objective quality levels have been adopted by the air, rail, and road transportation industries MathWorks will be presenting a case study of the Model-Based Design approach on 5/18 in at 8:00-8:45 12
From FDA Impact on US Medical Technology Innovation, A Survey of Over 200 Medical Technology Companies, Nov 2010, Josh Makower, MD et al 13