Integrated Environment for Development and Assurance

Transcription

1 Integrated Environment for Development and Assurance Software Engineering Institute Carnegie Mellon University Pittsburgh, PA Peter H. Feiler Feb 4, 2015

2 Copyright 2015 Carnegie Mellon University This material is based upon work funded and supported by the Department of Defense under Contract No. FA C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense. NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN AS-IS BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. This material has been approved for public release and unlimited distribution. This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu. DM

3 Outline Certification and Recertification Challenge Improving the Quality of Requirements Incremental Life-cycle Assurance of Systems 3

4 Certification & Recertification Challenges Certification: assure the quality of the delivered system Sufficient evidence that a system implementation meets system requirements Quality of requirements and quality of evidence determines quality of system Certification related rework cost Currently 50% of total system cost and growing Recertification Challenge Desired cost of recertification in proportion to change Improve quality of requirements and evidence Perform verification compositionally throughout the life cycle 4

5 High Fault Leakage Drives Major Increase in Rework Cost Requirements Engineering Aircraft industry has reached limits of affordability due to exponential growth in SW size and complexity. 70% Requirements & system interaction errors System Design Software Architectural Design Component Software Design Where faults are introduced Where faults are found The estimated nominal cost for fault removal Sources: 70%, 3.5% NIST Planning report 02-3, The Economic Impacts of Inadequate Infrastructure for Software Testing, May D. Galin, Software Quality Assurance: From Theory to Implementation, Pearson/Addison-Wesley (2004) B.W. Boehm, Software Engineering Economics, Prentice Hall (1981) 1x 80% late error discovery at high rework repair cost 20%, 16% 5x Code Development 10%, 50.5% 0%, 9% 80x 20x Major cost savings through rework avoidance by early discovery and correction A $10k architecture phase correction saves $3M Unit Test 20.5% x Integration Test System Test Acceptance Test Total System Cost Boeing 777 $12B Boeing 787 $24B Software as % of total system cost 1997: 45% 2010: 66% 2024: 88% Post-unit test software rework cost 50% of total system cost and growing 5

6 Mismatched Assumptions in System Interactions System User/Environment System Engineer Hazards Impact of system failures Operator Error Automation & human actions Hardware Engineer System Under Control Compute Platform Physical Plant Characteristics Lag, proximity Data Stream Characteristics Latency jitter affects control behavior Potential event loss Distribution & Redundancy Virtualization, load balancing, mode confusion Runtime Architecture Control Engineer Control System Embedded SW System Engineer Measurement Units, value range Boolean/Integer abstraction Air Canada, Ariane, 7500 Boolean variable architecture Application Software Application Developer Concurrency Communication ITunes crashes on dual-cores Embedded software system as major source of hazards Why do system level failures still occur despite fault tolerance techniques being deployed in systems? 6

7 Model-based Engineering Pitfalls The system Inconsistency between independently developed analytical models System models Confidence that model reflects implementation System implementation This aircraft industry experience has led to the System Architecture Virtual Integration (SAVI) initiative 7

8 Quality & Certification Improvement Strategy 2010 SEI Study for AMRDEC Aviation Engineering Directorate Architecture-led Requirement Specification Architecture-centric Virtual System Integration Static Analysis & Compositional Verification Incremental Assurance Plans & Cases throughout Life Cycle Mission Requirements Function Behavior Performance Survivability Requirements Reliability Safety Security Model Repository Architecture Model Component Models System Implementation System configuration Operational & failure modes Resource, Timing & Performance Analysis Reliability, Safety, Security Analysis Four pillars for Improving Quality of Critical Software-reliant Systems 8

9 Project Approach Architecture-Led Incremental System Assurance (ALISA) Approach Semantically Consistent Unification of Modeling Concepts from Different Perspectives and their Use in Existing Practice Standards Goal, Intent, Requirement, Assumption, Claim Obstacle, Fault, Defect, Hazard, Vulnerability, Challenge Verification Method, Activity, Result, Evidence, Counter evidence ALISA Workflow & Eclipse-based Workbench Architecture-focused Sources: CMU ECE Indirect Architecture-led Control Path Analysis, Contractbased Compositional Assurance Plan with Multi- CMU ECE/SEI System Verification Requirements & Hazard Manager, SEI/VDID Architecture Fault Analysis, AMRDEC/SEI SAE EMV2, SEI Analysis Analysis & Verification valued Argumentation Logic Assurance Case, OMG Structured Assurance Case Model, UBS RDAL, UO/UL KAOS Technical and Operational Validation in Actual Projects Access to Actual Project Information Early Life Cycle Requirements Case Study Late Life Cycle Certification Study Measurably Increased Assurance Confidence Credit for Analytical Evidence Measurably Reduced Defect Leakage & Assurance Cost Apply COQualMO and SAVI ROI Measurement-driven Assurance Cost and Confidence Improvement through Incremental Life Cycle Assurance Anticipated Improvement Thresholds 25% Higher Requirement/Hazard Coverage 35% Higher Evidence Confidence Assessment of Potential for Proportional Recertification Cost Benefit and Risk of Partial Verification 9

10 Project Theme and Focus System Requirements going forward Coverage: what all should be covered given a model (description) of a system interacting with its operational context Uncertainties and incremental development: explore and understand sensitivity points in proposed system architecture Incremental assurance through compositional verification Incremental as one layer and component at a time Subset of requirements, verification activities Phase based Criticality, change sensitivity driven Identify verification hotspots: Given the set how well are we doing in satisfying them Coverage argumentation: Do we have enough requirement, verification activities Scalability: take advantage of architectural abstractions 10

11 Outline Certification and Recertification Challenge Improving the Quality of Requirements Incremental Life-cycle Assurance of Systems 11

12 Stakeholder Needs and Requirement Categories Leveson System Theoretic Framework System, operational environment, development and V&V process 12

13 Quality of Requirements Generally, requirements are provided in a textual form. Guidelines exist for writing good requirements; they include recommendations about the syntax of requirements statements, wording (exclusions, representation of concepts, etc.), and characteristics (specific, measurable, achievable, feasible, testable, etc.). Refer to INCOSE (2011, Section ) and ISO/IEC (2011). 13

14 Mixture of Requirements & Architecture Design Constraints Requirements for a Patient Therapy System Typical requirement documents span multiple levels of a system architecture We have made architecture design decisions. We have effectively specified a partial architecture Adapted from M. Whalen presentation 14

15 System Specification and Requirements Developmental Coverage Requirements Modifiability Assurability Quality attribute utility tree Environmental Assumptions Requirements Guarantees Assumptions Precondition Postcondition Invariant Mission Requirements Function Behavior Performance Dependability Requirements Reliability Safety Security Interaction contract: match input assumption with guarantee Implementation constraints Exceptional condition 15

16 Architecture-led Requirement & Hazard Specification Error Propagation Ontology Output Behavior Control System State Input Leveson pattern Actuator Behavior System Under Control State Sensor 16

17 Exceptional Conditions Integral to Requirement Specifications Identify exceptional system states Failures, anomalies, in components and interactions Exceptional conditions that contribute to incidents/accidents Identify contributors to exceptional system states Control functions as error sources that result in exceptional condition/unsafe state inability to recover from exceptional condition Identify derived safety requirements Health monitoring, safety system Additional sensors, control conditions Integrate STAMP/STPA with SAE ARP4761 ES/Haz 1 ES/Haz 2 ES/Haz 3 Repeat for next architecture layer Leverage SAE ARP4761 automation in OSATE Sensors Actuators 17

18 Outline Certification and Recertification Challenge Improving the Quality of Requirements Incremental Life-cycle Assurance of Systems 18

19 Building the Assurance Case throughout the Life Cycle 19

20 Virtual System Integration & Compositional Verification Continuous Confidence Measure throughout Life Cycle that a System Meets its Requirements Incremental Evolution and Execution of Assurance Plans Architecture Requirements Led Verification Requirements Specification System&SW Architecture Verification Architecture-centric Virtual Integration Virtual Architecture Integration & Analysis Design Verification Requirements Deployment Engineering Build Early Discovery through Architecture Analysis leads to Assurance Related Rework Reduction System & SW Architectural Target Design Build Component Software Design Design Validation Code by Verification Virtual Integration Code Development Integration Build Code Coverage Testing Unit Test Integration Test Acceptance Test Flight Test System Integration System Test Lab Testing Auto-generation from verified models AADL&SCADE/Simulink Ada SPARK/Ravenscar MISRA C Design & Req Refinement RS VA RS VA RS Design & Req Refinement Incremental Architecture & Requirement Evolution RS Requirement Coverage RS VA RS VA RS Incremental Contract-based Compositional Verification Compositional Verification VA Compositional Verification VA Build the System Auto-generated Assurance Cases Build the Assurance Case 20

21 From SEBOK 21

22 Evidence throughout Life Cycle Evidence aligned with architecture models Impact of architecture changes on evidence Compositionality of analysis methods Scalability across architecture layers Fidelity of analysis throughout life cycle Adjustable uncertainty margins 22

23 Representing Assurance Plans & State Bridge to assurance plans and cases Draw on RDAL, Resolute, Agree, SACM (Certware), SVM Associated with requirements architecture, design, implementation artifact Intended verification actions Across life cycle on different artifacts Across layers of system architecture Argumentation Necessity and sufficiency of refined/decomposed requirements Necessity and sufficiency of verification actions Assurance state Multi-valued logic of verification action & results (pro, con, neutral, tbd) Acceptable risk factors (e.g., design assurance levels) Time phased execution of assurance plans 23

24 Quality Metrics for Verification Actions Credit for analytical evidence Effectiveness of verification method In discovery of defects In early discovery of defects Quality of the model Does model reflect actual system Does actual system implement the system specification Quality of the verification tool Cost and value of tool qualification (level A etc.) Alternate approaches to assessing tool quality 24

25 Measurement Driven Assurance Improvement Understand where the cost drivers are Sources of defects and their impact Potential for early detetion 25

26 Strategy for Improving Recertification Incremental certification without (unacceptable) loss of quality compared to full certification Understand scope of change impact on system and requirements Reduce scope of impact through integrated modular architectures with welldefined interfaces and interactions enforced by static analysis Reduce risk of residual anomaly impact through operational fault containment regions (resilience) Verification strategy Compositional (modular) verification leveraging architecture 26

27 Relationship of Concepts/Meta models RDAL GORE/KAOS AADL Hazards Obstacles Goals Requirements Architecture specification Architecture instance EMV2 Verification Activities SACM Claims & Arguments Verification Methods Assurance Results Resolute Agree 27

28 Alisa Modeling Objectives Represent goals, requirements, hazards Associated with an architecture model For each layer of the architecture Specify intended verification activities Across life cycle on different artifacts Across layers of system architecture Via verification methods (manual, automated) on models/implementations Specify verification plans Reasoning logic of how verification activities satisfy requirement Assumptions, preconditions on verification activities Dealing with dependencies between verification activities Manage assurance state and results Multi-valued logic evaluation of verification action & results Acceptable risk factors (e.g., design assurance levels) Time phased execution of assurance plans 28

29 Relationship of Concepts/Meta models ReqSpec AADL Hazards Obstacles Goals Requirements Architecture specification Architecture instance EMV2 Verification Activities Claims & Arguments Verification Methods Assurance Results Verify Assure 29

30 A Multi-dimensional Certification Assurance Dependency Model Incremental Certification: confidence in system quality Understand and reduce impact of change on system, certification evidence through analysis of certification assurance dependencies Requirement allocation dependencies Functional and quality attribute requirements System & SW architecture Architectural impact dependencies Requirement refinement dependencies Resilience requirement dependencies Defects & hazards affecting safety, reliability, security Requirement & hazard dependencies Analysis & test results Confidence Argumentation Source code implementation Implementation compliance dependencies Verification activity dependencies Evidence dependencies claim defeater evidence argument Argumentation dependencies 30

31 Multi-valued Assurance Logic in Verification Plan Verification Plan: verification activities demonstrate requirements are met and hazards/obstacles have been addressed Verification expression referring to verification activities Execution All [ va+]: a collection of independent Vas Va1 andthen Va2: Execute Va2 only if Va1 produces positive result Va1 failthen Va2: Execute Va2 only if Va1 produce negative result Va1 unknownthen Va2: Execute Va2 only if Va1 is unable to complete Evaluation All: success if All success [and without shortcut] Failthen/Unknownthen: success if Va1 success or Va1 not success and Va2 success [alternate] Andthen: success if both are successful [not quite implies] Argument: Rationale justifying the verification plan 31

32 Registry of Verification methods Content of Registry Existing OSATE Analysis plugins Resolute claim and computational functions Reflective execution of Java/Xtend functions Execution of Junit-based tests More to come (e.g., Agree) 32

33 Assurance Result State Execution status Tbd, complete, running, redo Result state Success, fail, unknown categories of success Fail: verification method specific Unknown: Timeout, Execution Failed, Assumption/precondition failed, Not selected, not performed yet Reference to method specific result data Used in verification, assumption, precondition results Aggregate result state Success, fail, unknown: count Used on case, claim, hazard results 33

34 Assurance Results 34

35 Summary Incremental life cycle assurance improves quality of critical systems Systematic coverage of requirements and exceptional conditions leads to improved requirement quality Incremental evolution of requirement analysis and architecture design focused on critical requirements reduces uncertainty Compositional verification throughout life cycle leads to early discovery of issues and early verification evidence Multi-valued assurance result logic provides insights on assurance state and hotspots throughout life cycle 35

36 References AADL Website and AADL Wiki Blog entries and podcasts on AADL at AADL Book in SEI Series of Addison-Wesley On AADL and Model-based Engineering df On an architecture-centric virtual integration practice and SAVI On an a four pillar improvement strategy for software system verification and qualification Webinars on system verification and on architecture trade studies with AADL 36

37 Contact Information Peter H. Feiler Principal Researcher RTSS Telephone: U.S. Mail Software Engineering Institute Customer Relations 4500 Fifth Avenue Pittsburgh, PA USA Web Wiki.sei.cmu.edu/aadl Customer Relations SEI Phone: SEI Fax: