V&V of Autonomy: UxV Challenge Problem (UCP) Integrity Service Excellence. July 2016

Transcription

1 V&V of Autonomy: UxV Challenge Problem (UCP) July 2016 Integrity Service Excellence Jon Hoffman Verification & Validation of Complex and Autonomous Systems Team Autonomous Control Branch (AFRL/RQQA) Air Force Research Laboratory 1

2 Agenda Motivation Trust and Certification Process V&V Challenge Problem Need Two-Tanks Challenge Problem UxV Challenge Problem (UCP) V&V Tools & Techniques Summary 2

3 Motivation Introduction, Discovery, and Cost of Software Faults 1,2,3 Identified Need 70% of faults are introduced 3.5% faults are found 1x estimated nominal cost for fault removal Requirement Development Opportunity to find faults as they are introduced when costs are low Architecture Development 1. NIST Planning report 02-3, The Economic Impacts of Inadequate Infrastructure for Software Testing, May D. Galin, Software Quality Assurance: From Theory to Implementation, Pearson/Addison-Wesley (2004) 3. B.W. Boehm, Software Engineering Economics, Prentice Hall (1981) Detailed Design Implementation Integration 20% of faults are introduced 16% faults are found 5x estimated nominal cost for fault removal Validation Verification Transition 20.5% faults are found x estimated nominal cost for fault removal 10% of faults are introduced 59.5% faults are found 20-80x estimated nominal cost for fault removal Rework and certification is 70% of SW cost. 4 3

4 Trust and Certification Products / Process New Autonomy Need 4

5 New Autonomy Need Trust and Certification Products / Process Design Requirements Architecture Models System Design and Safety Requirements (ARP 4761, ARP 4754/A, MIL-HDBK-882E) 5

6 New Autonomy Need Trust and Certification Products / Process Design Requirements Architecture Models Validation Simulation Testing System Design and Safety Requirements (ARP 4761, ARP 4754/A, MIL-HDBK-882E) Testable Requirements & Verification Plans (DO-178C/254, MIL-HDBK-516) 6

7 New Autonomy Need Trust and Certification Products / Process Requirement Formalization & Analysis Architecture Formalization & Analysis FORMALIZED SAFETY ASSESSMENT HAZARD MITIGATION REQUIREMENTS Validation Simulation Testing System Design and Safety Requirements (ARP 4761, ARP 4754/A, MIL-HDBK-882E) Testable Requirements & Verification Plans (DO-178C/254, MIL-HDBK-516) 7

8 New Autonomy Need Trust and Certification Products / Process Requirement Formalization & Analysis Analytical Proof Synthesis Architecture Formalization & Analysis FORMALIZED SAFETY ASSESSMENT HAZARD MITIGATION REQUIREMENTS Multiple V&V Technology Paths Modeling, Simulation, Test & Evaluation Run Time Assurance Assurance Validator System Design and Safety Requirements (ARP 4761, ARP 4754/A, MIL-HDBK-882E) Testable Requirements & Verification Plans (DO-178C/254, MIL-HDBK-516) 8

9 New Autonomy Need Trust and Certification Products / Process Requirement Formalization & Analysis Analytical Proof Synthesis Certified Assurance Case Architecture Formalization & Analysis FORMALIZED SAFETY ASSESSMENT HAZARD MITIGATION REQUIREMENTS Multiple V&V Technology Paths Modeling, Simulation, Test & Evaluation Run Time Assurance Assurance Validator System Design and Safety Requirements (ARP 4761, ARP 4754/A, MIL-HDBK-882E) Testable Requirements & Verification Plans (DO-178C/254, MIL-HDBK-516) 9

10 New Autonomy Need Trust and Certification Products / Process Compositionally Verified Systems of Systems Requirement Formalization & Analysis Analytical Proof Synthesis Certified Assurance Case Architecture Formalization & Analysis FORMALIZED SAFETY ASSESSMENT HAZARD MITIGATION REQUIREMENTS Multiple V&V Technology Paths Modeling, Simulation, Test & Evaluation Run Time Assurance Assurance Validator System Design and Safety Requirements (ARP 4761, ARP 4754/A, MIL-HDBK-882E) Testable Requirements & Verification Plans (DO-178C/254, MIL-HDBK-516) 10

11 Need for a Challenge Problem Transitioning V&V technologies Target isn t to design and build the tech in the challenge problem but rather to demonstrate the V&V technologies / techniques in comparable systems The V&V challenge problem is to do the V&V groundwork at the same time at the design and development of systems Early and Often Design for Certification 11

12 A Challenge Problem Two Tanks System Two Tanks Simplified Currently no faults, operation is instantaneous, flow rates are constant, boolean assumptions, etc. Language AADL with the AGREE Extension AADL Architecture Analysis Design Language AGREE Assume Guarantee REasoning Environment Tool OSATE

13 Two Tanks System Block Diagram Environment Two independent tanks with components Controller Comprised of two separate controllers Tank 1 informs Tank 2 that that flow out valve is open This allows Tank 2 knowledge that it has an in flow 13

14 Two Tanks Decomposition 14

15 Moving Past the Tanks Need for a more complex challenge problem Domain specific problem More complicated interactions lead to more challenging V&V problems Determine robustness of V&V tools Apply our Trust & Certification process Opportunity to utilize additional V&V techniques such as Run-Time Assurance (RTA) 15

16 UxV Challenge Problem (UCP) High Level Objectives Goals of the system A vehicle that moves around an area that avoids dangerous situations A vehicle with navigate to objective locations and return to a homebase Autonomous navigation V&V Goals of the system Formal requirements system of the UCP Formal architecture of the UCP system Run-time Assurance of autonomous mission planning algorithm provides software fault tolerance Build assurance case throughout the process to capture decisions to support certification arguments 16

17 UCP Map 17

18 High Level Requirements What requirements must hold throughout? The vehicle must remain safe A number of parts feed into this Restricted Operating Zone (ROZ) G-Load limit on vehicle Intruder location The vehicle must complete its mission Secondary to remaining safe A number of parts feed into this Remain safe Visit all objective locations Start/Return to home base 18

19 CONOPS Agenda Map Vehicle Safety ROZs G-Load Intruder Run-Time Assurance Mission Objectives Home Base 19

20 Map 20 km x 20 km Operating Region Roughly the size of the Indianapolis metro area One Homebase Square section in the map that denotes the start and endpoint for the UxV Multiple objectives locations Locations that the UxV must visit in order to complete the mission Multiple Restricted Operating Zones (ROZs) Limited UxV access due to physical obstructions or possible danger 20

21 UCP Map 21

22 Vehicle Notional vehicle size Length: 1.5 m Wingspan: 3 m Mass: 20 kg Max Speed: 40 m/s Flight Modes Dash Cruise Loiter Evade 22

23 Vehicle Safety Overall Vehicle Hazard Level: Hazard = T ROZ + T GL + T Int There is a current hazard level for a vehicle that is the sum of the three independent threats ROZ Threat (T ROZ ) G-Load Threat (T GL ) Intruder Threat (T Int ) The vehicle is considered safe if the hazard level is less than 20 23

24 Vehicle Safety ROZ Threat Threat Levels (T ROZ ) Measure of risk to the mission and to the vehicle safety as related to the ROZ Range from 0 to 9 or 20 0 : no threat to the vehicle 1-4 : Threat to mission Example: Vehicle detected by enemy sensors 5-9 : Threat to safety Example: Probable incoming fire (5-7) Example: Physical obstructions (8-9) 20 : Loss of vehicle Unavoidable physical obstruction Example: Collision with building/ridge 24

25 Vehicle Safety G-Load Threat Threat Levels (T GL ) Identifies the threat to the vehicle due to the current G-Load on the vehicle The current G-limit on the vehicle is calculated according to the following force equation F = Range from 0, 1, or 20: 0: No threat to the vehicle G-Load less than 3 m v 2 r 1: Minimal threat to the vehicle G-Load between 3 and 5 20: Loss of vehicle G-Load greater than 5 4G 6G 2G 25

26 Vehicle Safety Intruder Threat Threat Levels (T Int ) Identifies the risk of collision with the vehicle based on proximity and Vehicle (non-intruder) speed. Range from 0, 5 or 20 0 : no threat to the vehicle Intruder outside of Caution Halo 5 : Potential threat to collide Intruder inside Caution Halo and outside the Safety Halo 20 : loss of vehicle Intruder is inside the Safety Halo (collision is unavoidable) 26

27 Vehicle Safety Run-time Assurance The RTA s job is to prevent the system from a loss of vehicle event (threat = 20) Provide software fault tolerance through a system monitor and verifiably safe backup Three aspects to protection Safe Now Currently the vehicle (Hazard less than 20) Safe on the Horizon Look ahead to determine possible catastrophic events Switch to recovery if necessary 27

28 Vehicle Mission Overall The vehicle mission shall include the following: The vehicle shall prioritize safety and maintain acceptable hazard levels The UxV shall begin inside the Homebase The vehicle shall accomplish 100% objective acquisition The vehicle shall return to Homebase (RTB) after leaving the Homebase area and attempting objective acquisition 28

29 V&V Tools & Techniques Requirements formalization and analysis Specification & Analysis of Requirements (SpeAR) Architecture modeling and analysis Assume Guarantee Reasoning Environment (AGREE) Run-Time Assurance (RTA) for safe operation Monitor and protect system from software faults at run-time Assurance Case for certification Build argument for certification based upon evidence collected throughout system engineering process 29

30 V&V Throughout the Process CONOPS 30

31 V&V Throughout the Process CONOPS SpeAR 31

32 V&V Throughout the Process CONOPS SpeAR AGREE 32

33 V&V Throughout the Process CONOPS SpeAR AGREE Run-Time Assurance 33

34 V&V Throughout the Process CONOPS Assurance Case SpeAR AGREE Run-Time Assurance 34

35 Summary UxV Challenge Problem for V&V of Autonomy V&V Philosophies Design for Certification Early and Often 35

36 Topic Continuation Specification and Analysis of Requirements with SpeAR 2.0 Dr. Jennifer Davis, Rockwell Collins High-level Requirements and Arguments for the UCP Mr. M. Anthony Aiello, Dependable Computing UCP Compositional Requirements Structure Mr. Aaron Fifarek, LinQuest 36

37 External Resources ARFL\RQQA VVCAS GitHub Location UxVChallengeProblem TwoTanksExample 6UCubeSat SpeAR 2.0 Release Candidate Location 37

38 Questions? 38