Safe Testing and Execution of Autonomy in Complex, Interactive Environments. Safe and Secure Systems and Software Symposium (S5) 7/12/2016

Transcription

1 Safe Testing and Execution of Autonomy in Complex, Interactive Environments Safe and Secure Systems and Software Symposium (S5) 7/12/2016 David Scheidt William Van Besien

2 Autonomous Systems Tele-operation Automatic Autonomous UXV UXV UXV Sensor Control Sensor Control Sensor Control raw sensor data discrete actions raw sensor data maneuver commands raw sensor data abstract status operational objectives Operator interface similar to pilot to plane interface Operator interface similar to pilot to autopilot interface Operator interface similar to commander to pilot interface

3 Autonomy Test & Evaluation Challenge

4 Safety Concerns when Testing Autonomous Vehicles Autonomous Unmanned Vehicles (AUVs) inherit safety protocols from tele-operated unmanned vehicles. Additionally, AUVs produce the need for additional safety protocols associated with unsafe acts produced by autonomous AUV decisions. Safety Violations Caused by Autonomous Decisions Geospatial Faults The AUV is not where it is supposed to be Airspace deconfliction Airspace violation Sensor Coverage violation Propriaceptive Faults The AUV is performing an unsafe maneuver Actions risk exceeding the flight envelope of the physical system Interoceptive Faults The AUV s internal state is unsafe Failure to correct diagnose or manage recoverable component faults which causes a change to the AUV performance envelope (i.e., fuel leaks) Safety Protocol Violations Failing to report or respond to range safety personnel

5 Testing of Autonomous Systems in Complex Environments (TACE)

6 TACE System Architecture TACE Internal Messages over ZeroMQ TCP/IP Socket TENA over TCP/IP Socket TACE External Messages over JUAS? SUT Comms SUT Sensor Middleware BIT Middleware Autopilot Middleware Autonomy Middleware SUT Communication Server Stimulator Watchdog TACE GNC TACE Year 1-2 TACE Year 3 Non-TACE Components SUT Ground Stations Streaming Reliable Test Data DB SIMDIS Synthetic Force Generator Sensor Sensor Model Model Sensor Model Communication Model Ground Communication Server Blackbox Monitor Whitebox Monitor TENA Datalogger TACE TENA Execution Test Config DB R/W TACE Test Executive Test Test Executive TACE Executive Client Test Client (User (User Client Interface) Interface) (User Interface) Watchdog Manager Server Watchdog Manager Client (User Interface) LVC LVC LVC TENA to TENA Bridge External TENA Execution R/W Test Result DB R R/W LVC LVC LVC Safety Constraint DB

7 TACE System Under Test (SUT) On-board Architecture TACE processes (Watchdog, Stimulator, Guidance and Control (GNC) Manager) on one processing platform (Gumstix Overo) with autonomy processes on a separate processing platform Middleware provides interface between Autonomy and TACE processes Separate GPS receiver for independent TACE position information No camera or other sensors

8 First Generation TACE Watchdog Five active software modules in two classes, diagnosis and remediation Fault Diagnosis, three components operating in parallel Range boundary violation Airspace deconfliction Lost Link Violation Overbank Violation Fault Remediation Return to Loiter Point What could possibly go wrong?

9 TACE Flight Test Equipment Phase 2 TACE Product Demo Support Equipment 915 MHz RC link T2 Live Entity 3DR Aero or Quad Copter 433 MHz 3DR telemetry link TACE Comms Gateway 3DR Transceiver + Antenna 2.4 GHz Wave Relay link 2.4 GHz Antenna SUT TACE Demonstration Vehicle Procerus Unicorn Unicorn Ground Station 900 MHz safety link 900 MHz Antenna RC controller T1 Live Entity 915 MHz 3DR telemetry link 3DR Transceiver + Antenna Mission Planner Laptop Mission Planner Laptop Wave Relay Switch Laptop with Virtual Cockpit TACE Infrastructure Commbox

10 TACE Flight Tests at Aberdeen Test Center 2014 APL Test Team on the tarmac at Phillips Army Airfield (PAAF) Aberdeen Test Center (ATC) Hand launch of the Procerus research AUV controlled by JHU/APL s Autonomy Tool Kit (ATK)

11 TACE Watchdog 2.0 Design a Watchdog using Model-based Reasoning Methods Developed for Spacecraft and Ship Auxiliary Fault Management

12 ExecSpec Executable Model Specifications Intended to replace rule-based autonomy engine for spacecraft fault management Enhance three important properties to autonomy system design Clarity, flexibility, verifiability.

13 TACE Watchdog CONOPS Watchdog to serve as mediator between SUT autopilot and TACE remediation algorithms Watchdog determines whether the autonomy is at risk of placing the SUT in an unsafe state Considers situational environment, multiplexes between set of remediation algorithms, and selects appropriate one, assuring that the remediation algorithm does not itself cause a safety violation Re-enables SUT autonomy and inhibits remediation when complete Drafted Taxonomy of anomalies Each anomaly class has corresponding state machine Each state machine constitutes a lifecycle of a remediation

14 Modeling the system state and behavior as a probabilistic finite state machine Transition constraints P(t) Transition constraints P(t) State constraints State constraints Transition constraints P(t) Transition constraints P(t) Transition constraints P(t) State constraints Transition constraints P(t) State constraints Constraints are logical statements that: 1) May describe ontological relationships 2) May include assertions of resource variables or component states 3) May incorporate dependencies 4) May be links to methods in high fidelity models (i.e., dynamical models used to anticipate collisions)

15 TACE Watchdog 2.0 Example Geospatial Constraint-Violation State Machine

16 System Under Test (BLUE) triggers fault by violating spatial constraint (RED). System Under Test (BLUE) remains Test System and Under Evaluation/Science Test (BLUE) flies towards and Technology Program remediation zone (GREEN). in remediation zone (GREEN) until released by safety manager. System Under Test (BLUE) is tracking ground target (YELLOW). Distribution Statement C Distribution: Authorized to U.S. Government Agencies and their contractors only; Reason: administrative/operational use. Other requests for this document shall be referred to the Program Manager for Test & Evaluation/Science and Technology (T&E/S&T), Test Resource Management Center,1225 South Clark Street, Suite 1200, Arlington, VA DISTRIBUTION STATEMENT A. Approved for public release; distribution is unlimited.

17 Recent Flight Experiments Scan Eagle testing (2015) Embedded TACE Watchdog 2.0 in Scan Eagle Payload Bay Integrated TACE Watchdog 2.0 with Scan Eagle Avionics Bus Using I-MUSE Training Simulator to test Watchdog 2.0 Geospatial Deconfliction Airspace Deconfliction Overbanking Fault Failsafe Remediation Testing with Range Infrastructure (2016) Repurposed as Tactical Decision Aide Integrated with Atlantic Test Range infrastructure (DASR-11 Radar) Integrated with Atlantic Test Range Computing and Network Infrastructure Demonstrated an Ability to Alert on Geospatial Safety Violations Range Boundary Violations Test Boundary Violations Sensor Coverage Violations

18 Acknowledgements Thanks to our sponsors: Test Resource Management Center Unmanned and Autonomous Systems Test (TRMC UAST) Program under contract W900KK-13-C-0036 ONR Machinery Automation Program under contract N D-6400 and thanks to Atlantic Test Range (ATR), Range Systems and NAVAIR 5.4 for their transition support and leadership.

19