Increasing ATM Efficiency with Assistant Based Speech Recognition (ABSR)

Transcription

1 Chart 1 Increasing ATM Efficiency with Assistant Based Speech Recognition (ABSR) Hartmut Helmke, Oliver Ohneiser, Jörg Buxbaum, Christian Kern Supported by DLR Technology Marketing and Helmholtz Validation Fund German Aerospace Center (DLR) Braunschweig DFS, Langen Austro Control, Vienna

2 Chart 3 Emergency flight Without ASR support (Automatic Speech Recognition) green trajectory planned by AMAN

3 Chart 4 With ASR support

4 Chart 5 Command Recognition & Error Rates Feb./Mar Trials Recognition Rate Rejection Rate Error Rate With ABSR 91.6% 8.4% 3.0% 3 male + 3 female DFS controllers,1 male ANS-CR controller; 3 male + 1 female ACG controllers Arrival sequence is more stable AMAN trajectories match to controller s trajectories Controllers want the speech recognizer, but. integration into an AMAN is not a killer application

5 Chart 6 Contents 1. The Motivation of an ANSP to use Speech Recognition. 2. The Experiment 3. Our quantitative results 4. Assistant Based Speech Recognition 5. ASR: What is achieved and where research is still needed?

6 Chart 7 From Paper Flight Strips to Paperless Systems Today Austro Control operates ATC fully paperless. All flight information is on radar screen. Paperless Systems are very beneficial: - Additional safety nets - Additional monitoring aids - Full flight trajectory available - Silent coordination functionalities - Increased adaptability

7 Chart 8 From Paper Flight Strips to Paperless Systems - However, paperless Systems are demanding: - Manual input of each controller s instruction - This occasionally results in: - Increased workload - Increased cognitive load - Decrease of productivity in dense traffic - Austro Control soon identified voice recognition as a potential solution. - Radar label maintenance could be the killer application for Speech Recognition. How to validate this feeling by quantitative numbers?

8 Chart 9 AcListant -Strips: The Killer Application Using Speech Recognition to reduce controller s workload and to improve flight efficiency Validation Trials: Two input conditions were tested: 1. Commands entered by mouse into radar labels (condition Mouse ) 2. Commands entered by speech recognition, correction if necessary by mouse (condition ABSR )

9 Chart 10 Working with Mouse and Keyboard (movie) High Usage of Frequency (Heavy Workload) KLM29Y TURN_RIGHT 210 KLM29Y CLEARED_ILS 23R KLM29Y TURN_RIGHT 150 DLH6JT TURN_RIGHT 220 DLH6JT CLEARED_ILS 23R DLH3ER DESCEND 3000

10 Chart 11 ABSR Support Reduces Workload (movie) BEE5YX Handover Tower CFG7617 DESCEND 60 DLH4MA DESCEND 4000 QNH 1026 Just a click to confirm GWI9339 identified expect runway 23 right

11 Chart 12 Contents 1. The Motivation of an ANSP to use Speech Recognition. 2.The Experiment 3. Our quantitative results 4. Assistant Based Speech Recognition 5. ASR: What is achieved and where research is still needed?

12 Chart 13 Radarscreen (Radarvision) ISA Interface ABSR Log Radar overview VOIP Radio Weather display Secondary Task Pseudo Pilot stations DLR

13 Chart 14 Complete APP Scenario for Dusseldorf APP Tested with 8 controllers from Munich and Vienna Controller acts as both Initial and Final controller with medium traffic (35 arrivals / h, scenario Complete APP, 60 min duration) Runway closure for 5 minutes Emergency flight

14 Chart 15 Final APP Scenario for Dusseldorf Approach Controller acts only as final controller with very high traffic (60 arrivals / h, scenario Final APP, 45 min duration) Automatic coordination with Initial ( Holding of flights, if final leg is longer than 22 nm)

15 Chart 16 ABSR Support Reduces Workload (movie) DLH3RK DESCEND 3000 QNH 1013 SWR106L REDUCE 160 DLH6YW REDUCE :30min

16 Chart 17 Contents 1. The Motivation of an ANSP to use Speech Recognition. 2. The Experiment 3. Our quantitative results 4. Assistant Based Speech Recognition 5. ASR: What is achieved and where research is still needed?

17 Chart 19 Results with respect to Reduction of Controller Workload

18 Chart 20 Time of Scenario Busy with Mouse Input 35% 30% 25% 20% 15% 10% 5% 0% ABSR 9% Mouse 27% Complete App ABSR 11% Mouse 31% Final App

19 Chart 21 Time of Scenario Busy with Mouse Input 35% 30% 25% 20% 15% 10% 5% 0% ABSR 9% Mouse 27% Complete App ABSR 11% Mouse 31% Final App % more time available with ABSR support.

20 Chart 22 NASA-TLX and ISA 5 4 ISA: - Controllers have 10% less workload with ABSR support ABSR 2.6 Mouse 2.9 Complete App ABSR 2.6 Mouse 2.9 Final App NASA-TLX workload: - Controllers have 15% - 20% less workload with ABSR support ABSR 7.6 Mouse 9.5 Complete App ABSR 7.4 Mouse 8.7 Final App NASA-TLX mental demand: - Controllers have 32% - 45% less mental workload with ABSR support ABSR 21 Mouse 31 ABSR 16 Mouse 29 Complete App Final App

21 Chart 23 Cognitive Resources

22 Chart 24 Cognitive Resources / Time Needed to Sort Cards Scenario Condition Average [min] Min Max Complete APP ABSR 05:31 04:08 07:19 Complete APP Mouse 10:38 08:34 12:42 Final APP ABSR 04:52 03:38 05:54 Final APP Mouse 08:45 07:27 10: % more cognitive resources with ABSR support

23 Chart 25 Results with respect to Improvement of Efficiency

24 Chart 26 Missing Commands in Labels in [%] Scenario Condition Average Complete APP ABSR 4.9% Complete APP Mouse 12.1% Final APP ABSR 4.4% Final APP Mouse 6.7%

25 Chart 27 Distance Flown [NM] Scenario Condition Average Max Min Complete APP ABSR Complete APP Mouse Final APP ABSR Final APP Mouse Complete APP: - Reduction of 5.0 NM with ABSR support (-6.5%)

26 Chart 28 Additional Flight Time [seconds] Scenario Condition Average Complete APP ABSR Complete APP Mouse Final APP ABSR Final APP Mouse Complete APP: - Reduction of 77 seconds with ABSR support (-30.7%) 50 to 65 liters savings of kerosene with 250 CAS in 7000 feet.

27 Chart 29 Aircraft Landings per Hour Scenario Condition Average Flow # Medium # Heavy Complete APP ABSR Complete APP Mouse Final APP ABSR Final APP Mouse Final APP: - Increase of 2.1 mv/h with ABSR support (+4.6%)

28 Chart 30 Dependency between efficiency measurements - Reduced workload Increase in ATM efficiency - Reduced flown distance Reduced flight time - More Landings could result in increase flight time Scenario Condition Average Flow Flight Time Flown Distance Missing Label Complete APP ABSR % Complete APP Mouse % Final APP ABSR % Final APP Mouse %

29 Chart 31 Combination of 4 Measurements Scenario Condition Overall Efficiency Complete APP ABSR 79.4% Complete APP Mouse 60.1% Final APP ABSR 68.7% Final APP Mouse 52.2% - Best Value set to 100%, - Worst Value is 0% - Combination of - Weighted Flow (2), - Flight Time (1), - Flown Distance (1) and - Radar Label Deviations (2)

30 Chart 32 Combination of 4 Measurements for Different Controllers for Complete / Final APP Scenario Seq No Seq No ABSR Mouse % 75.8% % 79.6% % 73.2% % 77.6% % 68.9% % 6.1% % 19.5% % 80.0% Average: 79.4% 60.1%

31 Chart 34 Significance of Efficiency Hypotheses with and without considering Sequence Effects Hypotheses Complete Approach Final Approach Both without with without with without with Increased flow 23.8% 14.3% 3.8% 2.5% 3.7% 1.3% Flight distance 5.1% 3.2% 27.7% 24.1% 4.3% 2.6% Flight time 4.9% 2.9% 23.8% 19.0% 3.6% 1.9% Label info 1.9% 0.5% 9.3% 9.7% 0.7% 0.2% Overall effic. 6.1% 2.0% 7.7% 5.7% 1.6% 0.5% Average do not change, but standard deviations decrease.

32 Chart 35 Contents 1. The Motivation of an ANSP to use Speech Recognition. 2. The Experiment 3. Our quantitative results 4. Assistant Based Speech Recognition 5. ASR: What is achieved and where research is still needed?

33 Chart 36 Assistant Based Speech Recognition (ABSR) Developed by our partner Saarland University 21:30min

34 Chart 37 Assistant Based Speech Recognition (ABSR)

35 Chart 38 Assistant Based Speech Recognition (ABSR) Hypotheses Generator

36 Chart 39 Assistant Based Speech Recognition (ABSR) Hypotheses Generator Plausibility Checker

37 Chart 40 Assistant Based Speech Recognition (ABSR) Hypotheses Generator Plausibility Checker Command Monitor

38 Chart 41 Assistant Based Speech Recognition (ABSR) AMAN Hypotheses Generator Speech Recognizer Assistant Based Speech Recognition is much more than using a Speech Recognizer.

39 Chart 42 Command Recognition & Command Error Rates Nov./Dec Trials Recognition Rate Rejection Rate Error Rate With ABSR 95.2% 4.4% 1.7% 4 male DFS controllers, 2 female + 2 male ACG controllers Feb./Mar Trials Recognition Rate Rejection Rate Error Rate With ABSR 91.6% 8.4% 3.0% 3 male + 3 female DFS controllers,1 male ANS-CR controller; 3 male + 1 female ACG controllers

40 Chart 43 More Numbers Nov./Dec. 2015: AcListant -Strips Final Trials Without the Green Part of ABSR Command Recognition Rates were between 58% to 83% instead of 95% - Approx. 11,280 commands given wrong call signs per scenario of minutes duration - Context Error rate 0.9% Assistant Based Speech Recognition

41 Chart 44 Contents 1. The Motivation of an ANSP to use Speech Recognition. 2. The Experiment 3. Our quantitative results 4. Assistant Based Speech Recognition 5. ASR: What is achieved and where research is still needed?

42 Chart 45 ASR: What is achieved and where Research still is needed? Support of ATM Tools - Clearances usable as input (AMAN, SMAN) - Conformance Monitoring also with respect to security - Highlight of aircraft (just speaking to) - Checking clearances against conflicts - AcListant - GAMMA project - AcListant, Usability Aspects - AcListant, Integration into Conf Detection

43 Chart 46 ASR: What is achieved and where Research still is needed? - Bridge technology between radio transmissions and datalink (CPDLC) - Controlling all display functionality by Voice - Check of manual against voice input - Ideas, Concept? Roadmap? - Usability aspects, CBA - AcListant Pilot utterances - Consistency check of read backs - Pilot keyword triggers attention guidance ( wake, Go-Around, TCAS ) - Highlight of heavy aircraft from initial call - ASR challenge

44 Chart 47 Many possibilities for cooperation 95% recognition rate possible We have a business case Believe it or come to Braunschweig How we can adapt it to other applications areas? How to ease maintenance?

45 Chart 48 Learning of Models for Controller Assistance Reduces Implementation Costs Data Base with Radar Data - DFS Data Mining and Machine Learning Algorithms Improved Controller- Models Data Base with Voice Recordings Project Partners: DLR, UdS, Idiap, ACG, ANS CR

46 Chart 49 Summary Results of the Dusseldorf trials*** are with respect to: Airlines: 50 to 65 liters savings of kerosene per flight in medium traffic situations* Airports: Increase of flow by 1 to 2 landings per hour ANSPs: Reduction of controller workload needed for advisory input by a factor of 2 to 3 Reduced head down times increase safety Relevance for society: Saving of about 130 kg of CO 2 per flight** * A320, ** 0.8 kg / l, 1 kg kerosene results in 3.15 kg C02; 35 landings per hour *** extrapolation of results of 60 minutes scenarios for 23R, 8 controllers, see previous slides

47 50 > AcListant >Chart H. Helmke > 12th ATM Seminar > Seattle > to 29 DLR, DFS, ACG Supported by DLR Technology Marketing and Helmholtz Validation Fund Thank you very much for attention. Listening Participating in discussion and decision making