Toulouse A-SMGCS verification and validation results

Transcription

1 Contract No. TREN/04/FP6AE/SI / Toulouse A-SMGCS verification and validation results MONGENIE Olivier & PAUL Stéphane DSNA & Thales ATM Document No: D6.4.1 Version No. 1.0 Classification: Public Number of pages: 115 Project Funded by European Commission, DG TREN The Sixth Framework Programme Strengthening the competitiveness Contract No. TREN/04/FP6AE/SI / Project Manager M. Röder Deutsches Zentrum für Luft und Raumfahrt Lilienthalplatz 7, D Braunschweig, Germany Phone: +49 (0) , Fax: +49 (0) Web page: , - All rights reserved - EMMA Project Partners The reproduction, distribution and utilization of this document as well as the communication of its contents to other without explicit authorization is prohibited. This document and the information contained herein is the property of Deutsches Zentrum für Luft- und Raumfahrt and the EMMA project partners. Offenders will be held liable for the payment of damages. All rights reserved in the event of the grant of a patent, utility model or design. The results and findings described in this document have been elaborated under a contract awarded by the European Commission.

2 Distribution list Member Type No. Name POC Distributed 1 Web Internet Intranet 1 DLR Joern Jakobi 2 AENA Mario Parra Martínez 3 AIF Marianne Moller 4 SELEX Giuliano D'Auria 5 ANS_CR Miroslav Tykal 6 BAES Stephen Broatch 7 STAR Max Koerte 8 DSNA Thierry Laurent 9 ENAV Antonio Nuzzo 10 NLR Luc de Nijs 11 PAS Alan Gilbert Contractor 12 TATM Stéphane Paul 13 THAV Alain Tabard 14 AHA David Gleave 15 AUEB Konstantinos G. Zografos 16 CSL Libor Kurzweil 17 DAV Rolf Schroeder 18 DFS Klaus-Ruediger Täglich 19 EEC Stéphane Dubuisson 20 ERA Jan Hrabanek 21 ETG Thomas Wittig 22 MD Phil Mccarthy 23 SICTA Claudio Vaccaro 24 TUD Christoph Vernaleken Sub-Contractor CSA Karel Muendel N.N. Customer EC Morten Jensen Additional EUROCONTROL Paul Adamson 1 Please insert an X, when the PoC of a company receives this document. Do not use the date of issue! Save date: Public Page 2

3 Document control sheet Project Manager Michael Roeder Responsible Authors MONGENIE Olivier, PAUL Stéphane DSNA, TATM Additional Authors SAINI Luca TATM MARCOU Nicolas DSNA MONTEBELLO Philippe DSNA Subject / Title of Document: Related Task('s): WP6.4 Deliverable No. D6.4.1 Save Date of File: Document Version: 1.0 Reference / File Name D641_Results-TLS_V1.0.doc Number of Pages 115 Dissemination Level Public Target Date Change control list Date Release Changed Items/Chapters Comment Initial draft Comments by Holger Neufeldt and by Marianne Moller and Update of Airbus analysis by Marianne Moller New 2.4 Contribution by Holger Neufeldt New 2.5 Contribution by Luca Saini and Paolo Gervasoni New Answer of FAA on RTCA issue New 2.6 Temporary internal results, to be updated Parallel D641 document created by DSNA for validation results. Initial draft validation results by DSNA. Release v0.01 is delivered to TATM for review New 2.7 Time stamp issue The TATM editorial comments to the parallel D641 document created by DSNA, release v0.01, are implemented. The results of the shadow-mode trials are completed. A fast-time simulations chapter is added. By DSNA. Release v0.02 is delivered to TATM for review Editorial and content improvement. By DSNA. Release v0.03 is Save date: Public Page 3

4 Integrated parallel D641 document created by DSNA for validation results in its release v0.03. Transfer of the analysis tables for the shadow-mode validation trials in an annex. Editorial improvement. Check of acronym list. Suppression of figure 15. New verification tests, re-organisation of chapters and consolidation with V&V test plan Small comment about MOGADOR. Editorial improvements. delivered to DLR for review and to TATM for integration of the verification results. Submission to DSNA and Luca Saini for review. Submission to DLR for general assembly review and/or submission to EC Update of all the document by DSNA following EC review Update of all the document by Luca Saini following EC review Processing of remaining comments from EC by Philippe Montebello and Stéphane Paul Approval by EC. Submission by DSNA to TATM. Save date: Public Page 4

5 Table of contents Distribution list... 2 Document control sheet... 3 Change control list... 3 Table of contents Scope Identification Project overview Document overview Purpose Intended audience Document structure Relationships to other EMMA documents Verification Mosquito position precision System configuration Results On-the-fly analysis Conclusion RTCA DO-260A vol-1 bug on ADS-B surface position reporting ADS-B ground station coverage Results On-the-fly analysis Unstable track positions for some aircraft Results On-the-fly analysis Conclusion MAGS preliminary verification results System configuration Result of step (e) system calibration Result of step (f) localisation of test transmitter Results of step (g) localisation of real targets Surveillance latency and position precision First test: latency & position verification with a vehicle Second test: latency verification with an aircraft Third test: dispersion verification On-the-fly analysis of the 3 tests Conclusion ADS-B stations synchronisation: time drift test The test set-up On-the-fly analysis Detailed analysis and issue solving Conclusion and more (interesting) results ADS-B latency test Test of the probability of detection of a stationary vehicle System configuration Findings ADS-B cost-benefit test System configuration Findings Consolidation of verification results Surveillance Control Save date: Public Page 5

6 Human machine interface Validation session shadow mode trials Introduction Data description and data collection methods Raw data Additional data Data analysis Data analysis method Analysis tables for the shadow-mode validation trials Results Operational feasibility: acceptance of technical performances Operational feasibility: procedures Operational improvements Validation session fast-time simulations Introduction Data description and data collection methods Raw data Additional data Data analysis Data analysis method Measured indicators Results Operational improvements Digest and conclusions Annexes Annex A: analysis tables for the shadow-mode validation trials Operational feasibility: acceptance of technical performances Operational feasibility: procedures Operational improvements Annex B: references Applicable documents Referenced documents Annex C: abbreviations Save date: Public Page 6

7 1 Scope 1.1 Identification This document provides the. Document Name: EMMA No.: D6.4.1 Revision: 0.13 File Name: d641v013.doc 1.2 Project overview The project is named European Airport Movement Management by A-SMGCS with the acronym EMMA. The duration of the project is 2 years, with a follow-up in EMMA-2 (another 2 years). The project is organised in six different sub-projects. There are three ground-related sub-projects and one onboard-related sub-project. Based on an advanced operational concept, three functional level III advanced surface movement, guidance and control systems (A-SMGCS) will be implemented at three European airports: Prague-Ruzynĕ, Toulouse-Blagnac and Milano-Malpensa. The systems are to be tested operationally (i.e. with live traffic). The three ground-related sub-projects and the onboardrelated sub-project are autonomous, but are inter-linked with the sub-projects concept and validation to guarantee that the different systems are based on a common A-SMGCS interoperable air-ground co-operation concept and that all are validated with the same criteria. On-site long-term trials (i.e. 6 months to one year) are to ensure the assessment of benefit estimations. The results of the test phase shall feed back to the concept of operations, and are intended to set standards for future implementation in terms of: (a) common operational procedures, (b) common technical and operational system performance, (c) common safety requirements, and (d) common standards of interoperability with other ATM systems. These standards shall feed the relevant documents of international organisations involved in the specification of A-SMGCS, i.e. mainly ICAO, EUROCAE, and EUROCONTROL. This document is produced in the scope of work package 6.4 entitled "Verification and Validation at Toulouse". The activities performed in this work package 6.4 include: preparation of the infrastructure at Toulouse-Blagnac airport for verification and validation; technical tests (i.e. verification) of the A-SMGCS installed in Toulouse-Blagnac; training of Toulouse-Blagnac controllers to the A-SMGCS; shadow-mode trials with trained controllers on the A-SMGCS HMI installed in the Toulouse- Blagnac tower to evaluate the operational feasibility of the A-SMGCS and assess the operational improvements brought by the system; fast-time simulations of several traffic and environment conditions in Toulouse-Blagnac to quantify the operational improvements potentially brought by the A-SMGCS. 1.3 Document overview Purpose The ICAO manual [15] on Advanced Surface Movement Guidance and Control Systems (A-SMGCS) and the EUROCAE WG-41 MASPS [16] for A-SMGCS contain operational and performance requirements that are considered to be necessary in the process of selection, development and introduction of A-SMGCS. These manuals have been defined for those aerodromes where current SMGCS needs to be upgraded, or for aerodromes which currently have no SMGCS, but where the traffic density and/or aerodrome layout requires so. The objective of this document is to report on the collected data aimed at supporting the assessment of the surveillance, surface conflict alerting and routing functions as implemented in Toulouse-Blagnac Save date: Public Page 7

8 against the aforementioned ICAO and EUROCAE requirements, in conformance to [3], [4], [7] and [14]. Due to some delays in the implementation of the Toulouse-Blagnac A-SMGCS test bed (cf. [2]) it was not possible to run all the tests foreseen in [5]. The corresponding implications for the overall project plan, as well as the plans foreseen by the project, in general, to cope with such a non-conformance are described within each test description. However, many tests were run, and some significant results were obtained, as documented within Intended audience The dissemination of this document is public Document structure Chapter 1 defines the scope of this document, the intended audience, and the document structure. Chapter 2 presents the verification results. Verification is testing against predefined technical specifications, i.e. technical functional testing: "did we build the system right?" Each section of this chapter presents a short description of the experiment set-up, an overview of the raw data that has been gathered, and different stages of analysis by the different partners involved. Chapter 3 and 4 present the validation results obtained respectively through shadow-mode trials and fast-time simulations. Validation is testing against operational requirements as defined by stakeholders and written down in [6], i.e. "did we build the right system?" The chapters describe the data, data collection methods, data analysis and results measured in terms of operational feasibility and operational improvements. Chapter 5 provides the conclusions drawn from the V&V activities in Toulouse-Blagnac airport. Chapter 6 provides details on referenced documents and some definitions. 1.4 Relationships to other EMMA documents Apart from the EMMA documents already referenced above, the author would like to bring the reader's attention to: similar result collection reports for Prague-Ruzynĕ [8] and Milano-Malpensa airports [9], a similar result collection report for airborne systems [10], the verification and validation analysis report [11], the verification and validation recommendation report [12]. Public EMMA documents can be found on Save date: Public Page 8

9 2 Verification Due to delays in the A-SMGCS implementation at Toulouse-Blagnac, it was not possible to thoroughly follow the V&V test plan, as documented in D6.1.3 [5]. However, some of the verification tests provided below in 2.1 to 2.9 and the site acceptance tests performed for each of the A-SMGCS sub-system elements (cf. D4.1.1, D4.12, D4.1.3, D4.2.1, D4.3.1, D4.4.1 and D4.4.2) allow us to take a stand on a number of technical hypotheses, as described below in 2.10, in particular where the surveillance, control and human-machine interface hypotheses are concerned. The D6.1.3 Toulouse-Blagnac V&V Test Plan [5] planned that the MOGADOR tool would be used in order to assess detection and identification indicators on a long-term basis from EMMA A-SMGCS recorded data. The recorded EMMA A-SMGCS SDF data (i.e. one week sample recorded in February 2006) was input into MOGADOR. However, the results obtained were abnormally bad and a very low number of tracks could be actually analysed. This is explained by the important number of lack of identifier and the poor continuity of the correlation function (i.e. MOGADOR uses the flight call sign as a key element to associate and identify tracks) and the need for tuning of MOGADOR to Toulouse-Blagnac platform that would necessitate additional time and effort. Consequently, since the results of the MOGADOR analysis do not reflect the reality, it was decided not to publish them in this document. The impossibility to perform a long-term analysis has a significant impact on the verification phase. In particular, the detection and identification performance of EMMA A-SMGCS will not be fully verified in Toulouse-Blagnac. Therefore, an additional campaign of measures will have to be performed as a preliminary to EMMA2 validation activity. 2.1 Mosquito position precision In June 2005, Laurent Volkmann (DSNA) calculated the A-SMGCS reported position accuracy (RPA), by comparing the Mosquito reported position to the D-GPS reported position System configuration For the test, the following set-up was prepared: full automatic dependant surveillance broadcast (ADS-B) system installed, up and running; ADS-B ground station (GS) configured to emit ASTERIX reports as soon as they receive 1090ES message updates from ADS-B targets; Mosquito installed on a vehicle, up and running; the same vehicle mounted with a D-GPS independent receiver, used as a reference; ELVIRA recording system connected to the system LAN Results When both Mosquito and D-GPS track trajectories are superimposed on the ELVIRA display (cf. Figure 1), the reported position accuracy (RPA) appeared to be very good, but the calculation gave bad results because of the ADS-B GS time stamp. See picture below On-the-fly analysis Fact 1: The D-GPS has an update rate of 5 per second. The GPS inside Mosquito has an update rate of 1 per second. When the mobile is moving, it is assumed that the position sent by the GPS inside Mosquito is an average position over the last second. This introduces some latency. Save date: Public Page 9

10 Fact 2: Mosquito does not time stamp as no time stamp can be sent in ADS-B position messages. The first time stamp is performed during the ADS-B ASTERIX message formatting by the ADS-B ground station and reflects the ASTERIX message emission date. Fact 3: During the tests and in accordance with the ADS-B standard, each Mosquito/GPS position data was sent twice per second (with 0.5s interval), whilst the odd and even coding/decoding of the position data differed. This explains on Figure 1: Mosquito RPA on ELVIRA display the presentation of Mosquito positions as tight-couples (i.e. in fact the same data) separated by 0.5 seconds. Fact 4: In June 2005, the Mosquito equipment did not perform any position extrapolation to compensate the latency Conclusion Thales ATM implemented a position prediction algorithm in order to extrapolate the 1 st message position and the 2 nd message position by different factors. The position prediction algorithm was derived from the ADS-B standard (RTCA-260A). Figure 1: Mosquito RPA on ELVIRA display RTCA DO-260A vol-1 bug on ADS-B surface position reporting Thales ATM implemented inside the Mosquito vehicle ADS-B transmitter an algorithm suggested by RTCA D0-260A for GPS position prediction by ADS-B transmitting mobiles. This algorithm aims at compensating some latency effects that can affect the reported position accuracy. During the implementation activities, Luca Saini and Paolo Gervasoni identified a bug on a formula in the RTCA D0-260A volume 1 document. The bug details are reported below. Document reference: Minimum operational performance standards for 1090 MHz extended squitter, automatic dependant surveillance broadcast (ADS-B) and traffic information service - broadcast (TIS- B), RTCA DO-260A volume 1, 10th April 2003, Section: : ADS-B Surface Position Message, Save date: Public Page 10

11 Sub-sections: and Surface Longitude Position, Extrapolation/Estimation, precision and non-precision case. In the above mentioned sections there is twice reported a prediction formula of a longitude value in order to compensate for the time latency between the time of last received update position fix and the time of the applicability of the ADS-B surface position message. In such a formula the variable called "approximate longitude" is defined as the "longitude at the time of the fix". The mistake is found in that this must be the latitude at the time of the fix and not the longitude. Note that the corresponding formulas, as reported in the sections for the airborne position messages longitude extrapolation/estimation are correctly typed. Two simple examples can describe some effects of the mistake. Case 1: a target moving along the equator in the E-W direction at a constant velocity. In such a case it's expected to obtain a constant longitude predicted delta for the same delta time. Applying the formula would result in delta longitude predicted values that strongly depend on the target longitude position. In world areas close to plus or minus 90 this delta becomes infinite! Case 2: a target is moving in world areas with longitude > + 90 or < In such a case the formula is predicting a negative deltas of longitude (i.e. longitude decreasing) when the target is moving towards east (longitude increasing) and vice-versa. RTCA and EUROCAE were contacted to communicate the finding. The following message was received from FAA, acknowledging our finding Original Message----- From: gary.ctr.furr@faa.gov [mailto:gary.ctr.furr@faa.gov] Sent: Thursday, January 19, :11 PM To: [ ] Subject: Re: DO-260A bug on ADS-B Surface Position Reporting Gentlemen, After a review of the problem that was initially reported below, we agree that there is a typographical error in the text of RTCA/DO-260A in the two places indicated for Surface Position reporting in the "Commentary" sections of and , where longitude is indicated in the definition of the symbol "Phi" when in fact the latitude should have been indicated. I deeply regret that these errors were entered into the DO-260A document, and cannot understand how a simple cut-andpaste from DO-260 to DO-260A could have occurred, but we are pleased that the error has been found and can be easily corrected. To the end of making the correction to these, and other errors identified in DO-260A, I have added the corrected text for these two errors into the proposed "Change 1 to DO-260A" document that was initially reviewed during Meeting #19 of the RTCA SC-186 Working Group 3 (WG3) in December 2005 as Working Paper 1090-WP After discussions of how the proposed changes to both DO- 260 and DO-260A might effect the 1090ES SARPs Technical Manual, during the ICAO SCRSP TSG meeting in Fort Lauderdale 2-10 February 2006, WG3 will finalize and submit "Change 1" documents for both DO-260 and DO-260A to RTCA SC-186 membership for their Final Review and Comment period, which ends with the SC-186 Plenary meeting in Washington on 20 April, where WG3 expects that both "Change 1" documents will be approved for publication on the RTCA web site. [ ] Gary Furr L-3 Communications / The Titan Group FAA Technical Center 2.2 ADS-B ground station coverage In June 2005, Luca Saini performed some ADS-B ground station output recordings. At that time, the equipment configuration was as follows: two ADS-B ground stations, one at the GBAS location and one at the top of the new tower. Save date: Public Page 11

12 2.2.1 Results The main results were as follows: Both ADS-B ground stations showed good surveillance coverage relating to aircraft; The new tower ADS-B ground station showed good performance, with a few systematic holes (probably due to reflections), concerning vehicle tracking with Mosquito; The GBAS ADS-B ground station showed very poor coverage performance concerning vehicle tracking with Mosquito On-the-fly analysis The poor performance of the GBAS ground station relating to vehicle tracking is difficult to understand. Indeed the GBAS ground station is more powerful and has an additional low noise amplifier compared to the new tower ADS-B ground station. It is currently thought that the low height of the antenna makes it more subject to reflection problems. 2.3 Unstable track positions for some aircraft In June 2005, Luca Saini performed some ADS-B ground station output recordings. At the time, the equipment configuration was as follows: two ADS-B ground stations, one at the GBAS location and one at the top of the new tower Results A surprising result was the unstable track positions reported for some aircraft, mainly Air France aircraft, whilst the EMMA Mosquito track showed very stable trajectories, as well as some other aircraft (EasyJet, Airbus A380, etc.) see figures below. Figure 2: Mosquito (stable) vehicle tracking Save date: Public Page 12

13 Figure 3: Unstable aircraft tracks during pushback Figure 4: Unstable aircraft tracks during taxi On-the-fly analysis For aircraft with unstable target reports, there seemed to be a systematic jump between odd and even target position reports. This is thought to be related to the encoding of the positions, using different algorithms for odd and even reports. But why is this phenomenon related only to some aircraft? Comments by Thales ATM Regarding the accuracy of aircraft, it is to be noted that within the ADS-B ASTERIX cat 21 message, the ground stations forward a Figure of Merit / Position Accuracy indicator (FOM/PA). The higher this number, the better the accuracy. A FOM/PA of 7 or above is good, 6 or less is poor. The reason Save date: Public Page 13

14 for poor quality is usually the current GPS reception quality or constellation as perceived by the onboard GPS receiver. Some older types do not deliver adequate position quality. It is also to be noted that there is a number of other misbehaving equipment sets flying around (including sudden jumps backwards along track, jumps to the north pole or another distant position etc.) These are known issues in some manufacturers equipment that are caused by, for example: onboard equipment that whilst outputting ADS-B formats is not certified for ADS-B use, a problem in the internal time synchronisation function of the transponder. When analysing ADS-B reports one must therefore be extremely careful in describing what actually happened. ADS-B is just a data link mechanism. The positions are measured by avionics and the ADS- B receiver depends on this information (hence the name "dependant surveillance") Comments by Airbus The above results were provided to Airbus, who gave the following preliminary analysis. Air France has not yet fully equipped all its airliners with the package enabling GPS data transmission via ADS- B. In such a case, the resolution of the position transmitted via ADS-B can lead to jumps around the aircraft trajectory. The observed jumps should progressively disappear 2 whilst Air France completes the installation Conclusion Thales ATM decided to suppress all ADS-B tracks whose Figure of Merit / Position Accuracy indicator (FOM/PA) was strictly lower than 7. In addition, this result is interesting on a safety assessment point of view. The observed behaviour is linked to an intermediate configuration in the frame of the ADS-B deployment. It would be interesting to know if an on-board equipment failure of the final configuration might lead to a similar fallback behaviour. If so, it might be interesting to analyse it in future releases of the EMMA functional hazard assessment [13]. 2.4 MAGS preliminary verification results An important part of the EMMA installation in Toulouse is the MAGS multilateration. Multilateration is a complex technology whose implementation on an actual airport requires a great deal of tuning. For the installation of MAGS at Toulouse-Blagnac airport within the frame of the EMMA programme, the initial tuning phase has yielded some preliminary verification results that are presented in this section. The system installation and tuning consists of the following steps: (a) installation of equipment; (b) setting up of communication links; (c) tuning of ground station sensitivity for the intended coverage area; (d) verification of probability of detection of ground stations; (e) configuration of calibration mechanism to establish a common time base for all ground stations; (f) parameter tuning to locate the fixed test transmitter; (g) parameter tuning to locate targets; (h) definition of areas and their properties; (i) optimisation of plot validation and tracking parameters; (j) integration with sensor data fusion. The results presented correspond to steps (e), (f) and (g). For this purpose after an initial on-site tuning session - data were recorded from the MAGS and taken to the lab. These data were then replayed in real time to the reference system in the lab and its parameters were configured to obtain the best possible performance. The following sections present some intermediate results System configuration For the MAGS system in Toulouse, a set of five ADS-B/MLAT ground stations were installed as depicted in Figure 5. One ground station was additionally equipped with a calibrator module to 2 If it were not the case, a new analysis would be required. Save date: Public Page 14

15 provide system synchronisation, and another ground station was additionally equipped with an interrogator module to allow interrogation of aircraft. A fixed test transmitter based on a Mosquito ADS-B transmitter was installed as a permanent, fixed test target. GSR (PC) GSC (TWR) 1 GSR (Old TWR) 5 GSR (GBAS) GSI (SMR) Test Transmitter (DF) Figure 5: MAGS Configuration at Toulouse Blagnac Result of step (e) system calibration Calibration results are best illustrated by locating the calibrator transmitter using multilateration. The result is depicted in Figure 6. The filled red square indicates the ground station calibrator (GSC) position, while the outlined red squares are 10m, 20m, and 30m, respectively, around the GSC position. Figure 6: Measured position of MAGS calibrator station (GSC) at control tower The GSC position measured by multilateration is shown as sequence of target plots on the MAGS technical display. Due to ASTERIX cat-10 quantisation, the position resolution is limited to 1m (i.e. distance between adjacent plots). All position reports are within 3 metres of the exact GSC position. Save date: Public Page 15

16 2.4.3 Result of step (f) localisation of test transmitter Figure 7 shows the resulting plot sequence of locating the fixed test transmitter (TT) by multilateration. Again the minimum distance between adjacent plots is determined by the ASTERIX position resolution of 1m. All position reports are within 4 metres of the exact TT position. Figure 7: Measured position sequence of fixed test transmitter Results of step (g) localisation of real targets The following two screenshots in Figure 8 and Figure 9 show real targets detected via multilateration together with the GSC and the test transmitter. Track continuity was not ensured. Indeed, no tracker was yet configured (as this is a task within the definition of area properties). Therefore tracks were neither smoothed nor fully regular in their update rate. It should be noted that, at that time, MAGS did not interrogate aircraft, so that multilateration relies purely on passive reception. Active interrogation of targets can be used to enhance update rates and to fill gaps. Save date: Public Page 16

17 Figure 8: Targets, GSC (label 3C3D0C ) and test transmitter (label 3C3D0D ) detected by MAGS Missing detections can be caused by coverage gaps. It is clear that five ground stations in this configuration are not ideal in terms of redundancy. For example, if one ground station (e.g. the one installed at the SMR, cf. Figure 5) does not contribute to a plot data set, the geometrical constellation is not good enough for parts of the coverage area. Figure 9: Targets, GSC (label 3C3D0C ) and test transmitter (label 3C3D0D ) detected by MAGS The next screenshot, in Figure 10, shows one target that taxies onto the apron area. It should be noted that no tracker was yet configured to smoothen the track and to provide regular updates. Save date: Public Page 17

18 Figure 10: One target picked out 2.5 Surveillance latency and position precision On Thursday 23 rd February 2006, a DSNA/DTI team (headed by Mr. Philippe Soleilhac) performed a surveillance latency and reported position accuracy verification session. The test scope included the vehicle tracking system (Mosquito), the automatic dependant surveillance broadcast sensor (AS-680) and the sensor data fusion (SDF) First test: latency & position verification with a vehicle The verification was performed using the latest Mosquito equipment delivered week 7 of year Tests were performed by driving a Mosquito-equipped car on service roads. The car was detected only by the automatic dependant surveillance broadcast (ADS-B) sensor and the surface movement radar (because the multilateration system was shut down). A very precise differential global positioning system (D-GPS) was used as time and position reference. Save date: Public Page 18

19 Figure 11: Mosquito latency & precision verification test at ADS-B sensor output The comparison between the D-GPS reports and the Mosquito reports as output directly by the ADS-B sensor located at the new control tower (cf. Figure 11) showed a 1.2s latency and a position error of 12 metres. Save date: Public Page 19

20 Figure 12: Mosquito latency & precision verification test at sensor data fusion output The comparison between the differential global positioning system (D-GPS) reports and the Mosquito reports as output by the sensor data fusion (cf. Figure 12) showed a 2s latency and a position error of 6 metres Second test: latency verification with an aircraft This test was performed with a flying aircraft. The approach secondary surveillance radar (called DACOTA) was used as reference system. Save date: Public Page 20

21 Figure 13: Air surveillance latency & precision verification test The comparison between the DACOTA target reports and the EMMA target reports as output by the ADS-B sensor located at the new control tower and the sensor data fusion showed a 0.3s latency Third test: dispersion verification The EMMA sensor data fusion (SDF) introduced some target report dispersion, which is not originating from any sensor. Save date: Public Page 21

22 Figure 14: Dispersion test with flying aircraft On-the-fly analysis of the 3 tests The offset of 12 m between the ADS-B output trajectory of Mosquito and the D-GPS recording is due to a bug inside the Mosquito software. The latency of 1.2 seconds is affected by the same error. A new software release has been produced. The "improvement" performed by the SDF output on the trajectory (from 12 m to 6 m) is only due to the fact that the SMR output is much more precise than the ADS-B output for Mosquito. Fixing the Mosquito problem should allow to reduce this offset. The crazy behaviour of the SDF output for the flying aircraft has an explanation and is due to two probable contributions: 1) The SDF does not fuse the ADS-B output (which is very good in this case) because the ADS-B emitting transponder does not deliver a figure of merit better than 6 (i.e. 7 or 8 or 9). 2) The EMMA SDF, as far as the DACOTA GTW tracker (alfa, beta and gamma filter) is concerned, is configured in such a way that aircraft entering pre-defined geometrical areas (cones) associated to each RWY are kept with a stronger weight on the current direction, at the time the aircraft enters the cone. This setting is performed in order, for a landing airborne aircraft, to "follow" the RWY centreline. The case described above was a test flight entering the cone in a perpendicular direction from what could be expected. As this setting is also creating problems in case aircraft land in "baïonnette" mode, the tracker's filter values were changed. The new proposed values are not giving any "preferred" direction on the tracker Conclusion In case a test aircraft will be used for EMMA verification, it should be better that this aircraft is delivering a good Figure of Merit, otherwise the SDF will not fuse the ADS-B output. Save date: Public Page 22

23 2.6 ADS-B stations synchronisation: time drift test On Thursday 9 th March 2006, a DSNA/DTI team (headed by M Philippe Soleilhac) performed timestamp tests related to Mosquito and ADS-B stations. With Mosquito, the position is elaborated onboard the vehicle (using the GPS), but the time is stamped by the ADS-B ground station. The ADS-B standard does not allow for the transmission of the time together with the position The test set-up Tests were performed by driving a D-GPS and Mosquito-equipped car on service roads. Positioning was performed via the D-GPS, the ADS-B ground stations and the mode S multilateration. GPS 1090ES ADS-B Precision GPS RX Vehicle TX GPS Position & Time SMR OTWR PCBLD NTWR GBAS Asterix 21 Asterix 21 Asterix 62 ADS-B GTW multilateration ELVIRA Asterix 10 NTP Synchro SDF Asterix 62 SAF Clock Figure 15: Position and timestamp test set-up ASTERIX-21 messages are time-stamped by ELVIRA with a tenth of a second accuracy. The following table shows timestamp discrepancies between the ADS-B ground stations. ADS-B ground station Time in the ASTERIX messages stamped by the ADS-B stations Time stamped by ELVIRA upon message reception New TWR PC SMR Old TWR Table 2-1: Timestamp measurements Times in the ASTERIX-21 messages, stamped by the ADS-B stations, are in advance, with a dramatic 20 seconds for the ADS-B ground station located at the new control tower. Time in the ASTERIX-62 messages, stamped by the SDF, is also in advance, but with a smaller amount (between 0.7s and 1s). Save date: Public Page 23

24 2.6.2 On-the-fly analysis The ADS-B ground stations are normally synchronised, between themselves and with the real time, using NTP. It seems that the implementation of NTP in this case was not effective, with a detrimental effect on latency compensation (at ADS-B gateway level) and position accuracy Detailed analysis and issue solving According to Luca Saini, the poor latency performances are essentially related to three sub-systems: Mosquito; the ADS-B GS; the SDF (at ADS-B GTW level). With respect to Mosquito, the main problem is due to the fact that the internal processing cycle is not synchronised with the GPS receiver output. TATM is working on this feature. However, to solve definitively the full latency problem, it is required to introduce additional features that at least compensate each known delay. Specific requirements have been added to the software requirements specification (SRS). See below for more details. With respect to the ADS-B GS, the main problem is the synchronisation using NTP (SDF being the master). After investigating the network time protocol (NTP) effect, it was discovered that the issue was in fact a human error: the NTP server was not reachable on the network. The NTP service, initially located on the ADS-B gateway PC had been transferred to the SDF PC. Thus, the NTP server IP address had changed. The ground stations were re-configured to synchronise with the current NTP server. With respect to the ADS-B GTW, one additional source of latency has been identified, i.e. the ADS-B GTW stamped anew the ASTERIX messages without making any change on the reported position. The proposed correction consists in forwarding the original time stamp given by the ADS-B ground station to the SDF and letting the SDF extrapolate the position at the time of SDF output as done for any other sensor track Conclusion and more (interesting) results Beyond the aforementioned test, which reflects human error more than system performance, one should not discard the results so easily. First, it is to be noted that the built-in test equipment did not make it evident, neither to the end-user nor to technical staff, that the ground stations were not synchronised. This is a typical case of slightly corrupted surveillance data, which unnoticed, results in a severe "misuse of surveillance data" hazard (cf. [13]). Secondly, the aforementioned test results provide us with the opportunity to validate another two requirements, which were not initially in our test plan: When not synchronised on external time signal, the automatic dependant surveillance broadcast (ADS-B) ground subsystem time shall have a maximum drift of 4.4 sec/day. When not synchronised on the time reference system, the system time shall have a maximum drift of 20 ms per day. 2.7 ADS-B latency test On Wednesday 5 th April 2006, Holger Neufeldt, Volker Seidelmann and Bernd Doleschal tested the internal latency of ADS-B processing under real load conditions at Toulouse-Blagnac. They found that the total latency between the reception of a radio frequency (RF) signal and the output of the corresponding ADS-B report on the network (in ADS-B pipeline mode) was between 6 and 35 ms. 2.8 Test of the probability of detection of a stationary vehicle The objective of this verification test was to measure the probability of detection of a stationary vehicle at the output of the following components: MLAT; Save date: Public Page 24

25 ADS-B GTW; SDF. This test was performed in January 2006 on the fixed test transmitter located at the direction finder shelter System configuration The initial equipment conditions were as follows: full ADS-B system installed, up and running; ADS-B system synchronised with the airport time reference; Mosquito unit, installed within the direction finder shelter, up and running; ELVIRA recording system connected to the system LAN. In a stationary condition the Mosquito emits the following 1090ES messages with associated emission rates: surface position: once every 5 seconds; identity and type: once every 10 seconds. The ADS-B ground stations emit ASTERIX CAT 021 target reports as soon as they receive updates from ADS-B targets. The ADS-B GTW (only for stationary targets) adapts the received input rate from ground stations to a one second output rate (ASTERIX CAT 062) towards the SDF. The SDF emits target reports (ASTERIX CAT 062) towards the CWP at one-second output-rate Findings The results are provided in Table 2-2. Toulouse ADS-B MLAT Test trasmitter System Unit MLAT ADS-B GTW SDF Total measure time sec Total detected events 1 sec Detection Probability % 99,75 100,00 100,00 X-pos average m -456,7-479,0-478,5 Y-pos average m -478,3-470,4-470,4 X-pos stdev m 0,85 0,27 0,27 Y-pos stdev m 1,25 0,77 0,77 2D average deviation m 1,30 0,63 0,63 2D standard deviation m 1,51 0,82 0,82 2D deviation 50%-tile m 1,31 0,37 0,37 2D deviation 95%-tile m 2,81 1,63 1,63 2D deviation 99%-tile m 3,78 2,01 2,00 2D deviation 99.99%-tile m 5,74 2,01 2,00 2D max deviation m 5,76 2,01 2,00 Table 2-2: Test transmitter measured probability of detection The detection probability at the output of the sensor data fusion is 100%. It has to be noted that the Toulouse-Blagnac implementation of the ADS-B sub-system (made of 5 ADS-B ground stations) is over-redundant. Please refer to 2.9 for more details. Note: the reported accuracies in Table 2-2 are referred to the average measured position of the test transmitter GPS receiver. They are not referred to any other external reference and therefore they represent more an indication of the position sample dispersions rather than an absolute accuracy measure. 2.9 ADS-B cost-benefit test The objective of this verification test was to assess the number of ADS-B ground stations required at Toulouse-Blagnac to provide adequate vehicle surveillance coverage and probability of detection. Save date: Public Page 25

26 2.9.1 System configuration The test was performed by measuring the probability of detection of moving vehicles and aircraft along taxiways and runways centrelines (and runways borders) at the output of the following components: each ADS-B ground station; ADS-B GTW; SDF. This test was performed in January The initial equipment conditions were as follows: full ADS-B system installed, up and running; ADS-B system synchronised with the airport time reference; Mosquito units installed, up and running; some real life traffic; ELVIRA recording system connected to the system LAN. Considering the Mosquito emission rate, for a recording period of N seconds of each Mosquito equipped target and 100% of detection probability the following maximum detected events can be obtained: each ground station: 2*N+N/5; ADS-B GTW: N; SDF: N Findings The overall probability of detection at SDF output was 100%. The 5 ground stations installed at Toulouse-Blagnac define an over redundant system, but one single ground station is not enough to have a complete and reliable coverage. For a single ground station, the detection probability (with exception of the one located at GBAS shelter) ranges from 70 to 90%, depending on the airport area. Table 2-3: Detection probability for triples and couples of ADS-B ground stations As shown in Table 2-3, a combination of 3 ground stations provides an adequate 99.88% probability of detection of moving vehicles and aircraft along taxiways and runways centrelines (and runways borders). Combined with other sensors (e.g. SMR), this performance is sufficient to reach a 100% probability of detection at the output of the sensor data fusion Consolidation of verification results In the tables below, the hypothesis numbers and hypothesis titles as documented in the Toulouse- Blagnac verification and validation test plan (D6.1.3) are recalled, followed by the verification result. The remaining hypotheses will be verified in the scope of EMMA-2, when the proposed system improvements will have been implemented. Please refer to the verification and validation results analysis document (D6.7.1) for more details on the proposed improvements Surveillance Save date: Public Page 26

27 Hypothesis Hypothesis title N Ver.sur.2 Coverage Verified. Verification result Ver.sur.3 Target position accuracy Partially verified, cf. D6.7.1 for an in-depth analysis. At the current date (i.e. March 2007), the vehicle position accuracy issue has been solved and the test can be considered "verified". Ver.sur.5 Update rate Verified. Ver.sur.6 Latency Requirements not met, cf. D6.7.1 for an in-depth analysis. At the current date (i.e. March 2007), the vehicle position latency issue has been solved and the test can be considered "verified". Ver.sur.7 Efficiency Partially verified, cf. D6.7.1 for an in-depth analysis. At the current date (i.e. March 2007), the efficiency issue has been solved and the test can be considered "verified" Control Hypothesis Hypothesis Verification result N title Ver.con.1 Capability Verified as part of D4.2.1 Site acceptance procedure for the Surface Conflict Alert (SCA) system at Toulouse airport. Ver.con.2 Boundary Verified as part of D Ver.con.5 Latency Verified as part of D4.2.1 to range between 0 ms and 2 ms (depending on the number of conflicts) Human machine interface Hypothesis N Hypothesis title Verification result Ver.hmi.1 Surveillance Verified. Ver.hmi.2 Control Verified. Ver.hmi.3 Update capability Verified. Save date: Public Page 27

28 3 Validation session shadow mode trials 3.1 Introduction The validation session subject of the current chapter aims at assessing both the operational feasibility of A-SMGCS in Toulouse-Blagnac airport and the operational improvements brought by this system. Although the operational feasibility and operational improvements are defined as two different stages of the validation activities, they have been evaluated during the same validation session. The validation session consisted in a series of shadow-mode trials performed with controllers of Toulouse-Blagnac airport. They were asked to fill in a questionnaire while assessing the A-SMGCS HMI installed in the tower. The questionnaire was derived from the one developed by the DLR for the Prague test site and adapted to Toulouse-Blagnac local specificities. It addressed two aspects of the operational feasibility (i.e. the acceptance of technical performance and the procedures) and the operational improvements. For each topic, a series of statements were presented to the reader who was asked to point out his / her level of agreement over 6 possibilities ranging from strongly disagree to strongly agree. All the controllers of Toulouse-Blagnac airport (i.e. 65) have followed a 45 minutes training course on the SMGCS HMI installed in the tower for operational trials. Since the installation of the SMGCS in September 2005, the controllers had many opportunities to use and evaluate the HMI during operations. The A-SMGCS HMI has the same look-and-feel as the SMGCS HMI. However, in order to inform the controllers working at Toulouse-Blagnac about the differences between both systems, 44 of them have followed a one-hour briefing about the A-SMGCS between the end of January and the beginning of February Then a questionnaire was distributed to all the controllers. They were asked to fill it in by themselves whenever they wanted and, if possible, in front of the A-SMGCS HMI in the tower. Some of them were also asked to participate to interviews during shadow-mode trials on a voluntary basis. The interviews took place in front of the A-SMGCS HMI in the tower and lasted 1h30 on average. The questionnaire was filled in with each controller individually. In all cases, emphasis was put on the importance of collecting controllers feedback and remarks in order to better understand their answers. The shadow-mode trials took place between the beginning of February and mid March Data description and data collection methods Raw data Thirty-five questionnaires in paper format and in French have been collected, representing more than 50 percent of the 65 controllers in Toulouse-Blagnac. Amongst them, 17 were filled in during interviews with controllers in 4 different sessions and 18 controllers filled them in on their own. The initial version of the questionnaire, used for Prague test site validation activities, had to be adapted to the trial constraints and local particularities. Unlike the Prague airport A-SMGCS, Toulouse A- SMGCS was not used in operations. The questionnaire was adapted to take this into account. A human factor expert reviewed the questionnaire in order to delete some minor inconsistencies and redundancies. However, in order to allow comparison between the different test sites, it was decided to minimize the differences between the questionnaires. Three different versions of the questionnaire have been used during shadow-mode trials: The first version (V003) was too long (116 statements). After the first questionnaires had been filled in, it appeared that some questions were not relevant for Toulouse-Blagnac or were redundant, and therefore needed to be deleted or modified. That version was not distributed to all the controllers and only 6 V003 questionnaires were filled in, both during interviews and directly by controllers. Save date: Public Page 28