MEDITERRANEAN FREE FLIGHT PROGRAMME

MEDITERRANEAN FREE FLIGHT PROGRAMME ASMT TO INFORM MFF RISK ASSESSMENT DELIVERABLE 3 REPORT OF THE COMPARISON OF THE ASMT DATA COLLECTED FROM REAL TIME SIMULATIONS WITH MFF OHAS Final Edition 3.0 10/09/2004 Programme co-funded by European Commission DG-TREN

DOCUMENT IDENTIFICATION SHEET DOCUMENT DESCRIPTION DOCUMENT TITLE from Real Time Simulations with Mff OHAs (Hazard Analysis) DELIVERABLE REFERENCE NUMBER PROJECT REFERENCE NUMBER EDITION: 3.0 D3 EDITION DATE: 10/09/2004 Abstract This document presents the comparison of ASMT data collected during MFF RTS with MFF hazard analysis (OHA) of ASAS Spacing. Keywords ASMT ASAS Spacing Hazards Assessment Contact Person: Gordon Rachael Tel : (+33) 1 6988 7867 Organisation : EEC Email : rachael.gordon@eurocontrol.int Fax: (+33) 1 6988 7890 Address: Eurocontrol Research Centre (EEC) BP-15 91222 Bretigny-sur-Orge FRANCE DOCUMENT STATUS AND CLASSIFICATION STATUS Working draft Draft Proposed Issue Released Issue CLASSIFICATION Public Internal Confidential Restricted MDIR MREP TREP TSPEC SWC SYC 10 September 2004 Page 2 of 480

AUTHORS AND REVIEWING PROCESS ORGANISATION AUTHORS REVIEWING Deep Blue Simone Pozzi Marinella Leone Alberto Pasquini Eurocontrol --- Rachael Gordon DOCUMENT CHANGE RECORD The following table records the complete history of the successive editions of the document. ED. DATE REASON FOR CHANGE SECTIONS/PAGES AFFECTED 0.1 22-05-2004 Outline --- 0.2 25-06-2004 First Draft All 0.3 10-07-2004 Internal Review Second Draft Section 2, 4 & 5 1.0 26-07-2004 Released Draft All 2.0 31-08-2004 3.0 10-09-2004 Comments received by EEC and Deep Blue Released Draft Comments received by EEC Final Version Section 2 & 6 Minor editing 10 September 2004 Page 3 of 480

LIST OF PROJECT DELIVERABLES TITLE DATE NOTES D1 - Report about the ASMT data collected during Real Time Simulation and the possible methods and tools for the analysis and classification of the events D2 - Interim Report of analysis and comparison of ASMT data from Real Time Simulations with operational data D3 - Report of the comparison of the ASMT data collected from Real Time Simulations with MFF OHAs (hazard analysis) D4 - Final Report of the comparison of the ASMT data collected from Real Time Simulations with operational data 27 January 2004 22 April 2004 10 September 2004 06 September 200 D2 is the interim version of D4 D4 is the final version of D2 D5- Dissemination Workshop in EEC 17 September 2004 10 September 2004 Page 4 of 480

TABLE OF CONTENTS 1 GENERAL INFO 8 2 EXECUTIVE SUMMARY 9 3 OVERVIEW OF THE MAIN FINDINGS 10 3.1 Methodological Findings 10 3.1.1 Objectives of the analysis 10 3.1.2 The interrelation between the ASAS introduction and the ATM system 11 3.1.3 Limitations 12 3.2 Additional Information about Operational Failures 12 3.2.1 New Operational Failure 12 3.2.2 Scenario-Based Refinement of Operational Failure 13 4 MFF OPERATIONAL HAZARD ASSESSMENTS 15 4.1 OHA Methodology 15 5 ASMT ANALYSIS 22 5.1 RTS3b Methodology 22 5.2 RTS3b Findings 23 5.2.1 Workload Issues 24 5.2.2 Risk Homeostasis 24 5.2.3 Cognitive Clustering 24 5.2.4 Non-Intended Epistemic Use of the ASAS Procedure 25 5.2.5 Delegated a/c Monitoring 25 5.2.6 Traffic Structure 26 5.2.7 HMI Clutter 27 6 COMPARISON OF ASMT DATA WITH MFF OHA 28 6.1 Workload Issues 28 6.2 Risk Homeostasis 32 6.3 Cognitive Clustering 32 6.4 Non-Intended Epistemic Use of the ASAS Procedure 33 6.4.1 No determination of an appropriate traffic configuration (for all incoming traffic) - OF 1.1.1 & OF 1.1.1a 34 6.4.2 Erroneous spacing value determined/input by controllers - OF 1.1.2 & Selection of a pair of a/c that have incompatible speed/performance - OF1.1.2b 34 6.4.3 The required speed falls outside the flight envelope - OF 4.9.2 & The aircraft falls outside the flight envelope - OF 4.10.2 34 6.4.4 Discussion 35 6.4.5 Severity Comparison 35 6.4.6 Selection of a pair of a/c that have incompatible speed/performance - OF 1.1.2b 38 6.5 Delegated a/c monitoring 39 6.6 Traffic structure 40 6.7 HMI clutter 40 7 DISCUSSION OF THE OHA-ASMT COMPARISON 42 10 September 2004 Page 5 of 480

7.1 Different Methodological Approaches 42 7.2 The ATM system after the ASAS Introduction 42 7.3 Operational Failure Label Definition 43 7.4 Scenario-Based Refinement of OHA Labels 44 7.4.1 No determination of an appropriate traffic configuration (for all incoming traffic) - OF 1.1.1 & OF 1.1.1a 44 7.4.2 Erroneous spacing value determined/input by controllers - OF 1.1.2 45 7.4.3 Selection of a pair of a/c that have incompatible speed/performance - OF 1.1.2b 45 7.4.4 Erroneous positive check of the ASAS spacing applicability conditions for a sequence of a/c 45 7.4.5 Loss of ATC monitoring of one/all ASAS spacing instruction(s) - OF 4P.1 & OF 4P.2 45 7.4.6 Interruption of the ASAS manoeuvre by multiple flight crew - OF 5.1a.1 46 7.4.7 An aircraft is interfering with a pair of aircraft involved in an ASAS Spacing instruction - OF.P.2 46 8 CONCLUSIONS 47 REFERENCES 48 10 September 2004 Page 6 of 480

ABBREVIATION A/C Aircraft ACC ASMT ASOR ATC ATCo ATM ENAV FL Ft HMI MFF NM OF OHA R/T RTS Area Control Centre ATM Safety Monitoring Tool Allocation of Safety Objectives and Requirements Air Traffic Control Air Traffic Controller Air Traffic Management Ente Nazionale di Assistenza al Volo (Italian ATC Service Providers) Flight Level Feet Human Machine Interface Mediterranean Free Flight Nautical Mile Operational Failure Operational Hazard Assessment Radio/Telephony Real Time Simulation 10 September 2004 Page 7 of 480

1 GENERAL INFO This document presents progress report of ASMT to inform MFF risk assessment project. It is delivered by Deep Blue as D3 of contract C/1.057/CE/SS/02-TRS 078/2003. The objectives of the work are to corroborate the ASMT data collected in simulations with (i) event data collected in an operational environment and (ii) with MFF (OHA) hazard analysis. The first point has been addressed in Deliverable 4 [1], whereas this document will cover the comparison with MFF OHAs. This document consists of five sections. The first two provide a general overview of the report: - Executive summary; - Overview of the main findings; All the details of the analysis are presented in the last three sections: - MFF OHA: method of analysis, selected examples of findings; - ASMT analysis: method of analysis, summary of the main findings; - Comparison of ASMT data with MFF OHAs. This part consists of Section 6 where the full detailed comparison can be found, while a summary of the main findings and their discussion is reported in Section 7. 10 September 2004 Page 8 of 480

2 EXECUTIVE SUMMARY The objectives of this project are to corroborate MFF risk assessment with ASMT data collected in simulations and in the operational environment. This document presents (i) an overview of the methodology adopted in the MFF OHA, (ii) the main safety findings on ASAS impact obtained form the analysis of ASMT data, (iii) comparison of the ASMT findings with the OHA findings. Section 3 presents an overview of the main findings. Section 4 presents the method of analysis exploited for the MFF OHA and some selected examples of the findings. Section 5 includes a brief presentation of the findings obtained from ASMT data. It will describe the method of analysis and main findings. Findings about ASAS impact were all obtained from the MFF RTS3b on ASAS Spacing, since ASMT provided less relevant data in the other simulation sessions. Section 6 & 7 compare the findings of the ASMT analysis with the MFF OHA. The aim is to establish correspondences with the OHA findings and to complement them with new data from the ASMT analysis. Section 6 reports the full detailed comparison, while a summary of the main findings and their discussion can be found in Section 7. General conclusions are offered in Section 8. 10 September 2004 Page 9 of 480

3 OVERVIEW OF THE MAIN FINDINGS This section reports the main findings of the comparison between ASMT data collected in simulations and the MFF ASAS Spacing OHA [2]. Findings can be summarised in two categories: - methodological differences in the scope and objectives of the analyses; - additional information about Operational Failures. 3.1 Methodological Findings The comparison of OHA and ASMT analyses highlighted significant differences in the methodologies, scope and objectives. Nonetheless ASMT findings proved to constitute a complementary approach and to bring a different perspective to the OHA analysis. The different sources of information can account for the methodological difference. OHA started from the ASAS procedures phases and tried to list the operational failures. On the contrary, ASMT started from the set of events recorded in the RTS setting, with the aim of identifying causal factors and defining ASAS impact. The directions of analysis were almost opposite: OHA moved from the written procedure to specific events, whilst ASMT started from concrete events to obtain more abstract categories. The two analyses can be said to be complementary as ASMT point of view was more grounded in the complexity and subtleties of concrete cases, and proved effective to better articulate some of the results achieved by the OHA. 3.1.1 Objectives of the analysis The analyses aimed at two different objectives. OHA intends to provide an almost comprehensive list of operational hazards, without identifying possible causal factors. The identification of causal factors is instead to be conducted in the ASOR which is currently being undertaken. On the contrary, the ASMT analysis focused on the causal factors as a means of categorizing all the recorded events and to separate ASAS impact from RTS issues or non-asas factors. As a result, ASMT findings are not fully comparable with the OHA analysis. For instance in the ASMT analysis, workload issues (see Sections 5.2.1 and 6.1 for more details) was considered as a causal factor, as it was one of the causes of some losses of separation. In the terminology adopted in the OHA, workload caused some operational failures that lead to hazardous events. On the contrary, OHA considered workload only at the opposite side of the causal chain, that is workload was one of the hazards caused by an operational failure. In a similar way the risk 10 September 2004 Page 10 of 480

homeostasis (see Sections 5.2.2 and 6.2 for more details) could be considered as a causal factor of some OHA operation failures and hence was not addressed in the OHA. 3.1.2 The interrelation between the ASAS introduction and the ATM system The OHA analysis focused on failures that affect two a/c in any of the ASAS procedure phases. While this is coherent with the OHA scope and goal, it nevertheless fails to consider ASAS impact on the whole ATM system. The question of the interaction between ASAS a/c and the rest of the ATM system can be articulated in three interrelated points. - ASAS impact on the traffic: ASAS procedures will not only be a tool to be applied in a stable environment, but ASAS will contribute to radically change that environment. Bringing the argument to a paradox, ASAS procedures may even be fit for a non-asas environment, but rarely adaptive in an ASAS environment. - Chain of ASAS a/c: some hazardous interactions may be caused by the application of ASAS to multiple a/c, where some targets are also delegated in another ASAS pair. Those configurations were very frequent during RTS, and they significantly shaped traffic structure. On the contrary, OHA analysis main focus is on one single pair of a/c. Only two operational failures refer to a multiple ASAS chain, but are related to a single delegated. Operational failures at the level of the whole ASAS chain were not addressed. - Interactions between ASAS pairs and normal traffic: OHA analysed this point in OF.P.2 an aircraft is interfering with a pair of aircraft involved in an ASAS Spacing instruction. The case identified with ASMT (see Section 5.2.3) is not perfectly described by this operational failure. In more details, the OHA fails to reflect that the cognitive clustering can be considered an operational failure only in relation to a specific traffic situation. To address this type of event, the OHA should have also considered situations where the ASAS procedures were conducted correctly, but in traffic conditions that could cause hazardous interactions. This point represents one of the most problematic and widely discussed issue in safety analysis, that is where the boundaries of the system under analysis should be traced and which is the appropriate point of view. The present section do not intend to solve this complex issue, rather it highlights main drawbacks of some OHA assumptions. 10 September 2004 Page 11 of 480

3.1.3 Limitations Given these methodological differences, the comparison between the ASMT findings and the OHA required ad hoc reasoning for each specific case. The comparison methodology can thus be considered still to be refined, since it could not be completely standardised. An additional major limitation refers to the organising principles of the two analyses. Whilst ASAS phases are the organising criterion of the OHA, the ASMT cases are always identified by the consequences (i.e., a loss of separation). This observation refers to the analytical problem of distinguishing concrete events in clear-cut phases and referring ASMT cases to an unique phase of the ASAS procedure. But it also addresses the practical limitation that ASMT could not store data on the ASAS status. All the relevant information were gathered from controllers, and their feedback cannot be easily structured according to the different ASAS phases, as these phases can be said to mostly represent the ASAS designer s view and possess far less significance for operational people. To ensure a more consistent comparison, it is recommended to improve ASMT coordination with OHA criteria, for instance by recording whether a/c were involved in ASAS procedure and in which phase. 3.2 Additional Information about Operational Failures The main objective of the ASMT-OHA comparison was to find some additional information that could be used to inform the MFF safety assessment. This aim was achieved in two senses: - ASMT analysis identified one additional operational failure, that had not been considered in the OHA; - alternative interpretations for some OHA operational failures were defined, thus bringing implicit interpretations and assumptions to the front. This point also highlights important methodological issues. In this sense the comparison was successful as RTS data provided further operational failures, most of all by highlighting OHA interpretations that were too limited. 3.2.1 New Operational Failure OHA could include Erroneous positive check of the ASAS spacing applicability conditions for a sequence of a/c as an additional operational failure. This would ensure that at least one operational failure at the level of a chain of ASAS a/c is taken into account. This operational failure will affect phase 1 determination of the appropriate configuration and in particular the sub-phase 1.2 check 10 September 2004 Page 12 of 480

of the ASAS spacing applicability conditions. Operational failure ID would then be OF 1.2.2 (for details on this failure see Sections 6.2 & 6.4). 3.2.2 Scenario-Based Refinement of Operational Failure This point refers to additional information to complement OHA operational failures, but it also addresses a methodological issue. The OHA defines every operational failure with a textual label (see Tab. 1, page 18), which in some cases was too generic and even misleading. Broadly speaking, some of the operational failures were too underspecified and corresponded to multiple different events. Additional OF labels should be added to consider all the possible cases, most of all because otherwise the ASOR is likely to consider only one specific interpretation of that operational failure. For the same reason, the detailed definition of an operational failure (as reported in the detailed analysis, that is by considering also Examples of causes, Existing detection means, Fall-back actions after detection, Operational consequences & Recommendations) sometimes happened to coincide with only one particular point of view of that operational failure. This may be taken as an almost direct indication that the OHA analysis dealt with some operational failures as they could represent a whole category, but then introduced a specific interpretation and point of view when detailing the analysis, without keeping any trace of that assumption. Textual labels also caused a misleading resemblance with some ASMT findings, that proved to be incorrect at a deeper analysis, since the OHA adopted a different interpretation. From a methodological point of view this should question whether simple textual labels are appropriate means to define (and report/communicate) an operational failure. For instance, the severity of some operational failures could be defined only by considering the specific chain of event. Thus single operational failures appeared too abstract from a context (in this case defined by previous events) on the basis of the textual labels and one specific interpretation had to be assumed to perform the detailed analysis. A comparison of the ASMT findings with the ASOR could then be recommended to determine whether operational failures as defined in the OHA could effectively inform the ASOR. As a result, the comparison highlighted whether the ASMT and the OHA analyses were referring to a different interpretation of operational failures, and brought hidden interpretations to the front. These alternative interpretations has been defined with the help of scenarios, which could represent the situation involving the operational failure from a broader/different perspective. Operational 10 September 2004 Page 13 of 480

failures for which an alternative interpretation was defined are listed below (see Section 7.4 for more details): - OF 1.1.1 & OF 1.1.1a - No determination of an appropriate traffic configuration (for all incoming traffic) - OF 1.1.2 - Erroneous spacing value determined/input by controllers - OF 1.1.2b - Selection of a pair of a/c that have incompatible speed/performance - OF 4P.1 & OF 4P.2 - Loss of ATC monitoring of one/all ASAS spacing instruction(s) - OF 5.1a.1 - Interruption of the ASAS manoeuvre by multiple flight crew - OF.P.2 - An aircraft is interfering with a pair of aircraft involved in an ASAS Spacing instruction 10 September 2004 Page 14 of 480

4 MFF OPERATIONAL HAZARD ASSESSMENTS This section will include a brief presentation of the MFF OHAs. It will describe the analysis methodology and some selected examples of the findings. This document will consider only the ASAS Spacing OHA [2], because ASMT provided valid data only for RTS3b, where only ASAS spacing application was simulated. Other OHAs were reviewed only to verify methodological consistency. No significant differences were found. 4.1 OHA Methodology This section will present the analysis methodology adopted in the MFF OHAs, in particular in the ASAS spacing OHA. All the methodological information are taken from Section Applicable approach in MFF programme ([2,] pag. 8-9). Some selected findings are also reported for explicative purposes. The ASAS spacing application has been decomposed temporally in phases. Each phase groups a number of underlying operations, which in turn are composed by actions (lowest level of decomposition). A phase is defined as a group of operations contributing to a high level objective [ibidem]. An operation is defined as a set of actions that fulfils one or more of the following criteria: - an operation is triggered by the result of an evaluation or by an event - an operation is characterised by one end to end communication exchange between two domains - an operation has a specific operational goal (for the activity of the ASAS spacing procedure) [ibidem]. An action is the lowest breakdown level for the application chosen for the OHA. An action is performed by one single Domain [i.e., ground or air]. An action is supported by a function. [ibidem] Phases identified for the ASAS spacing are (see Figure 1): 1. Determination of the appropriate configuration 2. Target designation and identification 3. Delegation instruction 10 September 2004 Page 15 of 480

Fig. 1. Phases of the ASAS spacing procedure. 4. Implementation of the instruction 5. Normal end of the ASAS spacing instruction In parallel to phase 4: sector transfer In parallel to all the phases: issue of compatible ATC clearances/instructions with ASAS spacing instructions. The OHA analysis proceeded by articulating each phase in operations and by extracting operational failures at this level. For each operation, possible modes of failure correspond to operations erroneously or not executed, or performed with delay. The MFF OHA did not address the lower action level, as functions supporting actions are not yet defined. Once an operational failure has been identified the analysis looks for existing detection and fallback means. Then severity is assessed by analysing consequences (ESARR4 matrix [3] provides the relation between consequences and severity) and by considering if mitigation means are in place. If the direct effect of the operational failure is not immediate, a pointer to a future operational failure is identified. 10 September 2004 Page 16 of 480

An operational hazard is defined (consistently to EUROCAE ED-78A [4]) as the ultimate consequence of each operational failure, and it provides the rationale for assigning a severity level to an operational failure. Then, according to ESARR4 matrix, the operational hazard could be - Increasing workload of the air traffic controller - Increasing workload of the aircraft flight crew - Slightly degrading the functional capability of the enabling CNS system - Minor reduction in separation with flight crew or ATC controlling the situation and fully able to recover from the situation - Minor reduction in separation without flight crew or ATC controlling the situation and hence jeopardising the ability to recover from the situation - Major reduction in separation with flight crew or ATC controlling the situation and fully able to recover from the situation - Major reduction in separation without flight crew or ATC controlling the situation and hence jeopardising the ability to recover from the situation One or more catastrophic accidents ([2], pag. 8-9). In the MFF safety assessment methodology the OHA is intended to list the hazardous events that can affect the ASAS procedures. In practical terms, this means that OHA does not intend to address the causal factors, nor detail the causal path leading from causes to hazards. After the OHA, an ASOR is currently being developed and will identify the basic building events for each OHA hazardous events, by reconstructing in a backward sense the causal path. The identified basic events should be on the causal factors level. For instance the third phase ( delegation instruction ) was articulated in three operations: 1. specific instruction of the applicable operation to perform 2. entry of the ASAS parameters in the HMI 3. check of the procedure feasibility Operational failures for each operation are then described and consequences assessed. Operation 1 ( specific instruction of the applicable operation to perform ) presents 8 different modes of failures. A limited selection of them is presented in Tab. 1 below. 10 September 2004 Page 17 of 480

Tab. 1. Operational failures and severity levels (with and without mitigation means). OF ID. Operational Failures Severity with mitigation means Severity without mitigation means OF 3.1.1 Lack of the delegation If the procedure The controller does not send the message message continues 5 AND does not detect it, no impact on safety The controller thinks the message has reached the aircraft AND s/he does not realise the lack of read-back by the flight crew leading to OF 4.1.3, OF 4.5.3, OF 4.9.3 & OF 4.10.3 OF 3.1.2 OF 3.1.3 OF 3.1.3a Wrong delegated call sign in the delegation instruction message (issued by the controller on misunderstood by pilots) Erroneous specific instruction issued ((heading then) merge instead of (heading then) remain, and vice versa) No heading provided in the specific instruction received in the flight deck 5 Input to : An unexpected aircraft performs a heading change (OF 4.1.1a/OF 4.5.1a) Inappropriate speed adjustments by an unexpected aircraft (OF 4.9.1b) the ATCO may think that the correct delegated is going to carry out the manoeuvre, whereas it may be not: OF 4.1.3, OF 4.5.3, OF 4.9.3 & OF 4.10.3 for this A/C 5 OF 3.2.5 5 OF 3.2.6a 10 September 2004 Page 18 of 480

OF ID. Operational Failures Severity with mitigation means Severity without mitigation means OF 3.1.6 OF 3.1.8. Erroneous Spacing value in the instruction message Delayed delegation message 5 OF 3.2.2 / OF 3.2.3 4 / 5 N/A The above table summarises analysis findings about each operational failure. More detailed hazard analysis tables are included in the Annex A of OHA [2]. Detailed analysis of OF 3.1.3a is represented in Tab. 2. Tab. 2. Detailed analysis of operational failure 3.1.3a. PHASE OF OPERATION: PHASE REF: Delegation instruction P3 ACTION: Specific instruction of the applicable operation to perform OPERATIONAL FAILURE: No heading provided in the specific instruction received in the flight deck EXAMPLES OF CAUSES: ACTION REF: P3.1 OPERATIONAL FAILURE REF: OF3.1.3a CAUSES REF.: Air traffic controller (mistake in the message transmission) Flight crew (misunderstanding in the message reception) Voice communications interference Erroneous traffic configuration determination (ATC determines no heading is necessary while current a/c performance does not allow speed reduction) OF1.1.2a 10 September 2004 Page 19 of 480

Existing detection means 1. Aircraft 2. ATC 3. Airspace Fall-back actions after detection 1. Aircraft 2. ATC 3. Airspace 1 None Operational Consequences 1. Aircraft 2. ATC 3. Airspace SREQ/PROC 04 The flight crew shall read-back to the air traffic controller safety-related parts of the ATC clearances and instructions which are transmitted by voice. SREQ/PROC 05 the controller shall listen to the read-back to ascertain that the clearance or instruction has been correctly acknowledged by the flight crew. The controller can detect there is an error in the message read-back when s/he receives as it does not contain the heading read-back (unless the error is due to an erroneous manoeuvre assessed on ground) 2 None 1. SREQ/PROC 06 If the read-back message is incorrect the controller shall take immediate action to correct any discrepancies revealed by read-back. 2. None Detected: If the procedure continues the only effect is a slight delay in the confirmation message and procedure initialisation, but there is not any direct effect on safety provided applicability conditions are still met as the application is not time critical If the procedure is cancelled: 1 The aircraft is controlled by current practices. No effect on safety but a slight increase in pilot s workload 2 Increase in controller s workload as s/he has to control the aircraft conventionally 3 No effect on safety Undetected: - The pilot will not select a heading while it is necessary Ref. Severity 5 5 OF3.2.6a 10 September 2004 Page 20 of 480

Related Operational Hazard Detected: If the procedure continues: no effect If cancelled: no effect otherwise Undetected: leading to OF3.2.6a Recommendations Severity Airborne side SREC/TEC 27 The airborne In order to allow an erroneous 5 system shall take into account the positive feasibility check is own a/c performance to allow efficient feasibility check. performed and the pilot confirms ASAS spacing instruction A key part of any safety process is the assessment of operational consequences. Unfortunately the OHA document does not report explicitly on the main sources of that assessment, apart from stating that hazards were obtained from two separate lists, i.e. from RTS2 results and from OHA validation workshop, where it is known that operational experts, ASAS expert, safety experts and controllers were involved. Given domain common practices, it can be assumed that consequences (and severity) were derived on the basis of subjective ratings of safety and operational experts. 10 September 2004 Page 21 of 480

5 ASMT ANALYSIS This section will include a brief presentation of the ASMT analysis. It will describe the method of analysis and main findings. This document will consider only the MFF RTS3b, since ASMT provided less relevant data in the other simulation sessions. Further details on the ASMT analyses performed in the MFF project and the full details of the RTS3b analysis can be found in ASMT Deliverable 4 Final report of the comparison of the ASMT data collected from Real Time Simulations with operational data [1]. 5.1 RTS3b Methodology The initial plan for the analysis was to thoroughly interview controllers on each loss of separation to identify main causal factors, after controllers had taken part in normal HF and safety debriefings. This plan was refined after two simulation days, due to: - the larger than expected number of events (22 in first 2 days) were unsuitable for individual controller debriefing on each ASMT event. Time available in the RTS schedule for dedicated ASMT debriefings was only enough for one single event for each simulation run. - separate ASMT debriefings appeared to re-enforce an unfruitful blame-culture. Controllers that were asked twice (the former by HF/safety observers, the latter in front of ASMT) about the same loss of separation sometimes were defensive because they perceived the analyst was emphasising their errors. Obtaining relevant information about the event afterwards proved almost impossible. An indirect indicator of this is the ASMT console informally named by controllers the hits position. The data gathering and analysis process was then restructured as follows: - ASMT was monitored during exercises (almost in real time) to detect relevant events. - Information on losses of separation that happened in sectors observed by HF or safety observers were gathered during standard post-exercise debriefings, where ASMT was not involved at all. ASMT was used as a support only in case replay was needed. - Relevant events that happened in non-observed sectors were reviewed by the analyst to identify the most interesting ones. Debriefings with involved controllers were then performed during that day or on the next one. Some of these debriefings were performed without ASMT support, while in most of the cases the replay function was used to refresh 10 September 2004 Page 22 of 480

controllers memory and to stimulate further discussion. In addition it was also possible to replay pilots input and traffic status by using the ESCAPE platform replay function. This process ensured a smoother integration of ASMT in the RTS environment, hence profiting from controllers commitment and complementing traditional observational methods. Another method refinement was made at the end of the first two simulation weeks (when 68 events had been collected). At this point some recurring event typologies had emerged (8 in total, defined on the basis of phases of flight and conflict types; e.g., a/c in approach phase, bad estimate on parallel routes, same direction ), whereas a more in-depth analysis on single events had become too situation specific, as it could not provide further information on the general causes. Thus during the last week of simulation, the analysis of causal categories of ASMT events was revised, but most of the time was dedicated to prepare wide-scope debriefings on event typologies. A limited set of significant events for each category was shown to controllers in individual interviews, prompting them to provide general explanation that could account for the whole set. Thus event categories were validated and more information acquired on general causes. In comparison with safety analyses conducted during the previous RTSs, this enabled the collection of more general feedback and information. Furthermore this feedback proved crucial to distinguish between simulation issues, ASAS impact and other causal factors, since one of the main drawbacks of debriefing sessions on one event is that controllers often focus on the most apparent problems, which are likely to be those caused by the specific RTS setting. It should be emphasised that the approach remained a bottom-up one, that is the analysis started from specific events to achieve a higher level of abstraction. A total number of 102 relevant losses of separation were collected and analysed during RTS3b. 5.2 RTS3b Findings ASAS impact on ATM during RTS3b can be summarised in the 7 items listed below: 1. Workload issues 2. Risk homeostasis 3. Cognitive clustering 4. Non-intended use of the ASAS procedure 5. Delegated a/c monitoring 6. Traffic structure 7. HMI clutter 10 September 2004 Page 23 of 480

These items are presented as separated only for analytical clarity, whilst they should be intended as interacting and mutually reinforcing. None of them was sufficient by itself to cause an incident during RTS, but their joint effect could significantly diminish the reliability of the ATM service. Their interactions and the full detailed analysis can be found in ASMT D4 [1]. 5.2.1 Workload Issues In many occasions controllers subjectively assessed a substantial increase of workload due to ASAS procedures. In addition ASMT events showed typical instances of workload induced effects, as reduced monitoring on peripheral sector areas, fixation of the attentive focus on key points, monitoring failures of non-expected information (e.g., controllers failed to detect non adequate rate of descent). Workload induced effects appeared to concentrate outside of the main traffic fluxes. 5.2.2 Risk Homeostasis Controllers may take riskier decisions because of the additional support provided by ASAS. In other words, controllers were observed to erode the perceived additional safety margins provided by ASAS to maximise productivity. 1 Even if this behaviour may be strongly related to the simulation setting, it brings into light the interaction between new procedures/tools and the rest of the ATM system. Safety analyses often focus only on the main changes introduced, assuming that the rest of the system will remain relatively stable. The theory of risk homeostasis instead predicts that ASAS induced failures might impact on traffic density conditions higher than expected, more complex than those normally managed by controllers in non-asas settings. 5.2.3 Cognitive Clustering ASAS procedures are expected to reduce controllers workload by linking together a/c with compatible performances, destinations, routes This also applies from a cognitive point of view, in the sense that cognitive resources use is optimised by grouping in a single cluster the various elements to be considered. Unfortunately the same mechanism can result in the oversight of relevant characteristics of one of the cluster members. Once clustered, items are hard to separate and single members peculiarities may be overlooked as they are not common to other cluster members. For instance, in the event depicted in Figure 2, the delegated a/c was considered at the same FL of the target a/c, even if it was still climbing to that FL. In other words, once the controller had established the ASAS configuration and cleared the delegated a/c for the target a/c FL, he 1 The theory of risk homeostasis is drawn from the automotive field, where it indicates modifications of drivers behaviour after the introduction of supporting technologies (e.g., braking systems, anti-collision alarms ). See Wilde [5]. 10 September 2004 Page 24 of 480

Fig. 2. RTS3b event 39: ELG was mistakenly considered at the same FL of the target a/c. started considering the two a/c as if they were on the same FL, hence forgetting about the temporary discrepancy between them. 5.2.4 Non-Intended Epistemic Use of the ASAS Procedure Rather than deciding the a/c sequence and then using ASAS tools and procedures to correctly implement it, controllers were observed to issue ASAS clearances to obtain information from pilots about the feasibility of a sequence. In that case the ASAS procedure was used as a probe by controllers, and a decision was taken depending on pilot s reply. This practice shifts from a pragmatic use of the ASAS procedure to an epistemic one [6]. It should be better clarified whether ASAS algorithm can be used for this purpose without any safety relevant side effects. For instance this use proved critical for the algorithm implemented in pseudo-pilots consoles during RTS, where ASAS clearances were firstly accepted, then rejected at a very late stage, where controllers had limited possibilities to recover. A likely explanation is that ASAS algorithms could not cope with the interactions between all the a/c in the ASAS chain and their mutual speed adjustments. 5.2.5 Delegated a/c Monitoring ASAS manoeuvre (i.e., the trajectory of an ASAS delegated a/c) is not part of the controller s traffic picture. Hence controllers could not effectively monitor the delegated a/c manoeuvres, since they 10 September 2004 Page 25 of 480

could not correctly project the delegated a/c trajectory. Even in cases when they were aware that the separation to the target was less than the prescribed one, they never acted as they did not know exactly which manoeuvre was going to be conducted: maybe the delegated is going to slow down significantly in the last miles. In some cases they just asked the pilot to confirm ASAS feasibility, but in case of positive answer they did not act. The delegation was usually cancelled only at a very late stage, when the situation was already compromised. This finding seriously questions the use of controller s monitoring as mitigation means. 5.2.6 Traffic Structure The traffic structure in the RTS setting was significantly different from the real operational traffic (see ASMT D4 [1] for more details on the comparison between RTS and operational data). The RTS traffic tended to fly on the same FL with trails of a/c on the same route, whereas in an operational environment traffic is distributed on various FLs and more FL changes can be observed. At least four reasons may account for the different traffic geometry: - RTS team request to have trail of a/c on the same FL. This request is due to the ASAS procedure characteristics, as some of the ASAS benefits are expected to come from smoother profiles of descent. On the contrary, nowadays descents are usually performed with trails of a/c on different FLs. One a/c is cleared to descend only when the preceding a/c has left the lower FL; - RTS sectors were simulating one arrival sector and the sectors just preceding it. Most of the traffic was then to be sequenced for landing, resulting in trails of a/c on the same route; - Controllers apply personal preferences in managing the air traffic. Some of them may prefer using vertical separations, whilst others exploit the horizontal distance. In the ACC where the ASMT experimental live use took place, controllers are known to work mostly with vertical separations, while RTS controllers were from different ACCs and may have managed traffic using less FL changes; - In the real operational environment, controllers receive a considerable number of requests of FL change from pilots, since fuel consumption is minimised at higher FLs. It can be concluded that traffic structures were sharply different in the two environments, with noticeable consequences on the controllers activity: different tasks, action/error opportunities and constraints. For instance, some traffic situations that appeared to be error prone in RTS (as trails of 10 September 2004 Page 26 of 480

a/c on the same FL) were not faced (and/or deliberately avoided) by controllers in the operational setting. 5.2.7 HMI Clutter Controllers reported that ASAS links could significantly increase visual information displayed on the radar screen, thus negatively affecting clarity. Combined with a label overlap problems in high density areas (a very relevant problem with the RTS HMI), this was found to contribute significantly to less efficient monitoring, as visual information discrimination tasks were more difficult to do. Noticeably, HMI overload worsened the effects of the above mentioned workload increase and cognitive clustering, as exemplified by their interaction in two events: - one single a/c was involved in three serious conflicts, because its unexpected low rate of descent was not visually detected by controllers; - a/c was wrongly perceived and considered as delegated in on-going ASAS delegation, which was instead flying in the opposite direction. 10 September 2004 Page 27 of 480

6 COMPARISON OF ASMT DATA WITH MFF OHA This section will compare the findings of the ASMT analysis with the MFF OHA. The aim is to establish correspondences with the OHA findings and to complement them with new data from the ASMT analysis. This section reports the full detailed comparison, while a summary of the main findings and their discussion can be found in Section 7. 6.1 Workload Issues Workload is considered as a hazard in the OHA, consistently with ESARR4 [3] and EUROCAE ED-78A [4]. This means that workload is one of the ultimate consequences of operational failures. Operational failures that can lead to a workload increase are those marked by a severity rating of 4 (see Tab. 3). Tab. 3. Operational Failures where workload was listed as a potential consequence. Operational Failures No determination of an appropriate traffic configuration for all incoming traffic Selection of a pair of aircraft that have inappropriate trajectories Selection of a pair of a/c that have incompatible speed/performance Erroneous determination of the equipment of the pair of aircraft Erroneous negative check of the ASAS Spacing applicability conditions Misdirected target identification message Erroneous target A/C ID in the target identification message Lack of target A/C selection on the airborne HMI Erroneous target A/C selection on the airborne HMI An erroneous delegated aircraft selects a target on the airborne HMI The target is not displayed on board Display of an erroneous target on board. Erroneous target aircraft position display on board Display of a target in an erroneous delegated A/C Erroneous negative check of the target A/C ID Erroneous negative check of the target aircraft position in case position information was issued by the controller OF ID. OF1.1.1a OF1.1.2a OF1.1.2b OF1.1.2b-1 OF1.2.1 OF2.1.2 OF2.1.3 OF2.2.1 OF2.2.2 OF2.2.2a OF2.3.1 OF2.3.2 OF2.3.2a OF2.3.2b OF2.4.1 OF2.5.3 10 September 2004 Page 28 of 480

Operational Failures Lack of target identification confirmation message by delegated A/C Erroneous position information in the target identification confirmation Erroneous delegated A/C call sign in the target identification confirmation message Delayed target identification confirmation Erroneous negative check of target A/C position on ground Lack of delegation instruction message Wrong delegated call sign in the delegation instruction message or another delegated A/C takes the instruction Erroneous specific instruction issued (heading then) merge instead of (heading then) remain (and viceversa). No heading provided in the specific instruction received in the flight deck Erroneous WPT in the delegation instruction message for until WPT instruction or merge behind, or heading then follow proceeding direct to WPT, which is different from the one selected by the target a/c Erroneous heading instruction in the delegation instruction message Erroneous spacing value in the instruction message Lack of additional information in the specific instruction message (until further advice until WPT or at least Delayed delegation message Lack of entry of the ASAS parameters on the HMI spacing selected by flight crew lower than the cleared one Spacing value selected greater than the cleared one Erroneous WPT introduced on the HMI Erroneous application selection on the HMI Erroneous heading introduced on the HMI No heading introduced on the HMI Feasibility check performed with an erroneous target selected on-board the delegated aircraft Feasibility check performed while target parameters are not correct Erroneous negative feasibility check Loss of ATC monitoring of all ASAS spacing instructions (given WPT, limit of the manoeuvre, spacing value, merging or remain.) Issuance of incompatible clearances for a pair of aircraft involved in an ASAS instruction (climb, descend, or reduce speed ) during P4 An aircraft is interfering with a pair of aircraft involved in an ASAS Spacing instruction OF ID. OF2.6.1 OF2.6.4 OF2.6.5 OF2.6.6 OF2.7.1 OF3.1.1 OF3.1.2 OF 3.1.3 OF3.1.3a OF3.1.4 OF3.1.5 OF3.1.6 OF3.1.7 OF3.1.8 OF3.2.1 OF3.2.2 OF3.2.3 OF3.2.4 OF3.2.5 OF3.2.6 OF3.2.6a OF3.3.1 OF3.3.2 OF3.3.3 OF4P.2 OF.P.1 OF.P.2 10 September 2004 Page 29 of 480

Operational Failures Loss of delegated flight crew communication capabilities during an ASAS spacing instruction execution Loss of target flight crew communication capabilities during an ASAS spacing instruction execution R/T failure affecting a part of airspace loss of normal radar display in the ATCU centre Total loss of normal and fall-back radar displays on the ground Erroneous heading change execution Early Flight Crew report ASAS spacing at the merging point, is erroneously maintained by the delegated flight crew in case an exactly spacing was provided (ASAS spacing greater selected on-board than the value selected on the ground or erroneous Spacing value provided by ATC) ASAS spacing at the merging point, is erroneous maintained by the delegated flight crew in case an exactly spacing was provided (ASAS spacing value selected on-board is smaller than value selected on the ground or erroneous Spacing value provided by ATC) Inappropriate speed adjustments by an unexpected aircraft the required speed falls fall outside the flight envelope Lack of ASAS spacing monitoring on-board during Merge Behind operation Unexpected movement of the target during Merge Behind operation (trajectory deviation or speed variation) Merge behind at an erroneous WPT The two aircraft of the delegation under different A/C categories/performance during Follow operation (the target aircraft is larger than the delegated aircraft) Unexpected movement of the target during In-Trail operation No reception by ATC of the Flight Crew s report at the required WPT or of the unable delegation. No Flight Crew s report at the required WPT. Interruption of the manoeuvre for/by multiple flight crews Erroneous or no update of the ASAS spacing value in the ATC instruction message Provision of the Cancel ASAS spacing instruction at an inappropriate WPT by ATC No reception in the flight deck of the instruction message ending up the ASAS spacing operation No de-activation of the delegation by the controllers OF ID. OF.P.3 OF.P.4 OF.P.5 OF.P.6 OF.P.7 OF4.1.1 OF4.5.1 OF4.4.2 OF4.8.2 OF4.9.1 OF4.9.1a OF4.9.1b OF4.9.2 OF4.9.3 OF4.9.4 OF4.9.5 OF4.9.6 OF4.10.4 OF5.1.2 OF5.1.2a OF5.1a.1 OF5.2.1 OF5.2.3 OF5.2.4 OF5.3.2 10 September 2004 Page 30 of 480

Operational Failures ASAS instruction transfer with inconsistent ASAS spacing maintained between the delegated and the target aircraft: no respect of the receiving sector s separation minima or loss of spacing Reception of ASAS instruction including erroneous target or delegated aircraft during sector transfer No co-ordination exchange Erroneous frequency transfer transmitted to one aircraft of the pair (target or delegated) Erroneous ASAS spacing value in the Flight Crew report to next sector Report from an unexpected delegated flight crew to next sector the target is required to change SSR code during a sector transfer OF ID. OFp.1.1 OFp.1.2 OFp.1.3 OFp.2.1 OFp.3.2 OFp.3.3 OFp.3.4 A significant difference exists between the ASMT analysis and the OHA. Workload is considered as a consequence of operational failures in OHA, while the ASMT analysis identified workload as a causal factor of operational failures (e.g., loss of monitoring of non-asas a/c ). This sharp difference probably results from the different methodological approaches: - The OHA analysis focused on identifying the operational failures that could directly affect the ASAS procedure. In other words, operational failures were analysed inside the framework of the ASAS procedure phases. On the contrary, ASMT analysis was conducted in a RTS setting and the whole simulated ATM system was considered, thus including the side-effects of ASAS on other parts of the system. This does not necessarily entail that analysis conducted in a RTS setting are more comprehensive, as one major limitation is that a systemic view risks to evaluate the simulation setting rather than the procedure. For instance, workload issues were found to affect particularly those geographical areas where some operational procedures had been modified for simulation reasons. Distinguishing between ASAS impact and RTS factors is then of primary importance, but often requires extensive work to an accurate understanding. - The OHA analysis is performed before functions and tasks are allocated to the system s components, whether humans or hardware ones. Operational failures were analysed as if they had not yet been allocated. Hence from a strict theoretical point of view, workload effects on operators could not be assessed. As a result no operational failure identified in the OHA corresponds directly to ASMT workload issues, nor workload is considered as a causal factor. 10 September 2004 Page 31 of 480

6.2 Risk Homeostasis Risk homeostasis mainly affects the first of the OHA phases, that is determination of the appropriate configuration (see Figure 1), as controllers may use the ASAS procedure extensively to maximise the throughput of their sector. No corresponding operational failures could be found among those analysed in the OHA, as all of them refer to errors in determining the appropriate configurations. Instead risk homeostasis addresses ASAS configurations that are correct in theory, but that result in a more complex airspace and increased controllers activity. Risk homeostasis may be combined with non-intended use of the ASAS procedure to obtain a failure comparable with the OHA ones. For instance it may be named erroneous positive check of the ASAS spacing applicability conditions for a sequence of a/c. Probably bringing the argument to a limit, risk homeostasis questions the validity of the assumption that ASAS procedures will be exploited with air traffic situations comparable to today s ones. ASAS procedures will not only be a tool to be applied in a stable environment, but ASAS will also contribute to radically change that environment. This consideration is probably even more important in case of unexpected interruption of the manoeuvre by multiple flight crews (OF 5.1a.1, phase end of the ASAS spacing instruction ) as controllers may be faced by an abrupt workload increase in a high density traffic airspace. In conclusion risk homeostasis may well reinforce the OHA recommendation ([2], see Section 6) to better specify fall-back actions for the interruption of the manoeuvre by multiple flight crews. 6.3 Cognitive Clustering This factor is related to compatible ATC clearances/instructions issued during ASAS spacing (parallel to phase 4, while the ASAS is being implemented). Operational failure identified in the OHA is OF.P.2 An aircraft is interfering with a pair of aircraft involved in an ASAS Spacing instruction. The relation between cognitive clustering and the operational failure appears to be the cause-effect one, in the sense that cognitive clustering may cause controller s error and inappropriate clearance to a third a/c. In fact controller human error is also listed as one of the possible causes of OF.P.2 in the OHA. The OHA operational failure appears to represent the common form that many causal factors may take and hence cannot be used to identify one specific causal factor. The a/c interference discussed in OF.P.2 can originate from many causes, of which the cognitive clustering represents only one. In the same way it cannot be ruled out a priori that cognitive clustering may be the cause of other failures, maybe directly affecting the ASAS pair. Thus no straightforward comparison between 10 September 2004 Page 32 of 480