RESULTS OF THE PD/1++ TRIAL

Transcription

1 RESULTS OF THE PD/1++ TRIAL APPENDIX D D1. INTRODUCTION This technical Appendix presents the results from the Programme for Harmonised ATM Research in Eurocontrol (PHARE) Demonstration 1++ (PD/1++) relating to controller workload, sector capacity and service to aircraft. A separate appendix details the results from the usability analysis carried out. It includes comments made by the controllers during the trial and at debrief sessions, as well as the results of statistical tests on the recorded numerical data. This appendix only presents the results from the trial and does not attempt any interpretation or discussion thereof. Occasionally examples are given to assist in reading the graphs. The results are discussed in section 1 of the main body of the report and conclusions are drawn in section 11. This Appendix assumes familiarity with Appendix C which describes the measurements made, the planned analysis, and the methodology applied during the analysis phase. Section D gives an indication of the scale and range of each measure employed in the PD/1++ trial analysis. Appendix C gives a thorough description of these measures. Section D3 describes the statistical significance testing and the results for each of the detailed objectives relating to controller workload, quality of service, and capacity are given in sections D, D5, and D6 respectively. Relevant observations made by the specialist observers and appropriate controller comments from the questionnaires, debriefs and comment sheets have been included throughout. Where comments originated from the questionnaires, the relevant question has been included to provide the context of the comment. The role of the controller making the comment and the relevant organisation (ORG) have also been included, however it was not possible to determine the role of the controller from comments made during the debrief sessions as these were audio recorded. Each low-level objective of the PD/1++ trial relating to controller workload, sector capacity, and quality of service (see Appendix C) is addressed in turn. DOC Version 1. / April 1999 D-1

2 PD/1++ Trial Report D. DESCRIPTIVE STATISTICS Descriptive statistics, usually in the form of a graph, were produced for each analytic measure prior to the statistical tests being carried out. This gave an indication of the size and range of each measure; however, a statistical difference cannot be inferred from the graphs alone. early use of descriptive statistics in the analysis process avoided further statistical analysis if a measure was of little descriptive value; for example, there were so few telephone calls in the measured exercises that statistical analysis was not warranted. The graphs that were produced are given in this section. Unless stated otherwise, the vertical bars on the graphs represent the 5th to the 75th percentile of the measure in question, and the centre point the median. The median is robust to skewness and can also be used for data which is ordinal. The mid-spread is robust to the influence of outliers. Data were collected from 7 measured exercises out of a planned 3. It was planned to complete eight exercises for each ORG, however only six exercises were completed for and, and seven exercises completed for. In each of the above cases, the exercises with the 75% growth traffic samples were not completed. All eight exercises were completed for. The descriptive statistics use the data collected from all 7 measured exercises, therefore the graphs for ORGs and 3 use data from more of the high volume samples than those for ORGs and 1. To avoid any bias due to more exercises being completed in some ORGs, the statistical analysis compared only data collected from the same exercises completed for each ORG (ie matched pairs 1 ). Each sector was staffed by a team of one tactical controller and one planning controller. The controllers were allocated randomly to either a planning role or tactical role and in each case remained in the same role and in the same team for the duration of the trial. The controllers did not remain in the same sector for the trial, but rotated through sectors 1, 11, and the feed sectors. Controllers a, b, c and d formed one group and controllers e, f, g, and h a second group. Each group of controllers completed exercises with two different levels of traffic growth. Controllers a, b, c and d experienced different traffic growths to that of controllers e, f, g and h. Controllers a, b, c and d did not complete any exercises during and with the 75% growth traffic samples. 1 A matched pair is a pair of observations, measured under identical or similar circumstances except for the factor under investigation. D- Version 1. / April 1999 DOC

3 D.1. Traffic volume 8 Number of aircraft on frequency Figure D1 Number of aircraft on frequency per hour for each sector in each ORG Figure D1 shows the number of aircraft on frequency achieved per hour for each sector in each ORG over all exercises. For the purposes of this report, the number of aircraft on frequency over the hour will be referred to as throughput. Note that traffic samples with a higher growth were completed for ORGs and 3. The throughput achieved in each sector in each ORG must be taken into consideration when interpreting results presented in the remainder of this section. DOC Version 1. / April 1999 D-3

4 PD/1++ Trial Report D.. Instantaneous Self Assessment (ISA) Percentage of scores 1% 9% 8% 7% 6% 5% % 3% % PC - a n = 17 X Percentage of scores 1% 9% 8% 7% 6% 5% % 3% % TC - b n = 17 X % 1% % % Percentage of scores 1% 9% 8% 7% 6% 5% % 3% % PC - c n = 17 X Percentage of scores 1% 9% 8% 7% 6% 5% % 3% % TC - d n = 17 X % 1% % % Percentage of responses 1% 9% 8% 7% 6% 5% % 3% % 1% PC - e n = 38 X Percentage of scores 1% 9% 8% 7% 6% 5% % 3% % 1% TC - f n = 38 X % Experimental organisation % Percentage of scores 1% 9% 8% 7% 6% 5% % 3% % 1% PC - g n = 38 X Percentage of scores 1% 9% 8% 7% 6% 5% % 3% % 1% TC - h n = 38 X % % Figure D Percentage of ISA scores for each controller over all exercises Figure D shows the percentage of ISA scores recorded for each controller in each ORG. ISA responses of different controllers are not directly comparable because different controllers experienced different traffic and also use the ISA scale differently. For ORGs and, the percentage of each ISA response was combined for both sector 1 and sector 11. Where a controller failed to mark any ISA value, an x was recorded. D- Version 1. / April 1999 DOC

5 Number of aircraft on frequency ISA 3 All ISA Figure D3 Number of aircraft on frequency for each sector in each ORG, planning controllers 18 Number of aircraft on frequency ISA 3 All ISA Figure D Number of aircraft on frequency for each sector in each ORG, tactical controllers Figure D3 and Figure D show the number of aircraft on frequency corresponding to scores of ISA 3 and over all ISA scores in all exercises for each sector in each ORG for the planning and tactical roles respectively. The ISA 3 score is defined as when a controller is busy but comfortable, and therefore may be thought of as representing a sustainable workload. For example, considering all those occasions in when the tactical controller pressed ISA 3, the median number of aircraft on frequency at the time was 15. Note that the number of aircraft on frequency over all ISA scores is the same for the planning controller (PC) and the tactical controller (TC) since they both controlled the same aircraft. DOC Version 1. / April 1999 D-5

6 PD/1++ Trial Report D.3. NASA Task Load Index (TLX) Overall TLX Planning role Tactical role Figure D5 Overall TLX for each sector in each ORG Figure D5 shows the overall TLX scores for each controller role. The overall TLX scores range from a possible minimum of 15 to a possible maximum of 3. A high TLX score indicates a high level of controller workload. TLX score Physical Demand Planning role Tactical role TLX score Mental Demand Planning role Tactical role TLX score ORGG Time Pressure Planning role Tactical role TLX score Frustration Experienced Planning role Tactical role TLX score Effort Expended Planning role Tactical role Figure D6 TLX factor scores for each sector in each ORG D-6 Version 1. / April 1999 DOC

7 Figure D6 shows the scores for the individual TLX factors for each sector in each ORG, for each controller role. For example the lower quartile TLX time pressure score recorded by the tactical controllers in was 3.5. The TLX factor scores range from 1 (minimum contribution to controller workload) to (maximum contribution to controller workload). The controllers made the following comments when asked what contributed to their frustration: When you get busier the Highly Interactive Problem Solver (HIPS) lets you down which is why frustration is high ; Planning over boundaries, interaction with the HIPS, [and the] Communication List Window ;(, planning) The following comments were made when asked about what contributed to each of the other TLX factor scores: Time Pressure: Larger area, less time pressure (, tactical). Mental Demand: monitoring traffic off route and working out which exit fix was closest to their route. (, tactical) Physical Demand: None (All ORGs, planning and tactical) D.. Number of release alerts 18 Number of release alerts Figure D7 Number of release alerts per exercise for each sector in each ORG Figure D7 shows the number of release alerts per exercise in each sector in each ORG for the tactical controllers only. A release alert occurs if the tactical controller fails to release an aircraft before it crosses the sector boundary. The graph shows that release alerts occurred most often in sector 11 during exercises. DOC Version 1. / April 1999 D-7

8 PD/1++ Trial Report D.5. R/T usage 3% Percentage of time R/T in use 5% % 15% 1% 5% % Figure D8 Percentage of time R/T in use per exercise for each sector in each ORG Figure D8 shows the percentage of time that the R/T was in use, including transmit and receive time, for each sector in each ORG for the tactical controllers only. D.6. Time taken to register each aircraft 1 1 Time to first register (seconds) Figure D9 Time to first register each aircraft for each sector in each ORG Figure D9 shows the time taken to register aircraft over all exercises for each sector in each ORG for the planning controllers only. For example, for each sector in each ORG, at least 5% of aircraft were registered within 13 seconds of entering an advanced information state and over 5% of aircraft were registered within 6 seconds. For and sector 11, it took over two minutes to register 5% of aircraft. D-8 Version 1. / April 1999 DOC

9 D.7. Inter-sector telephone co-ordination time 8 Number of calls transmitted Figure D1 Total number of telephone calls transmitted for each sector in each ORG over all exercises 8 7 Number of calls received Figure D11 Total number of telephone calls received for each sector in each ORG over all exercises Figure D1 and Figure D11 show the total number of telephone calls received and transmitted by the planning controllers for each sector in each ORG over all exercises. The number of telephone calls was considered to be too low to warrant statistical analysis. DOC Version 1. / April 1999 D-9

10 PD/1++ Trial Report D.8. Safe separation infringements Number of infringements Distance at closest approach (Nm) Experimental Organisation Figure D1 Total number of infringements in each ORG Figure D1 shows the total number of infringements and the closest lateral approach for each ORG. An infringement is defined to be where aircraft are within 5nm laterally and 1ft vertically. In there were a total of 1 infringements, four of which had less than 1nm lateral separation. The number of infringements over both sectors is presented since five of the infringements occurred close to the boundary between sector 1 and sector 11, and it was therefore inappropriate to assign the infringement to a single sector. Controllers made the following comments with respect to separation infringements: I think we are getting more of that type of conflict (at sector boundary) when we are using the free route structure than we are getting when we are on the organised route structure ; When off route a lot of aircraft clip the corners so are only with you for a short time. Trying to separate aircraft especially around SPRAT with Europe and London South was long winded ; off-route stops people converging, and that s good news ; Conflict (genuine) caused by HIPS not showing any conflict and not giving a full set of 3D instructions ; [I] Planned separation between UKA85 and CMM3, based on Conflict Risk Display (CRD) derived conflict alert. No alert in the Augmented Dynamic Flight Leg (ADFL) or HIPS. CRD was right and would have been in conflict if nothing done ; Interaction between tools regarding conflict must be better, often only some tools saw conflict - sometimes all, sometimes none. D-1 Version 1. / April 1999 DOC

11 D.9. Number of times ADFL invoked and used to modify a trajectory 1 Planning role Number of times ADFL invoked Tactical role Figure D13 Number of times ADFL invoked per exercise for each sector in each ORG Number of times ADFL modified Planning role Tactical role Figure D1 Total number of times ADFL used to modify a trajectory over all exercises for each sector in each ORG Figure D13 shows the number of times the ADFL was invoked per exercise, for each sector in each ORG, for both controller roles. Figure D1 gives the total number of times that the ADFL tool was used to modify a trajectory over all exercises, for each sector in each ORG for both controller roles. During the trial the ADFL was used only six times by the planning controllers and once by a tactical controller to modify a trajectory. DOC Version 1. / April 1999 D-11

12 PD/1++ Trial Report D.1. Number of times HIPS invoked and used to modify a trajectory 1 9 Planning role Tactical role Number of times HIPS invoked Figure D15 Number of times HIPS invoked per exercise for each sector in each ORG Number of times HIPS modified Planning role Tactical role Figure D16 Total number of times HIPS used to modify a trajectory over all exercises for each sector in each ORG Figure D15 show the number of times the HIPS was invoked for both controller roles. Figure D16 give the total number of times that the HIPS tool was used to modify a trajectory for both controller roles. Analysis of the HIPS usage data showed that one of the planning controllers rarely invoked the HIPS tool. The HIPS data for this controller was not included in the descriptive or statistical analysis of this measure as it was felt that the controller s use of the tool was atypical based on the results obtained for the other planning controllers in this trial and previous trials. D-1 Version 1. / April 1999 DOC

13 Figure D16 shows that over all exercises, the planning controller used the HIPS tool a total of 33 times to modify a trajectory. DOC Version 1. / April 1999 D-13

14 PD/1++ Trial Report D.11. Number of modifications in each of the HIPS displays Percentage use of display 1% 9% 8% 7% 6% 5% % 3% % Route display Profile display Speed display 1% % Figure D17 Use of each of the HIPS displays for modifications over all exercises Figure D17 shows the percentage use of each of the three HIPS displays over both controllers in each sector in each ORG. For example, in sector 11, 6% of the trajectory modifications made in HIPS were made in the profile display, 39% were made using the route display, and 15% were made using the speed display. In and in sector 1, 5% of the trajectory modifications were made in the profile display and 5% in the route display. D.1. D.13. Number of trajectory re-plans (using both ADFL and HIPS) The number of trajectory re-plans could not be calculated within the analysis period. However, the total number of trajectory re-plans in each ORG would have to be less than, or equal to, the total number of trajectory modifications, as reported in Paragraph DD.1. The total number of trajectory modifications is the sum of the total number of trajectory modifications that did not have the initial trajectory registered, and the total number of trajectory re-plans. Percentage of time datalink-equipped aircraft are tactically controlled It was not possible to determine the percentage of time that D-FMS datalinkequipped aircraft were controlled for each sector. Figure D18 shows the total number of tactically controlled D-FMS aircraft in each sector in each ORG. Over all exercises no more than D-FMS aircraft were controlled in each sector in each ORG. No inference can be made using statistics since there were too few observations. D-1 Version 1. / April 1999 DOC

15 Total number of controlled D aircraft 1 Figure D18 Number of D-FMS datalink-equipped aircraft controlled over all exercises D.1. Percentage of time that aircraft are at their maximum altitude It was intended to show the percentage of time aircraft spent at their requested cruise flight level. However this was not possible within the analysis period. Instead, the cruder measure of the percentage of aircraft that achieved their requested cruise level for each exercise was calculated. Figure D19 illustrates this for each ORG. For example, in three-quarters of the exercises, the percentage of aircraft with maximum altitude corresponding to requested cruise altitude was greater than 8%. 1% 9% 8% Percentage of aircraft 7% 6% 5% % 3% % 1% % Figure D19 Percentage of aircraft at requested cruise flight level D.15. Percentage of time spent de-conflicting radar labels This measure was calculated for each sector in each ORG. Preliminary analysis showed the time spent de-conflicting labels was up to 5% of the measured exercise time. Investigation of the data logging showed that the actual time recorded was the total time a label was highlighted by the controller having the mouse pointer placed over the label. This meant that, in addition to the time de- DOC Version 1. / April 1999 D-15

16 PD/1++ Trial Report conflicting labels, the time included all instances where the controller used the label to: select and de-select the ADFL tool, interact with the flight levels within the label, or simply to highlight the label. As a result, no further analysis was carried out on this data. D-16 Version 1. / April 1999 DOC

17 D3. TRIAL RESULTS The following sections present the results from the statistical tests and questionnaires associated with each low-level objective. These are summarised in the form of tables. In addition, both the Performance and Usability Modelling in ATM (PUMA) and usability analysis results are presented in full. The statistical tests employed in this analysis determine whether a statistically significant difference exists between any of the measurements. The probability given for a hypothesis test is the smallest significance level at which the null hypothesis can be rejected with the observed sample data. A 5% significance level has been used throughout to determine whether the observed difference is statistically significant. If the probability was less than the significance level (ie.5), the null hypothesis was rejected and it was concluded that a difference exists between the two populations studied. If the null hypothesis was not rejected, then the power of the test was estimated to provide a figure for the confidence that can be placed in accepting the null hypothesis correctly. The power of a statistical test is the probability that the null hypothesis was accepted when it was in fact true. The 5% significance value has been shown over many NRF trials to be a useful threshold for filtering out noise results. No such value for power has yet been established. Here it will be assumed that a value of 95% or higher gives an appropriate level of confidence in the null hypothesis. For values lower than this, it will not be possible to say with confidence that the null hypothesis has failed to be rejected simply, say, through lack of data. Table D1 summarises the above procedure. Significance Power Conclusion <.5 - evidence to reject null hypothesis >.5 >.95 evidence to accept null hypothesis >.5 <.95 no evidence either way Table D1 Statistical outcomes Where statistical results are given for a certain measure, the notation ORGa > ORGb indicates that the scores for ORGa were significantly greater than those for ORGb, with a significance level of 5%. Likewise, ORGa < ORGb indicates significantly lower scores for ORGa. By ORGa = ORGb, it is meant that there was no evidence of a statistically significant difference between ORGa and ORGb. Where a statistical difference between the two systems has been observed, the difference is also presented alongside the result as a percentage of the lower value where appropriate, to suggest the scale of the difference. This is important as it may be possible to detect statistically a difference which is so small as to have no operational importance. The scale of the difference is not given for the ISA results as the ISA scale is not necessarily linear. In other words, the magnitude of controller workload changes between ISA and ISA 3 is not the same as (say) that between ISA 3 and ISA. Thus a percentage change in ISA is not a useful measurement. Where there was no evidence of a statistically significant difference between the two, a dash (-) is shown. As the Friedman test requires matched pairs of data, only the same exercises completed for all ORGs were included in the statistical comparisons. Note also that only these exercises were included in the Kruskal-Wallis test. Including the exercises for which there was no matched exercise would have biased the DOC Version 1. / April 1999 D-17

18 PD/1++ Trial Report comparisons between the ORGs. Not all possible comparisons have been made, (eg with sector 1). All those made have been reported. D. DETAILED OBJECTIVES RELATING TO CONTROLLER WORKLOAD ANALYSIS D.1. For each controller role, examine whether alternative airspace structures ie sector 1 and sector 11 versus, had a positive impact on controller workload, given the same operating practices. Comparison of ISA scores Role Comparison Conclusion Power Planning sector 1 vs < - < - sector 11 vs =.85 =.99 Tactical sector 1 vs < - < - sector 11 vs =.95 =.99 These results show that for both controller roles there was an increase in perceived controller workload between sector 1 and, for both routing scenarios. However there was no evidence to suggest that there was a change in perceived controller workload between sector 11 and for either role. On the contrary, for the tactical and for the planning controller with direct routes ( versus ), the power of the test is high enough to suggest that there is evidence that the controller workload was the same. Comparison of overall TLX scores Role Comparison Conclusion Power Planning sector 1 vs =.19 sector 11 vs = = = Tactical sector 1 vs =. sector 11 vs = = = These results show that for both controller roles there was no evidence to suggest that there was a change in controller workload between sector 1 or 11 and for either routing scenario. Although there was no evidence to suggest that there was a difference between any of the ORGs, the power of the test is small. This indicates that there is no evidence to suggest that the workloads were the same. The following comments were made by the controllers with respect to the individual TLX factors: D-18 Version 1. / April 1999 DOC

19 of the TLX categories, only Mental Demand was a significant item in the Overall workload. (, PC) larger area, less time pressure. (, TC) Number of release alerts Role Comparison Conclusion Power Tactical sector 1 vs =.99 =.98 sector 11 vs =.99 =.96 These results show that there was no evidence to suggest that there was a difference in the number of release alerts for the tactical controller between sector 1 or 11 and for either routing scenario. The estimated power shows that there is evidence to suggest that there is no difference in the number of release alerts between sectorisations. R/T usage Role Comparison Conclusion % difference of medians Power Tactical sector 1 vs < 68% - < 7% - sector 11 vs = -.97 = -.91 These results show that there was a significant increase in the R/T workload of the tactical controller between sector 1 and under both routing scenarios. The median percentage of time that the R/T was in use per exercise increased by 68% between sector 1 and with structured routes, and 7% between sector 1 and with direct routes. However, there was no evidence to suggest that the R/T workload was any different between sector 11 and under either routing scenario. The estimated power of the test suggests that there is evidence that the R/T workload was the same for sector 11 and for structured routes. Time taken to register each aircraft Role Comparison Conclusion % difference of medians Power Planning sector 1 vs < 3% - < 85% - sector 11 vs = -.88 = -.97 These results show that there was a significant increase in the time taken for the planning controller to register each aircraft between sector 1 and under both routing scenarios. The median percentage of time taken to register each DOC Version 1. / April 1999 D-19

20 PD/1++ Trial Report aircraft increased by 3% with structured routes, and 85% with direct routes. However, there was no evidence to suggest that the time taken to register each aircraft was different between sector 11 and under either routing scenario. The estimated power of the test suggests that there is evidence that the time taken to register each aircraft between sector 11 and was the same for direct routes. Questionnaires The following results have been taken from the questionnaires issued at the end of each ORG. Following each question the possible responses are given How did your average workload rate in each ORG? Too Low, Low but Acceptable, High but Acceptable, Too High Role Controller Planning a Low, but Low, but c Low, but e Too High g Tactical b Low, but d Low, but Low, but Low, but Low, but f Low, but Low, but Low, but Low, but h D- Version 1. / April 1999 DOC

21 Was your workload ever too high to allow you to cope? Always, Regularly, Rarely, Never Role Controller Planning a rarely rarely rarely rarely c never never never never e regularly rarely rarely rarely g never rarely rarely never Tactical b rarely rarely rarely rarely d never never never never f rarely rarely rarely rarely h rarely rarely rarely rarely Did you feel frustrated with the system at any time? Always, Regularly, Rarely, Never Role Controller Planning a regularly regularly regularly regularly c regularly regularly regularly regularly e always regularly regularly regularly g regularly rarely rarely rarely Tactical b (no response) rarely rarely regularly d rarely rarely rarely rarely f rarely rarely rarely rarely h regularly rarely regularly rarely What did you find frustrating in each ORG? All ORGs Planning HIPS failures. Tactical Trying to use tools around the boundary. Planning Planning over boundaries. Planning It was frustrating as before - less than with 1/11. Reduction in problems at sector boundaries. Tactical When off-route a lot of aircraft clip the corners of the sector so are only with you for a short amount of time. DOC Version 1. / April 1999 D-1

22 PD/1++ Trial Report Did you feel a great deal of mental demand with the system at any time? Always, Regularly, Rarely, Never Role Controller Planning a rarely regularly rarely regularly c regularly regularly regularly regularly e regularly rarely rarely regularly g rarely rarely rarely rarely Tactical b rarely rarely regularly regularly d rarely rarely rarely regularly f rarely rarely rarely rarely h rarely rarely rarely rarely What factors contributed to your mental demand in each ORG? All ORGs Tactical Remembering to press which mouse button on which part of the label. Trying to work out why something hadn t worked and getting confused. Planning Planning for a larger area. Tactical Mental demand was less than on 1/11. Tactical Monitoring traffic off routes and working out which exit fix was closest to their route. Tactical Less demanding due to reduced conflict. if more than two 3D aircraft being managed on the sector the workload increased rapidly. What effect did the larger sector have on your workload? Planning Actually made easier workload than smaller sectors. Eased planning task, more subtle and less complex solutions due to removal of 1/11 boundary. Made the use of the tools easier, therefore less workload. Tactical less pressure to achieve targets and more room. Debriefs, observations and comment sheets The following comments were made during the debriefs: Well, is a better sector job rate with a large tool system on it, you don t want to be mucking about handing over at the boundary, you want to be sorting out problems. D- Version 1. / April 1999 DOC

23 I think the larger sector was a very big plus. D.. For each controller role, examine whether alternative airspace structures ie structured routes versus direct routes, had a positive impact on controller workload, given the same sectorisation, but different operating practices. Comparison of ISA scores Role Sector Conclusion Power Planning sector 1 < - sector 11 < - < - Tactical sector 1 =.93 sector 11 < - < - These results show that with the exception of the sector 1 tactical controller, when comparing direct routes with structured routes there was an increase in perceived controller workload. The estimated power of the test suggests that the perceived controller workload of the sector 1 tactical controller was the same under each routing scenario. Comparison of TLX scores Role Sector Conclusion Power Planning sector 1 =.19 sector 11 = = Tactical sector 1 =. sector 11 = = These results show that when comparing structured routes with direct routes there was no evidence to suggest that there was a difference in controller workload for any sector. The estimated power indicates that there is no evidence to suggest that the workloads were the same. Number of release alerts Role Sector Conclusion % difference in median Power Tactical sector 1 = -.91 sector 11 < 313 % = -.5 These results show that when comparing direct routes with structured routes there was a significant increase in the number of release alerts for the sector 11 tactical controller. The estimated power indicates that there is no evidence to suggest that the number of release alerts was the same under each routing scenario for sector 1 and sector. DOC Version 1. / April 1999 D-3

24 PD/1++ Trial Report R/T usage Role Sector Conclusion % difference in median Tactical sector 1 < 69% sector 11 < 5 % < 75 % These results show that when comparing direct routes with structured routes there was a statistically significant increase in the tactical controllers R/T usage for all sectors. Time taken to register each aircraft Role Sector Conclusion Power Planning sector 1 =.99 sector 11 =.89 =.98 These results show that when comparing direct routes with structured routes there was no evidence to suggest a difference in the time taken for the planning controller to register each aircraft for any sector. The estimated power of the test suggests that for sector 11 and the time taken to register each aircraft was the same. Questionnaires The following results have been taken from the questionnaires issued at the end of each ORG. What effect did direct routes have on your workload? Planning Less intensive problem sequences. Easier solutions could be planned. Tactical Slight reduction in conflicts overall. Made the use of HIPS essential for conflict detection. Monitoring traffic was harder. D- Version 1. / April 1999 DOC

25 Would you agree that the balance of task sharing between the planning and tactical controller with direct routes is right? Planning Agree, provided the two work as a team and occasionally take on the other s tasks to relieve pressure. Tactical No. Planning seems to have higher workload. Maybe some of the simple plans can be registered by the tactical. Planning Role did not change, we worked as a team in all ORGs, irrespective of 3D, D or [traffic] level. No. I don t think the planning/tactical relationship is valid in PHARE, two controllers maybe, but task design is unequal and inappropriate - needs a lot of study. How did you split the responsibilities? Planning We split them tactically, ie the one who was least busy helped the one who was most busy. We worked as a team and shared work. Planning and tactical was used as a loose basis, but it s probably not a long term solution. Debriefs, observations and comment sheets The following comments were made during the debrief sessions: If as a planning controller you start moving the 3D s around, it doesn t half generate a lot of work for the tactical controller ; It comes back to the fact that the actual mixes is wrong in this environment. It s following the NERC-style planner and tactical break in responsibilities and its not actually appropriate, I don t think. There s different roles in there that need to be thought out ; When we ve done these 3D/D splits, the segregated airspace, I mean it may well be that the planning/tactical relationship is different in the lower and upper sectors ; I am amazed actually how simple it is to do the direct routings in D airspace ; I felt it was a bit easier when they were in direct routes because you had less problems of overtaking or opposite direction. DOC Version 1. / April 1999 D-5

26 PD/1++ Trial Report D.3. For each controller role, examine whether alternative operating practices and alternative airspace structures have a positive impact on controller workload. Comparison of ISA scores Role Comparison Conclusion Planning sector 1 vs < sector 11 vs < Tactical sector 1 vs < sector 11 vs < The comparison of ISA scores shows that for both controller roles there was an increase in perceived controller workload between sectors 1 and 11 with structured routes and the sector with direct routes. Comparison of TLX scores Role Comparison Conclusion Power Planning sector 1 vs =.19 sector 11 vs = Tactical sector 1 vs =. sector 11 vs = The comparison of overall TLX scores shows that for both controller roles there was no evidence to suggest that there was a difference in controller workload between sectors 1 and 11 with structured routes and the sector with direct routes. The estimated power indicates that there is no evidence to suggest that the workloads were the same. Questionnaires The following results have been taken from the questionnaires issued at the end of each ORG. Any other comments on how changing from structured routes to direct routes or from sectors 1 and 11 to the sector affected your workload? Planning Direct routes in sector was easier to handle. Tactical Workload decrease when changing from structured routes to great circle and from 1 and 11 to. Debriefs, observations and comment sheets The following comments were made during the debrief sessions: is easy because you ve got more room and you can be more subtle in what you do but the tools don t make a resounding difference between on route/off route/big sector/small sector ; The difference is in how we work together, I think, and the 3D/D to some extent which does impose a greater workload and is less predictable. I don t think other than that there was a lot of differences ; D-6 Version 1. / April 1999 DOC

27 I think [the worst aspect of the system] is the 3D/D mix. I think that s the worst bit as far as I m concerned. D5. DETAILED OBJECTIVES RELATING TO QUALITY OF SERVICE ANALYSIS D5.1. For each measured sector, examine whether alternative airspace structures ie sector 1 and sector 11 versus, had a positive impact on the quality of service provided to airlines, given the same operating practices. Percentage of time aircraft are at their maximum altitude As explained in paragraph DD.1, the percentage of time aircraft were at their maximum altitude, in cases where this corresponded to their requested cruise level, could not be determined. However, the percentage of aircraft in each exercise that achieved their requested cruise level was compared. Comparison Conclusion Power sectors 1 and 11 =. vs = These results show that there was no evidence to suggest that there was a difference in the percentage of aircraft that achieved their requested cruise level over sectors 1 and 11 than in with either routing scenario. However, the estimated power indicates that there is no evidence to suggest that the percentage of aircraft that achieved their requested cruise level was the same. Questionnaires The following results have been taken from the questionnaires issued at the end of each ORG. Planning It was easier to give direct routes and better profiles in the large sector. Tactical Large sectors means less changes. Easier and more continuous climbs and descents. Planning More violent heading changes required to resolve conflicts in a small sector. Not much change to, significant improvement to. DOC Version 1. / April 1999 D-7

28 PD/1++ Trial Report D5.. For each measured sector, examine whether alternative airspace structures ie structured routes versus direct routes, had a positive impact on the quality of service provided to airlines, given the same sectorisation, but different operating practices. Percentage of time aircraft are at their maximum altitude As explained in paragraph DD.1, the percentage of time aircraft were at their maximum altitude, in cases where this corresponded to their requested cruise level, could not be determined. However the percentage of aircraft in each exercise that achieved their requested cruise level was compared. Sector Conclusion Power sectors 1 and 11 =. = These results show that when comparing structured routes with direct routes there was no evidence to suggest the percentage of aircraft that achieved their requested cruise level was different for sectors 1 and 11 or. However, the estimated power indicates that there is no evidence to suggest that the percentage of aircraft that achieved their requested cruise level was the same. Questionnaires The following results have been taken from the questionnaires issued at the end of each ORG. Planning Aircraft receive less intervention, so D better. 3D, no benefit. Tactical Great circle tracks less flying time therefore less fuel used. With direct routing there are many advantages to the airline - time and distance particularly. In the aircraft get what they want. D6. DETAILED OBJECTIVES RELATING TO CAPACITY ANALYSIS D6.1. For each measured sector, examine whether alternative airspace structures ie sector 1 and sector 11 versus, had a positive impact on sector capacity, given the same operating practices. Number of aircraft on frequency corresponding to scores of ISA 3 Role Comparison Conclusion Planning sector 1 vs < < sector 11 vs < < Tactical sector 1 vs < < sector 11 vs < < D-8 Version 1. / April 1999 DOC

29 The above results show that for both controller roles, there was an increase in the number of aircraft on frequency for scores of ISA 3 between sector 1 and, and sector 11 and, for both routing scenarios. D6.. For each measured sector, examine whether alternative airspace structures ie structured routes versus direct routes, had a positive impact on sector capacity, given the same sectorisation, but different operating practices. Number of aircraft on frequency corresponding to scores of ISA 3 Role Sector Conclusion Power Planning sector 1 =.99 sector 11 < - < - Tactical sector 1 =.99 sector 11 < - < - The above results show that when comparing direct routes with structured routes there was an increase in the number of aircraft on frequency for scores of ISA 3 for sector 11 and sector for both controller roles. The estimated power of the test suggests that the number of aircraft on frequency for scores of ISA 3 was the same for sector 1, for both controller roles. Questionnaires The following results have been taken from the questionnaires issued at the end of each ORG. Planning Less concentration of problem at exit points in level flight. (compared to ) Tactical Through easier separation and Reduced Vertical Separation Minima (RVSM), more levels available. (compared to ) Planning Less opposite direction conflicts. (compared to ) Tactical Less separation problems. (compared to ) Probably higher capacity in, particularly with RVSM. (compared to ) DOC Version 1. / April 1999 D-9