Situation awareness implications of adaptive automation for information processing in an air traffic control-related task

Transcription

1 ARTICLE IN PRESS International Journal of Industrial Ergonomics 36 (2006) Situation awareness implications of adaptive automation for information processing in an air traffic control-related task David B. Kaber a,, Carlene M. Perry b, Noa Segall b, Christopher K. McClernon c, Lawrence J. Prinzel III d a Institute for Automation, College of Informatics and Electrical Engineering, University of Rostock, Rostock-Warnemünde, Germany b Edward P. Fitts Department of Industrial Engineering, North Carolina State University, Raleigh, NC, USA c Department of Behavioral Sciences and Leadership, United States Air Force Academy, USA d NASA Langley Research Center, Hampton Roads, VA, USA Available online 13 March 2006 Abstract The objective of this research was to assess the effectiveness of adaptive automation (AA) for supporting information processing (IP) in a complex, dynamic control task by defining a measure of situation awareness (SA) sensitive to differences in the forms of automation. The task was an air traffic control (ATC)-related simulation and was developed to present four different modes of automation of IP functions, including information acquisition, information analysis, decision making and action implementation automation, as well as a completely manual control mode. A total of 16 participants were recruited for a pilot study and primary experiment. The pilot assessed the sensitivity and reliability of the Situation Awareness Global Assessment Technique (SAGAT) for describing AA support of the IP functions. Half of the participants were used in the primary experiment, which refined the SA measure and described the implications of AA for IP on SA using the ATC-like simulation. Participants were exposed to all forms of automation and manual control. AA conditions matched operator workload states to dynamic control allocations in the primary task. The pilot did not reveal significant differences in SA among the various AA conditions. In the primary experiment, participant recall of aircraft was cued and relevance weights were assigned to aircraft at the time of simulation freezes. The modified measure of SA revealed operator perception and Total SA to improve when automation was applied to the information acquisition function. In both experiments, performance in the ATCrelated task simulation was significantly superior when automation was applied to information acquisition and action implementation (sensory and motor processing), as compared to automation of cognitive functions, specifically information analysis. The primary experiment revealed information analysis and decision-making automation to cause higher workload, attributable to visual demands of displays. Industry relevance The results of this research may serve as a general guide for the design of adaptive automation functionality in the aviation industry, particularly for information processing support in air traffic control tasks. r 2006 Elsevier B.V. All rights reserved. Keywords: Adaptive automation; Situation awareness; Air traffic control; Workload 1. Introduction Automation in air traffic control (ATC) can have significant effects on controller performance, including reduced workload (Laois and Giannacourou, 1995), Corresponding author. Tel.: ; fax: address: dbkaber@ncsu.edu (D.B. Kaber). increased capability to perform complex computations and data management (Wickens and Hollands, 2000), and increased performance reliability (National Research Council, 1998). However, high-level, static automation in ATC can also present many disadvantages (Dillingham, 1998), including a loss of controller situation awareness (SA), as a result of substituting the human in system control loops (Endsley and Jones, 1995). Endsley (1995a) /$ - see front matter r 2006 Elsevier B.V. All rights reserved. doi: /j.ergon

2 448 ARTICLE IN PRESS D.B. Kaber et al. / International Journal of Industrial Ergonomics 36 (2006) defined SA as, the perception of elements in an environment, within a volume of space and time, comprehension of their meaning and projection of their status in the near future. As machines perform more and more ATC functions and operations, controllers have less interaction with the system and may become less aware of system operations. Out-of-the-loop (OOTL) performance may reduce a controller s ability to detect problems (e.g., conflicting aircraft), determine the current state of the system, understand what has happened and what courses of actions are needed, and react to the situation (Endsley, 1996) Adaptive automation for ATC New, advanced forms of automation, including adaptive automation (AA), are being considered for ATC to moderate operator workload and preserve SA by facilitating a better match between task demands and operator cognitive resources (Bailey et al., in press). In identifying forms of ATC automation, Hopkin (1998) suggested that automation may not only be applied statically to different types of system functions at different levels, but it can also be dynamically applied to functions to switch control between the human and machine. AA refers to complex systems in which the level of automation or the number of system functions being automated can be modified in real time based on task/contextual information or operator states for the purpose of optimizing system performance and managing operator workload (Scerbo, 1996; Kaber and Riley, 1999). By level of automation, we mean the extent to which machines replace human operators in control of systems. Prior research (Hilburn et al., 1997; Parasuraman et al., 1993b) has demonstrated AA to improve performance in both simple monitoring tasks and tasks requiring cognitive aspects of information processing (IP) when control dynamically shifts from human manual performance to some form of automation. The study by Hilburn et al. (1997) involved actual controllers and a high-fidelity simulation of a European ATC system to determine the effects of AA on decision making in descent advising. They developed three different automation schemes including constant manual control, constant automation and AA, which introduced automation during high air traffic periods. They found the AA condition resulted in the smallest increase in workload compared to fully manual and automated control. This is one example of how AA may benefit ATC, as compared to static automation and manual control. More recent work has been conducted to compare the differential effects of AA applied to a range of IP functions, including psychomotor and cognitive functions, in the context of an ATC simulation (Clamann et al., 2002; Clamann and Kaber, 2003). Clamann et al. (2002) assessed the performance and workload effects of AA applied to information acquisition, information analysis, decision making, and action implementation functions in a lowfidelity simulation of ATC. They found that operators were better able to adapt to AA when applied to lower-order IP functions, such as information acquisition (sensing) and action implementation (motor performance), as compared to AA applied to higher-order, cognitive functions. By higher-order, cognitive functions, we mean those aspects of ATC requiring planning and decision making and use of complex, long-term memory structures. Results indicated that operator performance was greatest when automation was applied to the action implementation aspect of the task. Workload results revealed AA of the information acquisition function as part of the ATC task simulation to relieve some temporal demands (time pressure) on subjects. The results of this research indicate the effectiveness of AA in ATC may be dependent upon the type of automation (form of IP assistance) provided to a controller. In general, historical work has evaluated AA for supporting psychomotor and cognitive functions in terms of performance and workload and not measures of operator cognition. If potential benefits of AA include reductions in OOTL performance problems (e.g., automation induced complacency (Parasuraman et al., 1993a)) and support for specific IP functions in dynamic control tasks, then this needs to be assessed in terms of measures of operator cognitive states, such as SA, in order to develop more complete rationales for future systems design. Outcome measures, like performance, may not provide direct insight into whether AA for IP is facilitating a better match of task demands and cognitive resources than would otherwise be possible with static automation AA and SA It has been observed by review and theoretical studies of AA (e.g., Kaber et al., 2001; Parasuraman, 2000) that facilitating dynamic allocations of system control functions to a human operator or computer over time can moderate operator workload and, at the same time, may facilitate SA, or operator preparedness for unexpected system states, by maintaining some level of operator involvement in control loops. Parasuraman et al. (1993a) observed that AA could be beneficial for reducing decrements in SA during sustained monitoring performance under static automation. Other research has suggested that SA might be improved in aviation system operations through the use of AA (Prinzel, 2002). Unfortunately, little work has empirically assessed the SA implications of AA. Two recent studies (Kaber and Endsley, 2004; Kaber et al., 2002) measured SA under AA in dynamic control tasks, including radar monitoring and remote control of a telerobot. These studies used direct, objective measures of SA, following the Situation Awareness Global Assessment Technique (SAGAT; Endsley, 1995b), to assess the effects of dynamic changes in automation states (Kaber and Endsley, 2004) or changes in adaptive interface content (Kaber et al., 2002). The SAGAT is an on-line SA query

3 ARTICLE IN PRESS D.B. Kaber et al. / International Journal of Industrial Ergonomics 36 (2006) technique (Endsley, 2000) in which a simulation of a complex control task is frozen (stopped) at random points in time in order to administer questionnaires to operators on the current state of the system, the importance of system states to their task goals, and their projections of system states in the near future. The responses to queries are graded based on ground truth information recorded by the simulation system at the time of each stop. Kaber and Endsley (2004) found that the form of task automation adaptively applied to simulated radar monitoring played a significant role in operator SA measured using SAGAT. They observed that automation, involving computer assistance in monitoring (sensory processing) and action implementation functions of the task resulted in worse operator SA, as compared to automation primarily applying computer assistance to decision-making aspects of the task and fully autonomous system operation. However, the sensory and action forms of automation had significant positive effects on operator performance in terms of dealing with dynamic changes in control modes, as part of AA. This research suggests that the SA impact of AA for supporting IP may be dependent upon the IP functions to which automation is applied. Although Kaber and Endsley s (2004) performance results agree with those of Clamann et al. (2002), their results on SA should be of interest to designers because it may mean that applying AA to psychomotor functions to increase performance in the near term may undermine SA over extended periods. (Clamann et al. (2002) trials lasted 20 min; Kaber and Endsley s (2004) trials were 60 min in duration.) One drawback of Kaber and Endsley s (2004) study, in terms of providing insight into AA design for actual systems, is that they employed a model-based approach; that is, the control mode changes during task performance were pre-programmed based on anticipated changes in operator workload due to scheduled task events (see Scerbo (1996) for further definition of various approaches to AA). Although model-based approaches have been demonstrated to be effective for supporting monitoring task performance (e.g., Parasuraman et al., 1993b; Hilburn et al., 1993), pre-programmed system control allocations cannot be considered to represent truly adaptive systems that change states based on, for example, real-time task information or operator workload states Current research needs Research is needed to evaluate the effectiveness of actual adaptive systems for supporting complex task IP functions, in terms of SA. Other approaches to investigating AA in complex systems have included monitoring operator workload states and triggering control allocations based on workload fluctuations (e.g., Kaber and Riley, 1999), or triggering control mode changes on the basis of critical events (e.g., system failures), but this research has not assessed operator SA. It is possible that workload-matched AA of certain IP functions in complex systems control may promote operator SA and yield higher levels of SA than observed under manual control conditions by providing a better match of operator cognitive resources for performance to workload demands (Bailey et al., in press). In order to evaluate this thesis, there is a need for sensitive SA measurement techniques in the context of complex, dynamic control tasks, like ATC. Table 1 presents a categorization of the empirical studies of AA reviewed here in terms of the types of tasks investigated, the types and levels of system automation explored, and measures used to evaluate the impact on human operators. The majority of these studies have been conducted in the context of aviation-related simulations primarily because of the breadth and type of cognitive requirements imposed on operators (ranging from monitoring to tracking to decision making and real-time planning). In addition, the complex tasks as part of these simulations often pose multiple competing goals for operators, complex decision requirements, and simultaneous attention demands under time pressure. Consequently, performance may be critically dependent upon operators achieving good SA for recognition-primed decision making (Endsley and Jones, 1997; also see Klein, 1998). In general, this is why SA is important in the ATC context and other aviation-related tasks. Table 1 supports the need for further research to describe the SA implications of AA for supporting various IP functions in complex, dynamic control tasks, like ATC. The intent of this study was not to demonstrate AA specifically in the ATC context, but instead to show how AA may support various aspects of IP in a dynamic control task, in general, and the implications on SA. We believe that the ATC paradigm is particularly conducive for this type of research; however, such work could also be conducted using, for example, a process control simulation (e.g., Moray, 1986; Vicente, 1991) or manufacturing systems simulation (e.g., Sharit and Salvendy, 1987) also posing operator performance requirements across stages of IP. Our focus here was to use the ATC paradigm as an application of AA for studying human IP and SA effects. We addressed this focus by defining a measure of SA sensitive to differences in forms of automated assistance in an ATC-related task. We observed the measure when AA was applied to the IP functions of information acquisition, information analysis, decision making, and action implementation (as described in Parasuraman et al., 2000) Link of SA to forms of AA and hypotheses The historical approach to AA has been a binary one, involving periodic allocations of completely manual (human) control of a system during full automation in order to reintegrate operators in the control loop and to maintain vigilance and support system state awareness. In this way, AA has been applied to entire tasks, such as tracking performance (e.g., Scallen et al., 1995). Only few prior

4 450 ARTICLE IN PRESS D.B. Kaber et al. / International Journal of Industrial Ergonomics 36 (2006) Table 1 Classification of empirical research on AA covered in review Studies (in chronological order) Type of task Type/function of automation Level of automation Response measures used to evaluate AA Parasuraman et al. (1993b) Piloting tasks Monitoring Full for function or manual control Hilburn et al. (1993) Piloting tasks Monitoring Full for function or manual control Hilburn et al. (1997) ATC simulation Decision making Full for function or manual control Kaber and Riley (1999) Radar monitoring and Decision making Full for function or target elimination manual control Kaber et al. (2002) Telerobot control System supervision Full for function or manual control Clamann et al. (2002) ATC task simulation Information acquisition; Partial automation of information analysis; various functions or decision making; action manual control implementation Clamann and Kaber (2003) Kaber and Endsley (2004) ATC task simulation Radar monitoring and target elimination Information acquisition; information analysis; decision making; action implementation Monitoring; planning; decision making; and action implementation Partial automation of various functions or manual control Full for function or manual control Performance and workload Performance and workload Performance and workload Performance and workload Performance, workload and SA Performance and workload Performance and workload Performance, workload and SA studies have investigated AA applied to specific IP functions in the context of a task. In the present study, AA was designed to provide assistance with IP functions of the ATC-like simulation and, consequently, we expected SA to be affected in different ways. As in real ATC, SA in the context of our simulation can be defined as an up-to-date assessment of the changing locations of aircraft in a designated airspace, projection of their future locations relative to each other, and knowledge of pertinent aircraft parameters (destination, speed, etc.). Endsley and Rodgers (1994) said that controllers have historically called this the picture ; that is, their mental model of the traffic situation at any moment in time upon which all decisions are based. According to Endsley s (1995a) general definition of SA, operators achieve system/state awareness on three hierarchical levels, including perception (Level 1 SA), comprehension (Level 2 SA), and projection (Level 3 SA). These levels are reflected in the contextual definition of SA for the ATC task. She also stated that these levels may build on each other in a hierarchical manner; that is, good Level 1 SA may lead to good comprehension and so on. According to Parasuraman et al. (2000), information acquisition automation includes methods for sensing the task environment, delivering information to human operators, as well as providing for some organization/categorization of information. This form of automation allocates aspects of the perceptual stage of IP to machine control and may support operator achievement of Level 1 SA. Information analysis automation involves filtering of information and highlighting particularly relevant items for task performance, as well as providing integrated and predictive displays of information. Consequently, this form of automation may support operator achievement of states of task comprehension (Level 2 SA) and facilitate more accurate projection of system states (Level 3 SA). One potential problem, however, is that integrated displays may draw operator attention away from primary-task displays, like the radarscope in an ATC task, and cause operators to focus on automation projections of data (e.g., aircraft trajectories) versus data on the current state of the system (degraded Level 1 SA). Decision-making automation is intended to assist or replace operators in response selection by providing prompts/alerts to critical system states and recommendations for resolving problems. This type of automation may also support accurate operator comprehension and projection of system states (Level 2 and 3 SA). However, it often supplants the human operator from system control loops to a great degree and can lead to passive decision-making behavior (cf., Endsley and Kiris, 1995). Operators may come to rely on computer recommendations on, for example, resolving aircraft conflicts in traffic management, with little suspicion of the automation. Such automation may also lead to operators spending little time on basic control functions, including directly querying aircraft for information, which may be important to achieving Level 1 SA. Finally, action implementation automation is intended to assist or replace operators in response execution once a decision has been made. This form of automation may primarily impact operator psychomotor workload and free-up additional attentional resources for perceiving states of the environment, or achieving fundamental Level 1 SA.

5 ARTICLE IN PRESS D.B. Kaber et al. / International Journal of Industrial Ergonomics 36 (2006) In general, because of the potential for OOTL performance problems under automation (Endsley and Kiris, 1995; Wickens, 2000), we expected that operator SA would be superior when AA was applied to lower-order IP functions, including information acquisition and action implementation, than automation applied to higher-order IP functions, providing computer assistance in decision making. This speculation is supported by prior empirical work (Endsley and Kiris, 1995; Kaber et al., 2000). Following the same logic, we also expected that a completely manual control condition, allowing operators to play an active role in system functions (e.g., aircraft clearances, etc.) would support greater SA, as compared to decision automation supplanting operators from formulation of response alternatives. Based on Clamann et al. s (2002) research, we expected performance in the ATC-related simulation to be worse during completely manual control as compared to any mode of AA. Related to this, Leroux (1993), and Laois and Giannacourou (1995) found that humans performed better in ATC simulations using automation providing assistance with sensory/response functions. We hypothesized that performance in our low-fidelity ATC simulation would be superior during trials in which AA was applied to lowerorder sensory/response functions, including information acquisition and action implementation. We suspected that decreases in performance under AA of the decision function might be attributable to the higher cognitive complexity of the automation (i.e., the relative complexity of an operator s mental model for comprehending the impact of the automation function) versus the transparency of automation for the lower-order IP functions and the need to smoothly transition between the use of mental models for automated and manual control during AA (Kaber et al., in press). Based on Hilburn et al. s (1997) research, we expected trials involving manual control to produce greater workload than trials involving automated control. However, on the basis of observations made by Parasuraman et al. (2000), it was expected that automation of higher-order IP functions, including information analysis and decision making, presenting complex displays for operator interpretation might demand high levels of visual attention and increase workload. 2. General methodology A pilot study and primary experiment were conducted as part of this research. The objective of the pilot was to define and validate a direct objective measure of SA for assessment of AA conditions for supporting IP in the ATCrelated task simulation. The pilot was to demonstrate whether the measure was sensitive to differences in the various levels of operator SA during completely manual versus automated control of the simulation. We started with an existing SA measurement methodology and assessed its use in the low-fidelity ATC simulation. The objective of the primary experiment was to use the validated measure of SA to describe the differential effects of AA applied to the various IP functions, including information acquisition, information analysis, decision making and action implementation, in the ATC-related task simulation. In the primary experiment, the performance and workload implications of the various forms of AA were also assessed to provide further evidence on the potential dependence of the effectiveness of AA on the IP function to which automation is applied. Since our simulation was of a limited fidelity, the results presented here may not be directly applicable to actual ATC, but they do shed light on how AA supports IP in terms of SA in complex, dynamic control systems, in general Participants and experimental design We used samples of eight participants for both the pilot and primary experiments. The participants ranged in age from 21 to 29 years and were naïve (i.e., none of the participants had prior flying or ATC experience). In order to control for intra-individual variability in cognitive and psychomotor behavior during the experiments, we elected to use within-subjects designs with limited sample sizes versus between-subjects designs with many participants serving as repeated observations on each of our automation conditions. We blocked on subject through the experimental designs (presented the manual control and automation trials in random order) in order to limit the potential for individual differences to load on any one experimental condition. Repeated measures of individual participant SA, performance and workload were collected during AA of the low-fidelity ATC simulation in multiple trials (2) under each control mode Independent variable The independent variable manipulated in both experiments was the mode of one of the IP functions information acquisition, information analysis, decision making, or action implementation. (The modes of automation are described below in detail in the context of our task.) The automation conditions were compared with the manual control condition in both studies and the full experiment focused on the differential effects of the various forms of AA for supporting IP. Each AA trial consisted of both manual and automated control periods. The number of manual and automated control allocations and minutes during a trial depended on participant workload fluctuations during the ATC-related task simulation Tasks Both the pilot study and primary experiment used a dual-task scenario involving participant performance of the ATC-related simulation (Multitask r ) and a secondary gauge-monitoring task. The Multitask r simulation

6 452 ARTICLE IN PRESS D.B. Kaber et al. / International Journal of Industrial Ergonomics 36 (2006) presented functionality that could be resident in AA to support ATC IP. Since we used naïve participants, the fidelity of the Multitask r was intentionally limited. Many other historical studies have taken a similar approach (e.g., Itoh and Inagaki, 2004; Moray et al., 2000) to demonstrate the effects of forms of automation on IP in low-fidelity simulations of complex processes with naı ve subjects. The secondary task served as a measure of primary-task workload, which was used as a basis for computer mandated control allocations (manual or automated control of primary-task functions) during AA trials. There was no advance warning of the control mode changes for operators, but a salient automation-status display was updated when mode shifts occurred. They were instructed to allocate their attention to the ATC-related task and to the secondary task to the extent they could Primary ATC-related simulation Multitask r is a PC-based simulation with a radarscope (see Fig. 1) presenting approximately 15,400 square nautical miles (nm) of airspace and revealing the position of various types of aircraft to operators, including military, commercial, and private vehicles. Near the center of the radarscope are two airports with approximately 20 nm between them (e.g., Washington Dulles International and Ronald Reagan Washington International). Each airport has two runways for landing aircraft. There are also eight approach, holding fixes represented on the display by small circles located approximately 30 nm from the center of the radarscope. When an aircraft first appears on the scope, it is represented by a white flashing triangular icon and data tag, including the vehicle s call sign. In the pilot study, all aircraft were initially positioned on the periphery of the scope and traveled toward one of the two airports, destined for one of two runways. During the primary experiment, aircraft initially appeared at the edge of a predefined control sector (see the hexagon overlay (dashed line) on the radarscope near the 50 and 60 nm bands), which reduced the total area of control for participants and accelerated the action of the scenario. Once an aircraft is contacted by an operator, the icon becomes solid white and the aircraft s predetermined clearance (including speed, destination airport, and destination runway) is displayed in a data box on the left side of the Multitask r display (see upper-left corner of Fig. 1). If an aircraft is placed in a holding pattern, the icon becomes yellow until the (virtual) pilot is advised to resume the approach to the destination airport, at which time the icon becomes white again. Fig. 1. Multitask r display screen under information acquisition automation.

7 ARTICLE IN PRESS D.B. Kaber et al. / International Journal of Industrial Ergonomics 36 (2006) Military aircraft were generated 10% of the time and traveled at approximately 200 knots. Commercial aircraft were generated the majority of the time (70%) and traveled at approximately 170 knots. Private aircraft were generated 20% of the time and traveled at approximately 100 knots. In real time, all aircraft required between 21 and 42 min to reach an airport during the pilot study and about half this time in the primary experiment. The simulation was programmed to present seven aircraft on the display at any given time and this traffic load only varied by one vehicle during each minute of the task. The goal of the operator in the task was to contact aircraft, make any necessary changes to pre-existing clearances based on their potential to cause conflicts, and safely land aircraft. Clearing an aircraft required two general steps, establishing a communication link and issuing a specific clearance. Participants used a mouse controller to point to, and click on, an aircraft icon. They then selected a query command button from a control box, as part of the Multitask r interface (upper-left corner of Fig. 1), in order to request that a pilot provide aircraft flight information. Subsequently, an operator could decide whether to issue a revised clearance to the aircraft. The control box included eight control command buttons (in total) to facilitate five types of clearances, including reduce speed, hold, resume, change airport, and change runway, as well as two action commands, submit and cancel. A history box, located below the control box, displayed all clearance communications with an aircraft (in text form), simulating ATC-like communications, and pilot responses to clearances. It should be noted here that there were specific clearance constraints built-in to the simulation that would result in a pilot not accepting a clearance and maintaining course (e.g., pilots would not execute hold clearances if an aircraft was within less than 35 nm of its destination airport; change airport clearances could only be issued to vehicles in holding patterns; change runway clearances could only be issued to aircraft inside the band of holding fixes (less than 30 nm from airports)). The steps as part of communicating with and clearing aircraft did not have to be completed in sequence and participants could, for example, query multiple aircraft and then provide multiple clearances. During experimental trials, participants completed the task under one of five different control modes: a manual mode, and four AA modes (one for each IP level as described in Table 2). We used the general descriptions of each form of automation presented by Parasuraman et al. (2000) as a guide for defining the modes of Table 2 Modes of multitask r control (including manual) Mode Manual Information acquisition Information analysis Decision making Action automation Description Manual control no automated assistance was provided. (Operators performed the task, as described in the simulation overview.) A scan line rotated clockwise around the radar display and briefly revealed a trajectory projection aid (TPA) as it passed over aircraft icons (see Fig. 1 and TPA for Aircraft GA2 ). A line connecting the vehicle and the airport (or holding fix) revealed the aircraft destination and route. The aircraft speed (in knots) and destination airport and runway were affixed to the center of the TPA (see lower-right quadrant of radarscope). This form of implementation allocated aspects of the perceptual processing stage of IP to machine control Flight information for each aircraft appearing on the radarscope was displayed in a table in an automation aid box directly beneath the history box, as part of the Multitask r interface. The data included aircraft call signs, destination airports and runways, speeds, and distances (nm) from destinations. A final column in the table showed the call signs of any aircraft currently in conflict (traveling within 3 nm of one another, or two aircraft within 20 nm of the center of the radarscope, destined for the same runway at the same airport). This form of automation assisted operators with storage of aircraft data and integration of perceived information for projecting potential conflicts This mode of automation alerted operators to conflicts (visually during the pilot study and with the addition of audio during the primary experiment), and presented recommendations for conflict resolutions through a table in the automation aid box. (It was observed during the pilot that some aircraft conflicts were not salient to subjects based on their understanding of the conflict criteria. The use of auditory cues on conflicts was implemented for the information analysis and decision making modes in the primary experiment and was in-line with the theoretical descriptions of these forms of automation provided by Parasuraman et al., 2000.) Call signs for conflicting aircraft, recommended clearance changes (reduce speed, hold, resume, change airport, or no option ), and the identifier for a specific aircraft to advise of a necessary change, were all displayed in the table. (The automation was not capable of recommending change runway clearances.) Up to three automated clearance recommendations could be displayed at any time and they were listed in order of priority based on the separation distance and speeds of aircraft. This form of implementation assisted operators with decision and response selection aspects of the task This form of automation simulated the hand-off of aircraft control from approach control to local-tower control, and the tower automatically maintained full control and responsibility for aircraft within 20 nm of the center of the radarscope. The automation prevented any conflicts of aircraft after hand-off to tower control. A data table was presented in the automation aid box and summarized the classification and number of aircraft on the display. The automation assisted users with the response execution function as part of the simulation

8 454 ARTICLE IN PRESS D.B. Kaber et al. / International Journal of Industrial Ergonomics 36 (2006) Multitask r automation. In using ATC as an example system for application of their model of types and levels of automation, Parasuraman et al. (2000) identified specific ATC technologies that they considered to be representative of each form of automation. We modeled the modes of automation in the Multitask r simulation on the basis of descriptions of the various ATC technologies we found in the literature (e.g., Laois and Giannacourou, 1995; Leroux, 1993). Only one aspect of Multitask r performance was automated in each mode of automation Secondary (gauge) monitoring task This task presented a vertical, fixed-scale display with a moving pointer. Participants were required to monitor pointer movements and to detect when a deviation occurred from a central acceptable range on the scale (colored in green ) into an unacceptable region (colored in red ). The gauge display was presented on a computer monitor separate from that used to present the Multitask r simulation and positioned directly to the right of the Multitask r display. Participants were required to correct pointer deviations (return the pointer to the acceptable range) by pressing keys on a keyboard (the ctrl and shift keys if the pointer moved above and below the acceptable region, respectively). There were, on average, four unacceptable pointer deviations (signals) generated per minute in order for the task to constantly test an operator s ability to allocate residual attentional resources from the Multitask r simulation. The number of deviations varied by 71 per min. Once an unacceptable deviation occurred, the pointer remained in the red for no more than 3 s (if not corrected), at which time a miss was recorded and the computer system automatically returned the pointer to the center of the display. Participants were informed in advance of test trials that this would happen. Monitoring performance was measured in terms of the number of unacceptable pointer deviations detected, divided by the total number of deviations (the hitto-signal ratio) Approach to AA As previously mentioned, a workload-based approach to AA was used in this study (cf., Kaber and Riley, 1999). Under AA conditions, each mode of automation switched on and off depending upon operator workload states. To identify the point at which the automated mode would switch on or off during experimental trials, performance data were collected in one of the final training sessions. During this training trial, a cyclic automation pattern was applied to the Multitask r, including 2-min periods of manual control followed by 2-min periods of automated control in one of the four AA conditions. Two participants were randomly assigned to each specific mode of Multitask r automation for this training trial, and performed the gauge task simultaneously. The average participant gaugemonitoring performance level and the standard deviation (SD) for the hit-to-signal ratio on pointer deviations were recorded. The performance for the two participants in each mode was used to establish Multitask r overload and underload conditions for all participants exposed to the same mode of automation during test trials. When a participant performed an experimental trial, if gauge-task performance dropped below the average training trial performance minus 1 SD for 1 min of the trial (implying an increase in workload), the computer system would mandate a switch from manual control of the Multitask r to the mode of automation selected for the trial. Once the automation was initiated, if participant performance in the gauge task increased above the mean training trial performance plus 1 SD, or a hit-to-signal ratio of 1.0 was achieved, for 1 min of a trial, then manual control was restored. These criteria were developed by Clamann et al. (2002) for the closed-loop AA system suggesting operator overload (inability to address the majority of aircraft) and underload (no aircraft conflict situations) at 71 SD about mean gauge monitoring performance Dependent variables Situation awareness We decided to use the SAGAT for SA measurement because of prior research demonstrating validity of the method in various contexts (e.g., aircraft piloting; Endsley, 1995b). To define the measure for use in evaluating the AA and manual control conditions in our experiment, we reviewed cognitive task analyses of ATC operations. Endsley and Rodgers (1994) used goal-directed task analysis to develop a list of SA requirements for en route ATC. Endsley and Jones (1995) performed a similar analysis on Terminal Radar Approach Control center operators. The characteristics of our simulation were similar enough to these domains that we could use the SA requirements lists formulated by the researchers to develop a SAGAT-based approach to measurement. We followed Endsley s (1995b) methodology and conducted multiple simulation freezes (3) during each experiment trial. Trials were 30 min in duration and freezes were designed to occur within one-of-three 8-min windows of trial time to ensure a sampling of SA throughout testing. The first 6 min of a trial were not used for simulation freezes in order to allow participants time to acquire SA (Endsley, 1995b), and to allow aircraft time to move from the periphery of the radarscope towards the airports. Furthermore, no two freezes were scheduled within 2 min of each other in order to allow participants time to reacquire SA after a freeze (Endsley, 1995b). When a freeze occurred, the Multitask r displays were blanked, participants moved to an adjacent workstation and were required to respond to a memory test and six queries (selected at random from a pool of 18) targeting the three levels of SA defined by Endsley (1995a). The memory test during the pilot experiment required participants to recall the current locations of all (7) aircraft on the display

9 ARTICLE IN PRESS D.B. Kaber et al. / International Journal of Industrial Ergonomics 36 (2006) by marking-up a graphic of the radarscope with a pencil. In addressing the subsequent queries, participants used a database application to complete tables of responses for each question for all aircraft. There was no time limit on participant responding and once they completed all queries, they immediately resumed the ATC simulation. SA was quantified as the percentage of correct responses to Level 1, 2 and 3 SA queries and a Total SA score at each freeze. Participant answers were compared with ground truth information recorded by the simulation computer system at the time of each stop (or by comparison with expert experimenter responses to the same SA queries) Primary-task performance With respect to measurement of participant performance in the Multitask r simulation, the number of aircraft cleared (and safely arriving at an airport), conflicts and collisions (aircraft that come in contact with each other or that simultaneously arrive at the same airport destined for the same runway) were recorded. All of these measures were captured during each minute of a test trial Workload Operator workload was assessed in terms of the secondary-task performance. The hit-to-signal ratio on pointer deviations in gauge monitoring was considered as an objective indicator of primary-task (Multitask r ) workload. 3. Pilot study For the pilot study, participants were provided with a rigorous training protocol for both tasks, including experience under each mode of Multitask r automation. They were also trained on the SAGAT measurement method. We tested the participants over a period of 2 days. The SA response measure was determined for both automated and manual control periods as part of AA trials. Comparisons were made of the SA results under manual versus automated control. We found the general type of control to have an affect on higher (Level 3) SA, but the measure was not sensitive to revealing differences in the fundamental aspects of SA, including perceptual processing (Level 1 SA) or comprehension (Level 2 SA), due to the automation characteristics. Consequently, the Total SA score was also not sensitive. We concluded that the standard SAGAT metric might not be satisfactory for demonstrating AA support of IP in our ATC-related task. Since our SA queries as part of the SAGAT were based on actual ATC operations analyses, we were convinced of the construct validity of the measure for our simulation. However, we suspected there might be methodological issues relevant to the domain, in general. 4. Primary experiment 4.1. Revision to SA measure design Some recent work by Hauss and Eyferth (2003) suggested that SAGAT may not be a sensitive measure for SA in the ATC environment due to different aircraft having different relevance to controllers at different times in traffic management. They said that controllers may use event-based mental representations of air traffic situations in order to determine what information is currently relevant, what information will be relevant in the future, and what information can be neglected in an attempt to make the task manageable from a working memory perspective. With respect to the terminology event-based mental representations, Hauss and Eyferth (2003) said that aircraft that have recently been contacted by a controller and that have required recent control actions, or aircraft that are currently (or will soon be) in conflict, may demand more attentional resources than other displayed aircraft and this may drive states of controller awareness. Consequently, controllers may focus on certain aircraft to the exclusion of others at various times during control activities and they may more accurately recall the flight parameters of aircraft most relevant to current task performance in responding to SAGAT queries versus the parameters for aircraft that are notcriticaltoatcatthetimeofasagatfreeze. It is possible that in our pilot study, participants simply responded accurately to SAGAT queries for the aircraft they had recently dealt with and poorly for those not in the focus of their attention. Since each query was posed for all aircraft on the display at a freeze, the average percent correct responses to a single query could have remained relatively constant across conditions due to the aircraft relevance issue. Consequently, we attributed our lack of SA findings, and sensitivity of the measurement technique, to the approach of assessing participant recall of aircraft and the administration of the SA queries on all aircraft. We sought to modify our SA measure to promote sensitivity to differences in IP between the manual and automated ATC-related task conditions and to describe the differential effects of the various forms of AA on operator perception, comprehension and projection. We returned to the literature to find other studies for direction on refining the SA measurement approach. Aside from the Hauss and Eyferth (2003) study mentioned above, we found that Nunes (2003) assessed the performance and SA of ATC trainees in an experimental task by using the SAGAT. In his study, controllers were either aided or unaided when evaluating pilot requests for flight plan deviations. Trainees in the aided condition were given a data link trajectory of the proposed deviation, while those in the unaided condition had to extrapolate the trajectories. Nunes (2003) found that the SAGAT did not reveal differences between display conditions, while the performance measures of response time and response accuracy revealed a significant effect of display condition.

10 456 ARTICLE IN PRESS D.B. Kaber et al. / International Journal of Industrial Ergonomics 36 (2006) Nunes (2003) argued that SA might not have varied between groups because both aided and unaided controllers essentially processed the same information, but the processing occurred at different levels (greater for unaided controllers). However, Nunes provided no explanation on how IP could be affected by the aiding condition without an effect on SA, as operationally defined through the SAGAT. We suspected that the SAGAT results Nunes (2003) obtained, like ours, could be attributed to the implementation approach to the measurement technique. We presumed that his method, like that employed in the pilot study, placed equal emphasis on aircraft that controllers may have considered irrelevant at certain points in task performance, which may have limited sensitivity of Nunes measure to the condition manipulations. Hauss and Eyferth s (2003) concerns with the SAGAT for assessing controller SA led them to develop a new measure of SA called SALSA, which weighted aircraft based on their relevance to the current control scenario. The SALSA involves expert rating of replay of an air traffic management (ATM) simulation to determine the relevance of each task element (aircraft) to controllers. Only elements that are judged as relevant in the replay are considered for SA queries during subsequent test trials. In addition, rather than having participants recall aircraft positions on a blank radarscope, SALSA involves cued recall, in which participants are given the positions for the aircraft on which they are to be queried. Hauss and Eyferth (2003) contended that these changes in the SA measurement approach reduce the possibility of participants confusing two aircraft positions during free recall and take into account the air traffic controller s use of an event-based mental representation of their task. Hauss and Eyferth (2003) compared SALSA with SAGAT using a high-fidelity ATM simulation. Their empirical results confirmed that controllers used eventbased mental representations, since significantly more relevant parameters than irrelevant parameters were reproduced using the SALSA measure. Trends on the results of the SALSA measure also corresponded with changes in workload in intuitive ways. On this basis, we decided to investigate a modified approach to implementation of our SAGAT measure. We implemented cued recall of aircraft positions (in comparison to free recall), and we objectively weighted the relevance of aircraft as a basis for queries during simulation freezes. While Hauss and Eyferth (2003) used expert ratings on videos of their simulation scenarios to determine aircraft relevance at particular points in time, the approach we took involved real-time identification of aircraft in conflict, as well as those that had recently been issued clearances (e.g., hold, reduce speed, etc.) or queried during the simulation, as predictors of aircraft relevance to operators and selection as candidates for queries. It was expected that these modifications would lead to a more sensitive measure of the impact of AA of various IP functions on operator SA. It is important to note that although cueing operators on aircraft location during SAGAT freezes may promote sensitivity of the measure to the display condition, it is possible that cued recall of the presence of aircraft in a particular airspace and parameters of specific aircraft may impact operator SA and task performance following SAGAT freezes, as well as future SA. This was one potential drawback of the modified approach Procedures and variables The steps in the training procedure for the full experiment included: (1) 20 min of practice on the Multitask r and gauge monitor simulations; (2) 1 h of practice on the various modes of Multitask r automation; (3) 40 min of training on AA of the simulation, including a dual-task practice session in which participants performed gauge monitoring and the Multitask r under a randomly assigned mode of automation; and (4) familiarization with the SAGAT method followed by a 50-min dual-task practice session involving SAGAT freezes and SA queries (randomly selected from the pool of test queries). Testing was conducted on 2 days (separate from the training) during a 3-week period. Participants reviewed the simulation procedures on each day and were allowed to practice (15 min) the modes of automation to be tested. They completed five test trials on each day (5 modes of control 2 trials/mode) with each trial lasting 50 min and including SAGAT freezes. Test trial order was randomized across the 2 days. In total, each participant was trained and tested for 13 h. With respect to the SA measure, three SAGAT freezes were again conducted at random times during each experimental trial. When a freeze occurred, the Multitask r display was blanked and participants moved to the adjacent SAGAT workstation. An experimenter collected information on any aircraft in conflict from the Multitask r software by accessing a conflict detection aid, which was hidden from participants during testing. The aid also presented any recommended clearances. Based on these data, and position and speed information for each vehicle, the experimenter identified the three aircraft with the highest priority for clearances, or greatest relevance, at that point in time in the simulation. To facilitate the cued recall of aircraft as a basis for SA queries, the experimenter quickly sketched the locations of the high priority aircraft on a graphic of the Multitask r radarscope. The participants were given the drawing and required to respond to nine SA queries on each high priority aircraft, including three queries targeting each level of SA (perception, comprehension and projection). In total, participants answered 81 SAGAT queries (3 freezes 9 queries 3 high-priority aircraft) during each trial. The Multitask r and gauge monitoring performance measures were recorded during each trial of the study.