Predictors of Threat and Error Management: Identification of Core Nontechnical Skills and Implications for Training Systems Design

Transcription

1 THE INTERNATIONAL JOURNAL OF AVIATION PSYCHOLOGY, 14(2), Copyright 2004, Lawrence Erlbaum Associates, Inc. Predictors of Threat and Error Management: Identification of Core Nontechnical Skills and Implications for Training Systems Design Matthew J. W. Thomas University of South Australia Adelaide, Australia In normal flight operations, crews are faced with a variety of external threats and commit a range of errors that have the potential to impact negatively on the safety of airline operations. The effective management of these threats and errors therefore forms an essential element of enhancing performance and minimizing risk. Recent research has reinforced the need to examine a range of nontechnical or crew resource management skills that form threat and error countermeasures. This article provides an analysis of the predictors of threat and error management in normal flight operations within the context of a Southeast Asian airline. Through the structured observation of crews performance during normal flight operations, data were collected in relation to a set of contextual factors and nontechnical skills. Crews threat and error management actions were then analyzed in relation to these factors, and predictive models of threat and error management at various phases of flight were developed. The results of this study demonstrate the ways in which this type of data analysis can highlight the strengths and weaknesses of operational performance and suggest that this type of performance evaluation can offer individual organizations invaluable information for enhanced training system design through the further development of scenario-based training. For any organization involved in high-risk operations, the adequate performance of personnel is a crucial aspect of maintaining safety. It is now clearly understood that any failures of safety stem not simply from isolated incidences of human error Requests for reprints should be sent to Matthew J. W. Thomas, Flexible Learning Centre, University of South Australia, City West Campus, Y1 35 Yungondi Building (GPO Box 2471) North Terrace, Adelaide SA matthew.thomas@unisa.edu.au

2 208 THOMAS but rather from wide-ranging organizational factors. Encapsulating considerable research into organizational safety, models of accident trajectories have been developed that include both active failures of personnel and systems as well as latent conditions that may lie dormant in an organization s operational system for considerable time (Reason, 1990, 1997). Closely aligned to the concept of active failures and latent conditions are the terms error and threat, respectively, concepts that have recently been the focus of considerable research in the commercial aviation setting. Defined as situations, events, or errors that occur outside the flight-deck, threats are conditions that have the potential to impact negatively on the safety of a flight. In turn, defined as crew action or inaction that leads to a deviation from crew or organizational intentions or expectations, errors are taken to be an unavoidable and ubiquitous aspect of normal operations. As these two factors form fundamental causal components of incidents and accidents, it is argued that the management of threat and error must form the focus of any organization s attempts to effectively maintain safety in high-risk operations (Klinect, Wilhelm, & Helmreich, 1999). THREAT AND ERROR MANAGEMENT AND NONTECHNICAL SKILL DEVELOPMENT Threat and error management involves the effective detection and response to internal or external factors that have the potential to degrade the safety of operations (Helmreich, Klinect, & Wilhelm, 1999). From this new approach to safety has emerged a redefined emphasis on training that extends beyond merely the development of technical proficiency in areas such as system operation and specific psychomotor skills. Central to the emergent focus on threat and error management is the position that effective operational performance is dependent on the integrated use of specific technical skills and generic nontechnical skills such as cognitive and interpersonal skills. The specific development of crews nontechnical skills is not a recent addition to aviation training. In direct response to the realization of the contribution of human factors to incidents and accidents, training programs were developed to focus on nontechnical skills. Crew resource management (CRM), defined as the crews effective use of all available resources to achieve safe and efficient flight operations, has historically been an important focus toward the reduction of human error and the enhancement of safety (Lauber, 1987; Wiener, Kanki, & Helmreich, 1993). The primary focus of CRM was the development of discrete nontechnical skills such as communication, leadership, decision making, and conflict resolution as well as stress and fatigue management (Helmreich & Wilhelm, 1991). Evolving through a number of successive generations, CRM training has in recent times

3 PREDICTORS OF THREAT AND ERROR MANAGEMENT 209 been reconceptualized explicitly as the development of threat and error countermeasures (Helmreich, Merritt, & Wilhelm, 1999). The effective training of personnel in threat and error management will not be able to provide a panacea that eliminates the human contribution to incidents and accidents (Helmreich & Merritt, 2000). However, specifically tailored training programs can certainly make a critical contribution to the minimization of risk. It is now well understood that flight crew act as the last line of defense in what is often a flawed sociotechnical system within which aircraft operate (Reason, 1997). Therefore, by equipping crew members with skills in threat and error management, an organization can provide additional defenses against both active failures as well as latent conditions that may lie undetected within the organization or the broader operating environment. CURRENT DEFICIENCIES IN THREAT AND ERROR MANAGEMENT TRAINING The development of crews nontechnical skills through such mechanisms as Human Factors courses and CRM training is now a widespread regulatory requirement. However, the development of nontechnical skills remains a problematic area of aviation training. As Trollip (1995) argued, one of the major problems facing nontechnical skill development by flight crew is that the traditional approaches to training are generally ineffective for the development of nontechnical skills. It has been demonstrated recently that there is a lack of coherence in the approaches to nontechnical skill development across airlines. Significant difference exists in relation to whether the training adopts a focus on crews attitudes or specific behaviors as well as in relation to the specific labels, descriptions, and representations of the attitudes or skills that are the focus of training (Salas, Rhodenizer, & Bowers, 2000). Furthermore, it is evident that the instructional techniques employed in the development of nontechnical skills require further development. For instance, recent studies have indicated that nontechnical skills frequently remain neglected in the evaluation and debriefing of pilots in regular check situations (e.g., Hörmann, 2001). Current approaches to flight crew training have also been criticized for not sufficiently integrating crews technical and nontechnical skill development. Frequently, flight crew training involves a preliminary focus on aircraft technical knowledge and operational procedures, with the development of nontechnical knowledge and skills occurring in latter stages of training and in isolation from real-world operational contexts (Johnston, 1997). This lack of integrated training presents a major barrier to the development of effective threat and error management skills. In short, there is a lack of clarity about the relative importance of a wide range of attitudes and behaviors that contribute to effective threat and error

4 210 THOMAS management and a deficit in the industry s understanding of the most effective mechanisms for their training. TOWARD A MORE DETAILED UNDERSTANDING OF THREAT AND ERROR MANAGEMENT There is little doubt that understanding how threats, errors, and their management interact to determine the quality of operational performance is critical to safety in high-risk industries (Helmreich, 2000). Although the link between nontechnical skills and threat and error management is obvious, there remains a significant need to develop a more detailed understanding of the factors that contribute to effective threat and error management. Moreover, to design effective training curricula in this area, it is necessary to increase the depth of one s understanding of threat and error management in the environment of normal operations. New tools for operational performance evaluation provide unprecedented opportunities for the analysis of threat and error management during normal flight operations. In particular, the Line Operations Safety Audit (LOSA) methodology developed by the Human Factors Research Project at the University of Texas enables the collection of detailed data in relation to the occurrence of threats and errors during normal operations and details of the types of behaviors crews employ in response to these threats and errors (Helmreich, Klinect, et al., 1999; Klinect et al., 1999). The study I present in this article involves the analysis of the types of contextual factors and nontechnical skills that contribute to threat and error management by flight crews during normal flight operations. The aim of this research is to provide a more detailed understanding of threat and error management behaviors with the objective of better informing training system design. METHOD Participants The study was undertaken within a Southeast Asian airline operating both domestic and international routes as part of a broader project involving a structured evaluation of both normal operations and flight crew training (Thomas, 2003). Normal line operations from two fleets of the airline were examined, one comprised of Boeing and aircraft flying domestic short-haul operations and the other comprised of Airbus A aircraft flying medium-haul international routes. The primary focus of behavioral analysis was that of the flight crew, which was consistently comprised of a two person active crew of a Captain and a

5 PREDICTORS OF THREAT AND ERROR MANAGEMENT 211 First Officer. However, some additional specific data was collected in relation to the actions of individual crew members. Design and Procedure In the study, I adopted an observational design and employed a highly structured observational performance evaluation methodology for data collection and analysis. Data were collected by a group of 25 senior flight crew from the airline who were trained in using the LOSA methodology (Helmreich, Klinect, et al., 1999; Klinect et al., 1999). The observers collected data from the jump seat during normal line operations. A 2-day training session was held for the observers and covered all logistic and technical aspects of data collection. The standardization of observers was a major focus of the training sessions, and interrater reliability was established through a process of reflective analysis of videotaped examples of crew performance. In total, 323 sectors (individual airport-to-airport flights) of normal operations were observed (approximately 200 for the B737 fleet and 100 for the A330 fleet). Measures Dependent variables. The major focus of this study was the threat and error management of flight crew during normal line operations. The outcomes of crews threat management actions were simply recorded as a single dichotomous variable, namely, whether the crew had or had not effectively managed the threat. For instance, in relation to an encounter with adverse weather en route, effective detection of the weather, and appropriate actions undertaken to avoid weather penetration would be coded as an effectively managed threat. Conversely, if the crew failed to detect the threat or if their management actions lead to an error being committed, the threat would be coded as not effectively managed. The crews error management actions were initially coded under three categories according to the threat and error management model: (a) the error was trapped, which means that it was detected and managed before it became consequential; (b) the error was exacerbated, which means the error was detected, but the crew s action or inaction lead to a negative outcome; and (c) the crew could fail to respond to the error, which means the crew either failed to detect or ignored the error. Due to the limited numbers of errors that were exacerbated, data were recoded to form two distinct dichotomous dependent variables: first, whether the crew had or had not trapped the error and second, whether the crew had or had not failed to respond to an error. Independent variables. A set of contextual factors and crews nontechnical skills were examined as possible predictors of threat and error management.

6 212 THOMAS Although not specifically manipulated within the observational design, both the contextual factors and nontechnical skill ratings were considered as important independent variables, as they are frequently identified as essential mediators of operational performance and safety. First, during each flight, data were collected on a series of seven contextual factors that might impact on the crews management of threats and errors. These seven contextual factors are listed in Table 1. Crews nontechnical performances were evaluated using set of behavioral markers adapted from the existing NOTECHS and LOSA methodologies. A broad four-category structure of communication, situation awareness, task management and decision making was adapted from the NOTECHS system, which has been developed under European regulations (Flin, Goeters, Hörmann, & Martin, 1998). Under these four categories, 16 behavioral markers were developed to provide performance data on a wide range of competencies that have been specifically identified as threat and error avoidance, detection, and management actions (Helmreich, Wilhelm, Klinect, & Merritt, 2001). Observers recorded scores against each of the four categories of nontechnical skill for each flight observed to gain an overall impression of crew performance at the macro level. At a higher level of resolution, observers recorded scores against the 16 individual behavioral markers for each of five phases of flight for every flight observed. The four categories and 16 behavioral markers are found in Table 2. The observational methodology also involved the collection of qualitative data by the observers in the form of a written narrative. In relation to each quantitative measure embedded within the methodology, observers were required to write a descriptive narrative of the events they observed, which described crew actions and justified the observer s coding and rating of threat and error management and nontechnical skills. This information was used to develop a richer understanding of the crews actions and was used in the overall analysis and interpretation of data. Variable Length of flight (min) Departure time Late departure Number of threats Number of errors Experience of the Captain (years) Experience of the First Officer (years) TABLE 1 Independent Variables Identified as Contextual Factors for Threat and Error Management

7 PREDICTORS OF THREAT AND ERROR MANAGEMENT 213 TABLE 2 Behavioral Markers for Core Nontechnical Skills Category Communication Situation Awareness Task Management Decision Making Behavioral Marker Communication Environment Leadership followership Inquiry Assertiveness Cooperation Statement of plans and changes Vigilance Monitoring and cross-check Briefing and planning Workload management Workload prioritization Automation management Management of fatigue and stress Contingency planning Problem identification Evaluation of plans Statistical Analysis Once the data were collected, they were collated and subjected to both descriptive and multivariate statistical analysis. As the dependent variables describing threat and error management outcomes were recorded as dichotomous nominal values, logistic regression was utilized as the most appropriate multivariate technique. Logistic regression is used as an alternative to standard multiple regression techniques in describing and testing hypotheses about relations between a categorical outcome variable and one or more predictor variables (Cizek & Fitzgerald, 1999). Further, logistic regression was chosen as an alternative to discriminant analysis that involves assumptions of multivariate normality. According to standard procedures, independent variables that were not recorded as continuous values were recoded and analyzed as categorical variables (Peng, Lee, & Ingersoll, 2002). The logistic regression models were subjected to overall model evaluation and tests of goodness of fit to ensure the models were sound. One possible implication of the study design was the potential for common method variance (CMV) to result in a circularity of the relations between observers perceptions of threat and error management and their ratings of crews nontechnical skills. As Kline, Sulsky, and Rever-Moriyama (2000) suggested, CMV is a common problem facing self-report measures in which relations between a range of variables are detected in data collected using the same instrument. In these situations, the relations that emerge might be the result of a common relation with a

8 214 THOMAS third spurious and unmeasured variable rather than an independent relation between the two or more variables as measured. Within this study, it was suggested that CMV is unlikely to effect the statistical analysis and findings for two reasons: (a) significant independence between the variables measured by the observers was well defined within the measurement protocols, and (b) observers were required to justify their ratings with clear evidence of observed crew behaviors that were subjected to independent review by a data-cleaning roundtable. These research design strategies effectively minimized possible observer bias and reduced the likelihood of CMV effects. RESULTS Descriptive Analysis of Threat and Error Management From the 323 observations of line operations, 451 individual threats were observed and recorded. This was an average of 1.4 threats per observed flight operation. Klinect et al. (1999) reported a similar average number of threats per flight, with data from 314 sectors indicating an average of 1.9 threats per flight. Marked interairline variability was evident in the baseline study, with the average number of threats ranging from 0.3 to 3.3 per flight. This considerable range highlights the utility of threat analysis in identifying the unique operational characteristics of an airline or fleet s operating environment. On average, in the line operation environment, 57.4% of threats in this study were well managed by the crews. However, the remaining 42.6% of threats were poorly managed by the crews. As a threat, by definition, has the potential to be detrimental to the safety of the flight, it was hoped that a much larger proportion of threats would have been effectively managed by the crews. There were 22 different types of threats faced by flight crews during normal operations. Table 3 shows the distribution of the 10 most common threats, which accounted for 90% of all threats encountered. From the 323 line observations, a total of 508 errors were committed, representing an average of 1.57 errors per flight. Captains were responsible for the majority of the errors, with the First Officer responsible for less than one fourth of the errors. Although procedural errors were the most common type of errors committed by flight crew, a significant proportion of intentional noncompliance errors were committed, demonstrating a propensity for flight crew to violate standard operating procedures (SOPs) and regulations. In coding an intentional noncompliance error, observers were required to demonstrate demonstrable behaviors that indicated that the noncompliance was intentional. Although not suggesting that crew attitudes, values, or behavior are at fault, this error type highlights the need for further investigation in relation to the possible root causes of intentional noncompli-

9 PREDICTORS OF THREAT AND ERROR MANAGEMENT 215 TABLE 3 The Most Common Threats Encountered by Flight Crews and Their Management Threat Type Frequency (% of All Threats) a % Managed % Not Managed Weather Aircraft malfunctions Operational pressure Traffic ATC command Airport conditions Terrain Ground handling event Passenger event Communication event a N = 451. ance including procedure design and operational necessity. A summary of the errors and their management by flight crew is provided in Table 4. Nearly half of all errors remained undetected by flight crew. First Officers performed better than Captains in error detection, and air traffic control also detected a significant number of errors before flight crew were able to do so. Less than one fourth of all errors were effectively managed by flight crew, with the most common error management response being a failure to respond to errors. Predictors of Effective Error Management At a macro level, error management during all phases of flight was examined in relation to the seven contextual factors and the four major categories of nontechnical performance. Using logistic regression, a model was developed for effective error management, which examined the predictors of errors being trapped by the flight crew. As described in Table 5, it was found that three of the contextual factors and nontechnical skills were found to be significant predictors of errors being trapped by the flight crew. Of the nontechnical skills, it was found that, controlling for all other independent variables, an increase in the crews decision-making performance was linked to an increased likelihood of the error being trapped. Similarly, the more experienced the First Officer, the more likely errors were to be trapped. However, if the Captain committed the error, it was significantly less likely to be trapped by the flight crew. These findings demonstrate a delicate balance between the benefits of experience and the negative effects of power on crew interaction and performance. When examining crews failure to respond to errors, it was found that the experience of the First Officer and whether the Captain had committed the error were

10 216 TABLE 4 A Summary of Errors Committed by Flight Crews and Their Management Error Origin Frequency (% of All Errors) a Error Type Frequency (% of All Errors) a Error Detection Frequency (% of All Errors) a Error Response Frequency (% of All Errors) a Captain 60.8 Intentional 38.4 Captain 15.9 Trap 23.6 noncompliance First Officer 24.8 Procedural 41.3 First Officer 24.2 Exacerbate 13.2 Both crew 14.4 Communication 6.7 Both crew 3.3 Fail to respond 63.2 Proficiency 3.3 Air traffic 7.7 control Decision making 10.2 Nobody 46.9 Other 2.0 a N = 508.

11 PREDICTORS OF THREAT AND ERROR MANAGEMENT 217 TABLE 5 Logistic Regression Analysis Contributions of Contextual Factors and Nontechnical Skills to Errors Being Trapped by Flight Crew During All Phases of Flight Predictor β SE β Wald s χ 2 df p e β Experience of First Officer Decision making Captain as error origin Constant Note. Model χ 2 = , df =3,p <.001. Nagelkerke R 2 =.234; 79.4% correct classification. In reporting logistic regression, β refers to the regression coefficient. The statistical significance of the regression coefficient is measured using Wald s χ 2 with p <.05 used as the test of significance. The e β refers to the odds ratio that provides an indication of the change in the likelihood of the dependent variable associated with changes in the independent variable. significant in the prediction of ineffective error management. Another important predictor of ineffective error management as shown in Table 6 was the late departure of the flight. If the flight was late to depart, crews were 1.89 times more likely to fail to respond to an error. Although these results identify core contextual factors and nontechnical skills at the macro level, by examining errors committed during specific phases of flight, a more finer resolution analysis was achieved. Of the errors committed by flight crew, 85% occurred during the predeparture, takeoff, and descent-approach-land- TABLE 6 Logistic Regression Analysis Contributions of Contextual Factors and Nontechnical Skills to Crews Failure to Respond to Errors During All Phases of Flight Predictor β SE β Wald s χ 2 df p e β Late departure Experience of First Officer Captain as error origin Constant Note. Model χ 2 = , df =3,p <.001. Nagelkerke R 2 =.109; 66.6% correct classification. In reporting logistic regression, β refers to the regression coefficient. The statistical significance of the regression coefficient is measured using Wald s χ 2 with p <.05 used as the test of significance. The e β refers to the odds ratio that provides an indication of the change in the likelihood of the dependent variable associated with changes in the independent variable.

12 218 THOMAS TABLE 7 Logistic Regression Analysis Contributions of Contextual Factors and Nontechnical Skills to Errors Being Trapped by Flight Crew During the Predeparture Phase Predictor β SE β Wald s χ 2 df p e β Cooperation Constant Note. Model χ 2 = , df =1,p <.005. Nagelkerke R 2 =.379; 72.2% correct classification. In reporting logistic regression, β refers to the regression coefficient. The statistical significance of the regression coefficient is measured using Wald s χ 2 with p <.05 used as the test of significance. The e β refers to the odds ratio that provides an indication of the change in the likelihood of the dependent variable associated with changes in the independent variable. ing phases of flight. Accordingly, crews error management behaviors were examined against data from the seven contextual factors and 16 behavioral markers of nontechnical skills for each of these three phases of flight. During the predeparture phase, it was found that only one nontechnical skill emerged as a significant predictor of errors being trapped by flight crew. As the model in Table 7 indicates, an increase in crew cooperation resulted in a marked increase in the odds of an error being trapped. It was found that during the predeparture phase, higher levels of contingency planning by the flight crew significantly predicted a decrease in the likelihood that the crew would fail to respond to an error (Table 8). This finding suggests that crews who actively engage in predeparture contingency planning may have heightened expectation that unexpected errors might occur and are thus less likely to fail to detect or ignore them. During the takeoff phase, contingency planning was also found to be a significant predictor of effective error management, thus confirming its core function as TABLE 8 Logistic Regression Analysis Contributions of Contextual Factors and Nontechnical Skills to Crews Failure to Respond to Errors During the Predeparture Phase Predictor β SE β Wald s χ 2 df p e β Contingency planning Constant Note. Model χ 2 = , df =1,p <.005. Nagelkerke R 2 =.346; 75.0% correct classification. In reporting logistic regression, β refers to the regression coefficient. The statistical significance of the regression coefficient is measured using Wald s χ 2 with p <.05 used as the test of significance. The e β refers to the odds ratio that provides an indication of the change in the likelihood of the dependent variable associated with changes in the independent variable.

13 PREDICTORS OF THREAT AND ERROR MANAGEMENT 219 an error countermeasure. However, increased workload management, defined as the effective management of workload through the definition of roles and responsibilities and the appropriate sharing of operational tasks, was found to decrease the likelihood of errors being trapped and increase the likelihood of failure to respond to errors. A number of possibilities can account for this somewhat anti-intuitive finding. First, it is possible that heightened levels of workload management result in a diversion of cognitive resources that might be better focused on other areas that are more effective threat and error countermeasures. A second suggestion is that the often highly proceduralized tasks assigned to each crew member during this phase of flight might be working against error detection and management. In short, the script that each crew member must follow could leave little room for effective threat and error management. The predictive models for error management during the takeoff phase can be found in Table 9 and Table 10. The descent-approach-landing phase of flight is typically characterized by increased workload and an overall increase in risk. It was found that the nontechnical skills of vigilance and problem identification were significantly related to increased likelihood of errors being trapped by crews during the descent. Both these nontechnical skills are linked closely to the notion of situation awareness and refer to the perception and comprehension of critical operational cues. As shown in Table 11, it was also found that an increase in the number of errors committed by crews decreases the likelihood of errors being trapped. If the Captain was responsible for the error, it was again found that the likelihood of the crew failing to respond to the error increased. However, the demonstration of high levels of assertiveness by the crew and one would suggest the First Officer was linked to a decrease in the likelihood that the crew would fail to detect or ignore the error. These findings reinforce the important contribution of the climate and power TABLE 9 Logistic Regression Analysis Contributions of Contextual Factors and Nontechnical Skills to Errors Being Trapped by Flight Crew During the Takeoff Phase Predictor β SE β Wald s χ 2 df p e β Workload management Contingency planning Constant Note. Model χ 2 = , df =2,p <.001. Nagelkerke R 2 =.432; 77.6% correct classification. In reporting logistic regression, β refers to the regression coefficient. The statistical significance of the regression coefficient is measured using Wald s χ 2 with p <.05 used as the test of significance. The e β refers to the odds ratio that provides an indication of the change in the likelihood of the dependent variable associated with changes in the independent variable.

14 220 THOMAS TABLE 10 Logistic Regression Analysis Contributions of Contextual Factors and Nontechnical Skills to Crews Failure to Respond to an Error During the Takeoff Phase Predictor β SE β Wald s χ 2 df p e β Workload management Contingency planning Constant Note. Model χ 2 = , df =2,p <.005. Nagelkerke R 2 =.290; 73.5% correct classification. In reporting logistic regression, β refers to the regression coefficient. The statistical significance of the regression coefficient is measured using Wald s χ 2 with p <.05 used as the test of significance. The e β refers to the odds ratio that provides an indication of the change in the likelihood of the dependent variable associated with changes in the independent variable. structure on the flight deck and emphasize the need for assertiveness during high-risk situations. The model describing contributions of contextual factors and nontechnical skills to crews failure to respond to errors is provided in Table 12. Predictors of Effective Threat Management At the macro level, it was found that two contextual factors and two nontechnical skill categories were significant predictors of effective threat management by flight crews during all phases of flight. First, the more experienced the First Offi- TABLE 11 Logistic Regression Analysis Contributions of Contextual Factors and Nontechnical Skills to Errors Being Trapped by Flight Crew During the Descent-Approach-Landing Phase Predictor β SE β Wald s χ 2 df p e β Number of errors Vigilance Problem identification Constant Note. Model χ 2 = , df =3,p <.001. Nagelkerke R 2 =.515; 87.6% correct classification. In reporting logistic regression, β refers to the regression coefficient. The statistical significance of the regression coefficient is measured using Wald s χ 2 with p <.05 used as the test of significance. The e β refers to the odds ratio that provides an indication of the change in the likelihood of the dependent variable associated with changes in the independent variable.

15 PREDICTORS OF THREAT AND ERROR MANAGEMENT 221 TABLE 12 Logistic Regression Analysis Contributions of Contextual Factors and Nontechnical Skills to Crew s Failure to Respond to an Error During the Descent-Approach-Landing Phase Predictor β SE β Wald s χ 2 df p e β Captain as error origin Assertiveness Constant Note. Model χ 2 = , df =2,p <.001. Nagelkerke R 2 =.293; 70.8% correct classification. In reporting logistic regression, β refers to the regression coefficient. The statistical significance of the regression coefficient is measured using Wald s χ 2 with p <.05 used as the test of significance. The e β refers to the odds ratio that provides an indication of the change in the likelihood of the dependent variable associated with changes in the independent variable. cer, the more likely it was that crews would effectively manage threats. Conversely, an increase in the number of errors decreased the likelihood that threats were effectively managed. Both situation awareness and decision making were positively linked to effective threat management, highlighting the importance of these categories of nontechnical skills. The overall model of predictors of effective threat management can be found in Table 13. It was found that the majority of threats were encountered by flight crews during the predeparture and the descent-approach-landing phase of flight. Accordingly, the threat management performance of crews was analyzed for these two phases of flight against the seven contextual factors and 16 behavioral markers of TABLE 13 Logistic Regression Analysis Contributions of Contextual Factors and Nontechnical Skills to Crew s Effective Threat Management During All Phases of Flight Predictor β SE β Wald s χ 2 df p e β Experience of First Officer Situation awareness Decision making Number of errors Constant Note. Model χ 2 = , df =4,p <.001. Nagelkerke R 2 =.665; 87.3% correct classification. In reporting logistic regression, β refers to the regression coefficient. The statistical significance of the regression coefficient is measured using Wald s χ 2 with p <.05 used as the test of significance. The e β refers to the odds ratio that provides an indication of the change in the likelihood of the dependent variable associated with changes in the independent variable.

16 222 THOMAS TABLE 14 Logistic Regression Analysis Contributions of Contextual Factors and Nontechnical Skills to Crews Effective Threat Management During the Predeparture Phase Predictor β SE β Wald s χ 2 df p e β Number of threats Number of errors Statement of plans and changes Briefing and planning Constant Note. Model χ 2 = , df =4,p <.001. Nagelkerke R 2 =.825; 92.4% correct classification. In reporting logistic regression, β refers to the regression coefficient. The statistical significance of the regression coefficient is measured using Wald s χ 2 with p <.05 used as the test of significance. The e β refers to the odds ratio that provides an indication of the change in the likelihood of the dependent variable associated with changes in the independent variable. nontechnical skills. During the predeparture phase, a model of effective threat management emerged that included four factors. First, the nontechnical skills of statement of plans and changes and briefing and planning were found to contribute significantly to increased likelihood of effective threat management. An increase in the number of errors committed by crews was found to decrease the likelihood of effective threat management. However, if crews encountered an increased number of threats, they were more likely to engage in effective threat management. This model is shown in Table 14. A similar predictive model for effective threat management emerged in relation to threats encountered during the descent-approach-landing phase of flight. The experience of the First Officer was again found to be related to an increase in the likelihood of effective threat management, whereas an increase in the number of errors was related to a decrease in the likelihood of effective threat management as described in Table 15. Overall, the individual predictive models of threat and error management highlight a range of contextual factors and nontechnical skills that act as threat and error countermeasures. The difference between the models for each threat and error management task and each phase of flight have significant implications for training system design. DISCUSSION Predictors of Threat and Error Management In this study, I investigated the contribution of contextual factors and nontechnical skills to threat and error management during normal flight operations. Each of the

17 PREDICTORS OF THREAT AND ERROR MANAGEMENT 223 TABLE 15 Logistic Regression Analysis Contributions of Contextual Factors and Nontechnical Skills to Crews Effective Threat Management During the Descent-ASpproach-landing Phase Predictor β SE β Wald s χ 2 df p e β Experience of First Officer Number of errors Constant Note. Model χ 2 = , df =2,p <.001. Nagelkerke R 2 =.450; 71.9% correct classification. In reporting logistic regression, β refers to the regression coefficient. The statistical significance of the regression coefficient is measured using Wald s χ 2 with p <.05 used as the test of significance. The e β refers to the odds ratio that provides an indication of the change in the likelihood of the dependent variable associated with changes in the independent variable. four categories of nontechnical skills were represented as predictors of threat and error management behaviors across the different phases of flight. Further, a variety of contextual factors were also found to contribute to the predictive models of threat and error management. However, it was found that the contribution of the contextual factors and nontechnical skills were quite different for effective threat management when compared to effective error management. Similarly, differences in the predictive models were also found between the successive phases of flight. These differences can be best demonstrated by examining each of the four categories of nontechnical skill and the contextual factors in turn. In relation to the broad nontechnical skill category of crew communication, it was found that cooperative and interactive functions were essential for high levels of operational performance during the preflight phase. In this study, I found that the statement of plans of changes and cooperation were predictors of effective threat and error management, respectively. These aspects of nontechnical competence can be described simply as forms of communication that involved the exchange of information and the development of a shared understanding of situations. In contrast, during the descent-approach-landing phase when actions were time critical, assertiveness was found to be a major predictor of effective error management. This highlights the use of a different communication strategy in achieving effective performance, one that serves more direct and action-oriented functions. These findings suggest that a close relation exists between interactive group processes and effective threat and error management on the flight deck. A number of recent research studies have begun to explore the role of communication in error management and have focused on the identification of effective communication strategies (Bowers, Jentsch, Salas, & Braun, 1998; Fischer & Orasanu, 1999). However, the results of this study suggest that it is unlikely that any single communication strategy will be found to be most effective in relation to threat and error management, and stress that effective threat and error management involves

18 224 THOMAS skills in the adaptation of communication strategies to suit situational requirements. The nontechnical skill category of task management was the least represented in the models of effective threat and error management. This suggests that considerable further research is required to unpack the individual nontechnical skills found in this category. One specific skill, that of briefing and planning, was found to be a predictor of effective threat management in the preflight phase, again reinforcing the importance of communication strategies that assist in the development of a shared understanding of situations between crew members as a crucial aspect of threat and error management. The nontechnical skill category of situation awareness was found to be related to effective threat management across all phases of flight. This finding reinforces the perspective that maintaining an accurate mental model of the status of aircraft systems and environmental factors is essential to effective performance (Endsley, 1993). At a finer level of resolution, vigilance was found to be an important predictor of effective error management during the descent-approach-landing phase. As error detection is the primary action in effective error management, the benefits of increased vigilance during time-critical phases of flight are clearly supported by this research. The final category of nontechnical skill, that of decision making, was found to be related to effective threat and error management across all phases of flight. Specifically, in relation to effective error management, contingency planning and problem identification were found to be critical. This finding can be interpreted as highlighting the importance of remaining open to and being prepared for the occurrence of error. Crews that planned for a variety of eventualities and were able to identify problems as they occurred in normal operations were more likely to effectively manage error when it occurred. This finding reinforces the need for training to emphasis the ubiquitous nature of human error in normal operations and prepare crews to manage that error as it occurs. As Reason (1997) argued, at a broad level, both risk perception and hazard awareness are essential aspects of the organizational management of error. The findings of this study suggest that these approaches can easily be extended to the level of the individual operator in the commercial aviation environment, and therefore, an important step toward effective threat and error management is increasing pilot awareness of the potential for error. In this research study, I also found that a variety of contextual factors were linked to both effective and ineffective threat and error management during normal operations. First, a number of operational factors were found to be important predictors of threat and error management. Whereas the length of the flight and the time of day were not found to be linked to threat and error management, operational pressure through a late departure was clearly linked to poor error management. An increased number of threats during normal operations was actually

19 PREDICTORS OF THREAT AND ERROR MANAGEMENT 225 linked to improved threat management, a finding that might suggest the positive influence of increased cognitive arousal and an external frame of reference created by a number of external factors impinging on normal operations. In contrast, an increased number of errors impacted negatively on both threat and error management of crews. Together, these findings suggest that the quality of threat and error management by crews is particularly sensitive to the influence of both external and intracrew pressures. Second, a number of factors related to crew dynamics were found to be predictors of threat and error management. The experience levels of pilots was found to be a particular influence on threat and error management. Specifically, the more experienced the First Officer, the more likely crews were to effectively manage both threats and errors. Although this finding may initially indicate a simple relation between experience and operational expertise, the links to other influences such as assertiveness suggest experience is as much an influence on the ability of a First Officer to speak up and identify threats and errors as it is on technical expertise. Furthermore, the origin of the error was a significant predictor of error management. If the Captain committed the error, it was systematically less likely to be well managed across all phases of flight. These findings highlight the importance of crew dynamics and the influence of the flight deck authority gradient on effective threat and error management. Together, these findings provide significant empirical support for the importance of nontechnical skills in minimization of risk and the overall enhancement of operational performance. Further, the findings of this research stress the importance of taking into consideration contextual factors such as crew experience, operational pressures, and team structure in our understanding of effective threat and error management. Applications: Implications for Training System Design An explicit aim of this research study was to further develop the understanding of crews threat and error management during normal line operations to inform training systems design. Although this study has provided a robust analysis of the predictors of threat and error management during normal operations, there is an inherent danger in attempting to generalize these results to any airline or indeed any organization operating in high-risk industries. It is likely that the unique characteristics of an organization will produce significantly different models of effective threat and error management. In particular, organizational and national culture has recently been highlighted as a critical factor in accepting the validity of generalizations about models and concepts in the field of industrial and organizational psychology (Gelfand, 2000). Therefore, whereas this research does provide considerable insights into the types of contextual factors and nontechnical skills essential to effective threat and error management, the conclusions drawn from this work are

20 226 THOMAS more powerful in terms of the processes that an organization can undertake to effectively inform their own training systems design. In essence, this study demonstrates that an organization should utilize new forms of performance evaluation data in the creation of effective threat and error management training. The use of performance evaluation data is an essential component of training systems design, and it is paramount that training decisions are based on high-quality data (Goldsmith & Johnson, 2002). However, as Helmreich and Merritt (2000) suggested, training curricula developed in one organization is less effective when exported to another organization. Especially in relation to threat and error management, factors such as organizational and national culture, aircraft type, existing training regimes, SOPs, and the unique operating environments of individual airlines must be taken into account when designing effective training programs. Therefore, whereas it is desirable to develop detailed task analyses and a generalized understanding of job performance, an organization must utilize its own performance evaluation data to ensure that its training systems are effective in preparing crews for work in what is likely to be a unique operating environment. In the United States, the Federal Aviation Administration has explicitly acknowledged the need for airlines to design training systems that respond directly to their individual operational requirements. Through the Advanced Qualification Program (AQP), airlines are able to take advantage of flexibility in their exposition of regulatory requirements such that they can customize their flight crew training programs to reflect specific operational needs (Neumeister & Mangold, 1997). One of the major components of the AQP is the requirement for an airline to undertake a formal data collection and validation program to ensure that their training is meeting operational needs (Taggart, 1994). Accordingly, although this research does not profess to provide predictive models of threat and error management that can be generalized across any number of airlines, the research does highlight the utility of a new approach to training needs analysis that has the potential to overcome some of the limitations of traditional approaches. The forms of data collection and analyses discussed in this article highlight potentially new and more powerful forms of operational performance evaluation that have the potential to add considerably to the AQP process. Traditional approaches to aviation instructional design predominantly employ Cognitive Task Analysis (CTA) as the data source for development of training interventions (Oser, Cannon-Bowers, Salas, & Dwyer, 1999). Typically, CTA provides those responsible for training design with detailed analysis of both the knowledge and skills required for expert performance through the decomposition of work elements. However, it has been acknowledged that CTA has the propensity to provide decontextualized forms of information about effective performance that can be rather abstracted from the operational environment (Seamster, Redding, & Kaempf, 1997). Accordingly, although CTA is a robust empirical method for the detailed analysis of individual job performance, it does not provide sufficient in-