Paper Presented at the Human Factors and Ergonomics Society 48 th Annual Meeting. Collaborative Tools and Shared Mental Models

Transcription

1 Collaborative Tools and Shared Mental Models Mike Schneider, John Graham, Aaron Bauer, Katherine Bessiere Carnegie Mellon University Cheryl Bolstad SA Technologies Abstract This paper reports on an experiment conducted to measure collaboration tool usage and shared mental models in Army Command and Control. Several teams of soldiers participated in a twoweek simulation experiment. During the simulation, the run was stopped and participants completed a 5-minuted on-line questionnaire in which they were asked to rate their workload and other teammates workload as well as provide information on the collaboration tools used during the last hour of the exercise. Shared mental models was measured using congruency between each person s rating of their own workload and their teammates perception of their workload. Overall, using the same collaboration tool to communicate between team members lead to greater congruency of mental models, irrespective of the tool type. Introduction In many environments individuals work together to complete a shared task. These individuals are known as a team. Army Command and Control In general team members must continuously collaborate and share information in order to perform a common task. This dynamic process may be best exemplified by command and control (C 2 ) in the US Army. During C 2 soldiers must continuously seek out new information, integrating it with existing information and share this information with relevant officers all for the purpose of creating, executing, and modifying the Commander s plan. In current Army operations this process is done while the majority of participants are geographically co-located. However, the Army s vision for the future is the Future Force. In this new force structure soldiers will be..disposed across the battle space yet operationally integrated.. (Army, 2001). This represents a shift from legacy/current systems in which large TOCs (Tactical Operations Centers) composed of small teams of multiple personnel are used to manage conflicts, and plan and conduct operations. Instead, the Army envisions small groups working together in a distributed fashion. While this will give the Army more mobility as well as faster deployment times it provides a challenge as to how these teams will continuously collaborate together across large distances. Shared Mental Models In the Future Force, if soldiers are to function in a distributed fashion they will need similar mental models and a high degree of situational awareness (SA) to function effectively. Research has shown that when team members possess similar mental models their team performance is enhanced (Stout, Cannon-Bowers, Salas, & Milanovich, 1999) and when teams are not allowed

2 to generate shared mental models they perform significantly worse than teams with shared mental models (Bolstad & Endsley, 1999). Collaboration Taxonomy One possible way to ensure that team members possess similar mental models is through using the appropriate collaboration tool. In general, teams employ collaboration tools to help coordinate and share information. Selecting the right tool for each task is a challenge as there are a variety of collaborative tools available to perform a myriad of collaboration tasks. SA Technologies has created a taxonomy of collaboration to describe the different types of collaboration techniques and tool characteristics that are needed to support the differing types of collaboration (Bolstad & Endsley, 2003). The taxonomy can be use to select collaboration tools that are best suited to specific teams based on their collaboration needs. When determining which tools will work best for a team, the tasks that the team performs and the environment in which they perform these tasks need to be considered. Taxonomy and Army Command and Control. The collaboration that occurs for Army command and control is very complex and only a robust set of tools will fit the Army s needs. Using a description of the taxonomy and comparing it to Army C 2 activities a few tools emerge as potential candidates. These include: face-to-face collaboration, audio conferencing, white boards, domain specific tools and file transfer. While face-to-face collaboration is not always possible in the Army it has an ability convey emotional information that is not provided by the other three techniques. Other tools such as groupware, video conferencing and program sharing meet some of the needs for command and control, but they are poor at sharing spatial, photographic, video or emotional information. This paper reports on a simulated exercise in which soldiers were provided with a variety of collaboration tools to use to perform a C2 exercise. During this exercise both collaboration tool usage and mental model congruence were measured. Of interest was the type of collaboration tools selected by the soldiers and the mental model congruence of team members. Specifically, did soldiers use tools best suited for Command and Control and did soldiers who used similar collaboration tools achieve a higher level of shared mental model. Method Participants Fifty-six participants, consisting of both active duty officers and retired officers familiar with Brigade level operations participated in this study. Shared Mental Model Measure We used Entin s (1999) proximate measure of shared mental models in this study. He found congruence between participants mental models and their ability to rate other teammates workload (Entin & Serfaty, 1999). Using this measure, participants rate their own performance as well as their teammates. Congruence is measured by how similar the teammates rate each other s workload.

3 Workload was measured using the NASA TLX (Task Load Index) assessment consisting of six workload parameters on a Likert scale (Hart & Staveland, 1988). The parameters are mental demand, temporal demand, effort, own performance, frustration level, and physical demand. Participants were asked to rate themselves as well as five other people randomly selected from the other participants. When rating other people, participants had the option of selecting Don t Know for each of the six questions. Mental model congruence was determined by comparing each person s self-reported workload with the estimation of that person s workload by other participants. This measure was computed by summing the absolute differences between the selfreported ratings and the rater s estimations. For example, if person A s self report was a 5 for each question on the index, and person B estimated A s workload as a 3 for each question, person B s mental model congruence would be 12 (two multiplied by six). Congruence scores could range from 0 (indicating perfect congruence) to 36. Willingness to Rate Measure Willingness-to-rate was determined by number of the six NASA TLX items for which the participant chose to rate the other person divided by the total number of questions. Therefore if the participant chose to rate someone on every item, they would receive a willingness-to-rate score of 1, while if they chose to rate someone on no items they would receive a willingness-torate score of 0. A participant who answered one question out of six would receive a willingnessto-rate score of 1/6. However almost all participants (98.2%) chose to answer either all six questions, or to answer none of them. Collaboration Tool Usage Measurement During the simulation at random intervals participants were asked to complete several questions on their collaboration tool usage during the last hour. They were asked which tools they used most frequently and which tools were most effective for their job. Procedure Participants received one week of training on the new organizational structure and the positions they would be playing as well as the tasks they would be performing. They then completed three days of simulated battle runs in which they planned then executed their course of action using the collaboration tools provided. The simulation consisted of a mapping system (MC2) to display the common operational picture (COP) and a separate display for collaboration tools. Each participant was seated in front of a computer which could display the simulation (see Figure 1). They were also provided with collaborative software tools which included audio conferencing, instant messaging, chat, shared whiteboard, file transfer, and application sharing software. Partitions separated the teams, so that a participant could talk directly to members of his own team, but could only communicate with members of other teams using the communication tools.

4 Figure 1. Exercise Set-Up Every one to two hours throughout both planning and execution phases, activity was halted and the participants were asked to complete a 5-minute on-line questionnaire. In addition to collecting workload scores several questions were asked about collaboration tools used during the previous period of the simulation. Results Collaboration Tool Usage and Willingness to Rate Analysis showed that using the same collaborative tool did not have a significant effect on the willingness to rate another member s shared mental model. Participants could choose whether or not they wanted to rate another s workload. We introduced this capability to prevent random guessing at another s workload. Participants chose to rate others only 35% of the time. Therefore, if participants choose to rate another s workload, it appears they were selective and it was because they believed they had enough information to do a rating. Collaboration Tool Usage and Shared Mental Model Congruency If the participants chose to rate another s workload, we calculated a congruence index based on the sum of the absolute differences between the rater s workload estimation of the ratee on the six question NASA-TLX, and the ratee s self reported workload on the same measure. We analyzed this congruence index with the type of collaboration tool used to communicate with one another. The analysis showed that using the same collaborative tool had a significant effect, F (1, 539) = 2.83, p <.10, on shared mental model congruency. Collaboration Tool Usage Participants were asked which tools they used during the last hour of the exercise at each stop. On average participants selected the same collaborative tool as their primary tool 347 times (347/2254) = 15.4 %). It was most likely that same tool use would occur for users of face-to-face (27.1%), MC2 (COP) (22.5%), audio conferencing (20.5%), chat (15.6%), instant messaging (13.5%) then group systems (.87%) (see Figure 2).

5 Figure 2: Percent Same Collaboration Tool Usage Shared Mental Models While there was not a significant difference in answer congruence based on tool used, the highest congruence was achieved by participants who used audio conferencing (75.14%), followed by face-to-face (73.9%), MC2 (COP) (72.0%), instant messaging (70.83%), chat (68.58%), and group systems (55.56%). However, these values are much lower when participants are communicating using different tools (e.g. chat versus audio conferencing). Thus, highest congruency levels were achieved when the tool usage was similar between participants. Discussion The results from this study point out the need to select the appropriate collaboration tools for specific teams and task. In this experiment using the same collaboration tool to communicate with team members was the most important factor in mental model congruency. If participants were using the same collaboration tool to communicate with one another (regardless of the tool type) their mental model congruency (estimation of each other s workload) was increased. Providing soldiers with too many choices not only lowers their chances of achieving mental model congruency, but it also lowers their ability to communicate effectively and efficiently with one another and has the potential to impact team SA and team performance. Overall participants gravitated towards those tools that best support command and control activities as recommended by the Bolstad and Endsley taxonomy (2003). These results point to the need for the Army to further explore the use of collaboration tools in Army operations. Acknowledgements Prepared through collaborative participation in the Advanced Decision Architectures Consortium sponsored by the U.S. Army Research Laboratory under the Collaborative Technology Alliance Program, Cooperative Agreement DAAD

6 References Army, U. S. (2001). Concepts for the Objective Force: White Paper: United States Army. Bolstad, C. A., & Endsley, M. R. (1999). Shared mental models and shared displays: An empirical evaluation of team performance. Paper presented at the 43rd Annual Meeting of the Human Factors and Ergonomics Society, Houston, TX. Bolstad, C. A., & Endsley, M. R. (2003). Tools for supporting team collaboration. Paper presented at the Human Factors and Ergonomics 47th Annual Meeting, Denver, Colorado. Cannon-Bowers, J. A., Salas, E., & Converse, S. (1993). Shared mental models in expert team decision making. In N. J. Castellan (Ed.), Current issues in individual and group decision making (pp ). Hillsdale, NJ: Lawrence Earlbaum. Entin, E. E., & Serfaty, D. (1999). Adaptive team coordination. Human Factors, 41(2), Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In P. A. Hancock & N. Meshkati (Eds.), Human mental workload (pp ). Amsterdam: North-Holland. Salas, E., Prince, C., Baker, D. P., & Shrestha, L. (1995). Situation awareness in team performance: Implications for measurement and training. Human Factors, 37(1), Stout, R. J., Cannon-Bowers, J. A., Salas, E., & Milanovich, D. M. (1999). Planning, shared mental models, and coordinated performance: An empirical link is established. Human Factors, 41(1),