Multifaceted Approach to Situation Awareness Assessment Cheryl A. Bolstad and Haydee M. Cuevas SA Technologies GET CONNECTED to LEARN, SHARE, AND ADVANCE.
Human Performance Situation Awareness Decision Making Performance Situation awareness is key to good decision making and good performance.
What is Situation Awareness? World State Situation Awareness Level 1 Perception Level 2 Comprehension Level 3 Projection Action Decision Situation Awareness is the Perception of the Elements in the Environment within a Volume of Time and Space, the Comprehension of their Meaning, and the Projection of their Status in the Near Future. Endsley, 1988
Situation Awareness Is: Knowing what is going on around you The perception of important elements Seeing: display changes, missing parts Hearing: verbal communications, hearing alarms The comprehension of their meaning How this affects your objectives Compliance with procedures The projection of their status in the future Effects on schedule and operations Future impact on team safety
Goals of SA Measurement Investigations of SA in Ongoing Operations What are the critical skills/abilities that lead to high SA? What factors hinder SA the most? How do soldiers maintain SA under harsh operational conditions? What strategies lead to high SA? How does SA develop within and between teams? Evaluation of System Designs Do new technologies actually improve SA? Which aspects of SA are hurt by technology? Evaluation of Training Programs How effective are new training techniques?
Different SA Metrics Process Indices State of Knowledge Behaviors Performance Eye Movements Information Acquisition Communications &Verbalizations Questionnaires Post-test On-line probes SAGAT Subjective Measures Modeled Measures Actions Verbalizations System Performance Emergency Performance Assessment Processes System Interface Situation Awareness Decision s Performance
Subjective Measures Objective Measures Another View of SA Measures Process Measures Outcome Measures Pre-Post Questionnaires SA Communications SA Behaviors Workload Metrics Freeze Queries Real Time Queries
Diagnosticity A decision situation is said to be diagnostic if it provides better evidence for one hypothesis over than for others. In short: this is the informativeness of the information at hand. By utilizing different SA measures, we can not only tell that SA has changed, but HOW its changed and WHY it has changed.
Example Study: BCBL at Fort Leavenworth Simulation Exercise Objective Force Structure 3 Brigades (1 manned) 3 Battalions 1 Higher Headquarter Objective: To evaluate a new force structure
SAGAT - Team SA
SAGAT - Team SA
Why The Difference? Perhaps communicating more SA level one information? SAMTC Team Communication Measure Expert observer rating of individual statements made by soldiers in terms of representing good or bad SA statements
SAGAT - Team SA
SAGAT - Analysis
Why The Difference? Perhaps Build and Sustain Cell Members are not looking at or sharing the right information? SABARS Team Behavior Measure Expert observer rating of physical actions (behaviors) of a soldier for consistency with acquisition and dissemination of SA
What is MEGA-SA? Metrics of Global Assessment and Situation Awareness Comprehensive multi-faceted measurement system founded on SA theory. It enables researchers to objectively measure and analyze individual and team SA in real-time across Army command and control (C2) system evaluations.
What is Included in MEGA-SA? SAGAT Queries (SA) Random subset of queries are presented one at a time on their computer screen / display. Task performance is interrupted. Real Time Probes (SA) Queries are transmitted within the context of current operations and tasks as an audio message. Task performance is not interrupted. Team Behavior Measures Expert observer rating of physical actions (behaviors) of a soldier for consistency with acquisition and dissemination of SA Team Communication Measures Expert observer rating of individual statements made by soldiers in terms of representing good or bad SA statements Other Metrics NASA-TLX End of exercise questionnaire
MEGA-SA Characteristics Real Time Assessment - Provides near real-time SA assessment Valid and Reliable - Based on valid and reliable metrics - Uses a multi-faceted approach to SA measurement increased sensitivity and diagnosticity of the assessment Flexible and Robust - Measures both individual and team SA - Customizable metrics - Easily adapted to other DoD organizations - System database is built with other domain requirements Ease of Use - Results are stored and available for additional review and analysis. - Easy-to-use menu-based tool
Metric Modules Metric Module Interfaces With SA Database Experiment Set up Module creates query set and defines parameters to be used in data collection Run Module sends metric file to the clients Administers queries based on the set triggers and records responses to data files
MEGA-SA Modules Metric Creation Experiment Review MEGA SA Experiment Setup Experiment Execution
User Centered Design Process 1. Requirements Specification (Iterative) A. User Requirements B. Functional Requirements C. Data Requirements D. Environmental Requirements E. Usability Requirements 2. Contextual Task Analysis (Iterative) A. Operational Task Flows B. Use Case Sequences C. Use Case Scenarios 3. Conceptual Design (Iterative) 4. Initial Design Alternatives (Iterative) A. Physical Mockups B. Interactive Prototypes
Report Setup & Run Manage Metrics Metric Items Queries Behaviors Communications Metric Sets Workload (stock) Metric Set 1 + + Queries Behaviors Communications Metric Item Text This is a customer made metric item This is a customer made metric item This is a SA Tech made metric item This is a customer made metric item This is a SA Tech made metric item This is a SA Tech made metric item This is a customer made metric item This is a SA Tech made metric item SA Level Role Category Topic All of these filter and sort like Excel Branch/Specialty Domain Notes Metric Set 2 Metric Set 3 Experiment Files Experiment File 1 Experiment File 2 + Indicates this metric item is developer content and cannot be changed Experiment File 3 Metric Item Details Edit/Save Copy Preview Metric Type SA Level Role(s) Metric Item Text Search Terms Domain Image Response Type Browse Branch/Specialty Topic Category Response Options Add Remove Notes 3
Report Setup & Run Manage Metrics Metric Items Queries Behaviors Communications Metric Sets Workload (stock) Metric Set 1 + + Metric Item Text This is a customer made metric item This is a customer made metric item This is a SA Tech made metric item This is a customer made metric item This is a SA Tech made metric item This is a SA Tech made metric item This is a customer made metric item This is a SA Tech made metric item SA Level Role Category Topic Branch/Specialty Domain Notes Metric Set 2 Metric Set 3 Experiment Files Experiment File 1 + Experiment File 2 Experiment File 3 Metric Set Details Edit/Save Copy Preview Metric Set Type Metric Set Name Notes Role(s) Search Terms Domain Branch/Specialty Topic Category 3
Conclusion Situation Awareness is a complex component SA Measurement Approaches need to address these complexities MEGA-SA is being created to address these needs
Questions