Situation Awareness Metrics

Transcription

1 Situation Awareness Metrics An AutoUI 16 workshop Situation Awareness in Automotive Evaluation & Design 1

2 Task Interface Perception Prediction Different SA Measurement Approaches Situation Awareness Human Assessment Processes Comprehension Situation Model Decision Action Process / Indices Physiological Information Acquisition Direct Subjective/Ratings Questionnaires/Probes On-line probes Real-time probes (SAGAT) Behavior Assessmen ts Process Assessments Verbalization s Performan ce Performanc e Post Hoc 2

3 Process Indices Direct Behavior Assessments Performance P Physiological ; Information Acquisition Indices Possible Research Questions Is the driver s current state conducive to forming good SA? Are critical cues being perceived? Does the driver show expectancy of upcoming events? Potential Metrics Respiration Rate Eye Blinks Eye Tracking Brian Activity (EEG) Event-related potential (ERP) Heart rate variability (HRV) Galvanic skin response (GSR) Denso's driving simulator from (10/15/92) 3

4 Process Indices Direct Behavior Assessments Performance Physiological: Eye Tracking, Heart Rate, Brain Waves (ERPs) Information Acquisition Indices Pros Objective Indication of information access / utilization Data is unique to each person Can be used to identify possible times to perform other measures Cons Can not infer what is done with the information (processing) Can not tell whether information is registered correctly Creates large amounts of data Analyzed using computer algorithms Recommendations Best for examining SA processes rather than product Examine specific research questions 4

5 Process Indices Direct Behavior Assessments Performance Direct SA : Ratings, Possible Research Questions Does the driver observe differences in their SA between various conditions? What aspects of the task have the greatest influence on SA? Potential Metrics Situation Awareness Rating Technique (SART) Situation Awareness Subjective Workload Dominance (SA-SWORD) Situation Awareness Rating Scale (SARS) Subjective Situation Awareness (SSA) Mission Assessment Rating Scale (MARS) Questionnaires MIT's Age Lab (11/16/14) 5

6 Process Indices Direct Behavior Assessments Performance Direct SA : Subjective Ratings and Questionnaires Pros Face validity Non-intrusive Used during test and/or post-test Easy to administer and score Cons Do you know what you don t know? Multiple measures needed for reliability Maybe tainted by performance outcomes Confidence in own SA Recommendations Should be used to complement objective techniques 6

7 Process Indices Direct Behavior Assessments Performance On-line Probes and Real-time Probes Possible Research Questions Is the driver prepared for taking future actions? Does the driver understand how they need to adjust their driving based on the current road conditions? What is capturing the drivers attention at this specific moment within the test? Potential Metrics Situation Awareness Global Assessment Technique SAGAT Real time verbal probes Virtual Environment Situation Awareness and Review System (VESARS) ( SA Technologies) BCBL, Fort Leavenworth 7

8 Direct : On-line and Real-time Probes Pros Provides most detailed information SAGAT Heavily validated o Content Validity - Inclusive of SA elements (Endsley, 1990a) o Construct Validity - Does not impact on performance (Endsley, 1989, 1990) o Criterion Validity - Predictive of performance (Endsley, 1990b) Objectively collected and evaluated Sensitive and diagnostic Minimizes potential subject bias (when compared to post-test metrics) Cons Requires the interruption of the test -> may interfere with task performance Noise may interfere with real-time probes 8

9 SAGAT: Situation Awareness Global Assessment Technique Derived from SA Requirements (GDTA) 1.0 Employ Assets to Collect Needed Intelligence 1.1 Determine Information Collection Requirements 1.2 Determine Gaps in the Intelligence Data 1.3 Determine Suitability of Available Assts for Collection Needs 0.0 Provide timely, accurate, relevant intelligence information to the brigade commander 2.0 Support Friendly COA Development 2.1 Determine Effects of Enemy s COAs on Battle Outcome 2.2 Predict Likely Enemy COAs 2.3 Support Other Officers in Order Development 2.4 Utilize Counter Intelligence to Friendly Advantage GDTA 3.0 Provide Effective Communication 3.1 Communicate Data to Commander That Affects His COAs and Intents 3.2 Communicate Mission Critical Information Level 1 Location Fuel level Level 2 Speed Distance to next turn on route Speed limit Distance to next car Planned route Impact of hazard on own safety Current traffic Level 3 Diff. b/ w speed limit & speed Accidents Projected on route time Estimated to destination travel time on Current road current conditions route Automobile performance level Projected time to destination on Weather conditions alternate route Change in traffic density Projected hazard Rate of level closure of for vehicle weather directly behind Projected likelihood of getting a ticket Projected impact of new route on travel time SAGAT Probes 9

10 Percent Correct SAGAT Methodology Real-time man-in-the-loop simulation of system (rapid prototyping) At random times, freeze the simulation, blanking all displays Administer a rapid battery of queries to ascertain the subject's SA at that point in time Score the subject's SA on the basis of objective data derived from the simulation SA Score By Cell CO IS FE BS MS CI CoBn 10

11 Real-Time SA Probes Real-time audio questions asked during the scenario Questions are based on SA requirements Collect a more limited amount of information One question at a time Multiple measures needed for reliability Participants can easily look for needed information Measure response time as well as accuracy (SA and workload) Requires careful synchronization with mission tasks/events Sensitivity is a concern Limited by number of repeats of each probe Comparing SAGAT vs Probes: trends tend to follow that of SAGAT but significance difficult to attain (p value) Probe 1 Probe 2 Probe 3 Probe 4 Probe 5 TIME 11

12 Behavior Ratings, Process Assessments Possible Research Questions In order to update their big picture are regular scans or mirrors and windows occurring? Does the driver alert other passengers when they have to make a very abrupt stop? While driving, does the driver accurately respond to the imbedded task (e.g. adding three numbers together)? Potential Metrics Situation Awareness Behavioral Anchor Rating Scale (SABARS) Situation Awareness Measure of Team Communication (SAMTC) Imbedded Tasks (UK) (2015) 12

13 Behavior Ratings, Process Assessments Pros Objective SA ability momentary SA Non-intrusive Cons Observer ratings can be influenced by multiple factors Inter-rater reliability Sensitivity Diagnosticity Interpretability of measure (Assumes certain behavior gives SA) Must infer SA Recommendations Best used in conjunction with other measures Must cover a wide range of tasks/scenarios Need situations in which behavioral indicators are present Must provide good tools to support consistent ratings 13

14 Performance and Post Hoc Possible Research Questions How long does it take the driver to recognize that a pedestrian is about to jaywalk across the highway? Did the driver notice a change in the engine s temperature? If they noticed the change, what action did they take? Potential Metrics Situation Present Assessment Method (SPAM) (Response Latency) Performance End of Session Performance Evaluation How to pass your Texas driving test. (2016) 14

15 Performance and Post Hoc Pros Objective Non-intrusive Provides detailed assessment Cons Sensitivity and diagnosticity Must infer SA Interpretability of measure (Performance only) Rationalization/generalization after the fact (Post- Hoc only) Selected performance metrics can impact interpretation of your results. Recommendations Best used in conjunction with other measures Carefully select measures to use based on analysis plan of how will interpret the results 15

16 SA Measurement Summary SA Measurement in testing systems and design options provides useful diagnostic information Choice of measures is important. Must consider: What are your research questions? What type of data will best support your questions? Time and cost availability Validity type supported by the metrics Subjective or objective measures? 16