Going Fully Automated: A Case Study

Size: px

Start display at page:

Download "Going Fully Automated: A Case Study"

Alison Peters
5 years ago
Views:

Going Fully Automated: A Case Study October 23, 2014 Presented at: 38 th International Congress on Assessment Center Methods Presenters : Robin McBeth, Bureau of Alcohol,

1 Going Fully Automated: A Case Study October 23, 2014 Presented at: 38 th International Congress on Assessment Center Methods Presenters : Robin McBeth, Bureau of Alcohol, Tobacco, Firearms, and Explosives (ATF) Suzanne Tsacoumis, HumRRO 66 Canal Center Plaza, Suite 700 Alexandria, VA Phone: Fax:

2 Background Bureau of Alcohol, Tobacco, Firearms, and Explosives (ATF) Federal law enforcement agency within the U.S. Department of Justice Focused on the reduction of violent crime 1996 settlement agreement to change human resource practices for special agents 1 st -line supervisor promotion process was an assessment center 2002 = = = =

3 Assessment Center Candidate played the role of the supervisor In-basket Exercise Role Play Exercise: meet with an agent to help plan an upcoming operation Analysis Exercise: Review a case file and recommend how to proceed with investigation Oral Presentation or structured Interview Implementation 2 assessors (ATF managers) per candidate per exercise 1:1 assessor to candidate ratio Different team for each exercise; team changed each day Mechanical combination of scores 3

4 Implementation Realities AC > $1M Large pool of potential candidates Had to cut costs Need to assess the leadership competencies of our special agent population 4

5 New Process Variations to traditional AC methodology weren t going to work Needed a valid tool to evaluate leadership competencies Implement more often Reach a larger candidate pool Reduce overall costs 5

6 Electronic Promotion Assessment System (EPAS) Interactive, rich-media online simulations Situational Judgment Test Virtual Role Play In-basket Unadministered AC exercises as starting point for content ATF Director greatly supportive Mandated implementation begin in 6 months 6

7 Targeted Competencies Judgment and problem solving Lead others Plan, organize and prioritize Relate to others General investigative knowledge Knowledge of relevant laws, regulations and policies 7

8 Virtual Role Play Richness of a live role play Engaging format Realistic Branching Multiple data points 8 8

9 Virtual In-Basket Items/issues presented electronically 9

10 Animated Situational Judgment Test Scenarios simulate relevant job activities Response options describe realistic actions to situations Responses target important competencies 10 10

11 Demo 11

to interact with assessors Free responses Interactions are closer to a real life work sample

12 Live vs. Online: Primary Distinctions Live Read facts about a situation, make notes Applicants go to a room to interact with assessors Free responses Interactions are closer to a real life work sample Assessors evaluate interaction Online Information presented by characters and material Real time Closed-ended responses Interpersonal interactions measured indirectly Can interact with multiple people 12

y resembles the job (i.e., nature of the responses) More direct measure of some competencies

13 Live Simulations: Pros and Cons Advantages Rich open-ended responses Direct observation of behavior Closely resembles the job (i.e., nature of the responses) More direct measure of some competencies Interpersonal skills Oral communication Disadvantages Not completely standardized Resource constraints Money Personnel Time Restricted to single interaction when in reality different interactions may be warranted 13

Online Simulations: Pros and Cons Advantages Standardized can force key questions,

Insert interruptions Manipulate virtual environment to fit the scenario Increased

, email) Disadvantages Decreased realism of complex interactions Lose non-verbal

14 Online Simulations: Pros and Cons Advantages Standardized can force key questions, interactions Novel, engaging Automated scoring Able to: Interact with multiple people Insert interruptions Manipulate virtual environment to fit the scenario Increased realism for some tasks (e.g., ) Disadvantages Decreased realism of complex interactions Lose non-verbal behavior, inflection Nuances of response options (inferred tone, personality) Some heterogeneous items Bandwidth requirements Few validity studies conducted 14

15 Validation Evidence Content-oriented development process Simulated important tasks Measured critical supervisory competencies Given novelty and nature of assessment, not sufficient Criterion-related validity study Concurrent design Criteria: Supervisor ratings and job knowledge test N=160 Overall uncorrected EPAS r=.45 Very respectable given small sample of already-screened incumbents 15

EPAS Administration EPAS 2012 623 participated 510 passed (81.

16 EPAS Administration EPAS participated 510 passed (81.9%) 113 failed (18.1%) EPAS participated 364 passed (79.5%) 94 failed (20.5%) 16

17 Use of EPAS Scores Agent applies to supervisory job opening Applicants with top five EPAS scores form Best Qualified List (BQL) Merit Promotion Board evaluates BQL applicants Structured Interview Resume EPAS Competency Profile (not scored) 17

18 The Bottom Line Scalable and easy to administer ATF leadership loves it! Candidates split in terms of preference for live vs online simulations Custom, content-oriented development that adhered to SIOP Principles Evidence of criterion-related validity Substantial cost savings ATF received 2013 SIOP-SHRM HRM Impact Award 18

19 Questions?

Assessment Centers in the Public Sector: Best Practices and Lessons Learned Jennifer Geimer, Ph.D. Suzanne Tsacoumis, Ph.D.

Assessment Centers in the Public Sector: Best Practices and Lessons Learned Jennifer Geimer, Ph.D. Suzanne Tsacoumis, Ph.D. Presented to: 2 nd Annual IPAC Conference, Washington, DC July 19, 2011 1 Overview