Overcoming the Fear of Employment Testing: What Exactly is Testing? Presented by: Chris Atkinson, M. S. and Jim Kuthy Ph.D.

Transcription

1 Overcoming the Fear of Employment Testing: What Exactly is Testing? Presented by: Chris Atkinson, M. S. and Jim Kuthy Ph.D.

2 Visit BCGi Online If you enjoy this webinar, Don t forget to check out our other training opportunities through the BCGi website. BCGi Standard Membership (free) Online community Monthly webinars on EEO compliance topics EEO Insight Journal (e-copy) BCGi Platinum (paid) Membership ($299/year) Fully interactive online community Includes validation/compensation analysis books EEO Tools including those needed to conduct AI analyses EEO Insight Journal (e-copy and hardcopy) Members only webinars and training and much more

3 HRCI Credit BCGi is an HRCI Preferred Provider CE Credits are available for attending this webinar Only those who remain with us for at least 80% of the webinar will be eligible to receive the HRCI training completion form for CE submission

4 About Our Sponsor: BCG Assisted hundreds of clients with cases involving Equal Employment Opportunity (EEO) / Affirmative Action (AA) (both plaintiff and defense) Compensation Analyses / Test Development and Validation Published: Adverse Impact and Test Validation, 3 rd Ed., as a practical guide for HR professionals Editor & Publisher: EEO Insight an industry e-journal Creator and publisher of a variety of productivity Software/Web Tools: OPAC (Administrative Skills Testing) CritiCall (9-1-1 Dispatcher Testing) AutoAAP (Affirmative Action Software and Services) C 4 (Contact Center Employee Testing) Encounter (Video Situational Judgment Test) AutoGOJA (Automated Guidelines Oriented Job Analysis ) COMPare: Compensation Analysis in Excel 4

5 Contact Information Jim Kuthy, Ph.D. Principal Consultant (800) x 239 Chris Atkinson, M.S. Consultant catkinson@biddle.com (800) x 120

6 The Presenters Jim holds Masters and Doctorate Degrees in Industrial/ Organizational Psychology Jim has taught Psychology and Business-related courses at the University of Akron and California State University, Sacramento Chris holds a Masters Degrees in Industrial/Organizational Psychology More than twenty five combined years of experience in the employment selection field They have designed and validated a variety of employment tests for many employers

7 Outline What is testing? A Brief Overview of Adverse Impact A Brief Overview of Test Validation Commonly Used Employment Tests Using a Whole Person Approach to Testing Common Testing Myths

8 What is Employment Testing? Assessment Standpoint A test is a standardized series of problems or questions that assess a persons knowledge, skills, abilities, or other characteristics Legal Standpoint - Uniform Guidelines on Employee Selection Procedures defines a test as any method used to make an employment decision

9 U.S Employment Test Usage 41% of employers test job applicants in basic literacy and/or math skills. 34% of job applicants tested in 2000 lacked sufficient skills for the positions they sought. 68% of employers engage in various forms of job skill testing. 29% of employers use one or more forms of psychological measurement or assessment. 20% of employers use cognitive ability tests 8% of employers use interest inventories. 14% of employers use some form of managerial assessments 13% of employers use personality tests 10% of employers use physical simulations of job tasks ***This information is based on survey data collected in 2001 from 1,627 HR managers who are members of the American Management Association. The survey accurately reflects the AMA membership base, and is therefore more reflective of larger organizations than all employers in the U.S. economy.

10 Reasons for Testing 1. Testing leads to savings in the decision-making process 2. The costs of making a wrong decision are high 3. The job requires attributes that are hard to develop or change 4. Hard-to-get information can be obtained more easily and efficiently 5. Individuals are treated consistently 6. There are a lot of applicants

11 A Brief Overview of Adverse Impact

12 Adverse Impact A substantially different rate of selection in hiring, promotion, or other employment decision which works to the disadvantage of members of a race, sex, or ethnic group Men Pass (50) Women Pass (35) Men Fail (50) Women Fail (65) Men Passing Rate (50%) Women Passing Rate (35%) 80% Test Statistical Significance

13 According to the 1964/1991 Civil Rights Act(s) Adverse Impact (i.e., significant difference in rates) Sufficient Evidence of Job Relatedness /Validity Disparate Impact Discrimination

14 A Brief Overview of Test Validation

15 What is Validity? Traditionally, validation is making sure a selection practice, procedure, or test (PPT) appropriately measures what it is designed to measure In a legal realm, a selection procedure is valid if it can be proven by an employer that it is job related and consistent with business necessity It is the inferences regarding the specific uses of a test or other measurement procedure that are validated, not the test itself

16 What Needs to be Validated? Any practice, procedure, or test (PPT) exhibiting adverse impact Written tests Interviews Physical ability tests Resume screens Virtually any process that is used for making employment-related decisions

17 Validity Overview Content: A connection between the important parts of the job and the test Requires demonstrating a connection between the job (using a job analysis) and the content of the test Criterion: A mathematical study that proves the test predicts job performance: It ALWAYS requires: A statistical study and results that are statistically significant (<5% chance) Construct: A connection between a test, a trait, and job performance: It ALWAYS requires empirical evidence connecting the test to the trait and the trait to the job (all 3)

18 Content Validation Process Other Job Duties KSAs Operationally defined KSAs Selection Devices (e.g., application form, tests, Interviews, BQs)

19 Content Validation Process Other Job Duties Selection Devices (e.g., application form, tests, Interviews, BQs) KSAs Operationally defined KSAs Content Valid!

20 Performance Measure Criterion-related Validity Criterion-Related Study Test Score Score on some Criteria (e.g., job performance, days missed work, etc.) Score on a Test 20

21 Commonly Used Employment Tests

22 Commonly Used Employment Tests Assessment Centers Personality Testing Biodata Cognitive Ability Tests Job Knowledge Tests Situational Judgement Testing Employment Interviews Training Education and Experience (TEE) Work Sample Tests

23 Cognitive Ability Tests Cognitive ability tests assess abilities involved in thinking (e.g., reasoning, perception, memory, verbal and mathematical ability, and problem solving) Examples: General mental ability, math, verbal analogies, reading comprehension, spatial reasoning

24 Cognitive Ability Tests Validity: High validity for a variety of jobs. The more complex the job, the higher the validity Applicant Reactions: Tests developed for specific jobs have higher face validity than off-the-shelf tests Adverse Impact: High against minorities Development Costs: Purchasing off-the shelf tests is typically less expensive than developing custom tests Administration: Easy to administer to many applicants Use: Best used for jobs requiring specific cognitive skills and more complex jobs

25 Job Knowledge Tests Assesses technical or professional expertise in specific knowledge areas. Not appropriate when applicants will be trained after selection in the critical knowledge areas needed for the job. Examples: Basic accounting principles, computer programming, and financial management

26 Job Knowledge Tests Validity: High - Customized job knowledge tests have been shown to have slightly higher validity than off-theshelf tests Applicant Reactions: Tend to be favorable Adverse Impact: High against minorities Development Costs: Frequent updates to the test content and validation may be needed to keep up with changes in the job Use: Best used for jobs requiring specific job knowledge on the first day of the job

27 Situational Judgement Tests Situational judgment tests (SJTs) present a description of a work problem or critical situation related to the job and asks applicants to identify how they would respond SJTs measure effectiveness in social functioning dimensions such as conflict management, interpersonal skills, or problem solving, SJT questions and alternatives are typically based on critical incidents generated by job experts. Scores are based on subject matter experts' judgments of the best and worst alternatives.

28 Encounter Situational Judgment Testing

29 Situational Judgement Tests Validity: High content validity, moderately high criterion-related validity Applicant Reactions: Very positive Adverse Impact: Moderate against minorities Use: Typically used for managerial positions or other jobs requiring interpersonal interactions

30 Employment Interviews Interviews are subject to the requirements of the federal Uniform Guidelines on Employee Selection Procedures (1978), similar to any other type of test or selection procedure Unstructured Interviews No constraints on types of questions asked Global evaluation of applicant responses Structured Interviews Asking exact same questions to all applicants Applicant responses evaluated against rating scales

31 Interview Validity Least Validity Highest Validity Unstructured Single Rater Generic one size fits all Open Scoring/No Scoring Structured Multiple Raters Job Specific BARS

32 Structured Interviews Applicant Reactions Unstructured interviews are more favorable than structured interviews Adverse Impact Low Typically used late in the assessment process as a final screen

33 Training, Education & Experience (TEE) Requirements A systematic method used to assess previous experience, education, and training information provided by job applicants Often collected as a supplement to the application This information is evaluated against education and experience benchmarks to generate scores for selection purposes blank

34 Training, Education & Experience (TEE) Requirements Training requirements posted by employers as hiring or promotion criteria. Certifications, credentials, training institution completions, licensing requirements Educational requirements, including: Degrees, specific course requirements, etc. Experience requirements used as hiring or promotion criteria, including: Closed systems (e.g., time-in-grade requirements) Open systems (inside/outside experience counts)

35 Training, Education & Experience (TEE) Requirements Validity: Highly representative of the job, but do not relate well to performance on the job. Applicant Reactions: Typically more positive reactions from more experienced applicants Adverse Impact: Little to none against women or minorities Uses: Commonly used as a screening device prior to another selection tool (e.g., structured interview).

36 Work Sample Tests Work sample testing refers to hands-on performance tests in which a job candidate is required to perform (or describe) activities/tasks that are performed on the job Callinan, M., & Robertson, I. T. (2000). Work sample testing. International Journal of Selection and Assessment, 8(4),

37 Work Sample Test Examples Mechanic Repair a problem on a car Use a repair manual Clerical Typing Test Proofreading test Police Officer Writing test Climbing, running Computer Programmer Create a simple program Debug a current program Adapted from Selection Assessment Methods, Pulakos 2005

38 Work Sample Tests Validity: One of the most valid forms of testing Adverse Impact: Typically low depends on the competencies being assessed Applicant Reactions: Favorable Provides a realistic job preview Development and Administration Costs Can be high Best Used When: Measured competencies are critical for successful performance Limited number of applicants Small number of applicants being tested

39 Assessment Centers Employs multiple assessment methods and exercises to evaluate a wide range of competencies Applicant performance is usually observed and evaluated by multiple assessors (i.e., raters) Examples: In-basket exercises, leaderless group discussion

40 Assessment Centers Validity: Moderate to high Applicant Reactions: Favorable - Many exercises serve as a realistic job preview Adverse Impact: Low to moderate Development Costs: High in terms of time and money Administration Costs: High - Requires many raters Use: Particularly effective for assessing higher-level managerial and leadership competencies as well as sales and customer service

41 Personality Testing Personality tests elicit information about a person s motivations, preferences, interests, emotional make-up and style of interacting with people and situations Self-report inventories typically ask applicants to rate their agreement level on a various items designed to assess various traits Examples: 16PF, OPQ, Hogan Personality Inventory, Big 5 Specific traits are more predictive for particular jobs Personality tests designed to assess psychiatric conditions are considered medical examinations

42 Personality Testing Validity: Moderate levels of validity for a wide range of outcome measures in a variety of settings Applicant Reactions: Not very favorable, may appear as invasive Adverse Impact: Few, if any differences between men and women, or ethnicities Administration: Easy to administer to large groups as a screen Use: Often used when job performance requires a great deal of interpersonal interaction or work in team settings

43 Biodata Biodata measures (also known as biographical data, autobiographical information, and life histories) are instruments assessing an individual's previous life experiences, usually with a focus on targeting those previous experiences that predict future behaviors of interest. Cooper, A. L., & Stokes, S. G. (2003). Biodata. In Hersen, M., Thomas, C. J (Eds.), Comprehensive Handbook of Psychological Assessment, Industrial and Organizational Assessment (Volume 4). Hoboken, NJ: Wiley

44 The Many Faces of Biodata Biodata can tap into many different dimensions Cognitive Ability: What was your academic rank in your high school graduating class? Personality: How often did you attend social events during high school? Vocational Interests: How often do you read science related publications? Values: During high school, how important was it to you to be regarded as successful? Cooper, A. L., & Stokes, S. G. (2003). Biodata. In Hersen, M., Thomas, C. J (Eds.), Comprehensive Handbook of Psychological Assessment, Industrial and Organizational Assessment (Volume 4). Hoboken, NJ: Wiley

45 Biodata Validity: Moderate predictive of other criteria such as turnover attendance, and safety behavior Applicant Reactions: Typically not as favorable as other assessment methods Administration: Can be administered to many applicants at once Adverse Impact: Low against minority groups

46 Using a Whole Person Approach to Testing

47 As you add more devices or tests that measure different aspects of a person, you will learn more about that person The U.S. Department of Labor refers to this as the Whole Person approach to selection

48 Observable Behavior Performed During Work Sample Test Written Knowledge Test Imagine you had the time and resources to conduct multiple types of selection testing Test of Attitude, Personality, Biodata, Values, etc. Interview

49 Written Knowledge Test Test of Attitude, Personality, Biodata Values, etc. Interview Research shows work sample tests are typically among the most directly related to observable job performance Observable Behavior Performed During Work Sample Test These generally include samples of work performance (e.g., data entry, video-based tests)

50 Interview Written Knowledge Test Adding a test of attitude, personality, values, biodata, etc. can provide additional helpful information about the job applicant, but is typically somewhat less directly related to observable job performance Test of Attitude, Personality, Biodata, Values, etc. Observable Behavior Performed During Work Sample Test

51 Multiple Selection Devices can measure the same factors about an applicant in the same way, the same factors in different ways, and/or different factors in the same way Test of Attitude, Personality, Values, Biodata, etc. Observable Behavior Performed During Work Sample Test Interview Written Knowledge Test It is typically best to try to measure as many different factors about a person as possible during testing while balancing available resources

52 Combined Score without Hurdles Multiple-Hurdle Personality Test Work Sample Test Personality Test Work Sample Test Knowledge, Skill, Ability Test Knowledge, Skill, Ability Test Interview Interview

53 Use Test Scores Appropriately Combining personality test scores with scores from other selection devices may result in lower adverse impact You can combine scores from various types of tests, but you should Only use scores from those who scored at or above some minimum level (i.e., they must pass that section of the selection process) Weight scores from each selection device appropriately Statistically standardize the scores from different tests into a single metric, such as T-scores or Z-scores, before combining them together We suggest that you consider using personality test scores from attitude/personality tests as a pass or fail hurdle Giving a personality test earlier in the process can frequently reduce potential adverse impact during a testing process

54 Test Scores Use Applicant Score Tom 100 Stacy 100 Bob 100 Frank 100 Julie 99 Rozanne 99 Mark 98 Luke 98 Henry 97 Paul 97 Peter 96 Rebecca 96 Alyssa 95 Matthew 94 John 93 Annette 93 Ray 92 Thomas 91 Julissa 90 Ranking assumes one applicant is reliably more qualified than the other Banding considers the unreliability of the test battery and ties applicants Pass/fail cutoffs treat all applicants as either qualified or not qualified Weighting/combining test scores can be done using compensatory or using cutoff on each test then weighting results

55 Comparison Between Test Use Options Factor Ranking Banding Pass/Fail Cutoffs Validation Requirements High Moderate Low Adverse Impact High Moderate Low Defensibility Low High High Litigation "Red Flag" High Moderate Low Utility High Moderate Low Cost Low Moderate High Applicant Flow Restrictive/ Moderate/ High Controllable Controllable Development Time Low Moderate High Reliability Requirements High Moderate Low # Item Requirements High Moderate Low

56 Common Employment Testing Myths

57 57 Myth #1 A good test or interview will guarantee the people you hire will be successful Selection devices help you to improve the likelihood you are hiring the best person for the job They cannot guarantee success (no matter what the sales person says) This is why we should try to capture as much as possible about the job candidate using multiple devices, attempting to measure the whole person

58 Myth #2 The test publisher is responsible for making sure the test is fair and valid The federal Uniform Guidelines on Employee Selection Procedures (Section 7A ) say that it is your responsibility, NOT the developers of the selection devices you use, to ensure that they are fair ( Be proactive o Make sure the selection device you use is valid -- Obtain copies of the validity report o Have the validity report reviewed by professionals Many Universities have Industrial & Organizational Psychology programs that may be able to help you Ask the right people for assistance o Stay current in your research What was good yesterday, may not be good today

59 59 Myth #3 It is better not to test at all because then you cannot be sued 1. Your chances of obtaining qualified employees without testing is just that a chance (and an unlikely one at that!) 2. Well constructed and validated tests are used successfully by the best organizations with very few or no problems 3. You are actually more likely to have problems if you do not use appropriately constructed & validated selection devices because That allows for discrimination to sneak in You have no evidence to back your employment decisions o Without evidence the courts will infer discrimination

60 60 Myth #4 You should only use tests that have absolutely no adverse impact It is virtually impossible to eliminate adverse impact when attempting to measure some types of skills and abilities That being said, minimizing adverse impact is one factor to consider However, there are other factors to consider, such as making certain the job candidate can perform at least at a minimally qualified level Using only tests with no adverse impact means that you will not be measuring many of the important knowledge, skill, or abilities that are necessary to perform the job

61 61 Myth #5 It s Too Expensive to Test Testing can be expensive, but hiring the wrong person can be even more expensive Compute the costs of a bad hire You can actually reduce employee costs by choosing the most qualified person for the job Less supervision Less training Reduced turnover You can sometimes reduce testing costs through better planning Set and enforce valid basic qualifications to minimize the number of applicants to be tested Use Realistic Job Previews to have those who are not truly interested select themselves out of the testing process

62 62 Myth #6 Testing is Too Time Consuming How much time are you willing to spend to find the best people for the job? Balance the amount of time it takes to hire the best person for the job against how long you would like that person to STAY on the job

63 Conclusion Employment testing can be a valuable method to use in your hiring process Test validation is required if there is adverse impact with the use of a test Test validation makes good business sense regardless Consider many factors in test selection Use a Whole Person approach to testing

64 Questions?