Today s presentation will begin at 10 a.m. Pacific Time 11 a.m. Mountain Time 12 noon Central Time 1 p.m.

Transcription

1 Adverse Impact and Test Validation: A Practitioner s Handbook Steps for Developing a Validated Work Sample Physical Ability Test (PAT) Using Content Validity Today s presentation will begin at 10 a.m. Pacific Time 11 a.m. Mountain Time 12 noon Central Time 1 p.m. Eastern Time Chapter 6 Copyright 2012 Biddle Consulting Group, Inc. 1 Visit BCGi Online If you enjoy this webinar, Don t forget to check out our other training opportunities through the BCGi website. BCGi Standard Membership (free) Online community Monthly webinars on EEO compliance topics EEO Insight Journal (e-copy) BCGi Platinum (paid) Membership Fully interactive online community Includes validation/compensation analysis books EEO Tools including those needed to conduct AI analyses EEO Insight Journal (e-copy and hardcopy) Members only webinars and training and much more 2 HRCI Credits Available for Today s Presentation BCGi is a Human Resource Certification Institute (HRCI) Preferred Provider Continuing Education (CE) Credits are available for attending this webinar for no charge Only those who remain with us for at least 80% of the webinar will be eligible to receive the HRCI training completion form for CE submission 3 1

2 About Our Sponsor: Biddle Consulting Group, Inc. (BCG) Affirmative Action Plan (AAP) Consulting and Fulfillment HR Assessments Thousands of AAPs developed each year Audit and compliance assistance AutoAAP Enterprise software AutoGOJA online job analysis system TVAP test validation & analysis program CritiCall pre-employment testing for 911 operators OPAC pre-employment testing for admin professionals Video Situational Assessments (General and Nursing) EEO Litigation Consulting /Expert Witness Services Over 200+ cases in EEO/AA (both plaintiff and defense) Focus on disparate impact/validation cases Compensation Analysis Publications/Books BCG Institute for Workforce Development Nation-Wide Speaking and Training Proactive and litigation/enforcement pay equity studies COMPare compensation analysis software EEO Insight : Leading EEO Compliance Journal Adverse Impact (3 rd ed.) / Compensation (1 st ed.) 4,000+ members Free webinars, EEO resources/tools Regular speakers on the national speaking circuit 4 Listening to Today s Presentation To eliminate potential long-distance telephone charges, you can hear the audio to this presentation through your computer starting at 10 a.m. Pacific Time. We recommend using a headset. Make certain the volume on your computer is not muted and turned up sufficiently to hear the presentation through your computer If that does not work, you can call into the long-distance number provided in your webinar invitation to hear the sound portion of the presentation. Once connected via telephone, you must enter the Telephone Access Code listed in your invitation or Outlook schedule. 5 BCGi: Adverse Impact & Test Validation Book Series Steps for Developing a Validated Work Sample Physical Ability Test (PAT) Using Content Validity Chapter 6 Copyright 2012 Biddle Consulting Group, Inc. Today s presentation by Jim Kuthy, Ph.D. Principal Consultant Heather Patchell, M.A. Consultant 6 2

3 A Little About Presenters Jim Kuthy holds Masters and Doctorate Degrees in Industrial & Organizational Psychology Almost 20 years of training and experience in the employment selection field Heather Patchell has a Masters Degree in Social Psychology Five years of experience in employment selection Designed selection and promotion devices for dozens of employers Validated many selection devices, including conducting validation studies that have been successfully defended in court or passed review by federal agencies 7 Fine Print The information provided in the presentation is not intended as legal advice We note there may be differences in Equal Employment Opportunity law in the jurisdiction where your organization is located. Please seek additional information regarding questions you may have regarding specific types of testing for your organization Information contained in this presentation is, to the best of our knowledge, current as of May 21, Developing/Validating Work Sample Physical Ability Tests (PAT) Using Content Validity Physical Ability Tests (PATs) are tests that measure an applicant s ability to perform the physical parts of the job They can be administered in pre- or post-job offer settings as long they are non medical Medically-related tests can only be administered post offer PATs are highly useful and necessary tools in a wide variety of employment situations Strenuous PATs can sometimes exhibit adverse impact against women, and must therefore be validated Using Work Sample style PATs to mimic the actual job are generally preferable to more abstract types of PATs Using a content validation strategy is preferable in most situations where PATs will be used 3

4 Why is this Topic Important to HR/EEO Professionals? Why do I need to know about this topic? PATs are typically the #1 cause of adverse impact against women in hiring settings PATs that have not been validated but exhibit adverse impact will quickly lose in litigation Physically demanding jobs need to have PAT, or false positives will enter the workplace What are the key essentials I need to know about this topic? Using a content validation strategy is fairly straight-forward for validating PATs The Uniform Guidelines and the 1990 ADA provide the key criteria for setting up defensible PATs What are the consequences surrounding these issues? Qualified workers will be unfairly screened out unless a PAT is valid The start up cost of a validation case is between $30k and $80k Presentation Overview Steps for Developing a Validated Work- Sample Physical Ability Test Using Content Validity Administering the Test Scoring Physical Ability Tests The Americans with Disabilities Act and Physical Ability Testing Sample Test Event Description: Ladder Removal/Carry Hints and Thoughts What is Validity as Specified in the Courts? The process of gathering and evaluating necessary evidence to support test score interpretations. A selection procedure is said to be valid if it can be proven by an employer that it is job related and consistent with business necessity 12 4

5 Why do we validate? It makes good business sense Validation often results in selection devices that are better at helping to choose the best people for the job Can help to reduce the amount of time spent training newly-hired employees Often helps to reduce turnover Sends the message that you care about doing the right thing Helps to minimize potentially-negative legal liability 13 Why do we validate? It is required under some circumstances If there is adverse impact against a protected group of test takers (federal Uniform Guidelines on Employee Selection Procedures, 1978) See Go to Section 14C and 15C for Content-Related Validity specifications For federal contractors: If selection testing is identified as a problem during an audit (Executive Order 11246, As Amended) Check with your AAP consultant to determine if validation is required if an audit should occur 14 Steps for Developing a Physical 1. Conduct a solid job analysis that focuses on physical aspect of the job Job analysis identifies the important and critical work behaviors (duties), as well as the abilities required to perform those duties Abilities should be operationally defined in terms of observable aspects of work behavior of the job Uniform Guidelines Section 14C(4): The closer the content and the context of the selection procedure are to work samples or work behaviors, the stronger is the basis for showing content validity. 15 5

6 Steps for Developing a Physical 2a. Identify parts of the job that are typically require continuous physical exertion for over 10 minutes This may include repeatedly performed tasks This may consist of unrelated tasks where a rapid pace is required 2b. Identify other physically strenuous parts of the job Identify the most strenuous in terms of weights, heights, distances, etc. 16 Steps for Developing a Physical 3a. Work with supervisors, trainers, and/or employees to assemble a continuouslytimed job simulation PAT The included events need to be those where rapid pace is required for physically-demanding job duties 3b. Also work to assemble discrete test events for measuring physically-demanding parts of the job that are not typically performed in rapid succession or continuously 17 Continuous vs. Discrete Events Continuous Events For tasks that are performed continuously and/or sequentially Combine events into a continuous test event only if supported by the job analysis Discrete Events Sort of like snapshots of strenuous tasks Allow time between events in a job-related fashion Focus on the most physically demanding aspects of the job 18 6

7 Steps for Developing a Physical 4. Run a representative sample of Job Experts (20-30, if possible) through the PAT attempt to oversample underrepresented groups to help insure fairness Then administer a validation survey to those experts that collects the following data: Record actual PAT completion time Total time if using a continuously timed test; each event s time if using discrete test events Ask Job Experts for their opinion Ask, given your time to complete the physical ability test, your current fitness level, and your level of job experience, what time should a minimally-qualified applicant use when taking this test? 19 Steps for Developing a Physical 4. Validation Survey (Yes/No Questions) Does the PAT measure skills/abilities that are important/critical (essential for the performance of the job)? Does the PAT measure skills/abilities that are necessary on the first day of the job (i.e., before training)? Does the PAT replicate/simulate actual work behaviors in a manner, setting, and level of complexity similar to the job? Do the events in the PAT need to be completed on the job in a rapid and safe manner (i.e., is speed important)? 20 Steps for Developing a Physical 4. Validation Survey (Yes/No Questions continued) Are the weights and distances involved in the PAT representative of the job? Is the duration that the objects/equipment are typically carried or handled in the PAT similar to what is required of a single person on the job? Is the PAT free from any special techniques that are learned on the job that allow current job incumbents to perform the PAT events better than an applicant could (that are not demonstrated to the applicants prior to taking the PAT)? Does the PAT require the same or less exertion of the applicant than is required on the job? 21 7

8 Steps for Developing a Physical 5. Analyze the survey data At least 70% of experts should respond Yes to each and every Yes/No survey question If not, go back and redesign the test as necessary Then re-survey until at least 70% endorse all survey items 22 Steps for Developing a Physical 5. Analyze the survey data Compute the average of the Job Experts opinion times gathered in Step 4B Use the Standard Error of Measurement (determined in Step 6) to adjust average opinion times to create final cutoff i.e., Add 1 or 2 SEMs to obtain final cutoff time 23 Steps for Developing a Physical 5. Analyze the survey data (continued) Compute the average of the Job Expert s actual time Add Standard Errors of Difference, which provides 90% confidence or 1.96 Standard Deviations, which provides 97.5% confidence This level constitutes the lowest score boundary of the normal expectations of acceptable proficiency in the workplace that is required when setting cutoff scores (Section 15H of the Uniform Guidelines) Ideally, the final cutoff score (i.e., required time) should be close to the adjusted actual time 24 8

9 6. Conduct a test-retest study of 60+ applicants or incumbents to determine the Standard Error of Measurement of the PAT =σ SEM ( 1 r) x σ x is the standard deviation of test scores r xx is the test-retest reliability 25 6 (continued). Use the SEM to determine the Standard Error of Difference SED = SEM * 2 SED = Standard Error of Difference SEM = Standard Error of Measurement 26 Warning Any pace, weights, or other limitations used during testing MUST be job related for the position in question (i.e., related to actual pace, distances, weights, or other limitations that are required on the job) EEOC v. Dial Corp. a unanimous decision by the U.S. Court of Appeals for the Eighth Circuit in Case Nos and Test was more difficult than the job because the pace did not include natural breaks between tasks that occurred in the workplace The decision affirmed an award of approximately $3.3 million to 52 rejected female job applicants 27 9

10 Administering the Test Train test administrators Run practice sessions; determine proficiency Prior to testing, inform candidates of the contents of the event(s) Give them sufficient opportunity to practice and prepare in advance of the test day Standardize instructions (i.e., read and/or show video) If appropriate, demonstrate proper and/or safe technique Make certain candidates understand the instructions 28 Administering the Test If candidate appears confused or frustrated, ask, Do you need me to repeat the instructions? Do not ask this if the ability to understand and follow the instruction is part of the test (as identified during the job analysis) Alternatively, you can consider the need for a rereading of the instructions when scoring There should be a penalty during testing only if there would be some penalty for that type of behavior on the job Allow multiple attempts to complete the test event(s) (if appropriate) 29 Administering the Test Your paramount concern should be safety safety Candidates should be provided safety rules and description of safe work practices in advance of the testing process Read those again prior to testing Immediately stop testing if candidate fails to follow a safety rule or safe work practice Explain to the candidate what they did incorrectly, and if necessary, demonstrate proper behavior Allow candidate to continue testing unless it is obvious injury or harm to the candidate, others, and/or company property will occur Document this! 30 10

11 Administering the Test If a candidate violates a safety rule or performs in a way the test administrator believes demonstrates that candidate does not possess the level of safetyrelated knowledge that a minimally-qualified applicant should possess, a complete and accurate description of the violation or unsafe practice should be recorded Have the information about the violation or unsafe practice reviewed by a panel of Job Experts Candidates who the experts believe do not possess the level of safety-related knowledge or ability that a minimally-qualified candidate should possess prior to training on the job will be disqualified 31 Administering the Test If candidate appears to be injured during testing, ask, Are you injured? If answer is yes, or if it is obvious the candidate has suffered an injury, stop the event and obtain assistance immediately Develop a plan in advance on how to handle injuries Clear mark the boundaries and start/end points of the test Base the limits on solid, job-related reasons for the position in question 32 Diagram the Test Event in advance to Minimize Potential Confusion (See book for narrative) 33 11

12 Scoring PATs Use only objective criteria Did the test taker successfully complete the task asked of them? Do not score on appeared to be struggling, was breathing hard, or even had to stand on toes to reach While these are observable, if the candidate successfully completed the task, they passed Carefully records times and distances It can be helpful to have multiple judges and/or timing devices Include timer in video, if appropriate Record information carefully 34 Americans with Disabilities Act Limit testing to abilities related to essential job functions employees are required to perform with or without reasonable accommodation 1. Function is the reason the positions exists 2. There are a limited number of employees available to perform that job function 3. The function is highly specialized so the person is hired for their expertise or ability to perform that particular function Medically-based tests can only be given after a bona fide offer of employment has been given Employers must provide reasonable accommodation during testing even if the employer knows they will not be able to provide that person with a reasonable accommodation on the job 35 Pre-Employment Medical Testing Pre-employment medically-based physical testing is allowed under the Americans with Disabilities Act Must be given post offer Must be given to all employees who enter that job category If employee is rejected after a medical exam, the EEOC will closely scrutinize whether the rejection was based on the results of the exam These can include measures of hearth rate and VO2 max If an employee is screened out because of a disability, the employer must demonstrate that the rejection is job-related and consistent with business necessity (i.e., valid) If person is screened out for safety reasons, the employer must demonstrate that the individual poses a direct threat (i.e., significant risk of substantial harm to themselves or others that cannot be reduced through reasonable accommodation) Make certain record-keeping requirements are addressed 36 12

13 What about Fitness Testing? Job candidates must run one-and-a-half miles in 15 minutes 20 seconds The candidate will complete a 3 minute step test with a 12 inch step. They will step to the beat of a metronome set to 96 beats-per-minute (24 steps per minute). Heart rate will start to be measured within 5 seconds of the end of the test, and measured for a total of 1 minute. Candidate s ranking must be greater than or equal to the 85 th percentile based on normative data from the YMCA 3 Minute Step Test. The candidate will do 15 pushups and 10 pull-ups 37 What about Fitness Testing? Fitness testing (e.g., measures of VO 2 max) requires a statistical examination of test performance (criterionrelated validity) unless test task mimics work behaviors Fitness tests cannot be validated using the content-related approach described in the current presentation In other words, the employer must be able to demonstrate a statistically significant relationship of test scores to job performance to use fitness testing Tests that are designed to measure strength in the abstract (being strong for the sake of being strong without any connection to job requirements) must be validated using a criterion-related study 38 Proceed Carefully Caution Tests that use arbitrary cutoffs, such as the fitness standards are based on the 40 th percentile for the general population are generally not defensible Caution Tests that measure physiological characteristics, such as heart rate, are considered medical tests under the Americans with Disabilities Act, and can only be given post offer Our experience has been that Step Test scores do not typically correlate with injury rates and/or occurrences This may be because employees who are more fit may also be more likely to take physical risks and not abide by company safety protocols Caution Some fitness testing requires age and gender to be considered. The issue of age and/or gender-norming fitness tests has not been settled in the courts and can result in liability Applicants generally consider testing that is face valid (i.e., that has a transparent relationship to the job) to be more fair 39 13

14 Fancy Machines Do Not Automatically Increase Validity Technical testing devices do not automatically result in more valid testing Frequently, they can actually make it less likely that testing is valid as defined in the Uniform Guidelines since they are represented to be scientific Machines that mimic work tasks can be content validated We suggest you have any claims of validity reviewed by an expert According to the EEOC: Many industrial and personnel psychologists validate selection procedures, review published evidence of validity and make recommendations with respect to the use of selection procedures. Many of these individuals are members or fellows of Division 14 (Industrial and Organizational Psychology) or Division 5 (Evaluation and Measurement) of the American Psychological Association. BCG has several Ph.D.-level Division 14 members on staff to assist you 40 Best Practices Consider video-taping the testing process This can minimize potential liability by demonstrating how the candidate actually performed during testing Have both administrator and test taker sign the scoring sheet, if possible Limit verbal feedback to candidates and to onlookers during testing If feedback is given, it should be similar across all candidates We recommend that spectators not be allowed Have test taker remain for a little while after testing to insure he/she has sufficiently recovered from the testing process 41 Best Practices Candidates should be asked to sign a waiver of liability (and indicate they are ready for testing) Liability waiver should be signed before testing begins Liability release forms should include a description of the test, so candidates can make an intelligent waiver If the employer so decides, candidates may be required to obtain a medical release prior to physical-ability testing If you require a medical release of one candidate, you must require the same release from all candidates Warning Do not use different rules for different candidates 42 14

15 Final Thoughts on Content- Related Physical Ability Testing The closer the content and the context of the selection procedure are to work samples or work behaviors, the stronger is the basis for showing content validity Uniform Guidelines Section 15C(4) Standardize administration and scoring of testing Abide by Americans with Disabilities Act Have fun! (But be professional) 43 Questions? Jim Kuthy, Ph.D. Heather Patchell, M.A. 44 BCGi: Adverse Impact & Test Validation Book Series Steps for Developing a Validated Work Sample Physical Ability Test (PAT) Using Content Validity Chapter 6 Copyright 2012 Biddle Consulting Group, Inc. Jim Kuthy, Ph.D. Principal Consultant Heather Patchell, M.A. Consultant 45 15