Title VII Case Study: Plaintiff v. U.S. City

Transcription

1 Title VII Case Study: Plaintiff v. U.S. City Defending Interviews Against a Title VII Adverse Impact Claim February 16, 2011 Visit BCGi Online If you enjoy this webinar, Don t forget to check out our other training opportunities through the BCGi website. BCGi Standard Membership (free) Online community Monthly webinars on EEO compliance topics EEO Insight Journal (e-copy) BCGi Platinum (paid) Membership ($199/year) Fully interactive online community Includes validation/compensation analysis books EEO Tools including those needed to conduct AI analyses EEO Insight Journal (e-copy and hardcopy) Members only webinars and training and much more HRCI Credit BCGi is an HRCI Preferred Provider CE Credits are available for attending this webinar Only those who remain with us for at least 80% of the webinar will be eligible to receive the HRCI training completion form for CE submission

2 About Our Sponsor: BCG Assisted hundreds of clients with cases involving Equal Employment Opportunity (EEO) / Affirmative Action (AA) (both plaintiff and defense) Compensation Analyses / Test Development and Validation Published: Adverse Impact and Test Validation, 2 nd Ed., as a practical guide for HR professionals Editor & Publisher: EEO Insight an industry e-journal Creator and publisher of a variety of productivity Software/Web Tools: OPAC (Administrative Skills Testing) CritiCall (9-1-1 Dispatcher Testing) AutoAAP (Affirmative Action Software and Services) C 4 (Contact Center Employee Testing) Encounter (Video Situational Judgment Test) AutoGOJA (Automated Guidelines Oriented Job Analysis ) COMPare: Compensation Analysis in Excel 4 Title VII Case Study: Plaintiff v. U.S. City Defending Interviews Against a Title VII Adverse Impact Claim February 16, 2011 Contact Information Dan A. Biddle, Ph.D. CEO (800) x 113 dan@biddle.com 6

3 Agenda Interviews in the Courts Characteristics of defensible Interviews Adverse Impact review Case Review Validation Process Case Outcome 7 8 Disclaimer Discrimination, in both allegation and court concluded forms, is a serious issue. BCG works on both sides of discrimination lawsuits and believes that Title VII currently provides the means for conducting a balanced investigation of adverse-impact based discrimination to be investigated. Validation should be a process used for building fair and effective employment devices, and not as a means for just defending tests that are potentially discriminatory. Interviews and the Courts The question is still sometimes asked Are Interviews really tests? Yes, they are really tests Any Practice, Procedure, or Test (PPT) that separates two groups (e.g., men/women) based on two possible outcomes (e.g., pass/fail) is classified as a test under the Uniform Guidelines

4 Interview Defensibility: Some General Characteristics Least Defensible Unstructured Single Rater Generic one size fits all Open Scoring/No Scoring Most Defensible Structured Multiple Raters Job Specific BARS Interview Validity: When Interviews are correlated to job performance, here s what we typically see: Low Validity High Validity Unstructured Structured r= = Litigation Involving Interviews Is there a connection between Interview type and success in court? Williamson et al. (1997). Employment interview on trial: Linking interview structure with litigation outcomes. Journal of Applied Psychology, 82 (6), Study involving 84 disparate treatment and 46 disparate impact cases where interviews were litigated 17 interview characteristics were evaluated (e.g., objective, subjective, standardized, etc.). Study resulted in clear findings that revealed the three primary ingredients for successful interview validity defense

5 Litigation Involving Interviews: Key Defensibility Characteristics The Three Primary Factors Are Interview objectivity and job relatedness, such as: o Objective and specified criteria o Trained interviewers o Validation evidence Standardized administration, including: o Scoring guidelines o Minimal rater discretion o Common questions o Consistency Multiple Interviewers o Implies a shared decision making process o Rater reliability What is Adverse Impact? Adverse Impact is defined as a substantially different rate of selection in hiring, promotion, or other employment decision which works to the disadvantage of members of a race, sex, or ethnic group 14 Adverse Impact: Selection Rate Comparison (1991 CRA) & UGESP Amends Section 703 of the 1964 Civil Rights Act (Title VII) (k)(1)(a). An unlawful employment practice based on disparate impact is established under this title only if: A(i) a complaining party demonstrates that a respondent uses a particular employment practice that causes a disparate impact on the basis of race, color, religion, sex, or national origin, and the respondent fails to demonstrate that the challenged practice is job-related for the position in question and consistent with business necessity; OR, A(ii) the complaining party makes the demonstration described in subparagraph (C) with respect to an alternate employment practice, and the respondent refuses to adopt such alternative employment practice. 15

6 Adverse Impact Selection Rate Comparison 2 X 2 Table Used to evaluate hires, promotions and terminations Statistical Significance: 5% chance in Standard Deviation 16 Now Let s Review the Specific Recent Interview Case Claim from a female minority member holding Position I that the city discriminated against her when she applied to be promoted to Position II The claim of discrimination was based on race using an Adverse Impact approach A structured oral interview process was used as a primary means of selection Plaintiff claimed selection process was not valid (i.e., job related and consistent with business necessity) While this case occurred in the public sector context, the process and concepts are identical for private sector employers covered by Title VII Four-Fifths (80%) Rule of Thumb Court indicated that a violation of the 80% rule of thumb alone is not sufficient to establish a prima facie case A statistically-significant difference in selection rates needs to be shown

7 Position II Selection History Data from different testing sessions needs to be combined appropriately Year Status Focal Group Reference Group 2001 Hired x x 2001 Applied x x 2003 Hired x x 2003 Applied x x 2005 Hired x x 2005 Applied x x 2007 Hired x x 2007 Applied x x How Should the Data Be Combined? Add the years together? Conduct adverse impact for each year and average? Use a special statistical analysis specifically used for combining different versions of interviews across year? Hint! Multiple Event Probability Test (MEEPT) Simpson s s Paradox Example Simpson's paradox is an apparent paradox in which a correlation (trend) present in different groups is reversed when the groups are combined.

8 Using Appropriate Statistics Hiring combined by simple adding Pooled Years Fisher Exact Mid-P Probability Value 2001, 2003, 2005, , 2005, The true relationship is different for different interviews across a period of time Multiple Events Exact Probability Test MEEPT Years Fisher Exact Mid-P Probability Value 2001, 2003, 2005, , 2005, Plaintiff s s Statistical Experts Mistakenly added a minority group member to the analyses who did not participate in the selection process BCG found Plaintiff s statistical expert s calculation error The Plaintiff s expert admitted they added the numbers incorrectly when conducting their analyses When correct data was used, they rested their claim on the fact that minority group members were promoted at a rate less than 80% of whites However, the difference in selection rates were not statistically significant What is Validation? Definition: Traditionally, validation is making sure a practice, procedure, or test actually measures what it is designed to measure. In a legal realm, a selection procedure is valid if it can be proven by an employer that it is job related and consistent with business necessity. 24

9 Why Validate? It is good business Validation often results in selection devices that are better at helping to select qualified applicants Can reduce the amount of time spent training newlyhired employees Sends a message to the community and your employees that you care about doing the right thing Minimizes potentially negative legal actions It is required under certain circumstances If there is adverse impact against a protected group of test takers If challenged in court or during a government audit 25 Content Validation Process Job Duties Selection Devices (e.g., Structured Oral Interview) Other KSAOs Operationally defined KSAOs Where these three overlap is content valid Job Analysis: The Foundation for CV Defensibility Low: Rely on job descriptions with no job analysis about what is required on the job; no links between duties, KSAOs, and PPTs High: Full & comprehensive job analysis, including input from job experts & supervisors; links between duties, KSAOs required at entry, and PPTs

10 How the Job Analysis was Conducted Reviewed Historical Documents Feedback from Incumbent Employees Individual Telephone Interviews Job Analysis Job Duties KSAPCs Surveyed Job Experts using AutoGOJA Important/Critical Job Duties Important/Critical KSAOs required upon entry Analyzed Data Job Analysis was Conducted Remotely A BCG consultant conducted telephone interviews of Position II incumbents (SMEs), their supervisors, and a Human Resource staff member A search of the literature was made regarding the positions Previous job postings and job descriptions were examined What Ratings Does BCG Collect During a Content-related Job Analysis? For Job Duties we collect ratings for: Frequency* (Guidelines, 14.C[2]) Importance* (Guidelines, 14.C[2]) Differentiating* (Guidelines, 14.C[2]) Relative difficulty Fundamental (Americans with Disabilities Act) Assignable (Americans with Disabilities Act) * = required under federal Guidelines

11 What Ratings are Collected during a Content-related Job Analysis? For knowledge, skills, abilities, and other characteristics (KSAOs) we collect ratings for: Frequency* (Guidelines, 14.C[2]) Importance* (Guidelines, 14.C[2]) Performance Differentiating Minimum v. Desirable Qualifications Level needed upon entry Level needed for success (knowledge only) * = required under federal Guidelines Targeting Key KSAOs for Test Building Only certain types of KSAOs can be tested when using content-related validity o The PPT measures and is a representative sample of a knowledge, skill, or ability and is o Used in and is a necessary prerequisite to performance of critical or important work behaviors (i.e., shown through a link to job duties) o Should closely approximate an observable work behavior, or its product should closely approximate an observable work product Skill, abilities, & other characteristics being measured should be operationally defined in terms of observable work behaviors Content Validity Flow Chart Identify Job Duties Identify KSAOs Rate Job Duties Rate KSAOs Link Important Job Duties and KSAOs Develop Selection Device Link Important Job Duties & KSAOs to Selection Device

12 Structured Interviews Interview questions asked of job candidates were based on an examination of the job Job Analysis and validation study confirmed the job-relatedness of the questions Job Analysis foundation rested on a review of city documents, telephone interviews, and ratings collected from a panel of Job Experts Interview panels were trained by City s Office Human Resource Agent Consensus discussions by the raters were held after each interview was completed Candidates were rank ordered based on scores Effectively used to band not strict rank order- candidates Different oral interview questions were asked of candidates during each year of the selection process Validity Findings % of Acceptable Questions # Testing Standard Is the question CLEAR and UNDERSTANDABLE? 100% 100% 100% 100% 2. Is the question written at an APPROPRIATE DIFFICULTY LEVEL? 100% 100% 100% 100% Is this question FAIR to all groups of candidates (based on their race, age, and/or gender)? Is the question job related (does it represent a situation that has occurred or is very likely to occur on the job)? Will applicants be able to provide a sufficient response without possessing job-specific knowledge that they will receive during training once they are hired as a Position II? 6. What KSA is measured by this question? 90.90% 100% 100% 88.89% 100% 100% 87.5% 88.89% 100% 100% 100% 100% All of the interview questions were linked by the Job Experts to at least one important or critical KSA. Don t t Questions Have to Linked to Important or Critical Job Duties? Section 14C(2) of the federal Uniform Guidelines indicates that the work behaviors being measured should be critical work behavior(s) and/or important work behavior(s) constituting most of the job. An interview question to important or critical work behavior (Job Duty) linkage can be inferred in the current study since each interview question was linked to KSAs and we had Job Experts indicate the strength of the linkages between KSAs and Job Duties

13 Plaintiff Also Claimed Claim: A separate validation study was needed before scores from interviews could be used for rank ordering of candidates Response: The process of ranking and/or score banding is permissible under the federal Uniform Guidelines and does not need to be validated using a separate validation study. Rather, the process can be validated using the job analysis and validation evidence that exists from the current study. The Uniform Guidelines allow for using a selection procedure for ranking job candidates if the user can specify evidence showing that a higher score on the selection procedure is likely to result in better job performance (see Section 14C9). Even More Claims Plaintiff moved to exclude BCG as the City s Expert based on the fact that the job analysis and validation was conducted after the litigation commenced Plaintiff claimed that since the consultant who was facilitating the job analysis did not visit onsite, it was invalid Plaintiff claimed interview questions were written to avoid knowledge and skills and to cater to the favorites who had been preselected Eve-of of-trial Study The job analysis and validation study was conducted after the claim of discrimination was made Even though courts and auditing agencies are likely to more strictly scrutinize validation studies that are conducted after adverse impact is determined, this court ruled that it was acceptable to conduct the study after a claim of discrimination was made

14 The Court Ruled Court ruled that to successfully challenge the validation study, Plaintiff would need to submit expert testimony or other objective evidence that tended to refute the findings of the study Plaintiff did not provide any expert testimony to rebut the City s validation study Plaintiff did not challenge qualifications of Dr. Biddle or the reliability of his methodology or analyses Based on these failures, the court denied the Plaintiff s motion to exclude this study Adverse Impact During a Specific Test or During the Entire Process? Court indicated Plaintiff must identify a specific test, requirement or practice that has an adverse impact on race Smith v. City of Jackson, Miss., 544 U.S. 228, 229 (2005) Plaintiff failed to identify a specific employment practice that had adverse impact For this reason, the court focused only on the Interviews given for the position, rather than the promotional process as a whole Take Aways Selection procedures should be job related (i.e., valid) and fair to all qualified applicants Evidence of validity should be available for review if challenged Written records of testing should be maintained The appropriate statistical tests should be used Having EEO experts in your corner can really help to defend your testing practices

15 Helpful Validation Resources Federal Uniform Guidelines on Employee Selection Procedures U. S. Department of Labor s Testing and Assessment: An Employer's Guide to Good Practices Adverse Impact and Test Validation: A practitioner s guide to valid and defensible employment testing Author: Dan A. Biddle, Ph.D