BCGi. Copyright 2009, BCGinstitute, inc. All Rights Reserved

Transcription

1 Advanced Adverse Impact Analysis Why the (uncorrected) Fisher Exact Test Should not be used for Most Adverse Impact Analyses ( ) Copyright 2009 Biddle Consulting Group, Inc. BCGi Institute for Workforce Development Visit BCGi Online While you are waiting for the webinar to begin, Don t forget to check out our other training opportunities through the BCGi website. Join our online learning community by signing up (its free) and we will notify you of our upcoming free training i events as well as other information of value to the HR community. 1

2 HRCI Credit BCG is an HRCI Preferred Provider CE Credits are available for attending this webinar Only those who remain with us for at least 80% of the webinar will be eligible to receive the HRCI training completion form for CE submission About Our Sponsor: BCG Assisted hundreds of clients with cases involving Equal Employment Opportunity (EEO) / Affirmative Action (AA) (both plaintiff and defense) Compensation Analyses / Test Development and Validation Published: Adverse Impact and Test Validation, 2 nd Ed., as a practical guide for HR professionals Editor & Publisher: EEO Insight an industry e-journal Creator and publisher of a variety of productivity Software/Web Tools: OPAC (Administrative Skills Testing) CritiCall (9-1-1 Dispatcher Testing) AutoAAP (Affirmative Action Software and Services) C 4 (Contact Center Employee Testing) Encounter (Video Situational Judgment Test) Adverse Impact Toolkit (free online at AutoGOJA (Automated Guidelines Oriented Job Analysis ) COMPare: Compensation Analysis in Excel Industry Leader 4 2

3 Contact Information Daniel Biddle, Ph.D. Biddle Consulting Group, Inc. 193 Blue Ravine, Ste. 270 Folsom, CA Questions? Should you have any questions during the webinar you have two options: Ask a question through the GoToMeeting screen console and we will try to address it at the end of the webinar. Should you have any questions regarding g OFCCP Audits, Testing and Selection, or Statistical Analysis, visit: 3

4 Presentation Overview Disclaimer: These are complicated topics! Adverse Impact Analyses Background: Is this really important? Issue #1: Marginal Totals Issue #2: Conservativeness Data simulation results Implications and recommendations Copyright 2009, Biddle Consulting Group What s the Big Deal? The issues we ll be discussing are a big deal and not a big deal at the same time. The big deal? Adverse impact is serious and no one wants to calculate liability statistics inaccurately. But it s not a big deal because most of the time, under most circumstances, when SS adverse impact is there, it s there! Most court cases and audits are typically only enforced when statistical evidence is strong. 4

5 How Did this Come About? For decades, EEO professionals have relied on Chi-Square type analyses for the 2x2 table question: Men Women Totals Pass Fail Totals Sometimes various corrections have been used (Yates, Cochran). Sometimes the Fisher Exact Test (FET) has been used How Did this Come About? But is that what Fisher intended? All 2x2 tables to be run through his exact test? Since about the 1950s, various challenges have been brought to the FET: What are the assumptions required for the FET results to be accurately interpreted? Is the FET too conservative? Are there other more accurate techniques when the strict FET conditions are not met? 5

6 How Did this Come About? Most recently, from the mid-90s to this year, a barrage of articles have been published in the bio- med field, theoretical statistics field and journals, and other fields that have criticized the FET, abandoned the FET, and recommended other less conservative replacements that are more applicable and accurate across a greater diversity of 2x2 situations The adverse impact has not been neglected in these discussions We ve reviewed over 80 such articles and chapters How Did this Come About? 2X2 analyses can be conducted in three situations: fixed, mixed, and free margins. While there is a consensus in the current literature that the FET is inappropriate in 2 of these 3 2X2 situations There is not a consensus regarding whether the uncorrected FET should be used in 1 of the 3 2X2 situations. When evaluating these 2x2 situations, it becomes clear that the FET should not be used in many AI circumstances, but may be used in some situations Let s take a look at the 2x2 situations 6

7 FET Issue #1: Marginal Totals The FET Requires Meeting Conditional Assumptions Not Always Met in Practice Are the margins FIXED before or after the event? Are the margins FIXED, CONSTRAINED, OR CORRELATED to the employer s previous decisions? Men Women Totals Pass Fail Totals FET Issue #1: Marginal Totals The Three Major 2x2 Models (see Collins & Morris, 2008) Men Women Totals Pass Fail Totals FIXED Men Women Totals Pass Fail Totals MIXED Men Women Totals Pass Fail Totals FREE Model 1: Independence Trial: The marginal proportions are assumed to be fixed in advance (i.e., proportion of each group and selection totals are fixed). Data are not viewed as a random sample from a larger population. Model 2: Comparative Trial: Apps are viewed as random samples from two distinct populations (e.g., minority and majority). Proportion from each population is fixed (i.e., the marginal proportion on one variable is assumed to be constant across replications). The second marginal proportion (e.g., the marginal proportion of applicants who pass the selection test) is estimated from the sample data. Model 3: Double Dichotomy: Neither row/column are assumed to be fixed. Apps are viewed as a random sample from a population that is characterized by two dichotomous characteristics. No purposive sampling or assignment to groups is used, and the proportion in each group can vary across samples. 14 7

8 FET Issue #1: Marginal Totals The Three Major 2x2 Models: Applied to EEO Analysis Men Women Totals Pass Fail Totals Men Women Totals Pass Fail Totals Men Women Totals Pass Fail Totals FIXED Terminations i / RIFs However shared/correlated with past practices; oftentimes not predetermined MIXED Some promotions However shared odds ratios FREE Apps are widely recruited; show up; unknown passing rates / hiring rates FET Issue #1: Marginal Totals Which of the Three 2X2 Models Apply to HR Decisions? Reviewing Three 2X2 Models to HR Decisions (Adapted from Collins & Morris, 2008) HR Practice 2X2 Model Comments Hiring with a fixed cutoff score Top-down selection Double Dichotomy None of the three models fit appropriately Selection decisions use a fixed cutoff score. The passing score typically set in advance or using normative data. MQs might be used. Candidates are selected top-down based on hiring criteria until a fixed number of positions are filled. Selection rate is fixed based on staffing needs. If a different sample had been used, the number passing would have been the same. However, each group's proportion is likely to vary across samples and is best treated as an estimate of an unknown population parameter. Further, because selection decisions depend on applicant rank position in a particular sample, the selected and nonselected groups are sample-specific and do not reflect two distinct populations as in the comparative trial model. 8

9 FET Issue #1: Marginal Totals Which of the Three 2X2 Models Apply to HR Decisions? Reviewing Three 2X2 Models to HR Decisions (Adapted from Collins & Morris, 2008) HR Practice 2X2 Model Comments Banding Promotion None of the three models fit appropriately. None of the three models fit appropriately. Banding is a combination of "ranking" and typically also involves a minimum cutoff score, so it is a hybrid method for which none of the sampling models are a perfect fit. Candidate pool is relatively fixed... If decisions were repeated, candidate set would be similar. In such cases, probabilities based on randomly sampling from a population, as in the comparative trial and double dichotomy models, would not apply. Similarly, probabilities based on random reassignment of participants (i.e., independence trial model), would not be appropriate. Without theoretical process for producing different data patterns (e.g., random 17 FET Issue #1: Marginal Totals Which of the Three 2X2 Models Apply to HR Decisions? Because the independence trial model ( Fixed ) does not represent typical personnel selection data, there is reason to question the appropriateness of the Fisher Exact Test for adverse impact analysis. The tendency of these tests to be conservative under the other sampling models indicates that the Fisher Exact Test and Yates s test will be less likely than other tests to identify true cases of adverse impact (Morris & Collins, 2008)

10 FET Issue #1: Marginal Totals Which of the Three 2X2 Models Apply to HR Decisions? In the EEO analysis field, The justification of conditional tests (those for fixed margins) depends on the assumption that the process determining the fixed marginal counts is not dependent on the process under study For example, When considering whether to use a conditional test (the FET) when conducting a promotional analysis, The number of minority members hired out of a labor pool should not provide information about the odds ratio of the promotion rates, the parameter of interest. Gastwirth advises checking this assumption before calculating conditional tests in situations where the available sample results from a previous selection process that may be affected by the same factors involved in the process being examined (because the odds ratio of the hiring rates and promotion rates would be related). For this reason, the unconditional tests may be a more accurate test across a greater number of AI cases (Gastwirth, J. (1997). Statistical Evidence in Discrimination Cases, Journal of the Royal Statistical Society, 160, Part 2, ) FET Issue #1: Marginal Totals Men Women HIRES PROMS Men Women Men Women TERMS In the EEO area, when using statistical ti ti tests, t it is important t to consider a crucial assumption underlying conditional tests. This assumption requires that one can condition on fixed marginal numbers that are not dependent on any factor related to the process being investigated. For example, if one examines the promotion data of a firm, the marginal sample sizes of minority and majority employees eligible for advancement clearly result from the hiring practice of that firm (Ibid)

11 FET Issue #1: Marginal Totals FET Requires Calling Out Marginal Totals Before the Analysis is Conducted When, if ever, is this really the case in AI analyses? The FET assumes that both of the margins in a 2X2 table are fixed by construction i.e., both the treatment and outcome margins are fixed a priori (Sekhon, 2005). Over decades there has been a lively debate among statisticians on the applicability of the conditional FET. The argumentation against the test mainly is that it conditions inference on both margins where only one margin is fixed by most experimental designs and the test is inherently conservative the row and column marginal totals are fixed by the researcher prior to data collection (p. 171, Gimpel, 2007). Fisher s 2 x 2 exact test requires that the marginal frequencies in both margins are fixed a priori (Romualdi, et al, 2001) FET Issue #1: Marginal Totals Which of the Three 2X2 Models Apply to HR Decisions? For fun it isn t truth today if it s not so on Wiki! FET assumes that the row and column totals are known in advance. In cases where this assumption is not met, FET is very conservative, resulting in Type I error which is below the nominal significance level. In practice, this assumption is not met in many experimental designs and almost all non-experimental ones. An alternative exact test, Barnard's exact test, has been developed and Proponents of it suggest that this method is more powerful, particularly in 2 by 2 tables

12 FET Issue #2: Conservativeness The FET is too Conservative Compared to other Methods The tendency of these tests to be conservative under the other sampling models indicates that t the FET and Yates s test t will illbe less likely than other tests to identify true cases of adverse impact (Morris & Collins, 2008). The exact test of Fisher gives tests which are both extremely conservative and is appropriate (Upton, 1982) (later endorsed mid-p). The traditional FET should practically never be used the FET is unnecessarily conservative with lower power than conditional mid-p tests and unconditional tests We do not recommend the use of FET. FET is conservative, that is, other tests generally have higher power yet still preserve test size Lydersen, et. al, 2009) FET can be conservative in the sense of its actual significance level (or size) being much less than the nominal level (Lin & Yang, 2009) Probability Theory Applied to 2X2 Tables DEMONSTRATION OF "DISCRETENESS" IN THE FET PROBABILITY DISTRIBUTION FET: Mid p: Uncond.: The FET has 4 "stopping places" below.05 Chi Square Theory has more Asymptotic "Best Estimate" Line Used by the Chi-Square

13 Probability Theory Applied to 2X2 Tables Actual Significance Level v. Desired (.05) Significance Level Mid-P Actual Significance Level v. Desired (.05) Significance Level FET (uncorrected) Important Questions for HR Professionals 25 What is the significance level used for testing whether a test is valid? What is the significance level used for testing Adverse Impact? Answers: Validity:.05 Adverse Impact:.05 What statistical tests are useful for answering these statistical questions? Validity: Pearson correlation is common Adverse Impact: Fisher Exact Test (under a variety of methods), Chi-Square, Z-test, etc. 13

14 Example Let s take two employers that use a physical test that has a standardized mean group difference of 1.0 (d) between men/women This difference is commonly observed on written tests (minority/non-minority) and physical tests (men/women). Each employer tests for 1,000 applicants per year One employer hires only the top 10%; the other only the top 40% Such a test will exhibit adverse impact, it s just depends on 2 factors: 1: the number of applicants tested and hired, and 2: the power of the statistical test used to detect the AI Example This example constitutes one where a substantial passing rate difference (required by the Guidelines) has been observed between the two groups in the population Using a 40% hiring rate, a standardized mean group difference of 1.0 (d) (between men and women) equates to: 58% male passing rate 22% female passing rate Using a 10% hiring rate, a of 1.0 (d) equates to: 18% male passing rate 3% female passing rate 14

15 Practical Implications of a Test with a 1.0 (d) How much overlap is there between two groups based on various d values?.25 d = 82% overlap.75 d = 55% overlap.50 d = 67% overlap 1.0 d = 45% overlap Summarizing the AI Evidence on a 1.0 (d) Test Evaluating the company that uses a 40% hiring rate: 58% of the men will pass and 22% of the women will pass Hiring ratio = 2.6 male hires for every 1 female hire The impact ratio is 38% (2X less than half the 80% test) Less than one-half (45%) of the male distribution overlaps with the female distribution Evaluating the company that uses a 10% hiring rate: 18% of the men will pass and 3% of the women will pass Hiring ratio = 6 male hires for every 1 female hire The impact ratio is 17% (4X less than half the 80% test) Less than one-half (45%) of the male distribution overlaps with the female distribution 15

16 Finding Adverse Impact Next let s investigate the usefulness in answering the AI question using three statistical tools: Fisher Exact Test (FET) Fisher Exact Test (mid-p) Chi-Square (or Z test) The sample sizes for both the 40% hiring rate and 10% hiring rate employers will be scaled and evaluated Sample sizes will be matched for both men and women First, Some Definitions 31 Type I Error (α): reject the null ( no difference ) hypothesis when the null hypothesis is true In other words, finding AI when it does not exist Type II Error (β): fail to reject the null hypothesis when the null hypothesis is false In other words, missing AI when it exists Type I Error Rate: The percentage of Type I errors made by a statistical test (i.e., the rate at which it falsely concludes AI). Type II Error Rate: The percentage of Type II errors made by a statistical test (i.e., the rate at which it misses AI that exists). Nominal Level: the p-value of significance, declared in advance (e.g.,.05) (in AI cases, the major concern is with answering the big.05 question ) 32 16

17 More Definitions Statistical power analysis evaluates the likelihood that a statistical test will find a meaningful difference at the specified level (e.g.,.05, or 2SDs). Adverse Impact tests that are more powerful are more likely l to find adverse impact when it exists. 100% 90% 80% 70% 60% 33 Power Analysis for 40% Hiring Rate Employer Power Curve for Detecting Adverse Impact on a 1.0(d) Test Used with a 40% Overall Passing Rate / Chart Answers the Question: What percent of the time will the test find adverse impact when it exists? FET FET (mid P) Chi Square 50% 40% 30% Gap Showing Increased Likelihood of Missing AI When it Exists 20% 10% 0% Sample Size (Equal for Each Group) 34 17

18 Power Analysis for 10% Hiring Rate Employer Power Curve for Detecting Adverse Impact on a 1.0(d) Test Used with a 10% Overall Passing Rate / Chart Answers the Question: What percent of the time will the test find adverse impact when it exists? FET FET (mid P) Chi Square 100% 90% 80% 70% 60% 50% 40% 30% 20% Gap Showing Increased Likelihood of Missing AI When it Exists 10% 0% Sample Size (Equal for Each Group) Power Comparison in Small Samples Average Statistical Power in Samples Between 10 and FET FET (mid P) Chi Square 80% 75% 76% 70% 64% 66% 68% 60% 56% POWER 50% 40% 30% 29% 39% 45% 20% 10% 0% 10% SR 20% SR 40% SR Selection Ratio 18

19 Accuracy of Tests for Answering the just.05 or less Question SD Value Comparison Between FET and Mid-P Based on Sample Size (Based on Monte Carlo Simulations from Cited Articles) FET SD Required for Significance Poly. (FET SD Required for Significance) Mid-p SD Required for Significance Poly. (Mid-p SD Required for Signif icance) AVERAGE "OVERAGE" OF FISHER EXACT TEST (AMOUNT HIGHER THAN 1.96 TO FIND AN ACTUAL 1.96 FINDING) AVERAGE "OVERAGE" OF MID P Sample Size Sample Size How Accurately do the Tests Answer the.05 Question? Actual FET/Mid-P Significance Levels (Compared to Desired.05 Level) Typical % Below Actual SD Alpha Desired.05 Required for Range Level Significance % % % % % % 2.02 Typical Estimate for % 2.19 n<50 (FET) Typical Estimate for n<50 (mid-p) %

20 Type I Error Rates Between Tests P V Value Generated by Test SRT 10% PMIN10% N 20 Type I Error Rate Comparison Between Three 2X2 Tests Across Sample Size/Selection Ratio Scenarios SRT 10% PMIN10% N 50 SRT 10% PMIN10% N 100 SRT 10% PMIN30% N 20 SRT 10% PMIN30% N 50 SRT 10% PMIN30% N 100 SRT 10% PMIN50% N 20 SRT 10% PMIN50% N 50 SRT 10% PMIN50% N 100 SRT 30% PMIN10% N 20 SRT 30% PMIN10% N 50 Z Test FET Mid P SRT 30% PMIN10% N 100 SRT 30% PMIN30% N 20 SRT 30% PMIN30% N 50 SRT 30% PMIN30% N 100 SRT 30% PMIN50% N 20 SRT 30% PMIN50% N 50 SRT 30% PMIN50% N 100 SRT 50% PMIN10% N 20 SRT 50% PMIN10% N 50 SRT 50% PMIN10% N 100 SRT 50% PMIN30% N 20 SRT 50% PMIN30% N 50 SRT 50% PMIN30% N 100 SRT 50% PMIN50% N 20 SRT 50% PMIN50% N 50 SRT 50% PMIN50% N 100 SRT 70% PMIN10% N 20 SRT 70% PMIN10% N 50 SRT 70% PMIN10% N 100 SRT 70% PMIN30% N 20 SRT 70% PMIN30% N 50 SRT 70% PMIN30% N 100 Selection Ratio (SRT), Minority Representation (PMIN), and Sample Size (N) 39 Summary The best AI test is one that balances the 3 concerns between: Being able to answer the.05 question Missing adverse impact when it exists, and Falsely concluding AI exists when it does not. The FET consistently undershoots the.05 level of significance: Drastically in smaller samples (n<50); substantially in samples The mid-p provides a correctly sized adjustment across various samples Type II error rates ( missing AI when it exists) differ substantially by test, especially in smaller samples where the FET is much less powerful All 3 common tests share similarly low Type I error rates, leaving the employer with very low odds of incorrectly concluding AI 40 20

21 Summary Using the FET unilaterally in all 3 conditions is unacceptable and should be discontinued in light of the recent findings just reviewed. The conditional FET may be appropriate in limited conditional settings, but there will always be an argument against such use: The FET is conservative regardless Does the situation analyzed truly meet conditional requirements? However, the mid-p has power advantages, adheres more closely to the.05 level, and is very closely aligned with the FET (where appropriate) 41 Summary Is either position the FET or mid-p aligned with a plaintiff or defense position? It depends on the question being asked If an employer is interested in knowing the exact p-value given in a clearly conditional situation where the margins were indeed fixed beforehand the FET will provide this answer. If an employer is interested in not missing adverse impact that may exist (i.e., wants strong power to detect AI), the mid-p will better answer the question, in both conditional and unconditional situations (i.e., all 3 models). For a test t to be useful, it should be reasonably accurate, reasonably powerful, versatile across wide situations The p-value from an AI test using a discrete distribution should be reasonably aligned with a P-value from a comparative continuous distribution

22 Summary The FET gives the actual conditional p-value, but will always go below the.05 nominal level, thus not answering the exact =<.05 or 2 SD question asked in Title VII situations. The Mid-p may be thought as assessing the strength of evidence against the null hypothesis (Barnard, 1989, p. 1474). This is not true regarding the exact p-value from the FET. The question being asked in Title VII situations is not necessarily what is the p-value ; but rather: Is the p- value less than.05? The mid-p answers this question more accurately Summary Advantages of Using the Mid-P (adapted from Hirji, 2006) Hirji jprovides the basis for endorsing the mid-p as the preferred exact method (for either conditional or unconditional situations): Statisticians who hold very divergent views on statistical inference have either recommended or given justification for the mid-p method. A mid-p version has been or can be devised for most of the statistics used in exact conditional and unconditional analysis of discrete data. The Confidence Intervals associated with Mid-ps are often preferred by statistical program because they are more narrow / accurate (e.g., StatXact). The shape and power function of the mid-p tests are generally close to the shape of the ideal power function an important distinction because it demonstrates that the power of the test is uniform, and able to detect AI when it exists across a variety of data sets both balanced and unbalanced)

23 Summary Advantages of Using the Mid-P (adapted from Hirji, 2006) In a wide variety of designs and models, the mid-p rectifies the extreme conservativeness of the traditional exact conditional method without substantially compromising the type I error. Empirical studies show that the performance of the mid-p method resembles that of the exact unconditional methods and the conditional randomized methods. With the exception of a few studies, most studies indicate that in comparison with a wide variety of exact and asymptotic methods, the mid- p methods are among the preferred, if not the preferred ones. The mid-p as good comparative small and large sample properties. Hirji concludes by stating: The mid-p method is thus a widely-accepted, conceptually sound, practical and among the better of the tools of data analysis. Especially for sparse and not that large a sample size discrete data, we thereby echo the words of Cohen and Yang (1994) that it is among the sensible tools for the applied statistician Summary EVALUATION FACTOR FET (conditional) FET (mid-p) FET-Boschloo (unconditional) Appropriate in Independent Trial? (Model 1, Fixed) Appropriate in Comparative Trial? (Model 2, Mixed) Appropriate in Double Dichotomy? (Model 3, Free) Average Distance from.05 Level in Small Samples Actual Significance Level Required in Small Samples Preserves.05 Nominal Sig. Level Average Power in Small Samples (n<50) MAYBE YES NO NO YES YES NO YES YES 41% 8% 5-10% YES NO NO 54% 62% 62%

24 How Do You Compute the Mid-P? It s rather simple Many Stat Packages will provide the mid-p If you already have an AI tool or stat program, just Compute the 2-tail FET Subtract ½ of the p-value from the first table from that value The hypergeomdist function can be used EXAMPLE If you want to avoid the hassle, just calculate mid-p values for FETs that are on the cusp of significance, such as 1.80 SDs (corresponding to p-values of about.07) Can easily be done for Mantel-Haenszel style analyses If the exact unconditional test is preferred: 47 Questions?