Assessment and the law These guidelines are intended to help you understand what your legal responsibilities are as a user of psychometric based assessments The most difficult thing in writing about the law is realizing that no sooner is the ink dry than the chapter must be rewritten. (Landy, 2005) The above reference was taken from a well-respected book on employment discrimination litigation in the USA and highlights the constantly changing landscape in this area. Add to that the variation across geographical territories and one can begin to understand why these guidelines cannot serve as a fully comprehensive guide to the law. We provide our clients with advice that enables them to operate in line with best practice across locations globally. However, we do not profess to provide legal advice or to replace the specific local advice that may be available from legally trained professionals operating within one s own locality. You must contact your organizations legal department for specific legal guidance. As a test developer, we uphold best practice standards in our approach and we provide our clients with a vast array of supporting documentation to support our assessment (all of which can be downloaded from the Talent Q Learning Centre). However, this is not sufficient to prevent legal claims from being made against those using our tests. Test users should be prepared to demonstrate that the process within which the test is used is fair and justified, and the onus lies upon the organization deploying the tests to ensure their practices are aligned with the legal requirements of the jurisdiction in which they operate. We firmly believe that if an organization knows their legal obligations and consistently follows them, test takers will feel they have been treated fairly, objectively and transparently. This leads to a positive test taker experience and creates a situation where they do not feel the need to question the use of the tests. Legal challenges Over the last 40 years there have been numerous cases brought against employers where an applicant felt that they had been unfairly discriminated against. Whilst a full review of all geographical territories is outside the confines of this document, a snapshot of activity is provided by the Equal Employment Opportunity Commission (EEOC) in the USA (Hartstein et al. 2012). In 2011, the EEOC received 99,947 discrimination charges the highest in its 46-year history, attributed to the economic situation in 2010-2011 and the fact that more and more employees have lost their jobs during this period. Of these, 261 lawsuits were deemed appropriate for filing 1. Whilst data is not easily available to inform whether these cases related specifically to psychometric testing, it is clear that when broken down by type almost 50% related to issues such as retaliation and harassment, leaving a further 50% being discrimination cases. Many of these discrimination cases are likely to be based on criteria outside of psychometric testing for example, some of the most noteworthy EEOC settlements in the same year relating to discrimination (outside of retaliation and harassment) included situations where employers had failed to make accommodation for employees with disabilities, failed to hire on the grounds of race and systematically discharged pregnant employees. Some, however, involve assessments. 1 They covered a variety of Acts including Title VII, Americans with Disabilities, Age Discrimination in Employment and Equal Pay. 1 Korn Ferry 2016. All right reserved.
A few examples of high profile cases from the UK and the US over the last 20 years are: International Brothers of Electrical Workers versus Mississippi Power (2006): The plaintiffs alleged that the employer had engaged in employment practices with a racially disparate impact. The complaint wasn t about the use of an ability test per se, rather the increase in the pass score that had been introduced. It was ultimately held that raising the pass score was justified as the employer showed that by doing so, the employees who passed demonstrated above average performance on the job. British Rail (1994): Eight guards brought an action against British Rail with the support of the Commission for Racial Equality (a public body in the UK which aimed to tackle racial discrimination and promote racial equality). The action alleged racial discrimination after they had failed the train driver assessment process. The process included psychometric aptitude tests (an ability test widely used in the transport industry at the time) and a personality questionnaire, followed by an interview. A review of test scores indicated that the tests, particularly those of verbal comprehension, had an adverse impact on ethnic minority applicants, which allegedly amounted to indirect discrimination. An out of court settlement was reached and British Rail agreed to review their assessment process. The Commission for Racial Equality had identified a similar problem with the same aptitude tests, which were also used as part of the assessment process used by London Underground for management appointments. Legal challenges may come from individuals or groups who claim that the processes and assessments being used are not legally valid. Employers should be prepared to prove that their assessment processes are defensible in a court of law. Whilst there is not one globally agreed legal standard that relates to testing, there are many commonalities between those that are in existence. Existing guidelines and principles Whilst we cannot give an exhaustive list across all geographical territories, examples of guidelines and standards that have been in place for a number of years include: Uniform guidelines on employment selection procedures. Equal Employment Opportunity Commission, U. S. Civil Service Commission, U. S. Department of Labor, & U. S. Department of Justice (1978). Standards for educational and psychological testing. American Educational Research Association, American Psychological Association & National Council on Measurement in Education (1999). International guidelines for test use. International Test Commission (2000). Guidelines for computer-based testing. Association of Test Publishers (2002). Guidelines for the development and use of computer-based assessments. British Psychological Society Psychological Testing Centre (2002). Principles for the validation and use of personnel selection procedures. Society for Industrial and Organizational Psychology (2003). International guidelines on computer-based and internet delivered testing. International Test Commission (2005). Using online assessment tools for recruitment. Psychological Testing Centre (2006). Psychological testing: A user s guide. Psychological Testing Centre (2007). Legal principles Despite the multitude of guidelines and principles outlined above, there are a number of common themes: Reliability and validity The International Test Commission (ITC) Guidelines on Test Use state that: Tests should be supported by evidence of reliability and validity for their intended purpose. Evidence should be provided to support the inferences that may be drawn from the scores on the test. All trained test users will have sufficient knowledge of where to seek such information on any test, but the key is that test users need to look for information on whether the test is a reliable measurement and whether it measures what it purports to measure. This information should be provided by test suppliers. In order for a selection test to be deemed relevant, the skill or attribute being tested must be critical or important to the job and the test should be a valid measure of the relevant skill or attribute. This is the key first point of defense in justifying the applicability of an assessment, because it will help identify people who will demonstrate superior performance in the role. Interpretation Whilst more and more tests are being delivered via technology it is still important that the test user understands how 2 Korn Ferry 2016. All right reserved.
raw scores are converted into standard scores and why. The supporting documentation for the tests should provide this information and should also provide the test user with guidelines on how to interpret the scores. Fairness and bias Common terms in test guidelines and standards are discrimination and adverse impact. The intention of all tests is to show up real differences between test takers, but to remain within legal guidelines they should do so fairly and in a manner which does not lead to unfair discrimination. Put simply, tests should not show differences where they do not exist. For many years test developers have been researching ways to evaluate whether a test score is biased against different groups of people. Suppliers of psychometric tests should include the methods they have used for evaluating potential bias and should provide evidence to show that there is no bias occurring. Direct and indirect discrimination Whilst test manuals should include information relating to evidence of bias within the test, the onus is still upon the test user to ensure that they do not use the test in a way which causes unfair discrimination. Unfair discrimination can be classed as direct and indirect. Direct discrimination is intentionally to discriminate unfairly purely on the basis of a protected characteristic 2. Whereas indirect discrimination consists of treatment which although apparently neutral cannot be justified, and in practice disproportionately disadvantages and is detrimental to one group more than another on any of the protected characteristics. If it is found that this occurs to a significant degree and one group is treated less favorably than another, then adverse impact may be said to have occurred. If an employer were to select women who score above 40 on an aptitude test, but only men who score 50 or more, then that would amount to direct discrimination. Whereas if an employer uses a particularly high cut-off score for all applicants but the nature of the test means that a considerably smaller proportion of women can pass, this could be indirect discrimination. This may be lawful if the employer can justify the requirement, for example, by demonstrating a correlation between scores on the test and job performance such as, by looking at whether higher test scores predict subsequently higher ratings on key performance measures in the role. The use of a firm cut-off should be justified and there are a number of ways to do so. A very strong way is to conduct a validation study to assess whether high test scores equate to superior job performance. Test publishers should also provide general evidence of validity that can be used to assess a test s suitability. Further advice on setting cut-offs can be found in the Dimensions and Elements Practitioner Guides (available to download from the Talent Q Learning Centre). Put simply, aptitude tests which require a benchmark score to be passed can sometimes be said to unfairly discriminate if inherently different passing rates are achieved by different groups and the test and passing score cannot be justified. Therefore, the first step is to look for tests that provide evidence that there are no or comparatively little group differences. Further, justification of why a benchmark score has been applied should be documented. Defining adverse impact In the US, Equal Employment Opportunity Commission guidance states that employment practices may be considered discriminatory and illegal if they have a disproportionate adverse impact on members of a minority group and the practices cannot be justified. The EEOC s Uniform guidelines on employment selection procedures define adverse impact as occurring when the selection rate for a lower scoring group is less than four-fifths (or eighty per cent) of the rate for the group with the highest rate. The guidelines state that if this occurs: [it] will generally be regarded by the Federal enforcement agencies as evidence of adverse impact, while a greater than four-fifths rate will generally not be regarded by Federal enforcement agencies as evidence of adverse impact. Smaller differences in selection rate may nevertheless constitute adverse impact, where they are significant in both statistical and 2 Protected characteristics vary from country to country for example, in the US the Civil Rights Act (1964) outlawed discrimination against anyone based on race, ethnic origin, nationality, religious affiliation or gender; this was closely followed by the Age Discrimination in Employment Act (1967) and then more recently the Americans with Disabilities Act (1990). The US Equal Employment Opportunity Commission has taken the position that the Title VII protection against sex discrimination extends to discrimination based on sexual orientation and sexual identity. Some states have also banned discrimination based on sexual orientation and some based on sexual identity. In the UK the Equality Act 2010 sets out the protected characteristics as being age, disability, gender reassignment, race, religion or belief, sex, sexual orientation, marriage and civil partnership, and finally pregnancy and maternity. 3 Korn Ferry 2016. All right reserved.
practical terms or where a user s actions have discouraged applicants disproportionately on grounds of race, sex, or ethnic group. Greater differences in selection rate may not constitute adverse impact where the differences are based on small numbers and are not statistically significant, or where special recruiting or other programs cause the pool of minority or female candidates to be atypical of the normal pool of applicants from that group. The general conclusion from the above is that whilst there are clear guidelines in the USA on what constitutes adverse impact there are of course exceptions. As a test user it is vital to consider the existing data available on a test, to fully understand the requirements of a job and where possible to conduct a validation study. A key determinant of unlawful adverse impact is whether or not the requirement that was applied can be justified. It is therefore important to document the rationale for the practices and processes you have in place. If the requirement cannot be justified the adverse impact is unlawful. If it is justified, the adverse impact may be due to a real difference between the groups capacity to do the job in question and hence justifiable on the basis of validity 3. A key point to note here is that factors such as variability in the quality of education that is received by different groups can lead to differences in performance on tests. It may be this issue rather than the test itself which is causing the observed differences in test performance, and similar effects may be seen across other assessment processes such as interviews and assessment centers. This diagram provides an overview of the possible steps leading to unfair discrimination: Requirement applied All groups have equal chance of meeting the requirement e.g. males and females score equally well on the test One group more able to comply than another E.g. males pass at a higher rate than females No adverse impact or discrimination Requirement justified? E.g. validation study shows high scorers perform better on the job Requirement unjustified? E.g. no validity evidence to show that the assessment is jobrelated or that high scorers perform any better on the job Evidence of adverse impact but not discrimination. The requirement is justified. Evidence of indirect discrimination. The requirement may not be justified 3 Under US law, in an age discrimination case, the user need not show validity or business necessity. A lower standard, of reasonableness provides a defence in disparate impact age discrimination cases. 4 Korn Ferry 2016. All right reserved.
How can we support you? The assessments: As previously mentioned, we pride ourselves on developing tests that are reliable, valid and fair. We document our approach and findings in both the Dimensions and Elements Psychometric Reviews, and we are undertaking an ongoing programme to ensure our products meet well respected external standards for example, the EFPA Review Model for the Description and Evaluation of Psychological Tests (Lindley et al. 2008). Advice: Our global team of experienced practitioners can support you locally. We have experts across the world that can help you to understand the implications of using tests within your own jurisdiction. Analysis: As has been discussed above, it is clear that to reduce the likelihood of a potential legal claim being made, our clients are encouraged to conduct upfront validation studies (where possible). Where this is not possible, we advise them to undertake other means of obtaining proof of validity and to continually monitor selection ratios for adverse impact. We have a team of experts who can support you in running a validation study, or can manage the entire study on your behalf. Furthermore, we can review data from your assessment processes to look for evidence of any bias. We have immediate access to all test data and we can extract and analyze it for you in line with data protection requirements. Where potential issues are found, we can work with you to address them. Consultancy: Our team of occupational psychologists can provide support for you across all aspects of the process surrounding test use. This can range from job analysis, through to designing an assessment process and defining decisionmaking criteria. We can also support you in the delivery of your assessments. Training: We can offer you training across many of the areas outlined above for example, job analysis and assessment design so that you are enabled to work proficiently and legally. Supporting materials: Everyone trained by Talent Q has access to all of the related online materials via the Talent Q Learning Centre. Also available to download are a series of more practical guidelines relating to fairness in testing. Expert witness testimony: In the unlikely event that one of our clients is required to defend the use of their selection process we can support you by providing the expertise of one of our experienced directors and consultants (many of whom were in involved in the development and deployment of our assessments). We will discuss your requirements with you and provide appropriate provision to suit the circumstances. Within the UK, for example, we work within the guidance set out by the British Psychological Society (and Psychological Testing Centre) in the Statement on the Conduct of Psychologists providing Expert Psychometric Evidence to Courts and Lawyers. References Hartstein, B.A., Coulter, J.G., Douglas, S.M., Gaur, J., McGovern, M.L., Mita, B.R., Murphy, T.F., Petesch, P.J. and Schuman, I.W. (2012) Annual report on EEOC Developments: Fiscal Year 2011. Landy, F.J. (2005) Employment Discrimination Litigation: Behavioral, Quantitative, and Legal Perspectives. San Francisco: Jossey-Bass. Lindley, P., Bartram, D. and Kennedy, N. (2008) EFPA Review Model for the Description and Evaluation of Psychological Tests. Psychological Testing Centre (2007) Statement on the Conduct of Psychologists providing Expert Psychometric Evidence to Courts and Lawyers. About Korn Ferry Korn Ferry is the preeminent global people and organizational advisory firm. We help leaders, organizations, and societies succeed by releasing the full power and potential of people. Our nearly 7,000 colleagues deliver services through our Executive Search, Hay Group and Futurestep divisions. Visit kornferry.com for more information. 5 Korn Ferry 2016. All right reserved.