HR Avatar Assessment Solution Technical Manual

Transcription

1 HR Avatar Assessment Solution Technical Manual August 31, Thaxton Lane Oakton, VA HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

2 Table of Contents Overview... 8 Testing Standards... 8 Solution Summary... 9 Cognitive Work Simulations Previous Research Content Development Attention to Detail Analytical Thinking Reliability Validity Fairness Attitudes, Interests and Motivations Assessment Previous Research Content Development Needs Structure Innovative and Creative Enjoys Problem-Solving Competitive Seeks Perfection Develops Relationships Expressive and Outgoing Corporate Citizenship Exhibits a Positive Work Attitude Adaptable Reliability Validity HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

3 Fairness Behavioral History Survey Previous Research Content Development Performance Tenure Unproductive Behavior Reliability Fairness Knowledge and Skills Tests Development Overview Sales Situation Analysis Test Development Reliability Fairness Typing Test Test Development Reliability Fairness Essay Test Test Development Descriptives Fairness Solution Scoring Construct Validity Evidence Technical Requirements Future Research Reliability Validity Fairness HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

4 Norms References Appendix A: Summary of the HR Avatar Solutions Appendix B: Historical Validity Evidence for the Cognitive Scales Appendix C: Historical Validity Evidence for AIMS Appendix D: Scoring Rubric for Essays Appendix E: Directions for Rating Essay Tests Appendix F: Automated Essay Scoring System Appendix G: Validation Studies Appendix H: Emotional Intelligence Test Appendix I: Work Competency Test HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

5 Tables Table 1. Demographic Characteristics of Sample Table 2. Participant Location Table 3. Descriptive Statistics and Reliability Evidence for the Cognitive Workplace Simulation Scales Table 4. Evaluation of Cognitive Score Differences by Gender Table 5. Evaluation of Cognitive Score Differences by Age Group Table 6. Descriptive Statistics and Reliability Evidence for the AIMS Scales Table 7. Evaluation of AIMs Score Differences by Gender Table 8. Evaluation of AIMs Score Differences by Ethnicity Table 9. Evaluation of AIMs Score Differences by Age Group Table 10. Evaluation of AIMs Score Differences by Race Groups: Asian and White Table 11. Evaluation of AIMs Score Differences by Race Groups: Black or African American and White.. 28 Table 12. Descriptive Statistics and Reliability Evidence for the Professional Behavioral History Survey Table 13. Descriptive Statistics and Reliability Evidence for the Entry-Level Behavioral History Survey Table 14. Evaluation of Biographical History Survey Score Differences by Gender Table 15. Evaluation of Biographical History Survey Score Differences by Ethnicity Table 16. Evaluation of Biographical History Survey Score Differences by Age Group Table 17. Evaluation of Biographical History Survey Score Differences by Race Groups: Asian and White 33 Table 18. Descriptive Statistics and Reliability Evidence for the Sales Situation Analysis Assessment Table 19. Evaluation of Sales Situation Analysis Score Differences Table 20. Descriptive Statistics and Reliability Estimates for the Typing Test Table 21. Evaluation of Essay Score Differences by Subgroup Table 22. Descriptive Statistics for Essay Scores Table 23. Evaluation of Essay Score Differences by Subgroup Table 24. Correlations between the AIMs Scales Table 25. Correlations between the Professional Behavioral History Scales Table 26. Correlations between the Entry-Level Behavioral History Scales Table 27. Correlations between HR Avatar AIMs and Behavioral History Scales Table 28. Correlations between the Two Cognitive Workplace Simulation Scales: Attention to Detail and Analytical Thinking Table 29. Correlations between the AIMs and the Cognitive Workplace Simulation Scales Table 30. Correlations between the Behavioral History Survey and the Cognitive Workplace Simulation Scales Table 31. Compiled Validity Evidence for the Original Content of the Cognitive Work Simulation Scales.. 72 HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

6 Table 32. Study 1 Results: Concurrent Validation Study for Insurance Consultants N= Table 33. Study 2 Results: Concurrent Validation Study for Inside Sales N= Table 34. Study 3 Results: Concurrent Validation Study for an Internet Services Order Processing N= Table 35. Study 4 Results: Concurrent Validation Study for an Internet and Cable Sales and Service Position N= Table 36. Study 5 Results: Concurrent Validation Study Auto Rental Sales Role N= Table 37. Study 6: Concurrent Validation Study for Paramedics N= Table 38. Summary of Relationships between Performance Measures and Original AIMS Scales HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

7 Figures Figure 1. Summary of HR Avatar Solution Figure 2. Cognitive Work Simulation Figure 3. Hypothesized Relationships between the AIMs Scales and the Big Five HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

8 Overview The HR Avatar Employment Assessment series was designed as a flexible assessment instrument for helping corporations obtain quality hires by measuring cognitive abilities, biographical data, personality characteristics and job knowledge related to performance and tenure in the workplace. To cater to differing individual client needs, HR Avatar offers both individual assessments as well as complete assessment solutions. By combining assessments and measuring multiple competencies, the assessment solutions allow for a broader evaluation of overall candidate fit with a particular job. This technical manual contains a summary of testing standards for the development of psychological assessments in the workplace and an overall summary of the development of the assessment solution. There are sections for each assessment within the solution including previous research, content development, and reliability evidence, validity evidence, and evidence for fairness. The technical manual concludes with a summary of HR Avatar s research agenda. Testing Standards The Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, National Council on Measurement in Education, 2014), the Principles for the Validation and Use of Personnel Selection Procedures, (Society for Industrial Organizational Psychology, 2003), the Uniform Guidelines on Employee Selection Procedures (Equal Employment Opportunity Commission, C. S. C. U. S. D. L. U., & Equal Employment Opportunity Commission, 1978), and Testing and Assessment: An Employer s Guide to Good Practices (Saad, Carter, Rothenberg, & Israelson, 2000) are all documents that govern the development and use of tests in employment settings. The developers of the HR Avatar Employment Assessment series have made, and continue to make, efforts to adhere to recommendations for test development put forth by these documents. The evidence produced in this technical manual is a summary of all of the theoretical and empirical research done on the tool to date. We anticipate frequent updates as we complete additional research. The following list summarizes the key test development standards from the above listed documents. The documents include additional standards governing test usage and we encourage test users to review and adhere to these standards. Define the psychological construct measured by the test Document the intended use of the test Document how test scores should be interpreted Demonstrate validity for the specific use of the assessment and interpretation of scores including: o Previous research on the construct o Previous empirical research that demonstrates the extent to which the validity evidence may be generalizable (e.g. meta-analytic studies) o Research supporting the link between the content of the assessment and the tasks, knowledge, skills, and abilities required for the job (i.e., content-related validity evidence) HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

9 o Research supporting a specific response process takes place if it is inferred in the use or interpretation of the test o Evidence that the internal structure of the assessment conforms to theoretical expectations o Relationships between scores and any outcomes the assessment claims to be able to predict, such as job performance (i.e. criterion-related validity evidence) o Evidence demonstrating that the test is statistically related to other measures in ways in which are consistent with theoretical expectations (i.e., convergent and divergent validity evidence) Document sufficient details related to validity research in order to allow the reader an opportunity to independently evaluate the quality of research Document reliability estimates and the Standard Error of Measurement (SEM) for each score, subscore, and combination of scores Use a type of reliability estimate (e.g., internal consistency, test-retest) that is appropriate for the test Document sufficient details related to reliability research in order to allow the reader an opportunity to independently evaluate the quality of research Design test in such a way that it allows all test takers an equal opportunity to demonstrate their standing on the construct regardless of subgroup membership (e.g., age, disability status, ethnicity, gender, age, or race) Provide empirical evidence demonstrating that the construct is measured the same way across subgroups Solution Summary The HR Avatar Employment Assessment series has been designed for use in identifying or screening out those job applicants with the lowest potential for success in a given role by assessing job-relevant cognitive abilities, personality characteristics, behavioral background, and knowledge and skills. There are currently 203 HR Avatar solutions (see Appendix A) in the HR Avatar Employment Assessment series. Each solution has been designed for specific job roles and contains several components: one of 15 different Cognitive Work Simulations, one of two different Attitudes, Interests, and Motivations assessment forms (AIMs), one of two different Behavioral History Surveys, and job-specific Knowledge and Skills assessments. The 15 Cognitive Work Simulations represent 15 distinct job families and are discussed in more detail below. The AIMs assessment and the Behavioral History Survey are both available in two forms. One form is designed for professional positions and the other is designed for entry-level positions. HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

10 Figure 1. Summary of HR Avatar Solution Cognitive Work Simulation Attitudes, Interests, Motivations Behavioral History Survey Knowledge and Skills tests Measures 1-2 specific cognitive abilities 1 of 15 job family-specific versions 10 personlity scales 2 versions (professional and entry-level) 2 biographical data scales 2 versions (professional and entry-level) 1 or more of 55 job-specific knowledges tests Sales Situation Analysis Typing Test Essay Test The assessments are delivered via a computer or other mobile device via the internet. Applicants are sent an with a link to take the assessment. Once the applicant clicks on the link in the , they are brought to a page in their web browser. The assessment begins with an animated host. The host guides the applicant through the process and provides instructions prior to the start of each assessment. The animated host is used to provide the applicant with a more engaging and positive assessment experience. Most solutions take about 40 minutes to complete. Prior to implementing the solution for selecting employees, HR Avatar recommends completing a job analysis to ensure that the competencies measured by the HR Avatar assessments are important for success in your organization. Additionally, HR Avatar recommends that the organization conduct a pilot study to examine the relationships between the assessments and relevant job performance criteria and to evaluate the potential for adverse impact. This pilot study should be done prior to using the assessment for screening out employees. Cognitive Work Simulations Previous Research Those individuals with higher cognitive abilities are more equipped to solve complex problems and learn new skills. Cognitive ability is necessary for successful performance on a broad range of tasks including written documentation, oral communication, identifying and managing details, solving problems, reading HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

11 and responding to messages, quickly learning and applying new information, quantitative computations and analyses, and making well-reasoned decisions The published research evidence consistently demonstrates a strong, predictive relationship between cognitive ability, or general mental ability, and job performance and training success. In fact, several metaanalyses have been completed, summarizing and combing the results of many research studies examining the validity of cognitive ability (Hunter & Hunter, 14; Schmidt & Hunter, 19) and the results generalize across geographical regions (Bertua, Anderson, & Salgado, 2005; Salgado & Anderson, 2003; Salgado, Anderson, Moscoso, Bertua, & De Fruyt, 2003a; Salgado et al., 2003b). The operational validity estimates from these meta-analytic studies tend to be in the.5-.6 range. Though cognitive ability predicts performance at all job levels, the relationship is moderated by complexity such that the relationship is stronger for more complex jobs (Bertua et al., 2005; Salgado et al., 2003b). Currently, the most commonly accepted taxonomy for cognitive abilities is the CHC model (Cattell, 1941; Horn, 15; Carroll, 1993). The emergence of this accepted hierarchical taxonomy came from the merging of two independent lines of research examining the taxonomy and structure of cognitive abilities (McGrew, 2008). The taxonomy posits general cognitive ability at Stratum I, broad abilities at Stratum II, and more specific abilities at Stratum III. For the prediction of job performance, we are most interested in measuring abilities at the second stratum as these are deemed to be broad enough to apply to multiple positions, but specific enough to yield incremental prediction in job performance. Furthermore, recent research has demonstrated that using specific abilities in selection decisions can reduce adverse impact without decreasing the validity of the selection process (Kehoe, 2002; Wee, Newman, & Joseph, 2014). The broad abilities listed in Stratum II consist of: fluid reasoning, comprehension-knowledge, short-term memory, visual processing, auditory processing, long-term storage and retrieval, cognitive processing speed, decision and reaction speed, reading and writing, quantitative knowledge, general (domain-specific) knowledge, tactile abilities, kinesthetic abilities, olfactory abilities, psychomotor abilities, and psychomotor speed (McGrew, 2008). Based on our understanding of these abilities, we believe that fluid reasoning, shortterm memory, reading and writing, quantitative knowledge, and general (domain-specific) knowledge would be the most generalizable across jobs. However, the general (domain-specific) knowledge was determined to be better measured by the knowledge and skills assessments. HR Avatar designed two scales for each job family: Attention to Detail and Analytical Thinking. The development of these scales is discussed further on the content development section, but, to summarize, Attention to Detail taps short-term memory, whereas Analytical Thinking is more closely related to fluid reasoning and quantitative knowledge. Both scales require that applicants read information, therefore, to some extent, the reading component of reading and writing is also measured by both scales. HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

12 Current meta-analytic research has examined the relationship of these specific abilities with overall job performance and training performance. In a European sample, Salgado et al. (2003b) found that perceptual ability and memory (similar to the competency measured by the Attention to Detail scale) had corrected mean validity coefficients of.52 and.56 with job performance and.25 and.34 with training performance, respectively. Additioanlly, they found that verbal ability and numerical ability (similar to the competency measured by the Analytical Thinking scale) had corrected mean validity coefficients of.35 and.52 with job performance and.44 and.48 with training perofrmance, respectively. In a UK sample, Bertua et al. (2005) found that perceptual abilities had a corrected mean validity coefficient of.50 with job performance and.50 with training performance. Verbal ability and numerical ability had corrected mean validity coefficients of.39 and.42 with job performance and.49 and.54 with training perofrmance, respectively. Content Development The Cognitive Work Simulations measure two cognitive competencies: Attention to Detail and Analytical Thinking. Attention to Detail most represents short-term memory from the CHC model and Analytical Thinking is a combination of fluid reasoning and quantitative reasoning. Attention to Detail The competency is defined by the ability to process information, recall information accurately, and identify appropriate resources for locating specific pieces of information. This competency is important for data entry, identifying errors in data, processing orders, working with tables, understanding policies and procedures, and working with numerical data such as the type found in financial reports. Analytical Thinking The competency is defined by the ability to understand language and the numerical relationships. This competency is characterized by the ability to process complex information, synthesize data, identify and solve problems, and make well-reasoned decisions. The Cognitive Work Simulation requires that candidates complete various job-related tasks in a virtual environment designed to replicate the workplace. There are 15 versions of the Cognitive Work Simulation and each is designed to represent scenarios common to a specific job family (listed in Table 3). Candidates are evaluated on how they respond to avatars representing customers and colleagues and how well they solve problems in various work-related scenarios. The realistic experience is designed to be more engaging for applicants and to provide a better measure of ability. During a simulation, the candidate might be required to read , listen to voic and perform basic keyboard and screen navigation tasks to solve typical business problems. HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

13 Figure 2. Cognitive Work Simulation The items used in the Cognitive Work Simulation were based on an existing library of ability items with a history of predicting job performance in various workplace settings. Both the original items and the new content were developed by examining the results of multiple job analyses. There are between 5 and 19 items in each competency measured by the simulation. Responses to assessment items are scored dichotomously. In some items, there is more than one correct response. In these situations, an applicant is awarded a point for each correct response selected. It takes approximately minutes to complete a simulation. The estimates were calculated using a sample of 264 individuals. Reliability To evaluate the properties of the cognitive assessments as well as other assessments in the solution, HR Avatar conducted a research study. Participants from the study were recruited via Mechanical Turk (MTurk). MTurk is a platform operated by Amazon.com and allows the user to pay workers for completing for individual tasks called Human Intelligence Tasks (HITs). MTurk is becoming increasingly popular for use in research in the social sciences including research related to pre-employment testing. Many researchers have found that MTurk provided adequate samples for the purposes of developing assessments (Buhrmester, Kwang, & Gosling; Johnson & Borden, 2012; Minton, Gurel-Atay, Kahle, & Ring, 2013; Miller, Gentile, Wilson, & Campbell, 2013). HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

14 HR Avatar recruited workers on MTurk to complete one of 15 versions of the solution one for each version of the Cognitive Work Simulation. The Cognitive Work Simulation was combined with an appropriate AIMs assessment form, and an appropriate Behavioral History Survey Form. The Knowledge and Skills assessments were not included in this study. Participants were allowed to take each version once. Instructions to participants warned them that if they responded to the questions randomly, their HIT would not be approved, and therefore they would not get paid. Workers were promised an assessment report upon completion of the assessments. Both of these approaches should have led to more genuine responses. Additionally, to ensure responses to the assessment were legitimate, the data used in the analyses were restricted to those individuals who, on average, took 2.5 seconds or more to respond to items (N=675). These individuals completed 1682 solutions. On average, participants took 2.49 solutions (SD=2.70). A summary of the demographic information for this sample can be found in Table 1 along with 2013 estimates (the most recent available) of the demographic breakdown of the U.S. population as reported by the U.S. Census Bureau (2015). As compared to the U.S. population, the sample in this research contains a higher proportion of individuals who are male, non-hispanic and White. The average age of the sample is years (SD=10.88) which is also slightly younger than the median age (37.5) of the U.S. population (U.S. Census Bureau, 2015). Given the differences between the sample and the overall population, the results in this study are to be regarded as preliminary an interpreted with caution. Table 2 provides a summary of the geographic areas represented by the sample. Table 3 contains the descriptive statistics for each scale and the alpha estimates of internal consistency. Please note, based on these findings, further analyses were done at the item and distractor level. These analyses led the developers to make substantial edits to these items and we anticipate that edits will substantially improve the reliability of the tools. Further research evaluating the effectiveness of these changes is forthcoming. HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

15 Table 1. Demographic Characteristics of Sample Gender N %* U.S. Only 2013 est. from U.S. Census** Female Male Unknown 8 Ethnicity N %* U.S. Only 2013 est. from U.S. Census** Hispanic Not Hispanic Unknown 55 Race N %* U.S. Only 2013 est. from U.S. Census* American Indian or Alaskan Native Asian Black or African American Native Hawaiian or Other Pacific Islander White Two or More Races Unknown 14 *Percentages calculated by summing those for whom demographic data were available. **U.S. Census Bureau, 2013 American Community Survey Table 2. Participant Location Country N % United States India UK Other HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

16 Table 3. Descriptive Statistics and Reliability Evidence for the Cognitive Workplace Simulation Scales Analytical Thinking Attention to Detail Cognitive Simulation # of # of N M SD Alpha N M SD Alpha Items Items Basic Entry-Level Face-to-Face Customer Service Administration Business Sales Business/Finance Entry-Level Administration N/A N/A N/A N/A N/A Entry-Level Business & Finance Entry-Level Office First-Line Supervisor General Office Workplace Information Technology Manager Remote Customer Service Technician Retail Sales Validity As mentioned above, the simulations were developed based on an existing library of content. The original developer has provided the results of historical validation studies done with the original content and these are provided in Appendix B. Validation evidence is available in two studies that demonstrate the validity of the assessments. The first is a study with 130 managers and the second is a study with 64 managers. In the first study, the assessment demonstrated that it predicts job performance (r=.25; p<.01), using a simple High or Marginal categorization of job performance, and a second study demonstrated the assessment predicted performance on a 100 point administrative performance appraisal, using a Spearman correlation (r=.25; p<.05). We expect validity evidence to be more robust in future studies with better criterion measures. A brief summary of the results is presented in Appendix G. HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

17 Fairness In order to evaluate how different demographic subgroups perform on the assessments, independent samples t-tests were performed to compare mean scores of the subgroups. The results are provided in Table 4 and Table 5. Included in the tables are the descriptive statistics for each subgroup, the t-statistic (used for evaluating whether or not the mean differences are statistically significant), and Cohen s d. Cohen s d is an effect size and an indication of how large the mean differences are. For reference, according to Cohen (1992), effect sizes of.2 are small,.5 are medium, and.8 are large. Although data were collected on ethnic group, the sample sizes for the Hispanic or Latino subgroup were too small for analyses (n<15). Data were also collected for racial group. However, the sample sizes for all but the White subgroup were too small for analysis (n<30). Future research will examine subgroup differences as more data become available. For gender, there were statistically significant differences in favor of males on three scales (Entry-Level Office: Attention to Detail, First-Line Supervisor: Analytical Thinking, and Information Technology: Analytical Thinking). However, there does not appear to be a strong trend in males scoring higher than females and it is possible that edits made to the items will diminish differences on the scales where differences have been detected. For age, there were statistically significant differences on three scales (Business Sales: Analytical Thinking, First-Line Supervisor: Attention to Detail, and Manager: Analytical Thinking), however these were all in favor of the protected group. Score differences will be further evaluated as more data become available on the new version of the assessments and subgroup differences will be evaluated for ethnic and race subgroups as well. HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

18 Table 4. Evaluation of Cognitive Score Differences by Gender Cognitive Simulation Basic Entry-Level Face-to-Face Customer Service Administration Business Sales Business/Finance Entry-Level Administration Entry-Level Business/Finance Entry-Level Office First-Line Supervisor General Office Workplace Information Technology Manager Remote Customer Service Technician Retail Sales Note. *p<.05 Scale Female Male n M SD n M SD t d Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail *.39 Analytical Thinking *.42 Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking *.46 Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

19 Table 5. Evaluation of Cognitive Score Differences by Age Group Cognitive Simulation Basic Entry-Level Face-to-Face Customer Service Administration Business Sales Business/Finance Entry-Level Administration Entry-Level Business & Finance Entry-Level Office First-Line Supervisor General Office Workplace Information Technology Manager Remote Customer Service Technician Retail Sales Note. *p<.05 Scale 40 or Older Less than 40 n M SD n M SD t d Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking * -.86 Attention to Detail Analytical Thinking Attention to Detail Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail * -.58 Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking * -.66 Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail Analytical Thinking Attention to Detail Attitudes, Interests and Motivations Assessment Previous Research The Big Five framework for understanding personality is the most widely accepted taxonomy of personality traits (Goldberg, 1992; 1993; McCrae & Costa, 1997; McCrae & John, 1992). The five factors, or traits, are Agreeableness, Conscientiousness, Emotional Stability, Extraversion, and Openness. Agreeableness is characterized as warmth, ability to get along with others, amd friendliness. Individuals who are HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

20 conscientiousness are characterized as being reliable, responsible, dependable, thorough, dutiful, achievement-oriented and competent. Individuals considered Emotionally Stable are more resilient, stable, and centered. Extraverted individuals are characterized as being sociable, outgoing, and high in energy. Openness describes indiciduals who are creativc, flexible, curious, and unconventional (Judge & Ilies, 2002) A substantial amount of research, including meta-analytic research, has demonstrated the relationship between personality and job performance (Barrick & Mount, 1991; Bartram, 2005; Hogan & Holland, 2003; Hurtz & Donovan, 2000; Mount & Barrick, 1995; Salgado., 1997; Tett, Jackson, & Rothstein, 1991). In a meta-analysis of 25 years of personality and performance research drawn from 117 studies of managers, professionals, sales people, skilled, and semi-skilled workers; researchers concluded that extroversion predicted success in management and sales; openness to experience predicted training ability; and, conscientiousness correlated with success in sales, management, professional, skilled and semi-skilled positions (Barrick & Mount, 1991). The findings were similar in the European community (Salgado, 1997). Personality traits are stable and slow to change (McCrae, Jang, Livesley, Riemann, & Angleitner, 2001). The stability of personality traits and their relationship to job performance makes them extremely important to measure before making a hiring decision. Theory on personality in the workplace has advanced and researchers have demonstrated that validity coefficients for personality assessments are stronger for those job performance criteria that can be theoretically linked to a given personality trait. For example, Borman and Motowidlo (1993, 1997) found support for their hypotheses that personality would have a stronger relationship with contextual rather than task performance. The researchers defined contextual performance as those areas of performance that are important to an organization but not explicity part of the job duties of an individual (e.g., following organizational rules, giving additional effort when needed, providing support to other organizational memebers, etc.). Hogan and Holland (2003) found that the Hogan Personality Inventory, a personality assessment based on the Big Five, was substantially related to multiple measures of job performance across a wide variety of positions. The relationships were stronger for personality traits that were theoretically linked to the job performance measure. Bartram (2005) had similar findings, supporting the notion that theoretically linked personality constructs and job performance domains had stronger relationships than personality traits with overall performance. The trait-activation model proposed by Tett and Burnett (2003) provides a theoretical model allowing for more detailed predictions regarding when a specific personality trait might be linked to job performance. Another approach to improving prediction with personality assessment is to measure narrower traits that are specific facets of the Big Five (Paunonen, Rothstein, & Jackson, 1999; Schneider, Hough, & Dunnette, 1996). Researchers who argue for this approach posit that some facets of a trait may relate to a particular job performance dimension, but that others may not - or may even be negatively related. Measuring at the broad HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

21 level may dilute the relationship between the narrower facet and the job performance dimension. For example, achievement and reliability are both narrow facets of conscientiousness. One might expect a strong positive relationship between reliability and retention, but there may not be a relationship between achievement and retention. Using an assessment that measures the broad trait of conscientiousness may under-predict retention. Researchers conducted a meta-analysis and demonstrated the superiority of using narrow facets to predict specific job performance dimensions (Woo, Chernyshenko, Stark, & Conz, 2014). Furthermore, many personality assessments used for pre-employment selection are actually compound traits or constellations of narrow facets (Schneider et. al., 1996). These constellations may come from one or more of the broader personality traits. In fact, instruments used in both the Hogan and Holland (2003) and Bartram (2005) meta-analyses contain measures of compound traits. Content Development A review of the literature on personality in the workplace allowed us to identify several competencies, or compound traits, that are related to successful job performance for many jobs. The ten scales, their definitions, and example items are listed below. Please note, example items are similar to, but do not reflect, actual items from the assessment. Needs Structure Often, following rules and procedures is critical to successful job performance. When employees do not follow established organizational policies, work processes may be performed incorrectly, work products may be flawed, customer service levels may suffer, the organization may become a victim to fraudulent wage and expense billings, and in some cases the organization may be liable for the actions of the employee. The Needs Structure scale was designed to evaluate an applicants tendency to adhere to organization rules and procedures and should be important in jobs where rule-abiding behaviors are related to job performance. Example: I prefer to make extensive plans before I start any project. Innovative and Creative For many positions, Innovation and Creativity is critical to successful job performance. Individuals with high levels of this competency should be more likely to generate new products, develop improved methods for producing work, and develop solutions to problems. Example: Some of my suggestions are really eccentric. HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

22 Enjoys Problem-Solving Enjoys Problem-Solving is a competency designed to predict which applicants will be more successful in roles dealing with data, completing analyses, and conducting research. Individuals who enjoy problem solving should be more successful in roles that require analytical thinking. Example: I enjoy learning about how things work. Competitive The Competitive competency evaluates the extent to which in individual is likely to do what is necessary to accomplish their goals which may be of concern if the individual s goals are not aligned with the organization s goals. Individuals who score highly on this competency are characterized as having concern for outcomes rather than the feelings of coworkers. Example: I am not above using people to get my way. Competitive v2 The Competitive v2 competency evaluates the degree to which an individual is driven by a desire to achieve objectives and outperform their peers. Competitiveness is the tendency to evaluate one s performance in comparison to others; a desire to do better than others, an enjoyment of the situations that can lead to a clear winner; thrives in an environment where people are differentiated by accomplishments. Unless otherwise noted, the results in this technical manual refer to Competitive scale from the original AIMS test development. The Competitive v2 scale began being used in July 2016, and data is being collected. The decision was made to replace the original Competitive scale due to customer complaints and upon psychometric review of its continuing usefulness, based on research conducted through June, Example: I get satisfaction from beating out the competition. Seeks Perfection When the quality of work (as opposed to pace) is important for success in a role, the Seeks Perfection competency should predict performance. Those who score high on this competency are more likely to double-check their work and have higher rates of accuracy. Additionally, individuals high on this competency are more likely to be detail-oriented. Example: My work tends to be faultless. HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

23 Develops Relationships For many roles, working in teams and with others is critical for successful performance. The Develops Relationships scale was designed to predict which individuals are more likely to develop productive working relationships and be successful in a team situation. Example: I have never deliberately said anything that hurt someone's feelings. Expressive and Outgoing Individuals who are Expressive and Outgoing are more likely to engage with others. Being Expressive and Outgoing is also characterized by the ability to influence and persuade others. This competency is important for success in roles where influence is necessary such as leadership and sales positions. Example: I tend to take control in most work situations. Corporate Citizenship The Corporate Citizenship competency was designed to measure applicants tendency to conduct themselves ethically and responsibly in an organization. Individuals who score highly on this competency are likely to be honest with managers and colleagues and avoid taking advantage of people and/or situations just because it suits their self-interest. Example: I would never use manipulation as a tactic to advance my goals. Exhibits a Positive Work Attitude This competency evaluates the tendency for an individual to feel positively toward their job and the organization they work for. Individuals who score highly on this competency are characterized as feeling satisfied with their work and are more likely to put in extra effort when needed. Example: I volunteer for additional work. Adaptable The Adaptable competency measures an individual s tendency to adjust to changes in their work environment. Individuals who score highly on this competency thrive in fast-paced settings and respond well to variety. Example: I think organizational changes are fun and exciting. HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

24 The figure below outlines the hypothesized relationships between the AIMs scales and the Big Five traits. As mentioned above, the AIMs scales are compound traits and measure competencies related to multiple narrow facets of personality and these facets may be under one or more of the five broader personality traits. Therefore, an AIMs scale may be expected to relate, albeit moderately, to multiple Big Five traits. Needs Structure Innovative and Creative Agreeableness Enjoys Problem-Solving Competitive Emotional Stability Seeks Perfection Conscientiousnes Develops Relationships Expressive and Outgoing Openness Corporate Citizenship Exhibits a Positive Work Attitude Extraversion Adaptable Figure 3. Hypothesized Relationships between the AIMs Scales and the Big Five The original assessment consisted of 100, Likert-type items. Each item consists of a single statement and candidates are asked to indicate their level of agreement with the statement. Preliminary data analyses allowed us to shorten the scales such that the assessment contains a total of 65 items. The AIMs assessment is available in two forms. The professional form measures all ten competencies. The entry-level form measures eight competencies as it was determined that Expressive and Outgoing and Innovative and Creative would be less critical competencies for these roles. The time to take the professional form is HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

25 estimated to be between 6 and 7 minutes, and the time to take the entry-level form is estimated to be 5 and 6 minutes. The estimates were calculated using a sample of 402 individuals. Reliability The descriptive statistics and reliability estimates of the AIMs scales were estimated using the data collected via MTurk as described in the previous section. As can be seen from the table below, the AIMs scales have acceptable levels of internal consistency. Table 6. Descriptive Statistics and Reliability Evidence for the AIMS Scales Competency N M SD Alpha Needs Structure Innovative & Creative Enjoys Problem-Solving Competitive Seeks Perfection Develops Relationships Expressive & Outgoing Corporate Citizenship Exhibits a Positive Work Attitude Adaptable Competitive v2* *Competitive v2 replaced Competitive in July 2016 Validity A Confirmatory Factor Analyses (CFA) was run using R (version 3.1.2) and the sem package to evaluate whether or not the internal structure of the assessment conformed to the hypothesized structure. In other words, we assessed the extent to which the items loaded onto the particular scale they were assigned to. The Root Mean Square Error (RMSEA), an indicator of model fit, indicates that the model was satisfactory, RMSEA=.0675, χ2 (1970, N=567) = , p<.001. This analysis provides validation support for the internal structure of the assessment. As mentioned previously, the content for the AIMs scales were modified and adapted from a longer assessment that had been used in previous validation research. The developer of the original content provided the results of several criterion-related validity studies and tables with these results can be found in Appendix C. HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

26 Validation evidence is available in two studies that demonstrate the validity of the assessments. The first is a study with 130 managers and the second is a study with 64 managers. In the first study, the assessment demonstrated that it predicts job performance (r=.25; p<.01), using a simple High or Marginal categorization of job performance, and a second study demonstrated the assessment predicted performance on a 100 point administrative performance appraisal, using a Spearman correlation (r=.25; p<.05). We expect validity evidence to be more robust in future studies with better criterion measures. A brief summary of the results is presented in Appendix G. Fairness In order to evaluate how different demographic subgroups perform on the assessments, independent samples t-tests were performed to compare mean scores of the subgroups (Tables 7-11). There were too few cases (n<22) to run analyses for the American Indian or Alaska Native, Native Hawaiian or Other Pacific Islander, Two or More Races, and Other subgroups. There were significant mean differences on the AIMs scales between males and females. However, all differences were relatively small or they were in favor of the focal subgroup, females. There were no significant mean differences based on ethnicity. There were a few significant mean differences with respect to age group, but all but one of these were in favor of the focal subgroup, 40 and older. There were a few significant mean differences between the Asian and White subgroups. However, individuals in this sample were from multiple countries and it s possible that these differences might be reflections of cultural differences. Additional research is needed to further evaluate these differences, to calculate country-specific norms, and to evaluate subgroup differences within country. There were no significant differences between the Black or African American subgroup and the White subgroup. Table 7. Evaluation of AIMs Score Differences by Gender Female Male Scale n M SD n M SD t d Needs Structure * Innovative and Creative Enjoys Problem Solving * 0.36 Competitive * 0.33 Seeks Perfection * Develops Relationships Expressive and Outgoing * 0.28 Corporate Citizenship * Exhibits a Positive Work Attitude * Adaptable * Note. *p<.05 HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,

27 Table 8. Evaluation of AIMs Score Differences by Ethnicity Hispanic or Latino Not Hispanic or Latino Scale n M SD n M SD t d Needs Structure Innovative and Creative Enjoys Problem Solving Competitive Seeks Perfection Develops Relationships Expressive and Outgoing Corporate Citizenship Exhibits a Positive Work Attitude Adaptable Note. *p<.05 Table 9. Evaluation of AIMs Score Differences by Age Group 40 and Older Less than 40 Scale n M SD n M SD t d Needs Structure * Innovative and Creative Enjoys Problem Solving Competitive * 0.62 Seeks Perfection Develops Relationships Expressive and Outgoing * 0.46 Corporate Citizenship * Exhibits a Positive Work Attitude * Adaptable * Note. *p<.05 HR AVATAR ASSESSMENT SOLUTION TECHNICAL MANUAL - AUGUST 31,