CASE STUDY. Incremental Validity of the Wonderlic Motivation Potential Assessment (MPA)

Size: px

Start display at page:

Download "CASE STUDY. Incremental Validity of the Wonderlic Motivation Potential Assessment (MPA)"

Lindsey Watkins
6 years ago
Views:

1 Michael C. Callans, M.S.W. Daniel Nguyen, Ph.D. Brett M. Wells, Ph.D. Introduction The Wonderlic Motivation Potential Assessment (MPA) is a 30-item questionnaire that measures the extent to which employees will make a significant, positive contribution to the success of their workplace. In essence, motivated employees give 100% of their effort 100% of the time. Unfortunately, less than 30% of U.S. workers are fully motivated, engaged, and productive in their jobs (Gallup, 2013). To this end, the MPA represents Wonderlic s commitment to improve your organization s ability to select, develop, retain, and promote highly motivated individuals, which profoundly has an impact on your organization s bottom line. In this case study, we present evidence for the MPA s predictive and incremental validity in job performance, beyond that of cognitive ability. The validity of a test is often described as an index of how well the test measures what the publisher says it measures that is, a valid test does what it is supposed to. There are different types of validity evidence. Commonly used with ability tests, predictive validity measures the relationship between the test score and should measure of performance, such as supervisor s ratings of job performance. A perfect relationship, where each increase in the test score corresponds with a proportionate increase in performance, would result in a validity coefficient of r = The lack of any relationship would have a coefficient of r = No test is able to measure all of the characteristics necessary for job performance; therefore, validity coefficients are always less than What is a good validity coefficient? According to the U.S. Department of Labor (2000), validity coefficients of r =.21 to r =.35 are typical for a single test and are likely to be useful (p. 3-10). In general, higher validity coefficients provide more value and utility to organizations; however, validity coefficients rarely exceed r =.40. Closely related to predictive validity, incremental validity is the amount of predictive validity one assessment contributes above and beyond that of another. In other words, incremental validity seeks to answer if a new assessment provides additional value after accounting for what is captured by existing methods. Within incremental validity studies, general cognitive ability tests are often used as the baseline comparison because they are regarded as the most predictive of job performance across a variety of jobs and they are among the least expensive to administer (U.S. Office of Personnel Management, 2007). To this end, the MPA s validity was compared to that of the Wonderlic Cognitive Ability Pretest (WPT-Q). The WPT-Q measures how well a job candidate will be able to understand instructions, learn, adapt, solve problems, and handle the mental demands of the position.

2 Case Study Usable data were obtained from 841 employees at a Fortune 100 company. Specifically, these employees completed both the MPA and WPT-Q. Of the 841 employees, 828 also completed a 12-item Employee Engagement Questionnaire (e.g., All in all, I m satisfied with my job; I often think about quitting; I am tightly connected to this organization) using a 5-point agreement scale. Ratings across the 12 items were averaged to create an Employee Engagement (EE) score, which demonstrated strong internal consistency reliability, Cronbach alpha (α) =.91. Lastly, the direct supervisors for 176 employees provided job performance ratings across six domains (quality of work, efficiency, professional standards, ability to perform core job tasks, judgment when performing core job tasks, and job knowledge with reference to core job tasks) using the following rating scale: 1 = Below Average 2 = Somewhat Below Average 3 = About Average 4 = Somewhat Above Average 5 = Above Average Ratings across the six domains were averaged to create an Employee Performance (EP) score, which demonstrated strong internal consistency reliability, Cronbach alpha (α) =.93. Among all of those tested, 13 employees (1.5%) did not provide information concerning their demographics. Tables 1 through 5 summarize the demographic characteristics of those who provided this information. Employees had an average age of 40.1 years (SD = 9.2) and an average tenure at the organization of 9.7 years (SD = 8.4). Table 1 Gender of Case Study Employees Male Female Total

3 Table 2 Work Location of Case Study Employees Argentina Australia Belgium Brazil Canada Chile China Czech Republic France Germany India Indonesia Italy Japan Korea, Republic Of Mexico Netherlands Panama Peru Philippines Poland Portugal Russian Federation Singapore Spain Switzerland Thailand United Arab Emirates United Kingdom United States Total

4 Table 3 Work Function of Case Study Employees Accounting Engineering Finance and Lending Human Resources Information Technology Legal Marketing/Sales Material Movement/Logistics Operations/Manufacturing Other Functional Areas Purchasing Strategic Planning Total Table 4 Level of Education of Case Study Employees High School graduate Some College Associate Degree College Bachelors Degree Some Graduate Work Graduate or Professional Degree Professional Doctorate (JD, MD, etc.) Research Doctorate Total

5 Table 5 Ethnicity of Case Study Employees within the United States African American Indian Asian Hispanic White Two or More Races Unknown Total What is the Relationship between Cognitive Ability and Motivation? Results from a correlation analysis indicated a non-significant relationship between cognitive ability scores and motivation potential scores, r = -.059, p >.05 (see Table 6). This non-significant relationship provides evidence of the MPA s divergent validity. That is, motivation appears to be substantially discriminable from the most widely studied and used measure of general cognitive ability. The two share no relationship and are considered different from one another. Table 6 Descriptive Statistics of and Correlations among Cognitive Ability, Motivation Potential, Employee Engagement, and Employee Performance Scores Assessment M SD WPT-Q MP EE EP Cognitive Ability (WPT-Q) Motivation Potential (MP) Employee Engagement (EE) **.488** Employee Performance (EP) **.211**.189* - Note. *Correlation significant at p <.05, two-tailed; **Correlation significant at p <.01, two-tailed. For each assessment, higher scores indicate greater levels of the underlying trait (i.e., cognitive ability, motivation, engagement, job performance). The numbers above the diagonal are the sample sizes from which the correlations were computed. For example, 176 employees completed the Motivation Potential Assessment and had Employee Performance ratings, and their correlation was.211.

6 What Predicts Employee Engagement? Motivation Potential significantly predicted actual, Employee Engagement, r =.488, p <.01 (see Table 6). The strong correspondence between Motivation Potential scores and levels of employee engagement provides evidence of the Motivation Potential Assessment s criterion validity. That is, the MPA is not designed to measure engagement directly, but rather identify those potential applicants who would be more likely to become engaged because of their workplace involvement. This result provides evidence for the strong relationship between potential and actual experiences. Cognitive Ability also significantly predicted actual, Employee Engagement, r = -.227, p <.01 (see Table 6), but to a far smaller degree than Motivation Potential. This result indicates that those who score higher on cognitive ability tend to be less engaged. Although we expected to find a positive relationship between cognitive ability and employee engagement, the observed negative correlation has some empirical support. For example, Maltarich, Nyberg, and Reilly (2010) found that in jobs with low cognitive demands, higher cognitive ability employees appear less satisfied. To this end, the current organization may be simply hiring the highest scoring employees, which, at times, can be self-defeating. Higher scores on cognitive ability tests indicate the ability to learn more quickly and effectively with less organized training and become more quickly proficient on the job. Employers are naturally eager to minimize training expense and seek new employees who will become immediately productive on the job. This translates into hiring the highest scoring applicants who meet other job requirements. A conflict arises when, after a brief period of intense mental activity learning the job, the new employee finds the job itself not challenging a bore and is disengaged or quits. The reduced cost of training is replaced with the high cost of losing, and having to replace, productive employees. Importantly, each employer must seek a resolution of this training versus turnover expense that suits the job, applicant availability, and the resources of the organization. What Predicts Employee Performance? The Wonderlic cognitive ability test family has been the subject of hundreds of predictive validity studies conducted by companies, schools, non-profit organizations, and governmental agencies. In line with our library of research studies, cognitive ability significantly predicted employee performance, r =.215, p <.01 (see Table 6), such that employees with higher cognitive ability were rated by their direct supervisors as being better performers as compared to employees with lower cognitive ability. Also in line with previous research (e.g., Gibbons, 2006; Sirota, Mischkind, & Meltzer, 2005; Towers Perrin, 2009), we found that engaged employees were rated as better performers as compared to their disengaged counterparts, r =.189, p <.05 (see Table 6). Of main interest, motivation potential significantly predicted employee performance, r =.211, p <.01 (see Table 6), and this relationship was just as strong as the relationship between cognitive ability and employee performance, Z H =.04, p >.05. As Table 7 presents, cognitive ability had an estimated validity of.215 and motivation potential had an estimated validity of.211. When combined, the two methods have an estimated validity of.317, an increase in 47% beyond what a cognitive ability test used alone could provide.

7 Assessment Table 7 Validity of Wonderlic Assessment Tools Alone and in Combination Validity of method used alone Cognitive Ability (WPT-Q).215 Incremental (combined) validity % increase in validity from combining tool with cognitive ability Motivation Potential (MP) % Note. Multiple regression results indicated that Motivation Potential significantly predicted Employee Performance above and beyond Cognitive Ability, ΔR 2 =.054, F(1, 173) = 10.48, p =.001. Taken together, results from this study demonstrate that 1) motivation potential is distinguishable from cognitive ability, 2) motivation potential can identify those applicants who would be more likely to become actively engaged on the job, and 3) the Motivation Potential Assessment not only is able to predict job performance, but can also provide utility beyond what is captured by existing methods. Although it is commonly thought that, Other predictors may add only small amounts of incremental validity over cognitive tests (U.S. Office of Personnel Management, 2007, p. 23), our research suggests that motivation potential can add a significant and substantial amount of incremental validity.

8 References Gallup, Inc. (2013) State of the American Workplace Report. Retrieved 10/30/2013 from strategicconsulting/163007/state-american-workplace.aspx Gibbons, J. (2006). Employee engagement: A review of current research and its implications. New York, NY: The Conference Board. Retrieved 5/21/2013 from U.S. Department of Labor, Employment and Training Administration (2000). Testing and Assessment: An Employer s Guide to Good Practices. Retrieved 10/30/2013 from U.S. Office of Personnel Management. Assessment Decision Guide. (2007). Retrieved 1/7/2014 from ContentFiles/AssessmentDecisionGuide pdf Maltarich, M. A., Nyberg, A. J., & Reilly, G. (2010). A conceptual and empirical analysis of the cognitive ability-voluntary turnover relationship. Journal of Applied Psychology, 95, doi: /a Sirota, D., Mischkind, L. A., & Meltzer, M. I. (2005). The enthusiastic employee: How companies profit by giving workers what they want. Upper Saddle River, NJ: Pearson Education, Inc. Towers Perrin (2009). Employee engagement underpins business transformation. Retrieved 5/21/2013 from towersperrin.com/tp/getwebcachedoc?country=gbr&webc=gbr/2008/200807/tp_isr_july08.pdf Wonderlic, Inc. (2013). Development and Validation of the Wonderlic Motivation Potential Assessment (MPA). Vernon Hills, IL: Wonderlic.

WORKFORCE METRICS BENCHMARK REPORT

WORKFORCE METRICS BENCHMARK REPORT THE MORE YOU KNOW, THE SMARTER YOU WORK As an HR professional, you focus on helping your organization optimize its human capital, but how do you show that achievement?