Criterion-Related Validation Study of the TALENTx7 Assessment

Transcription

1 Criterion-Related Validation Study of the TALENTx7 Assessment ABSTRACT A sample of employees identified as high potentials was administered the TALENTx7 Assessment. Results indicate that the vast majority of employees had a high amount of overall learning agility (M = 66.94). The relationship between overall learning agility and a composite measure of performance was statistically significant (r = 0.31). Kenneth P. De Meuse, Ph.D. July 2016

2 Criterion-Related Validation Study of the TALENTx7 Assessment Sample During the spring of 2016, the TALENTx7 Assessment of learning agility was administered to 39 participants of a high potential leadership program at a national retail company with stores located throughout the United States. In total, 36 participants responded. However, four employees were deleted from this analysis due to concerns about the accuracy of their responses. 1 Thus, the final sample size was N = 32 high potential employees. Sixteen (50%) of the participants were female. Position titles ranged from general manager, district manager, regional manager, director, senior director, and vice-president. Ratings of Performance As part of its annual performance review, the company evaluates employees on the following two dimensions: (a) key deliverables and (b) value-based behaviors. The key deliverables rating assesses the extent to which employees meet performance objectives. A 3-point rating scale is used, ranging from exceeds expectations (3), meets expectations (2), to improvement needed (1). The second dimension measures the extent to which individuals exhibit the company s values through their behaviors, and the scale ranges from being a role model (3), consistently demonstrate it (2), to inconsistently demonstrate it (1). As expected for a group of high potential employees, all of their performance ratings were high. As one can see in Table 1, no one received a rating of 1 on either dimension. Since the two dimensions were virtually unrelated (r = 0.02), a composite performance rating was computed to capture each employee s evaluation with a single score. Thus, overall, 2 employees (6%) received a relatively low evaluation, 13 (41%) received a relatively moderate evaluation, and 17 (53%) received a very high evaluation. Table 1. Number of Employees Receiving Performance Ratings of 1, 2, or 3 Performance Dimension Rating of 1 Rating of 2 Rating of 3 Key deliverables Value-based behaviors Two respondents received a 1-star rating on the Overall Accuracy Index when they completed the assessment initially. They likewise receive a 1-star rating on the second administration and were deleted from the study. As an extra precaution, two other respondents who received a 2-star rating on this index and also were deleted from the analyses. Copyright 2016 by Kenneth P. De Meuse, Ph. D. All rights reserved. 1

3 Ratings of Learning Agility The psychological construct of learning agility was measured by the TALENTx7 Assessment. This online self-assessment measures an employee s Overall Learning Agility and seven different facets of it. Scores are converted into percentiles to provide employees their level of learning agility relative to others (De Meuse & Fang, 2015). Table 2 presents the mean scores of Overall Learning Agility and the seven facets of learning agility measured by the TALENTx7 Assessment. In addition, the number of employees with scores less than and greater than the 50 th percentile is provided. Table 2. Mean Learning Agility Scores and Number of Employees with Scores Less Than and Greater Than 50 th Percentile Learning Agility Facet Mean 1-50 th Percentile th Percentile Interpersonal Acumen Cognitive Perspective Environmental Mindfulness Drive to Excel Self-Insight Change Alacrity Feedback Responsiveness Overall Learning Agility As expected, the mean Overall Learning Agility score for this group of high potential employees was very high (M = 66.94). Only six of the 32 employees had scores less than the 50 th percentile. With regard to specific facets of learning agility, Drive to Excel (M = 86.63) and Cognitive Perspective (M = 72.94) were exceedingly high; whereas, Feedback Responsiveness (M = 49.94) and Self-Insight (M = 49.41) were relatively low. One might infer that an employee s drive (e.g., ambition, motivation, determination) and ability to think critically and strategically (i.e., Cognitive Perspective) were critical for being identified as a high potential in this company. In contrast, an employee s self-insight and responsive to feedback appeared to receive little consideration. An alternative hypothesis is that Drive to Excel and Cognitive Perspective are easier to observe than Self-Insight and Feedback Responsiveness. Therefore, these facets of learning agility received relatively more weight when selecting employees for the high potential pool. Copyright 2016 by Kenneth P. De Meuse, Ph. D. All rights reserved. 2

4 Relationship between Performance and Learning Agility Several studies have found learning agility to be significantly related to performance (Dai, De Meuse, & Tang, 2013; Dries, Vantilborgh, & Pepermans, 2012). For example, Dai et al. observed that learning agility was significantly related to career growth trajectory. High learning agile individuals were promoted more often and received higher salary increases than their low learning agile counterparts during a period of 10 years. Dries et al. observed that high performing employees were three times more likely to be identified as a high potential than employees with low performance. However, they discovered that being high in learning agility increased an employee s likelihood of being identified as a high potential by a factor of 18. The researchers concluded that learning agility is an overriding criterion for separating high potentials from non-high potentials (Dries et al., 2012, p. 351). A recent review of the research literature uncovered 17 empirical studies that examined the relationship between leader performance and learning agility. The median correlation coefficient was r = 0.40 (De Meuse, 2016). In the current study, the relationship between performance and learning agility was investigated in two different ways. First, learning agility scores were categorized into the three categories of performance low, moderate, and high. See Table 3. Table 3. Mean Learning Agility Scores for Three Level of Performance Ratings Learning Agility Low Evaluation Moderate Evaluation High Evaluation Interpersonal Acumen 77.00* Cognitive Perspective 92.00* Environmental Mindfulness 67.50* Drive to Excel 90.50* Self-Insight * Change Alacrity * Feedback Responsiveness * Overall Learning Agility * Note. The number of employees evaluated low, moderate, and high was N = 2, 13, and 17, respectively. An asterisk (*) denotes highest mean score among the three groups. Copyright 2016 by Kenneth P. De Meuse, Ph. D. All rights reserved. 3

5 Notice there is little relationship between performance ratings and learning agility for those employees identified as high potentials. In fact, the results suggest somewhat of the reverse trend. The group of managers who were evaluated lowest on performance often scored highest on learning agility. In only one instance, did the highest evaluated group of managers have the highest mean learning agility score (Self-Insight). Consequently, this company seemed to devote little consideration to learning agility when selecting employees for this high potential program. Rather, the identification of high potentials likely was much more related to their performance, in that 100% of those employees were rated the highest or second highest on the company s performance scale (see Table 1). This outcome is quite common in the identification of high potential leaders according to a survey conducted by the Corporate Leadership Council (2005). A second approach for investigating the relationship between performance and learning agility is to calculate a statistical correlation. When one inspects the performance scores for the 32 employees more closely, one discovers that four employees were evaluated very high (i.e., situated in the third column in Table 3) but had low overall learning agility scores (i.e., scored less than the 50 th percentile), suggesting they were selected for high potentials based primarily on their performance. If one removes them from the analysis, the relationship between performance and Overall Learning Agility is statistically significant at the p <.10 level (r = 0.31, N = 28). 2 Evidence of Criterion-Related Validity Two findings of this study support the criterion-related validity of the TALENTx7 Assessment. First, the mean score of Overall Learning Agility as well as the mean scores for most of the seven facets were very high. This observation is consistent with what one would expect from a group of employees identified as high potential talent. Second, the correlation between employee performance and learning agility was significant for those employees who were evaluated as high performers and who scored above the 50 th percentile on learning agility. Naturally, it will be important to re-examine the relationship between learning agility and performance during next year s performance review. If learning agility influences leader success, its relationship to performance ratings will be higher at that time for this group of high potentials. The four individuals with low learning agility scores (who were removed from the analysis in this paper) should be evaluated low. Those employees who were given feedback on their specific facet scores of learning agility and worked to develop their low scores should perform better. Such a predictive validation study would provide additional support for the utility of the TALENTx7 Assessment. Further, if this was the case, the implication would be for this company to systematically apply learning agility as a key factor for identifying high potential leaders in the future. 2 Performance ratings were corrected for unreliability and restriction of range. Such statistical corrections often are implemented to estimate correlation coefficients that more accurately reflect the true relationship between two sets of scores (Guilford & Fruchter, 1978). Copyright 2016 by Kenneth P. De Meuse, Ph. D. All rights reserved. 4

6 References Corporate Leadership Council. (2005). Realizing the full potential of rising talent. Washington, DC: Corporate Executive Board. Dai, G., De Meuse, K. P., & Tang, K. Y. (2013). The role of learning agility in executive career success: The results of two field studies. Journal of Managerial Issues, 25, De Meuse, K. P. (2016). An examination of the scientific linkage between learning agility and leader success. Minneapolis: Wisconsin Management Group. De Meuse, K. P., & Feng, S. (2015). The development and validation of the TALENTx7 Assessment : A psychological measure of learning agility. Shanghai: Leader s Gene Consulting. Dries, N., Vantilborgh, T., & Pepermans, R. (2012). The role of learning agility and career variety in the identification and development of high potential employees. Personnel Review, 41, Guilford, J. P., & Fruchter, B. (1978). Fundamental statistics in psychology and education (6 th ed.). New York: McGraw-Hill. Copyright 2016 by Kenneth P. De Meuse, Ph. D. All rights reserved. 5

7 About the Author Dr. Kenneth P. De Meuse is founder and president of the Wisconsin Management Group, a consulting firm specializing in leader identification, executive coaching, and research on high potential talent. Dr. De Meuse is a global thought leader on the assessment and development of leadership, and has presented his research on learning agility and leadership competencies at numerous professional conferences, including the Academy of Management, American Psychological Association, Society for Human Resource Management, The Conference Board, International Coach Federation, Society for Consulting Psychology, and the Society for Industrial and Organizational Psychology. His 2010 journal article on learning agility is considered the first scholarly publication on the construct and lays the foundation for its scientific exploration. Throughout his career, Dr. De Meuse has consulted on a variety of strategic and leadership issues at businesses such as Nestle USA, Lucent Technologies, Siemens, RMS McGladrey, Presto Industries, Ayres Associates, and Xcel Energy. Prior to establishing the Wisconsin Management Group, he was executive vice president of Research and Product Development at Tercon Consulting, a global consulting firm headquartered in Washington, DC. He also was vice president of Global Research at Korn/Ferry International for six years. In addition, Dr. De Meuse was on the faculties of Iowa State University and the University of Wisconsin (Eau Claire). He has published more than 50 peer-reviewed journal articles and authored five books. Dr. De Meuse earned his Ph.D. in Industrial/Organizational Psychology from the University of Tennessee and his Master s degree in Psychology from the University of Nebraska. In acknowledgement for his contributions to the science and practice of talent management, he was elected Fellow by the Society for Industrial and Organizational Psychology. Copyright 2016 by Kenneth P. De Meuse, Ph. D. All rights reserved. 6