Shoot out! What predicts Leadership Potential the best? Assessment Centers (Simulations), Cognitive tests, or Personality tests?

Transcription

1 Shoot out! What predicts Leadership Potential the best? Assessment Centers (Simulations), Cognitive tests, or Personality tests? The latest research on the topic COPYRIGHT: EICM

2 Abstract In 1998 Hunter and Schmidt published research that led assessment practitioners to believe that Cognitive ability tests are the most powerful determinant of potential and performance; in the process other constructs such as Personality and Assessment Center or Simulation Competencies were devalued. As a result most assessment practitioners put cognitive tests center stage in their assessment programs in identifying leadership potential. When putting a battery together the practitioner would focus more on Cognitive or Levels of Work tests combined with a measure of Personality. The latest research questions the wisdom of doing so. Two Meta-Analytical research studies namely Sackett, Shewach and Keiser (N=3865, University of Minnesota, 2015) and Meriac, Woehr, Matthew and Fleisher (N=52000, University of Tennessee, 2008) have debunked this long held belief. To this end OMT have conducted similar research. Using a sample of 700 managers across six levels of management, 179 companies and 10 industries we have found that the Assessment Centre (EvaleX Business Simulation) had the highest validity, followed by Cognitive then Personality and then Levels of Work tests. Can we fly in the face of such accumulative and corroborating evidence! Problem Statement and Research Purpose Based on the 1998 Hunter and Schmidt research publication that advocated the superior power of cognitive tests, many talent assessment practitioners used cognitive tests as the bedrock of an assessment battery augmented with personality and other tests. However, studies since then found that leadership competencies (AC simulations) have higher predictive validities than cognitive test. The objective of this research was to define the relative contribution of three popular constructs in identifying leadership potential in a South African context: Leadership competencies assessed through Leadership Simulations, Cognitive abilities assessed through more traditional cognitive tests as well as Levels of Work tests and finally, Personality as assessed through personality and other behavioral questionnaires. Taking poetic license, a Level of Work test is the name given to a range of assessment instruments that provide the candidate with a number of stimuli to which the subsequent responses are classified as exhibiting a level of thinking based on the levels of work theory of Elliot Jacques. Examples are the CPA, CPP, CNT and EvaleX Business Simulation. COPYRIGHT: EICM

3 Research Design and Methodology From a population of 3,550 managers who undertook the EvaleX35 leadership assessment as part of a selection process, promotional decision making, talent landscaping or skills audits, a sample of just under 700 managers for whom verified position level and performance data were available was drawn. The EvaleX35 consists of: EvaleX Business Simulation (EBS) consisting of six simulations: Problem analysis and decision-making: Digitally interactive response exercise. Function management: e-tray. Project management: Business case study. Staff management: Business case study. Client management: Business case study. EvaleX Psychometrics consisting of the following tests: Cognitive: Business Comprehension Scale (BCT). Cognitive: Organisation Insight Scale (OIS). Personality: Organisation Personality Construct Scale (OPCS). Work styles: Organisation Personality Construct Scale (OPCS). Values: Organisation Personality Construct Scale (OPCS). Interests: Work Orientation Scale (WOS). For each candidate, 210 data points across 5 management simulations (EBS) and 323 data points for psychometrics were analysed. A data point mostly refers to a score per dimension or competence but can also include bio-graphical data such as age, gender, etc. The statistical program Wizard was used for correlations and distributions. For the correlation matrix and regression analysis SSPS and M-plus was used. Three regression models were used to determine the contribution that each of the assessment instruments made in predicting leadership potential. Two criterion scores were used namely Position Level and Career Velocity. For each candidate the level of work was defined using a Level of Work calculator. Career velocity was calculated by moderating position level with age. COPYRIGHT: EICM

4 Results A stream of conclusions followed the analysis of the data, but for this document, the focus will only be on the regression analysis showing the relative power of each instrument in determining Potential. Stratifying the sample into 7 groups according to Level of Work, T-tests showed significant differences in how the 7 groups performed in the Competence Simulations (EBS), Cognitive and Personality tests. The 7 groups were achieved as a result of breaking a level, for instance 3 into an upper and lower category (1,5/2,0/2,5/3,0/3,5/4,0/4,5 upward). Correlation with both position level and career velocity showed that the Competence Simulations had the highest predictive validity, followed by Cognitive then Personality and Level of Work tests. The two graphs that follow show the distribution of the managers in the sample across Position level, Gender and Race. Distribu(on across Posi(on Level Team Leaders Supervisory Junior Mngmnt Middle Mngmnt Senior Mngmnt Execu(ve Mngmnt The number of assessments done across Posi(on Level shows the demand for leadership in roles from Junior (25) to Senior (35) Management. COPYRIGHT: EICM

5 Distribu(on according to Gender The sample consisted of a rela(vely even distribu(on of gender across posi(on levels from 15 (Team Leader) to 45 (CEO) Distribu=on according to Race For those candidates in a South African context, where race was correctly captured in the system, the results are shown above. It is interes=ng to note the comparison of 962 White candidates versus 2049 Black candidates. It shows that the ra=o of applica=ons and appointments have shiged to basically 70% Black versus 30% White. Another key aspect of this is that when benchmarking an assessment, the EvaleX Business Simula=on norm group mirrors this distribu=on. COPYRIGHT: EICM

6 Of all the assessment instruments researched (simulation, cognitive, personality, levels of work tests), the EvaleX Business Simulation (competence) showed the highest predictive validity by far. The graph below indicates the slope of EBS performance across position level. The other methods have a similar trend line, but not as strong. Average Competence across all 5 EBS simula3ons correlates with Posi3on Level at 0,60 This is a cri)cal finding. Competence correlates with Posi3on Level at 0,60. If Posi3on Level is an indica3on of the Poten3al or Talent to advance to ever more senior roles, then Leadership Competence is a strong determinant of this Poten)al. In order to determine relative strength of each of the instruments a Regression Analysis was performed. Due to the fact that the criterion was Job Level, a categorical/ordinal rather than a continuous variable, the M-plus methodology was used. Regardless of having done a Multivariate regression, a Hierarchical regression and a Step-wise regression, the results show that average competence, as measured in the EBS, explains the majority of the variance, no matter how you model it. COPYRIGHT: EICM

7 Taking the above data and converting it to a pie chart provides a clear visual picture of the superior power of Leadership Competence (as measured in the EvaleX Business Simulation) in measuring Potential. COPYRIGHT: EICM

8 The pie chart above shows the results of a multivariate regression analysis, where all four instruments were given an equal chance to compete for the best predictor slot. But, we also wanted to consider the results from another angle. Rather than looking at which instrument explains job level the best (the simulations as seen above), we wanted to see how much additional variance in job level can be explained when we add Simulations to a cognitive / personality assessment. So, if your assessment/selection model includes a cognitive and personality test, how much more value does a Simulation like the EBS add? For this a step-wise regression model was chosen. Step-wise allows the researcher to feed the instrument results in one at a time in a defined order. We chose to start with Cognitive, then Personality and then only Competence (EvaleX Business Simulation). The reason was to measure the exact additional validity added to a standard Cognitive + Personality test battery that Simulations would contribute. Stepwise Regression: Model Summary d Model R R Square Adjusted R Square Std. Error of the Es=mate Change Sta=s=cs R Square Change a,123,122 5,771, b,227,225 5,422, c,427,425 4,670,200 a. Predictors: (Constant), BCT Total Score b. Predictors: (Constant), BCT Total Score, OPCS PREDICTIVE VALIDITY c. Predictors: (Constant), BCT Total Score, OPCS PREDICTIVE VALIDITY, AVERAGE COMPETENCE d. Dependent Variable: POSITION LEVEL Conclusions: In this stepwise regression the order was forced star=ng with Cogni=ve (BCT) then Personality (OPCS) then Competence last, in order to see what addi=onal value AC simula=ons add to an assessment baxery consis=ng of Cogni=ve and Personality. The results are staggering: The EvaleX Business Simula=ons doubled the predic=ve validity. Cogni=ve and Personality declared 22% of the variance, then adding the EBS the declared variance moves up to 42% (See column R squared) By adding Leadership Competence as assessed through the EvaleX Business Simulation to Cognitive and Personality in an assessment solution, validity in explaining job level is DOUBLED COPYRIGHT: EICM

9 Conclusions Using multivariate, hierarchical and step-wise regression analysis, Simulations came out tops in all three models followed by Personality and then only Cognitive. From the analysis it would seem that Leadership Competence (Simulations) declares the majority of the variance in job level, no matter how you model it. Conversely, in only doing Cognitive and Personality less than half the story about an individual s potential is explained. It is like a pilot flying a plane with half the instruments hidden from sight. Clearly in assembling a Management or Leadership assessment battery, the foundational instrument, the starting point, should be Assessment Centre Simulations then augmented with Cognitive or Levels of Work assessments and Personality. Unfortunately most companies prefer using the cognitive / personality combination. These findings correspond to two other major research studies namely Sackett, Shewach and Keiser (University of Minnesota 2015) and Woehr, Matthew and Fleisher (University of Tennessee 2008). In the face of such of such accumulative and corroborating evidence the Talent Manager will find it difficult to explain why AC simulations are not part of a leadership assessment process. Implications Leadership must be one of the most critical challenges in South Africa in all spheres but particularly commerce and industry. The business organisation will be one of the main drivers in creating employment and eradicating poverty. In developing business leadership, it is crucial to use the correct assessment approaches to identify talent in order to reap the best return on subsequent development efforts. Using a less than optimal solution in assessing candidates for managerial and leadership roles can waste valuable time and resources. Unfortunately, many business organisations are not using optimally structured leadership assessment programs due to an incomplete appreciation of which tools to use. As the sample of 700 leaders researched closely resembles the demographics of the South African leadership landscape with approximately 30% white candidates and 70% Black, it is crucial for I/O psychologists to take note This is a critical issue on which I/O Psychologists need to be well informed. What predicts Leadership Potential best for a South African context? Assessment centers (Simulations), Cognitive tests, Personality tests or Levels of Work tests? Clearly AC Simulations. The research set out in this document provides a more holistic framework in assessment solution design avoiding the over reliance on one tool at the exclusion of others. COPYRIGHT: EICM

10 We at OMT / EvaleX advocate a holistic approach with a strong belief that all four assessment types / constructs add value and should form part of an all-inclusive integrated solution. The research provides sound evidence and justification why the EvaleX35 Leadership Assessment solution consists of Management Simulation, Cognitive and Personality tests, the results of which leads to a balanced statement about an individual s potential. For more information on the presented research or to arrange a presentation to your management team, feel free to contact: Organisation and Management Technology admin@evalex.com COPYRIGHT: EICM