An Examination of Assessment Center Quality 1
Assessment Centers Remain a Preferred Method for Leadership Assessment and Development For many years assessment centers have been identified as the preferred means to identify development issues for future leaders and to select candidates for leadership positions. An assessment center can be defined as an assessment process where candidates participate in multiple simulations while being observed by two or more trained observers and are evaluated on a number of dimensions or competencies relevant to the position. For purposes of clarity, we will refer to centers used for selection as assessment centers, and those for development as development centers. Two branches of the United States Government, The Office of Personnel Management and the Department of Labor, both recommend assessment centers in these different capacities. There is a reported resurgence of the use of assessment centers as a response to the economic downturn in 2008 and the resulting increased supply of labor and fewer job openings. In fact an article in the journal, Workforce Management (2009) identified the area of assessment centers as a growth market for many management development consulting firms, with many companies entering the market to satisfy the need. Therefore, compelling questions for the leadership development practitioner becomes, how do you determine the quality of an assessment center and its value to a company? How do you determine the quality of an assessment center and its value to the company? If you were buying a car you might kick the tires, look under the hood, or take it out for a test drive. If you were buying a big screen TV you might look at a floor model or check for online, independent, third-party consumer ratings and testimonials. Even a standardized training program would have some obvious means for assessing quality. You could request to look at previous training evaluations, contact references or see if any articles have been published on its effectiveness. Unfortunately, many of these evaluation strategies are not feasible or available to determine the quality of an assessment center, and most of the published literature focuses on validity and reliability concerns, which while very important, focus solely on a narrow aspect of the overall quality issue. There have been several attempts to codify best practices in assessment center design and administration in the United States (1989, 2000) and internationally (2009). scitrain Assessment Centers meet or exceed all of the standards in the Guidelines and Ethical Considerations for Assessment Center Operations published by The International Task Force on Assessment Center Guidelines. 2
We know from studies performed on leadership development centers that a number of factors outside of validity and reliability impact the overall effectiveness of a center. Many of these factors are influenced by the assessment center design and administration but occurred after the center has concluded. These include but are not limited to such phenomenon as feedback acceptance, self-efficacy or self-confidence, career motivation, perceived manager and organizational support, and development plan completion. Articles by Jones and Whitmore (1992, 1995) identify the power of these factors in predicting outcomes such as performance and promotion. At scitrain we strongly support the view that validity and reliability are minimal and necessary conditions for determining quality, but we also take a more comprehensive view to the issue of quality. This whitepaper will outline the scitrain approach to assessment center quality, present a quality model, and discuss research on its products and services. This model states that assessment center quality is determined by four dimensions: Relevance - customized and tailored to the company and the role, containing meaningful, realistic, and engaging simulation content engineered by: identification of competencies and critical success factors translation of competencies to behaviors: a micro analysis of behavior-level indicators and descriptors by competency design of exercises to engage, elicit, and measure targeted competencies review of exercises by internal and external subject matter experts Relevance Expertise Interaction Structure The scitrain Quality Model Structure - structured observation and assessment administration process created by: a methodical, documented assessment center administration process development of behavioral note taking and scoring methodologies calculation and design of formulas and summary data schema documentation of all aspects of the center in manuals Assessor expertise - highly experienced, trained assessors, who consist of external observers, internal decision makers, and internal HR leaders, accomplished by: targeted identification and selection of potential external and internal assessors where possible, a mix of internal and external perspectives behavioral observation assessor training, and assessment center orientation use of pilots, dry runs, and rehearsals facilitation of calibration and consensus sessions 3
Interaction - open and transparent two-way communication with participants, with an emphasis on respect and empathy established by: the value of creating trust, credibility, and self-confidence of participants transparency of the experiences understanding the scoring process preparing feedback at the behavioral level understandable, thorough feedback reports timeliness and immediacy of feedback When all four of these aspects of quality are present in a center, this leads to successful outcomes. The scitrain Quality Success Model 4
Prediction - this is an important outcome for assessment centers used for selection purposes. Statistically this would be referred to as criterion-related validity. Criterion-related validity is a method of establishing the validity of an assessment center by showing a substantial relationship between the assessment center scores and job performance scores, promotions, or other desirable organizational outcomes. scitrain has conducted criterion related validity studies and found that its centers predict promotability. A study with one of its largest clients involving 88 executives found that every aspect of the center predicted promotability at a statistically significant level.* A follow-up study conducted with the company s senior managers found that one year after the candidate s placement in the position, every manager selected was performing well in his or her position. Accuracy - this is important for both selection and development centers. For selection purposes, if the ratings are not accurate, they will not be predictive of organizational outcomes. Accuracy is also important for development centers. Inaccurate ratings can misidentify development strengths and mask development weaknesses. Accuracy can be associated with the overall fidelity of the assessment center. From a validation perspective, accuracy is most closely related to construct validity. Construct validity can be defined as the relationship between the measures of an assessment center and the underlying concepts and constructs it claims to measure. For example, if the center is intended to measure leadership abilities, do the assessment center measures actually measure underlying concepts of leadership? scitrain has conducted a construct validity study on a large client s Leadership Potential Center involving 100 emerging leaders and found that the center strongly supported Authentic Leadership Theory, which is closely aligned with the company s values and culture. Fairness - this is important for centers that have either development or selection objectives. Fairness in the center context can be defined as a participant s perception that simulations are an accurate representation of what would be required for the position, and that every participant is treated the same way, exposed to the same challenges and potential stressors, and provided with the same opportunities to demonstrate his/her potential. scitrain Assessment Centers promote perceptions of fairness through relevance of the simulations, standardization of the processes, and transparency of the assessment process and feedback the participants receive. 5
Feedback Acceptance - this is primarily important for development centers. Studies by Jones and Whitmore (1995) and others have found that in order for development to occur as a result of a development center, the feedback needs to be understood and accepted by the participant. Their study found that feedback acceptance is critical to participants following through with the recommendations in a meaningful way. They also found that managers and the organization are less likely to support development if they feel that the participant does not embrace the feedback. The transparency in scitrain s assessment process and the immediacy in the delivery of feedback help participant to understand the feedback and make it easier to accept constructive feedback. scitrain s Assessment Center process focuses a great deal of attention and time to the feedback process helping to promote a candidate s positive reaction to the feedback. Career Motivation - It is important that participants finishing a development center feel motivated to develop. For a development center, this is achieved by helping the candidate feel self-confident in their abilities to develop and the ease and understanding of translating the feedback information into meaningful development goals and actions. scitrain Development Centers promote career motivation by incorporating a future career focus to the center, providing participants with a clear guide to competencies they need to develop, and identifying meaningful development actions. scitrain s value of treating every participant with respect and empathy promotes a participant s feelings of self-confidence and self-efficacy. scitrain s commitment to quality ensures Assessment Centers produce results that are both statistically significant and meaningful to the organization and to leaders careers. A wise investment in a high-quality assessment center does accelerate the development and placement of strong leaders. Going back to our car quality analogy, companies who make this investment will put their leadership development programs in the fast lane and make sure that their high-potential leaders are driving in the proper direction to develop the right set of competencies in time to ensure future success. *A technical report on scitrain s validation studies may be requested by contacting Dr. Mark Whitmore, Chief Scientist, at mark.whitmore@scitrian.com. 6
References Assessment Centers. Personnel Assessment and Selection Resource Center, The U.S. Office of Personnel Management (website, http://apps.opm.gov/adt). Testing and Assessment: An Employer's Guide to Good Practices. (2000). Washington, DC: U.S. Department of Labor, Employment and Training Administration. Note: Article can be accessed at http:// www.onetcenter.org/guides.html. The International Task Force on Assessment Center Guidelines. (2009). Guidelines and Ethical Considerations for Assessment Center Operations. International Journal of Assessment and Selection. vol. 17, no. 3, pgs. 243-253. Jones, Robert G. & Whitmore, Mark D. (April, 1992). When will feedback from an assessment center make a difference in people s careers? In Employees reactions to alternative sources of development feedback. Symposium conducted at the Convention of the Society of Industrial and Organizational Psychologists. Jones, Robert G. & Whitmore, Mark D. (1995). Evaluating Developmental Assessment Centers as Interventions. Personnel Psychology. Vol 48, pgs. 377-388. Rafter, Michelle V. (2009) Assessment providers scoring well. Workforce Management. Vol. 88(1), pgs. 24-25. 7