Multi-situation, Multi-Dimension Assessment in Organizations: A Brief History. Chris Dewberry and Duncan Jackson

Transcription

1 Multi-situation, Multi-Dimension Assessment in Organizations: A Brief History Chris Dewberry and Duncan Jackson

2 XpertHR 2011 Survey of UK Assessment Centres Two-thirds (67%) of the 191 employers taking part in a 2011 XpertHR survey used assessment centres for recruitment. Organizations were most likely to use assessment centres for high-value roles, such as senior or specialist staff or when recruiting graduates.

3 The Origins of MS/MD Assessment: Nazi Germany The first identifiable multiple assessment approach to selection was in military officer recruitment in Germany, starting in Several competencies were assessed over several situations by two officers, a medical doctor, and three psychologists. Positive will - a habit of voluntary response to the command of the superior leader Determination - the creation of means to achieve the goal and the achievement of a goal Operative thinking - planning and executing a preconceived action Mental elasticity - the ability to live up to any situation in order to achieve a goal Mathematical thinking Character - integrity, selflessness, idealism

4 MS/MD Assessment Starts in the UK In 1942 the British set up War Office Selection Boards (WOSBs). These were based on the German multi-situation, multi-competency approach. The WOSBs included psychiatric interviews, intelligence tests and simulated command tasks. They lasted 3 or 4 days. Each candidate s performance was assessed against 17 areas of competence that were thought to be indicators of officer potential.

5 Initial Non-military Use of MS/MD Assessment The first non-military use of MS/MD assessment was also found in the UK. In 1945, the Civil Service Selection Board (CSSB) introduced a three-phase system for recruiting middle to high level civil servants. The second phase of this process included verbal and non-verbal tests, projective personality measures, interviews, an examination, individual exercises including a presentation, and group exercises. The assessors - two civil servants and a psychologist - would share information following the centre, and then independently rate each candidate s potential.

6 MS/MD Assessment Starts in North America in WW2 The Office of Strategic Services (OSS) was a forerunner of the Central Intelligence Administration (CIA) It was set up to organize and carry out espionage behind enemy lines spies and saboteurs With the aid of the British the Psychological Selection Program for spies was set up in the OSS.

7 Competencies Assessed in the OSS Psychological Selection Programme Motivation for assignment Energy and initiative Effective intelligence Emotional stability Social relations Leadership Security Physical ability Observing and reporting Propaganda skills

8 Example of an OSS Exercise The "Belongings Test," measured a candidate's practical memory and intelligence. A candidate was given four minutes to examine a room and all the items in it. The candidate would then take a written test which measured his/her memory of what was in the room and what conclusions could be drawn about the person living in it.

9 The Private Sector Start Using MS/MD Assessment In 1956 The American Telephone and Telegraph Company (AT&T) began measuring 25 management characteristics using: A 2 hour interview An in-tray exercise A leaderless group discussion A business game A range of projective tests and cognitive tests

10 The Original Assessment Centre Assessors were industrial and clinical psychologists, company personnel and university staff. At this time the phrase Assessment Centre was used for the first time: it was actually a physical building owned by AT&T in which the assessments took place.

11 The Growth of Assessment Centres and Competencies During the 1960 s and 1970 s there was a steady growth in the use of assessment centres to measure behavioural dimensions in the public and private sectors. This growth was further fueled by the publication of a book: Boyatzis R.E. (1982) The Competent Manager: A Model of Effective Performance. New York, Wiley.

12 Boyatzis Introduces Competency Models A competency model of managers should have two dimensions. One dimension should describe the types of competencies. The other dimension of the management competency model should describe the levels of each competency

13 Boyatzis Does Not Define Competencies Unambiguously Certain characteristics or abilities of the person enable him or her to demonstrate the appropriate specific actions. These characteristics or abilities can be called competencies. A job competency is an underlying characteristic of a person which results in effective and/or superior performance in a job. A competency may be a motive, trait, skill, aspect of one s image or social role, or a body of knowledge which he or she uses. Each competency may exist within the individual at various levels, with motives existing at the unconscious level, self-image at the conscious level, and skills at the behavioural level.

14 The Measurement of Competencies in Modern Assessment Centres

15 What are Competencies? Civil Service 2016 Competencies are the skills, knowledge and behaviours that lead to a successful performance. They are short statements, up to 250 words, describing a time in which you have displayed the behaviours needed to perform well in a particular job. It allows the job holder to understand what you are capable of, it shows that you can apply the same behaviours to the new role. The competency framework centres on how you achieved the outcome. Therefore it is important that you show how your behaviour led to the outcome. It should always be remembered that the competencies required of each job role differs from position to position.

16 Most Common Exercises in ACs Krause and Thornton (2009) Role play Presentation In-basket Planning exercise Situational interview Group discussion Case study

17 Most Commonly Assessed Competencies in ACs Krause and Thornton (2009) Communicating skills Problem solving Influencing others Organizing and planning

18 The Generality of Competencies Competencies can be very specific (e.g. can some perform a three-point-turn in a car?). They can also be very general (e.g. communication skills). Communications skills can be demonstrated in all sorts of ways (verbally, in writing, non-verbal), about all sorts of matters (serious, frivolous), in all sorts of contexts ( s, formal meetings, chats in the corridor), to all sorts or people (friends, bosses, subordinates). Is there a good level of correlational structure between all manifestations of general competencies such as communication skills? There is virtually no academic literature on the correlational structure of competencies!

19 Assessment Centres: Competencies Assessed with Exercises Role play Leaderless group discussion Case study Analysis problem Communication X X X Teamwork X X Problem solving X X Customer focus X X

20 AC Scores Reflecting a Person by Competency Interaction Role play Leaderless group discussion Case study Analysis problem Paul Communication Teamwork 4 4 Problem solving 3 3 Customer focus 1 1 Carol Communication Teamwork 2 2 Problem solving 5 5 Customer focus 3 3

21 AC Scores Reflecting a Person by Competency Interaction Role play Leaderless group discussion Case study Analysis problem Average Score Paul Communication Teamwork Problem solving Customer focus Carol Communication Teamwork Problem solving Customer focus 3 3 3

22 AC Scores Reflecting a Person by Exercise Interaction Role play Leaderless group discussion Case study Analysis problem Paul Communication Teamwork 2 3 Problem solving 5 1 Customer focus 2 1 Carol Communication Teamwork 3 1 Problem solving 4 5 Customer focus 3 5

23 AC Scores Reflecting a Person by Exercise Interaction Role play Leaderless group discussion Case study Analysis problem Average Score Paul Communication Teamwork Problem solving Customer focus Carol Communication Teamwork Problem solving Customer focus 3 5 4

24 Confounded Dimension Scores So.finding that AC candidates have different scores on a particular competency is NOT evidence that the competency is being measured. And finding that AC competency scores predict job performance is NOT evidence that these competencies are being measured either. This is because competency scores are confounded with several other AC variables including general AC performance, exercise performance, and assessor rating style.

25 So How Can We Find Out Where AC Scores Actually Come From? What we need to do is decompose all of the variance in AC ratings into their sources (e.g. exercises, dimensions, assessors etc.) and see how much variation in ratings is associated with each source. There are 29 potential sources of variance in ACs. Decomposing the total rating variance into 29 sources presents a considerable statistical challenge, but as a result of recent developments relating to G (or generalizability) theory, it is now possible.

26 Evidence Now Available Journal of Applied Psychology Putka and Hoffman (2013): 633 candidates evaluated over three AC administrations in the USA; G theory used to decompose AC variance into 15 sources. Jackson, Michaeledes, Dewberry and Kim (2016): 698 candidates evaluated over five AC administrations in South Korea; G theory used to decompose AC variance into all 29 sources. In both cases the ACs studied were real-world, high stakes, very well designed and executed.

27 Source of Variance Percentage of Variance Putka and Hoffman (2013) Jackson, Michaeledes, Dewberry, and Kim (2016) N=633 N=698 General performance Candidates x exercises Candidates x dimensions Candidates x exercises x dimensions

28 Source of Variance Percentage of Variance Putka and Hoffman (2013) Jackson, Michaeledes, Dewberry, and Kim (2016) N=633 N=698 General performance Candidates x exercises Candidates x dimensions Candidates x exercises x dimensions

29 Implications 1. When ACs are used to predict outcome measures (e.g. future job performance), the attempted measurement of stable competencies does not contribute anything worthwhile. 2. There is no justification for using AC or DC competency scores to evaluate candidates, or to provide feedback to candidates. 3. Organizations which use AC scores on specific competencies when making selection or promotion decisions will make unnecessary errors. 4. Candidates who are not appointed/promoted because of specific AC/DC competency scores are being treated unfairly. 5. Organizations which continue to evaluate candidates with AC/DC competency scores risk litigation in the future.

30 Three Assumptions Underpinning the Use of Competencies in ACs A. There are systematic differences between people on specific performance-related behavioural dimensions (e.g. some people have more customer focus than others). B. People manifest these differences when they perform AC exercises. C. Trained assessors are able to observe and measure these differences when they observe AC exercises.

31 Putka and Hoffman (2013) Jackson, Michaeledes, Dewberry and Kim (2016) High stakes selection Yes Yes Number of administrations of AC 3 5 Number of candidates Where administered USA South Korea Dimensions based on job analysis and interviews with SMEs Number of exercises 3 to 4 3 Number of dimensions 10 3 to 5 Assessors per participant per exercise 2 2 to 3 Number of rating items for each dimension on each exercise Yes Yes 1 7 to 11 Behaviour-descriptive rating scales Yes Yes Assessors rate candidates independently Assessors trained by expert IO psychologists Training time for each assessor 32 hours 2 days Yes Yes Yes Yes