Similarities and Differences in Managerial Judgement Around the World: Developing an International SJT

Transcription

1 Similarities and Differences in Managerial Judgement Around the World: Developing an International SJT Alan Howard, Quest Assessments Ltd Max Choi, Quest Assessments Ltd Nina Krig, SHL Group Rachel Austen, SHL Group Simon Raymond, SHL Group The Occupational Psychology Conference, Stratford-upon-Avon, 12 Jan 2011

2 In a business meeting, introductions are taking place Which one of these would be considered unacceptable?: 1. Gulf Arabic male and Western male shake hands 2. Gulf Arabic female and Western male shake hands 3. Gulf Arabic female and Western female shake hands 4. Gulf Arabic male and Western female shake hands

3 JOOP Dec 2010 paper by Hetty van Emmerik et al N= +12,000 participants in 32 countries 3 Predictors of leadership behaviour Individual Differences = 79 93% of variance Organisational Oga sato a Differences ee = 5 11% %of the variance Country Differences = 2 10% of the variance

4 The Scenarios Managerial Judgement Test The first UK SJT launched in 1998 Suite of 3 tests: t Graduate, Managerial, Executive The Managerial Test used around the world although developed within the UK

5 Research Objectives To what extent do managers around the world make similar judgements as UKbased managers? Are there enough similarities in judgement around the world to develop a reliable and valid international SJT?

6 Test Data Analysis & Research N=+8,000 across 70 countries 8 Regions: Middle East, Australasia, Europe, Far East, US, Africa, Indian Subcontinent, & UK Across all industry sectors Private & public sector

7 How the Scenarios Test Works Scenarios which describes a management age e issue e.g. A boss who s under pressure to suddenly get the department s yearly plan produced within 2 weeks. Items relating to the scenario e.g. Spend an hour on it as yearly plans are never followed anyway 16 Scenarios with 84 scored items

8 Ratings Candidates rate each item: -3 Highly undesirable -2 Undesirable -1 Slightly undesirable +1 Slightly desirable +2 Desirable +3 Highly desirable There will be a correct answer e.g. -3

9 Item Analysis 54 of the 84 items (65%) worked relatively consistently across the regions 30 of the 84 (35%)items did not work consistently across the regions Some of the more interesting items?

10 Sample Item: Implement A as it will assist with B Correct Answer = -3 Pattern 1

11 Sample Item: To progress C, I would do D Correct Answer = -3 Pattern 2

12 Sample Item: In order to deal with this situation effectively, support your team to do E Correct Answer = -2 Pattern 3

13 Regions Researched Regions showing the greatest difference from the UK sample: Indian Subcontinent Middle East Africa (to be conducted)

14 18 Items for Further Research Greatest mean difference from UK sample To understand the differences in: - managerial judgement - culture - language or interpretation issues

15 Attributional Analysis Highly structured approach Telephone Interviews with volunteer managers Based on the Leeds Attributional Coding System (LACS) Munton et al, 1999 Volunteer managers responded to 6 items each

16 Telephone Interview Collected biographical information Each item rated and rationale for rating discussed Attributes of language, culture, & managerial judgement explored as appropriate

17 Analysis of the Telephone Interviews Interviews recorded d & structured notes taken following the LACS approach Focused on attributes of language. culture, and managerial judgement.

18 For Language: 1 = Language issues don t arise; the item is understood d as intended d 2 = Language g issues have some impact on the understanding of the item 3 = Language issues have a great impact on the understanding of the item Similar ratings for Culture Differences and Similar ratings for Culture Differences and Managerial Judgement

19 Analysis of the Telephone Interviews If a manager scores differently from the correct response how can we understand their score based on these 3 attributes? 40 interviews conducted: - 21 for Indian Subcontinent - 19 for the Middle East

20 Interview Research Findings Reminder that many items work this research focused on potential probiem items Helps us to understand the challenges of applying a SJT across different regions

21 Is the Indian Subcontinent Different from the UK? Language: No major language issues Culture: Some culture differences Example: Competition with peers seen as more healthy than the UK

22 64: Underneath it all, if you want to progress in the organisation, realise that you need to compete against your team colleagues in every way

23 Indian Subcontinent Another Example: More emotional attachment, t social moral responsibility, tolerance and support for individual team members.

24 49: Tell the rest of the team to get on with their job and suggest that they be a little more tolerant when the pressure is on

25 How different is the Middle East? Language specific words were not familiar to some e.g. demoted, frosty, disgruntled meanings of phrases or sentences were misinterpreted by some e.g. tell the rest of the team to get on with their jobs

26 How Different is the Middle East? Culture competition with peers seen as more healthy than in the UK, it means drive / energy, being active, company benefits as a result judgements reflect more respect for and more obeying of seniors judgements reflect the need to be more accommodating and tolerant, saving face

27 How different is the Middle East? You are relatively junior but drawing up proposals for a significant organisational change; you re under pressure from the affected senior managers 55. Implement some of the senior managers suggestions on the basis that they could assist you with your career advancement in the future

28 Research Results Of the 18 items qualitatively investigated 6 items work in a similar way to the UK 7 items may need minor re-wording 5 items had significant cultural or language issues and will be removed

29 Towards an International SJT Selecting items is a delicate balance! Don t want to lose something significant. Around 30% of UK test items dropped. Drop the item Language issues Cultural differences? It increases Mean differences between world regions Keep the item A more challenging Judgement item Increases test reliability Valid within each Region

30 Towards an International SJT smaller Means represent better performance UK Version Mean 84 items UK 91 Australasia 90 (-0.05) 05) Europe 95 (0.21) US 97 (0.32) Far East 103 (0.63) Africa 106 (0.79) Middle East 113 (1.16) Indian Subcontinent 116 (1.32) International Version Mean 67 items (-0.07) 07) 61 (0.14) 61 (0.14) 64 (0.34) 67 (0.55) 71 (0.83) 69 (0.69)* some DIF impact International (all 8 Regions) 103 (0.63) 64 (0.34)

31 Towards an international SJT Differing levels of leadership & management development around the world Region-specific norms; compare Middle East managers with the Middle East norm International norm; e.g. a Global company comparing UK, US, and Indian colleagues: use the International norm

32 Towards an International SJT Reliability: across the 8 Regions Validity: Region Correlation with Managerial Level Far East n=46 Africa n= (8% sig. level) 0.27 (<1% sig. level) Middle East n=96 030( 0.30 (<1% sig. level) Indian Subcontinent n= (<1% sig. level)

33 SIMILARITIES and Differences in Managerial Judgement around the world As long as the right items are chosen we can see many aspects of Managerial Judgement happening in similar il ways around the world. (i.e. big picture thinking, delegation, one-to- one and team management, career management and corporate protocol)

34 Summary Through the Scenarios test we identified some similarities and differences in managerial judgement around the world We are confident we can achieve a psychometrically robust international version of the Scenarios Managerial Judgement Test