Attract pool of applicants. Job Analysis. Recruitment Select out unsuitable applicants. Choose selection methods. Create person specification

Transcription

1 GP and Specialty Selection Professor Fiona Patterson Work Psychology Group & University of Cambridge Professor Jane Dacre University College London

2 Overview Principles of best practice selection Defining selection criteria & the criterion problem Accuracy of different selection methods Overview of selection methods Situational Judgement Tests GP & Specialty Selection Selection centres AoMRC Clinical problem-solving test pilot

3 Designing selection systems the function of selection is to provide a means of estimating candidates aptitude so that the most suitable applicants are appointed. Selection uses a predictivist paradigm. In exams, end of training capability is tested & in theory, all applicants can pass Selection aims to predict trainability, i.e. identify those who will successfully complete training, before it commences. Parameters for designing & validating selection systems are different to other assessment settings

4 The Selection Process Job Analysis Create competency model Attract pool of applicants Tasks, roles, responsibilities Competency analysis Required level of competence Create person specification Knowledge, skills, abilities, characteristics Identify selection criteria Choose selection methods Interviews, Simulations, Selection centre Reliability, validity, financial utility, fairness Recruitment Select out unsuitable applicants Selection Select in and rank Aptitude? Attainment? Validate selection decisions Make selection decisions Patterson & Ferguson, 2007.

5 What attributes are important to be an effective clinician? How are they best tested at selection? Beyond some basic assessment is a lottery the best option?

6 Selection method Evidence for Validity Applicant Reactions Extent of Use Structured interviews High Moderate to positive High Cognitive ability High Negative to moderate Moderate Situational Judgement Tests High Moderate Moderate Personality tests Moderate Negative to moderate Moderate Biodata High Moderate Moderate Work sample tests High Positive Low Assessment centres High Positive Moderate Graphology Low Negative to moderate Low References Low Positive High Lottery N/A Negative Low

7

8 Innovations in selection methods Situational Judgement Tests

9 What s a Situational Judgement Test? Situational Judgement Tests (SJTs) are a measurement method designed to assess judgement in work-relevant relevant situations: Present challenging situations likely to be encountered at work Make judgements about possible responses Scored against pre-determined key Validity (predictive & incremental) is well-establishedestablished Recent examples in postgraduate specialty selection GP, CMT, Surgery, Acute specialties, Public Health

10 Example SJT item You are reviewing a routine drug chart for a patient with rheumatoid arthritis during an overnight shift. You notice that your consultant has inappropriately prescribed methotrexate 7.5mg daily instead of weekly. Rank in order the following actions in response to this situation (1= Most appropriate; 5= Least appropriate). A B C D E Ask the nurses if the consultant has made any other drug errors recently Correct the prescription to 7.5mg weekly Leave the prescription unchanged until the consultant ward round the following morning Phone the consultant at home to ask about changing the prescription Inform the patient of the error

11 Medical Education, 2009

12

13 Job families of SJTs Job family 1. Interventionist Surgery, Radiology, O&G SJT Pilot studies Surgery SJT pilot (Rowley et al) Radiology SJT evaluation (Lindsell et al) 2. Family/Community-based GP, Medicine, Psychiatry, Paeds GP SJT (Irish, Jones, GPNRO) CMT SJT evaluation (Berkin, Burr, Lazell et al) Public Health pilot (Pashynan, Williams) 3. Acute Anaethesia, Emergency medicine Acute specialties SJT pilots (Gale, Anderson, Carr et al) 4. Laboratory-based Histopathology

14 GP selection GP National Recruitment Office

15

16 GP Selection 6,500 applicants per year job analysis to develop selection criteria longitudinal validation studies analysis of applicant reactions Patterson F, Ferguson E, Lane P, Farrell K, Martlew J, Wells A. A competency model for general practice: implications for selection, training, and development. Br J Gen Pract 2000;50(452): Patterson F, Ferguson E, Norfolk T, Lane P. A new selection system to recruit general practice registrars: preliminary findings from a validation study. BMJ 2005; 330(7493):711-4

17

18

19 GP Specialty Selection National Deanery Longlisting Shortlisting Selection Centre Stage 2 Stage 3 Electronic Application Process Clinical Problem solving test Situational Judgement test Written Group Simulation Foundation Competency National panel Ranking Matching to Deanery

20 GP Selection System Predictive & construct validity studies Study 1. Predictive validity of short-listing Study 2. Comparing the CPS, SJT & cognitive ability tests Study 3. Supervisor ratings 1 year into training Study 4. MRCGP outcomes after 2-3 years in training

21 Study 1. Predictive validity Correlations N Correlations CPS & SJT ** CPS & Selection Centre ** SJT & Selection Centre ** Prediction of SC (N=6824) Adj R 2 CPS & SJT 0.28 R 2 change Incremental prediction of CPS over SJT.04** Incremental prediction of SJT over CPS.11** ** p<.001 CPS Clinical Problem-Solving test. SJT Situational Judgement test SJT is the best single predictor for short-listing Both tests are strong predictors & show incremental validity over the other

22 Study 2. Comparing the validity of CPS, SJT & cognitive ability tests

23

24 Study 2. Comparing the validity of CPS, SJT & cognitive ability tests Although the cognitive ability tests were correlated with SC outcomes, the current shortlisting tests remained the best predictors overall, particularly the SJT Candidate reactions to the cognitive ability tests were negative, especially for relevance & opportunity to demonstrate skills

25 Study 3. Predictive validity in training N=196 Mean Supervisor ratings after 12 months 24 items SD α CPS Correlations SJT SC **.50**.47** All methods (CPS, SJT & SC) are highly significant predictors of in-training performance Significant incremental validity accounted for by the SC for important domains (e.g. communication skills, empathy) Results indicate priorities for improving efficiency ** p<.001

26 Study 4. Long-term Predictive validity Mean Correlations N Mean SD CPS SJT Selection Centre* MRCGP Applied Knowledge Test 6 diets MRCGP Clinical Skills OSCE 5 diets 2, **.47**.29** 1, **.48**.35** * Uncorrected correlations for restriction of range ** p<.001 The GP selection methods are highly significant predictors of end-of-training competence

27

28 Speciality Selection

29

30

31 Specialty Specific Definitions of Communication Skills Paediatrics Communication Skills Is able to switch conversational style from adult to child, and to manage the doctor-patient-parent parent triad during consultation; engages child in conversation; positions self at child s level when speaking to child Anaesthetics Communication Skills Is able to verbalise intentions, explain actions to various people in a team (e.g. surgeons & ODAs); asks open questions at pre-med to ascertain the facts - rather than just running through a checklist; can offer clear explanation quickly to patients with establishing an on- going doctor-patient relationship Obstetrics & Gynaecology Communication Skills Is able to discuss psycho-sexual sexual problems with patient; uses open questions or puts feeler questions out to patients who may be shy / embarrassed to discuss symptoms; uses cues (such as fetal heart monitor) to illustrate explanations

32 Job analysis to define selection criteria for Surgery Technical Knowledge & Clinical Expertise Personal Attributes Leadership & Team Involvement Communication Skills Judgement under Pressure Decision Making Situation Awareness Problem Solving Organisation & Planning Professional Integrity Learning & Development Practical skills

33 Key messages 1. Research evidence shows there is more commonality than difference between the specialties eg. communication skills, professional integrity, problem-solving 2. Research shows different priorities between specialties at selection Paediatrics: O&G: Anaesthesia: Communication skills, Empathy Professional Integrity, Empathy Vigilance & Situational awareness

34 AoMRC MMT Pilot Investigating: 1. If a single test could work for a range of specialties 2. If several specialties & deaneries can collaborate 3. Which test format is best - a CPS, an SJT or both 4. If computer based testing makes sense with respect to cost logistics stakeholder acceptability

35 AoMRC MMT Pilot Multispecialty/multideanery Computer based CPS & Acute Specialties SJT Anaesthetics, ACCS, GP, Histopathology, Medicine, & Paediatrics 25 locations, over 2 days N= 682 approx

36 Selection centres

37 Origins & evidence When designed appropriately SCs are the best predictor of future work performance for non-cognitive skills Reliability & validity gains - combining different selection tools, standardised scoring systems SCs uses job-relevant exercises, observation by independent, trained assessors Recent reviews estimate over 95% of large organisations employing more than 10,000 individuals, use SCs for graduate recruitment & almost all high stakes selection used SC techniques

38

39

40 Acceptability & candidate reactions

41 Candidate Reactions: Percentage agreement reported for each method S Listing N=5866 Selection Centre N=4631 S Listing N=2894 Selection Centre N=3571 CPS SJT Sim Written Group CPS SJT Sim Written Group Relevant 84% 52% 89% 71% 78% 90% 64% 85% 64% 76% Fair 78% 50% 89% 75% 81% 86% 53% 86% 75% 81% Appropriate 78% 57% 91% 80% 83% 86% 69% Opportunity 56% 38% 75% 47% 49% 73% 35%

42 Candidate Reactions: Percentage agreement reported for each method S Listing N=5866 Selection Centre N=4631 S Listing N=2894 Selection Centre N=3571 CPS SJT Sim Written Group CPS SJT Sim Written Group Relevant 84% 52% 89% 71% 78% 90% 64% 85% 64% 76% Fair 78% 50% 89% 75% 81% 86% 53% 86% 75% 81% Appropriate 78% 57% 91% 80% 83% 86% 69% Opportunity 56% 38% 75% 47% 49% 73% 35% Patient simulation viewed positively

43 Implications High fidelity assessments (valid, but costly) are rated as significantly more relevant & fair by candidates Introduced methods beyond academic qualifications to select doctors (eg. interpersonal skills, integrity, empathy) Disjoint between operational validity vs. stakeholder acceptance ie. - Why shouldn t climbing Everest count? Political validity important concept in this context

44 Summary Undergraduate and specialty postgraduate selection may have different goals - undergraduate courses selecting primarily on academic ability, with a focus on passing the course and postgraduate selection focuses more on job-fit it cannot necessarily be assumed that those with high academic ability alone can be turned into potentially good doctors via medical training - other skills & qualities may need to be present from the start.

45 Thank you