Predictive validity of recruitment into public health specialty training in the UK

Transcription

1 Predictive validity of recruitment into public health specialty training in the UK Brendan Mason, Assistant Academic Registrar. Selena Gray, University of the West of England. Nora Pashayan, University College London.

2 Background GMC require recruitment process to be open, fair and effective Evaluation required to demonstrate effectiveness recruitment process Predictive validity of selection process measure of effectiveness Recruitment critical control point in speciality training Correct decision in best interest applicant / scheme / public health

3 Recruitment process Eligibility checking Assessment centre (AC) Rust Advanced Numerical Reasoning Appraisal (RANRA) Watson-Glaser Critical Thinking Appraisal (WGCT) Bespoke Situation Judgement Test (SJT) Selection Centre (SC) Interview panels x6; group exercise; written exercise Total Recruitment Score (TR)

4 Recruitment process Eligibility checking Assessment centre (AC) Rust Advanced Numerical Reasoning Appraisal (RANRA) 25% Watson-Glaser Critical Thinking Appraisal (WGCT) 25% Bespoke Situation Judgement Test (SJT) 50% Minimum score each component => weighted ranking Selection Centre (SC) Interview panels x6; group exercise; written exercise Minimum score each area person specification & total score Total Recruitment Score (TR) Weighted ranking 10%RANRA + 10%WGCT+10%SJT + 70%SC

5 Methods Cohort all registrars took up appointment Anonymous record linkage Interim analysis follow up to December 2014

6 Methods Predictor variables Standardised score Rust Advanced Numerical Reasoning Appraisal (RANRA) Watson-Glaser Critical Thinking Appraisal (WGCT) Situation Judgement Test (SJT) Assessment Centre score (AC) 25% RANRA + 25% WGCT + 50% SJT Selection centre score (SC) Total Recruitment score (TR) 10% RANRA + 10% WGCT + 10% SJT + 70% SC

7 Methods Outcome variables Full pass first attempt MFPH Part A exam Pass first attempt MFPH Part B exam Satisfactory outcome all Annual Review of Competence Progression (ARCP) No recorded ARCP outcomes 2, 3, 4 or 5

8 Methods - Analysis Logistic regression Predictor variables categorical above and below median Adjustment for potential confounders age, sex, ethnicity, professional background Predictor variables continuous Receiver Operator Characteristic (ROC) analysis

9 Results Predictor variable Number registrars Mean (Standard deviation) Median Range RANRA (7) WGCT (6) SJT (4) AC (6) SC (5) TR (4)

10 Predictor variable Results Outcome Part A examination n=236 Pass Fail Odds Ratio 95% CI TR< TR Adjusted OR 95%CI OR-trend 95% CI

11 Predictor variable Results Outcome Part A examination n=236 Pass Fail Odds Ratio 95% CI TR< TR Part B examination n=188 TR< TR Adjusted OR 95%CI OR-trend 95% CI

12 Predictor variable Results Outcome Part A examination n=236 Pass Fail Odds Ratio 95% CI TR< TR Part B examination n=188 TR< TR ARCP n=212 TR< TR Adjusted OR 95%CI OR-trend 95% CI

13 Logistic regression Predictor variable Outcome Part A n=236 OR-trend 95%CI RANRA WGCT SJT AC SC TR

14 Logistic regression Predictor variable Outcome Part A n=236 Part B n=188 OR-trend 95%CI OR-trend 95%CI RANRA WGCT SJT AC SC TR

15 Logistic regression Predictor variable Outcome Part A n=236 Part B n=188 ARCP n= 212 OR-trend 95%CI OR-trend 95%CI OR-trend 95%CI RANRA WGCT SJT AC SC TR

16 Receiver Operator Characteristic (ROC) analysis Predictor variable Outcome Part A Part B ARCP AUC 95%CI AUC 95%CI AUC 95%CI RANRA WGCT SJT AC SC TR

17 Discussion Gold Standard of performance as a public health specialist Limitations Range restriction data Sample size (SJT since 2011, length follow up) Missing data General Practice recruitment Clinical Problem Solving Skill and SJT both strong correlation with exam pass rates and assessment of clinical competence in the workplace

18 Conclusion Good predictive validity higher selection scores clearly associated with passing exams and satisfactory ARCP outcome Overall score better predictor than individual components Individual parts testing different skills & abilities Together parts are providing additive value

19 Acknowledgement Registrars & Selectors All Staff who contributed from: Faculty Public Health Health Education England Individuals: Celia Duff, David Williams, Anna Koczwara, Fiona Patterson, Grant Fisher, Marcia Reed, Russell Amprofo

20 Additional slides NOT for presentation

21 Categorical Odds Ratio (above or below median score) Predictor variable Outcome Part A n=236 Part B n=188 ARCP n= 212 OR 95%CI OR 95%CI OR 95%CI RANRA WGCT SJT n= AC SC TR

22 Categorical (above or below median score) Adjusted (age, sex, ethnicity and professional background) Odds Ratio Predictor variable Outcome Part A n=236 Part B n=188 ARCP n= 212 OR 95%CI OR 95%CI OR 95%CI RANRA WGCT SJT n= AC SC TR