Capacity to Train: An overview

Transcription

1 Capacity to Train: An overview 2017

2 Contents Contents... 2 Executive Summary... 1 Selection into Basic Training Project... 4 Findings and outcomes... 4 What s next... 5 Computational Modelling Project... 7 Findings and outcomes... 7 What s next... 8 Appendix

3 Executive Summary Purpose This paper provides a summary of the RACP Board Capacity to Train initiative ( ) aimed at addressing emerging capacity to train pressures. It summarises the objectives, approach taken, and key activities and learnings from two projects: the selection into training quality and feasibility study, and the computational modelling project. It also identifies the next steps to be taken to progress the outputs from this important work. Background For a number of years the RACP has grappled with how to best address emerging training capacity pressures. In July 2015 the Capacity to Train Strategy highlighted a wide range of stakeholder perspectives on factors affecting capacity to train from which two priorities were identified, both key components of the RACP Education Renewal Program: the need for the College to introduce a robust, merit-based selection process for entry into physician training the need to effectively balance the quality of training experiences and outcomes with the number of trainees in health services. Project objectives Initiated by the RACP Board in 2015 and continuing throughout 2016, the objectives of the Capacity to Train strategic projects were: 1. Selection into training quality and feasibility study: Develop a robust model of selection to identify candidates who are best suited for basic physician training. 2. Computational modelling project: Develop a decision support package to assist health services optimise Basic Physician Training capacity at the individual training setting level and provide evidence of training capacity at the College level. Our approach We engaged professional expertise to ensure that outputs were informed by sound industry practice, and adopted an evidence-informed approach to co-design solutions to capacity to train. Extensive stakeholder consultation was integral to the projects, with a large crosssector reference group convened to facilitate design input and consultation with key partners. Members of the expert panel for the Selection into Basic Training Quality and Feasibility Study are world leaders in the field, who were engaged for their experience with selection processes and understanding of contextual factors. KPMG was engaged to support the computational modelling project due to its demonstrated consultancy skills and workforce statistical modelling capabilities. KPMG was selected as the preferred consultancy through a competitive tendering process. The cross sector reference group included 80 participants representing each state and territory in Australia, and New Zealand. Representatives were drawn from key College committees, Directors of Physician Education and supervisors. Representatives from the wider health sector included state and Commonwealth Ministries of Health, other Medical Colleges, health consumer organisations, prevocational training organisations, medical student associations, and several state health representatives. 1

4 Selection into basic training quality and feasibility study We partnered with three experts in postgraduate selection in medical education partnered to explore possible models and methods of selection into physician Basic Training. Consulted stakeholders and the expert panel highlighted a preference for a three-stage model comprising: Stage one: RACP administered eligibility check Stage two: RACP selection assessment a situational judgement test combined with a clinically focussed problem solving test). Stage three: local training networks or programs deliver additional selection methods, for example a structured interview, to inform local employment and selection into training decisions. Any selection method or process used in stage three must adhere to the RACP Selection Policy and Standards for selection. The expert panel made a series of recommendations to guide further design, development and implementation of the new three-stage selection process. The Board has approved a detailed business plan to progress the design and development of the approved three-stage model and is now starting to deliver these activities. Important next steps include developing selection criteria and a guide for training providers on best practice local selection methods. Computational modelling project The RACP partnered with KPMG and a diverse range of stakeholders to explore the use of computational modelling in managing capacity to train. Drawing on insights yielded from literature, multi-level health service consultations and a comprehensive survey of supervisors, a conceptual framework for capacity emerged. This framework acknowledges that capacity to train is a complex interplay of six overarching factors with numerous levers and controls operating within and outside the day to day environment in which physician training occurs. Using KPMG s expertise and stakeholder input we developed a prototype computational model of Basic Training capacity. This site level modelling tool provided insight into trainee numbers among accredited training settings, and projected growth in clinical activity. There were a number of limitations identified with the computational model. In particular, the model did not provide a holistic view of capacity and was inconsistent when it was applied across different training settings. A substantial program of related work is now underway with the Education Renewal program, and the RACP will embed capacity within these projects. The Education Renewal program will also include an advocacy strategy to bolster and ensure changes are implemented effectively.. There are four priorities for this reframed approach: capacity guidance for training settings data development to support future capacity monitoring and appraisal 2

5 policy and standards to support capacity strategic education advocacy. Conclusion The RACP has made strong progress to manage capacity to train, with both projects yielding important learnings and practical outputs. Though closely related to each other, these outputs will be developed further and implemented separately. In the short term, the RACP will publish an interim Capacity to Train Guide for training providers. A training provider guide will also be developed for local selection activities. In the longer term, the new RACP accreditation system will include processes to monitor and manage selection into training and capacity to train. 3

6 Selection into Basic Training Project The RACP engaged three consultants with recognised expertise in selection into postgraduate medical training to collaborate on the Developing a Good Practice Model for Selection into Basic Training: A quality and feasibility study. The project: explored current recruitment and selection practices within the RACP carried out a literature review explored key design factors from a system change perspective developed two potential models of selection into RACP Basic Training included extensive, staged consultations with stakeholders throughout Identified a series of recommendations to inform future design and development of a selection process for Basic Training. Findings and outcomes Training is different to work. This is acknowledged by our accreditors who set rigorous standards for Colleges as training providers of specialist medical education. Regulatory requirements include standards on the establishment of fair and transparent methods of selection into training. In collaboration with employers and other key stakeholders, through this study the RACP codesigned a competency-based approach to selection. A three-stage model was chosen by stakeholder and the expert panel to select trainees best suited for Basic Training, following RACP requirements, as defined in the RACP Professional Practice Framework. The model comprises: Stage one: RACP administered eligibility check Stage two: RACP selection assessment (a situational judgement test combined with a clinically focussed problem solving test). Stage three: local training networks or programs deliver additional selection methods, for example a structured interview, to inform local employment and selection into training decisions. Any selection method or process used in stage three must adhere to the RACP standards for selection, once established. 4

7 The project resulted in a series of recommendations from the expert panel to guide the next steps in design, development and implementation of a new selection process for RACP Basic Training. Broadly these recommendations covered the following areas: What s next In October 2016, the RACP Board endorsed the recommendation to undertake detailed planning for the design, development and implementation of the three-stage model for Selection into Basic Training. A business plan to progress this work was approved in May The RACP is now delivering these activities. This includes forming a Project Control Group and Reference Group. It also involves developing the project scope, scheduling and communication. We have started developing selection criteria linked to our Curricula Renewal project. Once these criteria are developed we will consult with stakeholders. We also intend to develop a 5

8 guide to support implementation of a selection policy and principles during the stage three local selection methods. This will provide guidance on best practice on selection into training and the use of selection methods. As the RACP implements other assessment methods (situational judgement tests and clinical problem solving tests) the guide will be expanded to include how these sources of information should be used alongside other methods of selection (for example interviews and references). A systematic review of worldwide practice in selection into training in postgraduate specialist training led by the RACP will be published by Best Evidence Medical Education (BEME). This provides an evidence-base confirming that the proposed approach to selection into training aligns to best practice. 6

9 Computational Modelling Project The RACP contracted KPMG to conduct a Capacity to Train Computational Modelling Project, a novel and exploratory attempt to produce an evidence-based tool to determine optimal numbers of basic physician trainees at both the local training setting and College level. The project included the following: a literature review a supervisor survey focussed on capacity to train the development of a computational model of capacity to train the development of supportive guidance for training settings regarding capacity to train extensive, staged consultations with stakeholders throughout. Findings and outcomes Six key factors and local planning processes affect capacity to train: Clinical/work activities Educational leadership and supervision Organisational culture and administration Responsible planning Assessment practices Profile of trainees and support Educational resources and services Some of these factors are easy to measure and others are much more difficult to analyse in a valid and reliable way that can be incorporated into mathematical models. Similarly, some of these factors are easier to influence than others. The project findings suggest that capacity can be improved through smart, pragmatic education design and implementation, and systemically sustained through supportive health service cultures and resourcing. A prototype computational model for Basic Training capacity was developed. This site level modelling tool provided insight into trainee numbers among accredited training settings, and projected growth in clinical activity. 7

10 The modelling work suggests that the ratio of occupied bed days (one health service derived measure for clinical/work activity) to Basic Training rotation days varies across Divisions, states and countries, as summarised in Table 1. Hospital peer group was also found to be a significant mediator of results. Table 1: Summary results from capacity peer comparator modelling undertaken to date Country Coefficient Explained variation Adult Medicine Australia 37.8 training days/1000 OBDs 88.0% NZ 64.9 training days/1000 OBDs 92.9% Paediatrics & Child Health Australia training days/1000 OBDs 86.0% NZ 51.0 training days/1000 OBDs 80.5% There were a number of limitations identified with the computational model. In particular, the model did not provide a holistic view of capacity, and there were data discrepancies and inconsistencies when the model was applied across different training settings. As a result, the computational model is not yet fit for implementation. What s next The RACP has adopted an integrated approach to optimising capacity to train, and as part of the RACP Education Renewal Program we will include an advocacy strategy to bolster and ensure changes are implemented effectively. The four priorities for the project are: 1. Develop guidance for training settings The PRIME (Planning Responsibly in Medical Education) capacity guide will be developed to support health services to make informed decisions about the optimal number of Basic Physician Trainee positions they can support without compromising the quality of education standards and experiences. An Interim PRIME Guide is attached. 2. Develop data to support capacity modelling This work will be undertaken in conjunction with the Accreditation Review. 3. Ensure our policies and standards support capacity. 4. Advocate for education We will develop an advocacy plan to support activities that position medical education as a core function of health services. An At a Glance Guide to the Capacity to Train Strategy is included in Appendix 1. 8

11 Capacity to Train Capacity to train is the combined ability of healthcare and education systems to provide trainees with the skills they need to become competent independent physicians. Exploring the issues Magic numbers, wicked problems In 2016, we partnered with KPMG and a diverse range of stakeholders to explore the issue of capacity to train. Drawing on a literature review, multi-level health service consultations and a comprehensive survey of supervisors, we developed a conceptual framework for capacity, which acknowledges that capacity to train is complex. There are six overarching factors with numerous levers and controls operating within and outside the day-to-day environment in which physician training occurs. We were asked by stakeholders to help manage capacity and provide a tool to calculate the optimal number of trainees that training settings could train effectively with current resources. In response, we worked with KPMG to develop a prototype computational model. This provided insight into current trainee numbers among accredited training settings and projected growth in clinical activity. We encountered a number of issues with this approach which have informed our decision that a computational model is not the best solution to optimise capacity at this point: Education resources and services Profile of trainees and support Clinical and work activities Assessment practices data for modelling is scarce, variable in quality, and includes limited factors related to capacity because modelling is based on observed relationships, we could only model current practice, rather than an ideal picture of capacity local practices are heterogeneous, making outputs imprecise when applied at a local level an holistic model of capacity is essential, otherwise unintended negative effects may occur when outputs are used to inform health service funding mechanisms. Education leadership and supervision Organisational culture and administration Findings from the 2016 Supervisors Survey Supervision load Average supervisor formally supervises five RACP trainees and another seven informally. Supervision time Average supervisor spends two days per week supervising, with one day on non- clinical supervision activities. Training days per 1,000 Occupied Bed Days at hospital, on average Adult Medicine: 38 days (AUS), 65 days (NZ) Paediatrics: 140 days (AUS), 51 days (NZ) Employment Major city (77%) Acute hospital (87%) Public hospital (87%) Permanent employee (84%) Single training setting (75%) No admin support for education (64%) Supervision in position description (50%) Top three challenges 1. Insufficient time for supervision due to competing demands (70%) 2. Lack of hospital support or resources for education (45%) 3. Insufficient time due to number of trainees (28%)

12 Capacity to train: it s everyone s business It is clear that capacity to train has an impact on the quality of health service environments. The design and implementation of education, resources, funding and organisational culture all have a role to play. The RACP will do its part to address capacity through its Education Renewal Program, which includes renewed curricula that follow a competency-based medical education model, increased support for supervisors, renewed accreditation processes, and a selection into training system. Capacity to train is embedded in a number of our Education Renewal projects: Curricula Renewal The renewed curricula will define the competencies Basic Trainees must develop and identify clinical and work activities to support them, define milestones for training progression and outline trainee responsibilities commensurate with competence, and help make workbased assessment programs more meaningful. Accreditation Review The new accreditation system will clarify and monitor education activities and resources provided by training settings and networks to support leaders to advocate and meet accreditation standards; clarify and monitor the impact of training setting culture, governance, management and support of training on quality of training experiences and outcomes, through training setting visits and responsive feedback; and ensure that trainees are exposed to an appropriate range of clinical and work activities. Supervisor Support The new Educational Leadership and Supervision Framework will grow and retain the supervisor workforce and clarify what is expected of supervisors; define the resources required to provide supervision, including trainee to supervisor ratios and role FTEs ; and provide support and professional development for supervisors. Selection into Basic Training The new Selection into Basic Training system will ensure candidates best suited to physician training are selected. College Learning Series A new online learning series will enrich trainee learning with content aligned to the revised Basic Training Curricula. Priorities Capacity guidance Data development Policy and Standards Education advocacy A Planning Responsibly in Medical Education (PRIME) Capacity Guide is being developed to support settings to conduct selfappraisals, inform responsible planning, support discussions with health service managers, meet the new accreditation standards and identify ways to enhance capacity We will develop reliable, longitudinal, targeted data that allows us to monitor and predict capacity. New processes and technologies will be introduced through the Accreditation Review, which will allow us to gather data, including annual training setting reports, trainee and supervisors surveys. We will then re-evaluate the potential for quantitative capacity guidance. Capacity will be embedded in our renewed education policies and standards. Stakeholders agreed that training settings should contribute Clinical Examination capacity at the rate they use it. Policies and standards will be renewed to optimise examination capacity Decisions made outside the control of supervisors and the RACP have significant impact on capacity. In order to embed sustainable change, medical education needs to be positioned as a core function of health services. A strategic advocacy plan and approach to stakeholder engagement will be developed to support education renewal.