Competence by Design-Residency Education: A Framework for Program Evaluation

Transcription

1 Competence by Design-Residency Education: A Framework for Program Evaluation November 2017 Authors: Elaine Van Melle, Jason Frank, Stacey Brzeznia, Lisa Gorman To be cited as: Van Melle E, Frank J, Brzeznia S and Gorman L Competency by Design-Residency Education: A framework for program evaluation. Ottawa, ON: Royal College of Physicians and Surgeons of Canada. CBD RE: Program Evaluation Framework - 1

2 Changing Context of Residency Education Royal College Strategic Goal To improve the health and healthcare of Canadians Changing patient care needs & healthcare system KEY OUTCOMES Long-term 7-10 yrs Enhanced patient care outcomes Medium-term 4-6 yrs Enhanced readiness for practice Sustainability of program changes Short-term 1-3 yrs Assessment systems & practice that promote the progressive development of competence Teaching that is individualized to the needs of the Learners who demonstrate the capacity for self-regulated & life long learning OUTPUT A logic model can also be used to focus evaluation Specialty training programs where all curricular elements teaching, learning activities. and assessment Ca practices are explicitly & systematically sequenced to promote the developmental acquisition of competencies required for practice. Advances in educational theories & practice ACTIVITY Competence by Design: Competency based - Residency Education Multi-year change process aimed at implementing a competency-based medical education approach to residency training across Canada. INPUTS Royal College Key initiatives Specialty education design process Assessment strategies Change strategy Faculty development Accreditation reform CanMEDS 2015 framework Royal College has: Control Direct Influence Contributing Influence Figure 7 A Competence by Design Residency Education Program Logic Model CBD RE: Program Evaluation Framework - 2

3 Based on this logic model we can now explicitly state an overall Theory of Action or how the CBD-RE program model is intended to bring about the desired change [29]. Together with the logic model this statement provides a basis for creating a shared understanding of the general mechanism by which the CBD-RE program model is supposed to work. In order to create a shared understanding, both the logic model and theory of action statement must be treated as works in progress and therefore always open to scrutiny and revision as an increasing number of stakeholders engage in program evaluation. Theory of Action: CBD-RE Program Model By systematically aligning an explicit sequence of learning experiences, instructional methods and assessment practices, with competencies required for practice, Competence by Design Residency Education enables the development of learners who are better prepared to enter into practice & to provide quality patient care. CBD RE: Program Evaluation Framework - 3

4 CBD-RE program evaluation goals Using the logic model as a guide, there are three key goals of the overall CBD-RE program evaluation. 1. To foster successful implementation of CBD-RE Implementation affects outcomes [33] and so it is critical to build an understanding of both the strategies that support, and those that detract, from the capacity to successfully implement CBD-RE. Scaling up an intervention is not a simple process [34]. It will be important to study strategies at the individual program level and to use these findings to improve the system-wide change initiatives. Understanding organizational or program readiness for change is also an key aspect of fostering successful implementation. As a multi-faceted construct, organizational readiness for change refers to the shared resolve to pursue what is required in order to implement CBD-RE [35]. Shared resolve is critical in this change initiative since implementing CBD-RE involves collective action between the RC and the 17 medical schools including the postgraduate offices, specialty program leadership, residents and faculty. As illustrated in the logic model, the RC only has direct control over their own key initiatives. Full implementation of CBD-RE is dependent upon collaboration amongst all of the stakeholders. Using a multi-stakeholder, collaborative process, creating a readiness checklist has been an emergent focus of activity for the RC change initiative. Formalizing this activity as part of the evaluation framework could prove to be invaluable in understanding rates of adoption and degrees of implementation across different settings [36]. 2. To understand the influence of local contexts, adaptations and innovations Interventions are frequently modified or adapted during the adoption process [37]. These adaptations reflect the influence of local context such as the availability of resources and/or expertise to support the intervention. In fact, in a recent study of the implementation of competency-based education in family medicine across Canada, a recurring theme was the differences in implementation across sites [38]. In particular, size of the community, academic versus non-academic sites, and the extent to which competency-based features were already in place influenced implementation. Understanding the fidelity of an intervention, or the degree to which a program has been implemented as intended, is a critical step in making a viable assessment of the program contribution to outcomes. Without such an evaluation it cannot be determined if a lack of impact is due to poor implementation or inadequacies in the program theory [38]. Notably, CBD-RE is not intended to be a one size fits all intervention. It is expected that there will be adaptations and innovations created in to response to features of the local specialty program context. Assessing fidelity therefore, will need to take into account the extent of the changes being made to the CBD-RE program model, and if there is a point where an adaption or innovation has actually compromised the fundamental principles of CBD-RE [39]. Indeed, a study evaluating K-12 competency education in New England, described implementation as a developmental continuum with certain tipping points along the journey to competency based education [40]. Furthermore, a debate exists as to whether CBD-RE will actually transform residency education or if CBD-RE simply formalizes how faculty intuitively approach teaching and assessment. Consequently, documenting the CBD RE: Program Evaluation Framework - 4

5 extent and quality of the change being experienced through CBD-RE will be an important focus for program evaluation. Accordingly, it is important to study local configurations of CBD-RE as well as describing and disseminating leading innovative practices. 3. To build an evidence-base of the impact of CBD-RE over-time As described, the impact of CBD-RE is critically dependent upon the quality of implementation and local context. Under these conditions [41] calls for a shift from creating specific proof that an intervention works (or doesn t) towards a generation of rich understandings. Such understandings can then form a detailed and practical description of what is it about this kind of intervention that works, for whom, in what circumstances, in what respects and why? [42]. a. To document short, medium and long-term outcomes A goal then of the CBD-RE program evaluation framework is to build an evidence-base linking the activities of CBD-RE to short, medium and long-term outcomes. A key focus will be describing the contribution of a particular CBD-RE activity to an observed outcome [43]. Over-time, such empirical studies can be synthesized to enhance our theoretical understanding and to provide deeper insights into CBD-RE and how it can be made to work most effectively [42]. b. To describe unintended outcomes As previously described, given the number of interacting activities, there is a potential for unanticipated outcomes to emerge over the life of the RC CBD change initiative [26]. Indeed, it is speculated that implementing CBME could possibly lead to the need for additional time and resources [44]. A concern that detailed specification of competencies may result in an overwhelming number of assessments has also been expressed [45]. It is indeed possible that implementing CBD-RE could lead to unintended outcomes and so any program evaluation must be vigilant in capturing these impacts. In doing so, it will be important to acknowledge that unintended outcomes can be positive as well as negative. For example, a recent study found that faculty were more engaged in teaching as a result of implementing a competency-based program [46]. Ferguson et al (2013) [47] also found that residents were much more efficient while undertaking service work in a competency-based curriculum. Consequently, any emergent outcomes should be captured documented as part of the CBD-RE program evaluation initiative. CBD RE: Program Evaluation Framework - 5

6 CBD-RE program evaluation guiding principles and process Each of the program evaluation goals will require a series of projects to be undertaken over the life of the project. The following section describes the overall governing structure and process that will be used to identify and organize these projects. This section provides principles that should be commonly applied across all program evaluation efforts. Guiding Principles Collaborative approach There are numerous stakeholders involved in implementing the CBD-RE program model (see Appendix C). Collaborative evaluation is defined as an approach that actively engages program stakeholders as members of the evaluation team to the extent that they are willing and able [48]. A collaborative approach therefore, requires active engagement of stakeholders as members of the evaluation team in many different roles e.g., to design instruments, collect and analyze data and to share findings. A collaborative approach has many benefits such as enhanced buy-in and participation in the change effort. The primary goal however, is to ensure that evaluation efforts are used to improve programs. Given the effort required to evaluate the CBD-RE program implementation over the long term, it is critical to ensure that the findings are actually used. The importance of using evaluation findings is further reinforced in the next principle. Utilization-focused Ensuring that evaluation findings are actually used is the primary focus of utilizationfocused evaluation [26]. Any evaluation therefore, must explicitly involve primary intended users. Primary intended users are a specific sub-group of stakeholders. Stakeholders are broadly defined as individuals, groups, or organizations that can affect or are affected by an evaluation process and/or it s findings [49],p.1). Within this broad definition, primary intended users are those individuals or groups of people who personally care about the evaluation and the findings it generates and are in a positon to ensure that the findings can be put into practice [26] p. 65). When identifying collaborators for an evaluation priority, it will be particularly important to ensure that primary intended users are actively engaged. Theory-informed As described previously in this document, the CBD-RE program model is an evidence informed innovation in that it draws from advances in educational theories (see page 6). As a complex innovation with many activities, there is however, a range of theories to draw from. The challenge then is to identify the important theory or theories underlying the particular evaluation question under investigation and in doing so, make explicit how the CBD-RE program activities are intended to lead to outcomes. When examined against this backdrop, evaluation findings can serve to illuminate how CBME theories actually translate into practice and provide guidance for required changes in practice or point to areas where theory may be lacking [50]. In particular, the theory of action (see page 19) can be used as an umbrella statement to help shape more specific if-then propositions relevant to a particular evaluation project [51]. CBD RE: Program Evaluation Framework - 6

7 Driven by program evaluation standards This evaluation framework describes a targeted effort to gather data that will help to make decisions regarding the implementation, sustainability and impact of the CBD-RE program model over time. As described, the focus is on understanding process, outcomes and perhaps most importantly, the relationship between process and outcomes over time. The focus on systematic use of the findings differentiates this initiative from a research project (see Table 4). As well, the focus on both outcomes and process for a specific program over time differentiates this program evaluation from a quality improvement effort. Clearly, there can be overlap between research, program evaluation and quality improvement. For example, all three rely on the use of both quantitative and qualitative methods to gather data. The approach to study design however differs (see Table 4). Given the focus on decision-making, program evaluation allows for technically adequate designs in order to allow for timeliness of data collection. For example, if residents are graduating in one month and there is not time to create a perfect survey tool then tool that is technically adequate can be used. Accordingly, the program evaluation standards were created to maximize credibility of the evaluation process (see Table 5)(insert reference for PE standards here). These standards will be used to guide specific evaluation efforts. Table 4 Program Evaluation, Research and Quality Improvement Purpose Focus Research Program Evaluation Quality Improvement To generate new knowledge Theory-based hypothesis or research question To provide information for decision-making in relation program processes and outcomes Questions regarding program improvement, merit and worth Audience Other researchers Internal and external stakeholders Methods Designs that support generalizability or transferability of findings to other contexts Timeline Outcome Dissemination of findings Based on researchers timeline and available funding Contribution to general body of knowledge Research Publication Technically adequate designs that match the purpose of the evaluation Bounded by organization s requirements for data to support decision-making Improvement in program design and understanding of program outcomes Program Improvement/ Technical Report To improve internal processes for a specific intervention The assessment of an existing practice Internal stakeholders Designs that allow for clear and easy recording and interpretation of information Short timeline that supports immediate change Change to practice Report and/or feedback to those responsible for managing the practice adapted from: Department of Evaluation and Research Services, Frazer Health. research.fraserhealth.ca/media/ research_qi-program_evaluation _Differentiation.pdf CBD RE: Program Evaluation Framework - 7

8 Table 5 Program Evaluation Standards Standard Utility Feasibility Propriety Accuracy Accountability Description Ensures that program stakeholders find evaluation processes and products valuable in meeting their needs. Describes evaluation effectiveness and efficiency. Considers what is proper, fair, legal, right and just in evaluations. Outlines dependability and truthfulness of evaluation representations, propositions, and findings, especially those that support interpretations and judgments about quality. Describes adequate documentation of evaluations and a metaevaluative perspective focused on improvement and accountability for evaluation processes and products. Using a systematic, iterative proces The following cycle has been developed specifically to put these principles into practice (see Figure 8). The overall priorities of the evaluation effort will be determined by the governing body as described in the following section. Once determined, the list of stakeholders will be consulted (see Appendix C) and relevant primary stakeholders will be identified and engaged. In addition, as indicated in Appendix C how to best involve those stakeholders who play a specific role in linking the CBD-RE change initiative with changes in approaches to residency education will be considered. Described as boundary spanners these individuals are significantly influential in program implementation and therefore must be meaningfully engaged in program evaluation projects. Collaboratively, these stakeholders will describe theories informing the evaluation focus and create an explicit statement e.g., theory of action, or description that shows how theory informs the area of investigation. The evaluation design will be determined by the nature of the questions. In this fashion, the evaluation framework does not presuppose the strength of any particular methodology or (e.g., qualitative, qualitative or mixed) but rather subscribes to the notion that the evaluation design should maximize capacity to illuminate the question. Every effort should be made to ensure that the evidence gathered and lessons learned are disseminated in a timely and meaningful fashion. Consequently, multiple avenues should be used to communicate and disseminate findings in addition to formal reports. Ultimately, the effectiveness of the evaluation efforts will be determined by the extent to which findings are used to inform and improve the CBD-RE change initiative. CBD RE: Program Evaluation Framework - 8