A Roadmap for Authoring Performance Improvement Activities

Transcription

1 A Roadmap for Authoring Performance Improvement Activities

2 Introduction What Is Performance Improvement? The world of continuing medical education (CME) is changing. Not too long ago, many of those in the medical community were dissatisfied with the current state of CME. They complained that the types of education that were available did not actually help doctors become better at their jobs, as was the goal. The CME community responded by developing a new type of CME activity inspired by adult learning theory on performancebased learning, and even quality management practices in the manufacturing industry. These new performance improvement (PI) activities are designed to help doctors measure aspects of their performance and give them tools to improve that performance over time. The practical and relevant nature of PI activities has made them a popular choice for those needing to meet continuing medical education requirements. In this paper, we ll first discuss some different ways of thinking about performance improvement and look at what all PI activities have in common. Then we ll present a framework to help medical associations that are new to performance improvement to develop their own PI activities, spanning the development cycle from pre-development to evaluation. What do we mean when we talk about performance? According to the International Society for Performance Improvement, performance is equal to work activity plus results. In healthcare, the work of treating and caring for patients results in better patient health. So to improve performance, we need to either improve the efficiency of patient care or improve patient outcomes (or both). The modern medical performance improvement movement has its roots in work done in the 1920s and 30s by Walter Shewhart and W. Edwards Deming in an area they called applied science, which meant using scientific principles and methods to perform quality control and quality improvement in manufacturing. Shewhart and Deming s work in this area culminated in these three tenets for quality improvement: 1. Focus on meeting customer needs 2. Focus on processes 3. Use data to make decisions We can see the influence of these ideas on how we think about medical performance improvement today. For example, the American Medical Association defines performance improvement as the following (emphasis ours): a process by which evidence-based performance measures and quality improvement (QI) interventions are used to help physicians identify patient care areas for improvement and change their performance. In summary, good PI activities help physicians improve their performance by using practice data to identify process changes that will result in real benefits for patients.

3 What Is Performance Improvement? PI-CME Development Roadmap Performance improvement programs are now offered by various types of organizations, including medical associations, medical specialty boards, and individual hospitals. While the delivery format of these offerings can vary, the PI activities themselves all have the same general structure, which we can call the Three Stage Model: Based on our experience working with organizations implementing performance improvement activities, Web Courseworks has created the roadmap for developing PI-CME activities illustrated below. Our roadmap starts with pre-development, or deciding what you re going to teach. Then you develop and implement the three stages of the activity, including any educational materials used in Stage B. Finally there is postdevelopment, where you evaluate the effectiveness of the activity and identify any areas for improvement. Now that we have a sense of what performance improvement is all about and what PI activities look like, let s walk through the process of developing one of these activities. In the remainder of this paper, we ll walk through these steps in order and explain how to accomplish them. Note however that this exact roadmap may not apply to all projects, activities, or content areas we encourage you to adapt this model to fit your particular situation and needs.

4 Identify Competencies Select Competencies in your Field for your PI Activity The first step of developing a PI activity is deciding on a topic area for the activity. If you already have a large library of educational content, that will make this step easier. But keep in mind that your goal is to improve physician performance, not simply disseminate knowledge. Ideally, the topic you choose will be directly related to the patient experience. Once you have your initial list of competencies, the next step is to select competencies to target with your PI activity. This involves organizing the competencies into domains, or groups of related competencies. From there you can prioritize the competency domains by relevance to your audience: the domains with the most relevance to your audience should be the highest priority. Next, take a close look at the competencies in your two or three highest-priority domains. Go through the competencies one by one and ask yourself these questions: To ensure that your activity is adequately performance-focused, we recommend identifying (or if one doesn t exist yet, making) a list of competencies in your field. Competencies are all the abilities and/or skills that people in your profession need to be successful. You may know where to find such a list already. If you don t, here are some places you can look for information on competencies in your field: Certification/licensure organizations devote significant organizational resources to developing and publishing lists or taxonomies of professional competencies. They almost always make their lists publically available, often through their websites. Standards organizations can help you identify performance benchmarks (we ll touch on those later). Subject matter experts the top people in your field are another excellent resource. Interview them, or simply shadow them on the job to figure out what makes them excel. Is this compenency essential for success in this profession? Is this compenency directly related to patient health outcomes and/or satisfaction? Is this compenency easy to measure? How will I measure it? Will physicians be able to improve their performance on this competency in just a few months? If you can t answer yes to all of these questions, the competency probably isn t a good one to include in your activity. Completing this analysis for your highest-priority domains should give you a sense of which domain will be the strongest base for a PI activity. Keep winnowing the list of competencies in your chosen domain until you re left with between four and eight of the strongest. These are the ones you ll design your PI activity around.

5 Set Benchmarks Write Performance Objectives Now that you have your final list of competencies, determine benchmarks for performance. For each competency, what does competence look like? How well would someone need to perform to be considered competent? These may sound like subjective questions, but be sure to do your research; often there is good information out there on which to base benchmarking decisions. If there is an organization that publishes standards or guidelines for your profession, review this literature for relevant information. If not, recruit subject matter experts to define and approve benchmarks. SMEs will often turn to the scholarly literature in your field to define benchmarks. Setting benchmarks early in development will also help ensure that your competencies are sufficiently measurable. Consider what data you can collect from participants to determine whether a benchmark has been met. If you can t figure out how to collect the data you ll need, the competency won t work in your PI activity. At this point, you should have a list of the competencies you want to target and the performance benchmarks you want to shoot for. Now it s time to formalize your expectations for how your activity will improve learners performance by writing performance objectives. Write one objective for each competency. If you re an instructional designer, you may be familiar with a handy tool for writing learning and performance objectives called SMART. This guideline says that objectives should be: Specific Measurable Achievable Relevant Time-bound If you ve followed the advice in the previous sections and have strong competencies and reasonable, vetted benchmarks, you should have no trouble writing SMART objectives. Even so, check each one against the list above. This is your last chance to make your objectives as good as you can before you move into development.

6 Measure Performance Compare to Benchmarks The first stage of performance improvement involves taking baseline measurements of learners current performance. There are a couple ways to go about this: you can either have doctors collect data on their own practice specifically for your activity, or use data that already exists. Many PI activities use existing data in the form of patient case files; this approach may or may not be viable depending on the topic of your activity and the specific competencies you need to measure. If you choose to have your learners collect new data, be sure to give them enough time to accrue the necessary data points (a month or two should be sufficient). Whether you use new or existing data, you ll need a way to collect that data. While the most advanced PI-CME implementations pull performance data from health registries or EMRs, most online PI-CME collects data from participants via a web form. When designing your data collection form, think about what information you need to gather. You might ask doctors questions about their practice, or simply give them a way to upload data from patient charts. Maybe you want to give them a paper document they can use to interview their patients and then upload the responses later. The National Board of Osteopathic Medical Examiners (NBOME) provides a paper questionnaire doctors can use to interview patients. Their responses are then uploaded to a web form. Collecting data is not the same as measuring performance. Once you ve collected the raw data, you ll need to convert it to a form that will allow you to compare learners to each other, and to the benchmarks you determined earlier. This usually involves converting qualitative (descriptive) data into quantitative (numerical) data through the use of a scoring rubric. For example, if one of the questions in your data collection form is a multiple-choice question, you might assign a numerical value to each answer. This will allow you to compute the average of a learner s responses to that question, giving you a concise and powerful measurement of that learner s performance on the competency associated with that question. The other advantage of obtaining quantitative measurements is the ability to compare learners performance on individual metrics to standardized performance benchmarks. If a learner performs under the benchmark for a particular competency, you can target that competency for improvement during Stage B. This is the power of the performance improvement model: each physician comes into the activity with different strengths and weaknesses, and Stage A reveals those areas in need of improvement, making it possible to create a personalized education program for each learner. NBOME s Stage A report compares learners baseline performance to national benchmarks and recommends educational interventions.

7 Identify Interventions Educational Interventions The final piece of Stage A is to provide each learner with recommendations on ways to improve their performance. Many performance improvement activities provide educational materials that guide learners through simple but effective changes they can implement in their practice. These materials can include articles, videos, on-demand courses, and many others. For each performance objective, ask yourself: What educational or practice interventions could help learners improve their performance? Learners in NBOME s Medical Communication Safety module fill out an action plan detailing which presentations they will view in Stage B. Another thing you can do at the end of Stage A is to allow learners to create their own action plans detailing which interventions they plan to undertake. If you are following the ACCME model of performance improvement, the individual action plan is mandated. This will help encourage your learners to take their own initiative for improving the care they provide to patients and make it more likely that they see your activity through to the end. Once you ve figured out how you re going to help your learners improve their performance on each of your objectives, you can start developing those educational resources. This is the portion of your activity that stands to benefit your audience the most. Accordingly, most of your development time and money should be funneled into this stage. However, there s no getting around it: learning object development is expensive. You might consider ways to reduce development costs, such as repurposing existing educational materials instead of developing new ones from scratch. Educational interventions in NBOME s Health Literacy module include video presentations on several topics related to health literacy. Stage C Development Stage B Development

8 Practice Interventions Remeasure Performance If your educational resources recommend that doctors make changes to their practice, consider how you can assist learners with implementing those changes. Old ways die hard you want to make those transitions as painless as possible for your learners. What performance support resources, such as paper or electronic job aids, could you provide to make the change easier? Now is the time to develop those resources. For example, imagine that you are trying to improve your learners hand hygiene practices to prevent the spread of disease within hospitals. You might create a sign that doctors can print out and post next to exam room sinks to remind themselves to wash their hands before and after seeing patients. This job aid helps doctors keep track of prescribed medications and related information on a per-patient basis. Stage C Development Stage B Development The goal of Stage C is to remeasure performance, and then compare the results to those obtained in Stage A to see if the educational and/or practice interventions undertaken in Stage B have had any effect. In order to draw a fair comparison, it s important to use the same timeframe and assessment questions that you used in Stage A. This is fortunate for you because it means you won t need to develop a whole new assessment. Just make a duplicate of your Stage A assessment and update the name and any accompanying details or instructions. Stage C Development Stage B Development

9 Compare to Stage A Evaluate & B Benchmarks After learners have completed the Stage C assessment, analyze the new data. You want to determine the following: Compare Stage C performance to Stage A performance. Has performance improved? Compare Stage C performance to the performance benchmarks. Are the benchmarks met? Make these findings available to your learners, preferably in a graphically rich format such as an interactive bar graph or infographic. The easier it is for learners to see how their performance has improved, the more valuable your activity will be and the more likely learners will be to continue to implement changes long after the activity has ended. Learners post-scores appear on their performance improvement report after their Stage C assessment is complete. Stage B Development Stage C Development Your activity should conclude by requiring learners to complete an evaluation form. The purpose of this is twofold: to help learners reflect on their experience and solidify their learning, and to help you evaluate the effectiveness of your activity. Your evaluation should give learners an opportunity to provide feedback on the quality of your activity and ask for suggestions on how it could be improved. Upon completing their evaluation, learners should receive a certificate of completion (or other credential) with a statement of continuing medical education (CME) credits earned. Most performance improvement activities award CME credit regardless of whether or not the physician was able to improve NBOME s Medical Communication Safety evaluation asks learners to rate their final performance on the module s core competencies. their performance as measured in the activity. CME credit distribution and tracking can be greatly simplified through the use of a learning management system. Stage B Development Stage C Development

10 Analyze Impact Publish Results Once a significant number of users have completed your activity, it s time to take a critical look at your activity s effectiveness. Compile and review learner performance data from Stage C, as well as any feedback collected via the post-activity evaluation. On the whole, is the activity improving physicians performance? If not, why not? Are the instructions clear? Are learners participating fully in Stage B, or just going through the motions? Does completing your activity require an inordinate amount of time and effort? Do your educational materials provide real value to your audience? Could they be revised so that they are more effective? Do your learners have access to the performance support resources they need to implement the necessary practice changes? To put it generally: how can the activity be improved so that it improves performance? If you aren t seeing the results you expected, don t be discouraged, especially if this is your first foray into performance improvement. You should expect your experience delivering PI-CME to be one of continuous improvement that will yield long-term rewards. Finally, be sure to publish the results of your performance improvement initiative. You will be doing a huge service to the community of CME providers, regardless of the success of your initiative. If you were successful, your model will enable other organizations to follow in your footsteps. If you weren t, sharing the lessons you learned will be equally invaluable. Either way, you will be helping to improve the quality of continuing medical education (and by extension medical care) for everyone. 5 5

11 Conclusion About Web Courseworks Performance improvement is a relatively recent innovation in the long history of continuing medical education that presents exciting opportunities for CME providers. Launching a PI-CME program will let you provide more meaningful and valuable educational experiences to your members. It can also bring new funding and recognition to your organization and help position you as an educational leader in your specialty. As more organizations move to provide PI-CME and share their experiences with the larger community, best practices are emerging. Finally, new technology platforms can help streamline the development and delivery process. Web Courseworks is a learning technologies and consulting company. We help associations leverage technology and drive highly successful education initiatives that push them to be the leading provider of education in their field. Web Courseworks Performance Improvement platform enables organizations to cost-effectively implement performance improvement activities. info@webcourseworks.com Website: