Mentor to Item Writers Training. Part One The Vision

Transcription

1 Mentor to Item Writers Training Part One The Vision

2 Agenda Overview of the Mentoring Program Mentoring in EPIC Mentoring Tips Mentoring toward Higher-Level Items Evaluating Items and Mentees

3 Overview

4 We need many high-quality items. We need approximately 4,000 new items written each year across all our exams to keep them fresh and current. The value of our credentials depends on it. The better the original quality of an item, the faster it passes through the review process. This saves time, effort, and money in getting the items onto exams. High-quality items more accurately assess the potential for high-quality patient care.

5 Mentors to Item Writers help us reach our goals. New Item Writers receive training on writing items that follow our guidelines. Their skills are honed and developed by the one-on-one support they receive from a Mentor. As a Mentor, you have the opportunity to share your knowledge and experience by guiding others to a higher level, leading to assessments that are relevant and valid and predict quality patient care.

6 Vision for the Mentoring Program New Item writers Experienced Writers Mentors Advanced Item Types (including hot spots, Semi-Interactive Consoles, PACSims), higher-level items, and hard to write areas

7 Steps for New Item Writers Complete training. Become paired with a Mentor. Write 10 items per 3-month writing project. Send each item to Mentor for feedback. Incorporate Mentor s comments and send item back for additional review. Submit items after receiving Mentor s approval. Evaluate your mentor.

8 Steps for Mentors to Item Writers Complete training. Become paired with an item writer and contact her/him within a week. Review items within a week of receiving them. Encourage your Mentee throughout the writing project. Evaluate each of your Mentees.

9 Summary of Vision paired with Mentor leads to high-quality patient care. New Item Writer

10 Mentor to Item Writers Training Part Two Mentoring in EPIC

11 Mentoring in EPIC

12 Step 1: You will receive an . From: Sent: Thursday, October 23, :03 PM To: Your Name Subject: Notification - An item has been moved to your queue Dear Your Name, You are receiving this because Mentee s Name has sent item to you for review under project Q2 PE Project. Click on the link below or copy it into your browser to login to the Item Authoring and Review System. The item is available in the Mentor Review queue under the project. You do not need to reply to this message. The status of the item, along with necessary comments and information, is available at the web address listed above. Thank you.

13 Step 2: Keep track of information. From: Sent: Thursday, October 23, :03 PM To: Your Name Subject: Notification - An item has been moved to your queue Dear Your Name, You are receiving this because Mentee s Name has sent item to you for review under project Q2 PE Project. Click on the link below or copy it into your browser to login to the Item Authoring and Review System. The item is available in the Mentor Review queue under the project. You do not need to reply to this message. The status of the item, along with necessary comments and information, is available at the web address listed above. Thank you.

14 This is an example of how you might want to keep track. Keep a log of your Mentees names, item numbers, and your comments so that evaluation of your Mentees at the end of the writing project will be easier to do. You may use whatever system works for you.

15 Step 3: Log into EPIC. Click on the writing project that was noted in the you received. Check to see if you have more than one screen of projects.

16 Step 4: Click on the item to review. When you click the Training Project, it will show the items ready to be reviewed. This screen shows you have two items to review. Click on the item number to review the item.

17 Step 5: Review the item. Review Stem. Review Content (classification). Review Answer Choices. Scroll all the way down the page to review references and rationales.

18 Step 6: Add your comments. Insert your comments here. You might find it easier to write in Word and cut-and-paste your comments into EPIC. Save comments in Word for reference when evaluating your Mentee. Click Add Comment.

19 Step 7: Click Review Complete. Click Remove if you want to edit your comment. Rewrite it and re-enter it. When you are finished, click Review Complete. After clicking Review Complete, you will not be able to retrieve the item until your Mentee sends it back to you.

20 When mentoring in EPIC TRACK COMMENTS TO MENTEES YOU MAY ONLY COMMENT ON THE ITEM, NOT REVISE IT.

21 Mentor to Item Writers Training Part Three Mentoring Tips

22 Mentoring Tips

23 1. Communicate regularly. your Mentees within a week of being paired. Let your Mentees know in advance if you will not be available for a while. If you haven t heard from your Mentee in two weeks, contact her/him. Review an item within five days of receiving it. Remind Mentees about a week prior to the end of the writing-project.

24 2. Encourage Mentees to begin writing. Start writing no matter what. The water does not flow until the faucet is turned on. -Louis L Amour

25 3. Suggest how to turn on the faucet. 1. Select a task on the content outline. 2. Choose an image. 3. Then just start. Many people find editing is easier than creating.

26 4. Provide constructive feedback. Specific Clear Objective Productive Encouraging

27 Your stem is great, but one of your distractors is weak. This is not constructive feedback.

28 Be Specific: Define abstract terms. WHAT DOES GREAT MEAN? Stem is focused. Stem is concise and grammatical. Stem contains only necessary information. Stem follows ARDMS guidelines. WHAT DOES WEAK MEAN? A is not a plausible answer. A and B are too similar. The key is obvious because it is the longest answer. Distractors vary too much in length.

29 Your stem is great focused and follows all ARDMS guidelines, Specific but distractor A is weak not plausible.

30 Be Clear: Avoid ambiguity. Avoid although, but, however, etc. These words tend to negate the positive comments that proceeded them. Put each comment into a separate sentence.

31 Your stem is great focused and follows all ARDMS guidelines, but. Your distractor A is weak not Specific Clear plausible.

32 Be Objective: Critique the writing, not the writer. AVOID THESE WORDS You You need to You should REPLACE WITH THESE WORDS The stem or key or distractor, Perhaps try A suggestion for

33 Your The stem is great focused and follows all ARDMS guidelines, but. Specific Clear Objective Your Distractor A is weak not plausible.

34 Be Productive: Guide toward improvement. Propose suggestions of resources to reference. Give direction for improving the stem or options. Critique only those things that can be changed. Coach toward higher-level questions.

35 The stem is focused and follows all ARDMS guidelines. Distractor A is not plausible. Try looking in [Book Title] for another pathology that might be more plausible. Specific Clear Objective Productive

36 Be Encouraging: Build a feedback sandwich. Finish on a positive note. Add the meat of the message. Start with something good.

37 Examples of Encouraging Statements This item really improved since I last saw it. Good job! You have some great images of [xxx]. We can really use those in the [xxx] domain. Seems like you have a good grasp on what it takes to write a good item. The quality of your items keep getting better with each one.

38 The stem is focused and follows all ARDMS guidelines. Distractor A is not plausible. Try looking in [Book Title] for another pathology that might be more plausible. You are very close to having an excellent item! Specific Clear Objective Productive Encouraging

39 Mentoring toward Higher-Level Items

40 Bloom s Taxonomy Levels of Questions Questions can be written to assess different cognitive levels of learning. ARDMS and APCA exams focus on Remembering/Understanding and Applying/Analyzing.

41 Action Verbs Associated with Each Level

42 Coach toward application and analysis. ITEM SENT BY MENTEE What is this image of a car dashboard showing? A. The amount of gas in the gas tank B. The amount of air in the tires C. The amount of oil in the oil tank D. The amount of coolant in the engine MENTOR S COMMENTS The image is clear and the structure of the item is good. It assesses remembering, which is the lowest level on Bloom s Taxonomy. Try to write an item that assesses application, a higher cognitive level, by asking for an interpretation of the image or what the driver should do with the information.

43 Lower-Level versus Higher-Level Item REMEMBERING AND UNDERSTANDING APPLICATION AND ANALYSIS What is this image of a car dashboard showing? A. The amount of gas in the gas tank B. The amount of air in the tires C. The amount of oil in the oil tank D. The amount of coolant in the engine If a driver saw this, what should the next step be? A. Drive to a gas station and fill up B. Take it to a service station to find out what s wrong C. No problem, keep driving D. Buy some antifreeze

44 Mentor to Item Writers Training Part Four Evaluation

45 Evaluating the Items and the Mentees

46 Evaluate the overall quality of the items. Language and terminology are clear and concrete. Each item has one stem and only one key and three distractors. Items contain proper grammar, punctuation, and spelling. Items use proper nouns (e.g., the patient) rather than names (e.g., Paul) or pronouns (e.g., he, she). Each item has a reference. The majority of items require application rather than just recall. Each item is independent. The key is not given away: There are no similar words in the stem and in the key. The key is neither the longest nor the shortest response option. All distractors are plausible response options. Common wording is placed in the stem. Items are written in the positive voice, avoiding words such as no and not.

47 Evaluate the content of the items. Items are relevant to the content area for which they are written. Items focus on critical knowledge required in daily clinical practice.

48 Evaluate the stem. The question is focused. The question is succinct. It is not a fill-in-the-blank question. It is not a true/false question. If there is an image attached to the item, the image is referenced in the question.

49 Evaluate the response options. Distractors are plausible. Distractors are indisputably incorrect. Each response option contains only one answer, not a combination such as A and B. None of the above nor All of the above are response options. There is only one correct response (the key). The key is not given away because of how it is written. Common (repetitive) wording is in the stem, not in the response options. Response options do not include excessive use of measurements.

50 Is your Mentee ready to move on without a Mentor? At the end of the three-month writing project, you will be asked whether your Mentee is ready to write items without mentoring support. This is when it will be useful to refer to your tracking sheet of names, items, and comments.

51 About each of your Mentees, you will be asked How well did the Item Writer follow item-writing guidelines? Did the items written require the test-taker to demonstrate an appropriate level of knowledge and application? Should the Item Writer be invited to continue writing? Is the Item Writer ready to write without a Mentor?

52 and each of your Mentees will be asked about you. Did your Mentor provide feedback within a week of receiving an item from you? Were your Mentor s comments specific, clear, objective, productive and encouraging? SCOPE! Did your Mentor motivate you to continue writing and improving? Did your items improve after working with your Mentor?

53 Thank You! The Exam-Development Team