Portfolio Assessment. John Norcini, PhD, FAIMER ACHIEVING IMPROVEMENT THROUGH RESEARCH AND EDUCATION. TOGETHER.

Transcription

1 Portfolio Assessment John Norcini, PhD, FAIMER ACHIEVING IMPROVEMENT THROUGH RESEARCH AND EDUCATION. TOGETHER.

2 Overview Define and describe the types of portfolios and their purposes Steps in implementation Strengths and weaknesses General Medical Council (UK) Revalidation process Description of the portfolio Reactions to the process

3 Portfolio A portfolio is a purposefully selected collection of work and related material that depicts an individual s activities and achievements

4 Types of Portfolios Showcase portfolio Contains the best over a period of time Demonstrates potential Process portfolio Contains educational experiences and reflections Documents a learning process Assessment portfolio Contains a set of accomplishments Documents competence Hybrid portfolio Contains some mix of the other types

5 Types of Portfolios Type of portfolio chosen depends on purpose Showcase portfolio Exhibit of work that supports self-marketing Process portfolio Exhibit professional goals, growth, and achievement Assessment portfolio Exhibit materials that demonstrate competence Hybrid Speaks to a more than one purpose

6 Portfolio Contents for Physicians Educational workshops and courses Operative cases Formative, self-, and summative assessments Patient/peer feedback Credentialing items Bedside procedures BLS/ACLS Promotion and adverse action letters Medical staff privileges Boards Diplomas Presentations and manuscripts Reflections

7 Steps in Implementation Set a purpose What is the rationale? What are the costs and benefits? Decide on placement Where do they fit? What contents are required? Preparation When are they introduced? How are users prepared? Stolle, Goerss, Watkins, 2005

8 Steps in Implementation Assessment On what criteria will they be assessed? Who will assess them? How will reliability be assured? Policies What policies and appeals process support the portfolio? What types of remediation are available to those who are unsuccessful? Stolle, Goerss, Watkins, 2005

9 Strengths and Weaknesses Strengths Tailored to the work and learning preferences of individuals Encourage critical thinking, judgment, self-reflection Foster responsibility for learning Cover a broad range of skills and purposes Weaknesses Time-consuming for learners Time-consuming for scorers Scoring rubrics Lack standardization Comparability across learners Fairness

10 Revalidation GMC process for revalidation (2012) Physicians must have a connection with a Designated Body Provide annual appraisals and support Physician needs to maintain a portfolio Responsible officer makes a recommendation to GMC every 5 years Often the medical director of the designated body

11 Appraisals Based on four domains from Good medical practice Knowledge, skills, and performance Safety and quality Communication, partnership, and teamwork Maintaining trust

12 Portfolio Contents Basic information Personal details Scope of work Record of annual appraisals Personal development plans and their review Probity Health

13 Portfolio Contents Continuing Professional Development Doctor s responsibility Reflect regularly Employers should facilitate access to resources including time Related to practice Points not specified Quality improvement Clinical audit Review of clinical outcomes Case review or discussion Audit Evaluate impact and effectiveness

14 Portfolio Contents Significant events Unintended or unexpected event that could or did lead to harm Participation Lessons learnt Actions taken Both colleague and patient feedback Covers the whole of practice Independent of doctor and appraiser Often based on surveys

15 Portfolio Contents Complaints and compliments Awareness of the procedures Participation Actions taken Opportunities for professional development

16 Threats to validity Ability to include only favorable information Subjective and inconsistent interpretation of the portfolio by appraisers Important information for some specialists is not required (e.g., patient outcomes for surgeons) Variable administration of multisource feedback Challenges of assessing the quality of reflection Furmedge et al, 2016

17 Interim Survey by GMC Online survey of participants (17% response rate) Majority were positive <50% thought it was an effective way to improve clinical practice <33% believed it has had a positive impact on appraisal Majority (58%) said they had not made any changes to clinical practice, professional behavior, or learning activities

18 Interim Survey by GMC Online survey of participants (17% response rate) There was skepticism among respondents about whether it Led to improved patient safety Will identify doctors in difficulty at an earlier stage Majority found patient feedback to be most helpful

19 Summary There are four types of portfolios matched to purpose Strengths include tailoring to work and encouraging critical thinking, judgment, and self-reflection Weaknesses include time needed and lack of standardization GMC experience highlights both the strengths and weaknesses in portfolios for revalidation