Performance measurement Part 2. Designing and implementing a performance measurement framework. January 21, 2014

Transcription

1 Performance measurement Part 2 Designing and implementing a performance measurement framework January 21, 2014

2 Housekeeping notes This webinar will be recorded, and will be available on the Centre s website as an educational resource. The slides have been sent to participants. Log in with a phone whenever possible for optimal audio quality. We have staff online to assist with any technical difficulties. There will be a short evaluation survey for all webinar attendees at the completion of the webinar.

3 Housekeeping notes This webinar will be followed by a question and answer period, however questions are encouraged throughout the presentation. Questions can be submitted electronically or verbally.

4 Questions or comments? To submit questions electronically, use the question box located in your control panel. To submit questions verbally, use the raised hand icon also located in the control panel.

5 Welcome and introductions

6 Today s presenters Jana Kocourek, M.A. Knowledge Broker, Evaluation support services Ontario Centre of Excellence for Child and Youth Mental Health Stephanie Greenham, PhD. Staff Psychologist, Lead for Outcomes Management and Research, Inpatient Psychiatry Program Children s Hospital of Eastern Ontario

7 About the Centre We bring people and knowledge together to strengthen the quality and effectiveness of mental health services for children, youth and their families and caregivers. Three strategic goals: Learning Collaboration Leadership Foster a culture of organizational learning to support agencies in using evidence to improve client outcomes. Build and develop collaborative partnerships to sustain capacity within mental health services. Be a true learning organization and lead by example.

8 About CHEO Inpatient Psychiatry Program 19-bed acute care inpatient unit for children and youth under 18 years of age Brief stay, crisis stabilization and assessment model (currently evolving) with interdisciplinary team Youth with acute, severe and complex mental health needs served in a tertiary care hospital setting o o Majority (65% in FY ) mood disorder as most responsible discharge diagnosis 26% of patients admitted involuntarily under Mental Health Act

9 Goals of this webinar To provide practical tips on designing and implementing performance measurement systems To provide examples from the field

10 Overview 1. Recap of performance measurement (PM) 2. Designing a PM framework 3. Implementing a PM framework

11 Recap of performance measurement

12 What is performance measurement? The regular collection of information for monitoring how a policy, program or initiative is doing at any point in time. It can be used to report on the level of attainment of planned results and on performance trends over time. 01/05/13 01/12/13 01/19/13 01/26/13 02/02/13 02/09/13 Κια1 Us Goal Report Oct 22, 2013 The ongoing monitoring and reporting of program accomplishments, particularly progress towards a preestablished goal. Πµι 2 Ρεµ 3 Start J F M A M J J A S O N D

13 Why measure performance? To serve as a descriptive tool on how a project, policy, organization, or program is doing. To monitors use of resources. To contribute to accountability. Can lead to performance improvement.

14 Performance measurement Evaluation Has a program reached its goal? Measurable standards High-level snapshots Focus on the outcome What factors effect program outcomes? Open areas of exploration In-depth analysis How and why an outcome is occurring

15 Performance measurement sports analogy Performance measurement is like the game score in a hockey game Ongoing throughout game Quantifiable indicator of team progress Standardized measure that facilitates comparisons Answers the question: How much? Program evaluation is like the analysis of the game In-depth analysis at intervals Connects context of team and player achievement Discussion of why score is higher/lower Answers the question: What does this mean?

16 Poll Do you currently use a PM system within your organization? Yes No Unsure Is there a particular framework that informed your PM?

17 Poll At what stage is the implementation of your PM system? Exploring Designing Initial stages of implementing Implementing for over six months

18 Designing a PM framework that s right for your organization

19 1. Identify organizational champions and form a team Organizational leadership is important. Action: Create a terms of reference document McDavid et. al 2012

20 2. Manage and clarify expectations 2.1 Agency reach 2.2 Website reach 2.3 Identification of stakeholder needs 2.4 oengagement activities 2.5 Satisfaction with services Two main purposes: Performance improvement formative Public accountability summative 3.3 Increased evidence-informed practice (EIP) capacity 3.4 Enhanced sustainability for implementing evidence-informed practices (EIPs) o Measures that can alert you to problems in processes that can impact outcomes Action-Understand what performance measurement systems can and cannot do Measures that assess if a program or project is achieving its goals Action: Create a project charter/gantt chart McDavid et. al 2012

21 3. Create a workplan Action:Include responsibilities, activities, timelines for development and implementation. McDavid et. al 2012

22 CHEO example Outcomes management planning phase Comprehensive outcome literature review Establishing program goals Mapping outcomes onto program goals Identify appropriate standardized measures and informants Establish time points for assessment Identify strategies to embed outcomes management activities directly in to clinical care; make clinically salient

23 4. Identify available resources For developing, implementing, maintaining and renewing the performance measurement system For performance measurement systems to achieve their potential McDavid et. al 2012

24 5. Create a communication plan Action: Establish multi-facet ways of communicating Action: Include stakeholder engagement McDavid et. al 2012

25 CHEO example: Regular communication Sharing of clinical outcomes at patient rounds and planning meetings Integration into team documentation Regular sharing of outcomes with program leadership, administration, and the clinical team (sustains buy-in) Programming and resource decisions are informed by clinical outcomes

26 6. Stakeholders: Involve them in the development and review process o increases buy-in and credibility Action:Create an expert panel Action: Assess buy-in McDavid et. al 2012

27 7. Understand organizational history around similar initiatives Action:Inquire about past experiences involving large-scale initiatives McDavid et. al 2012

28 8. Review tools to help inform framework Strategic plan Mission and strategic goals Program logic model Operations plan

29 9. Ensure framework is adaptive

30 10. Choose your indicators wisely What is the strategic value? What is the data quality of the measure? Action:Ensure they are operationally defined and concise.

31 Indicators checklist Strategic Actionable Owned Aligned Correlated Relevant Available Timely Accurate Referenceable Game-proof Standardized Concise Kinark Child and Family Services 2013

32 CHEO sample indicators 1.5 Clinical indicators Decrease in level of acuity of illness (i.e. risk behaviours, symptoms) from admission to discharge % of patients with needs met by discharge 1.6 Process indicators % of patients discharged within target length of stay for the model Rate of readmission within 7 days,28 days, etc.

33 The final product

34 CHEO inpatientpsychiatry outcomes management approach Total Clinical Outcomes Management (TCOM; Lyons 2009) Provided conceptual framework for outcomes management approach in place on the unit since 2000 Central importance of using clinical information (i.e. needs and strengths of children and families, to inform and manage decision making at all levels of the system) patient, program, hospital, community, full-system levels Use of standardized assessments Childhood Acuity of Psychiatric Illness Child & Adolescent Needs & Strengths Mental Health, Acute

35 Standardized assessment Routine collection of clinical data at admission Child or Youth: CDI-2, YSR Parent/Caregiver: CBCL (Conner s, SSP) Staff: CAPI, CANS-MH Acute Review of medical chart (demographics, treatment history) Collection of outcome data at discharge Child or Youth: CDI-2 (if indicated) Staff: CAPI Review of medical chart (diagnoses, medications, length of stay, discharge treatment plan)

36 Total Clinical Outcomes Management Grid Family & Youth Program System Decision Support Service Planning Eligibility Resource Management Quality Improvement Case Management & Supervision Accreditation Transformation Outcome Monitoring Service Transitions & Celebrations Evaluation Performance Contracting

37 Tips for implementing a PM framework

38 Measure indicators of the PM system Do selected tools measure the key indicators? Are resources available to sustain measurement process? Do your measures satisfy internal and external stakeholders? McDavid et. al 2012

39 Record, analyze, interpret and report performance data Comparisons over time Comparisons across organizational services or departments Comparisons with benchmarks or targets McDavid et. al 2012

40 Review regularly Balance between maintaining continuity of measures (facilitates comparisons over time) and maintaining relevance (changing measures periodically) Action: Review and revise McDavid et. al 2012

41 Create feedback loops Action:Ensure formal mechanisms are in place for sharing data. McDavid et. al 2012

42 Other important considerations Fostering a culture that embraces change o o Often requires shift in mind set Value of incremental change Understanding the importance of outcomes management to service delivery (e.g. buy-in) Securing champions Adequate financial resources Ongoing training requirements

43 Everyone plays a role in PM Leadership team Program manager Evaluator Quality assurance coordinator Staff

44 Poll Who manages the performance measurement system within your organization? Leadership (e.g. Director) Program manager Evaluator Quality assurance coordinator Other

45 Key points

46 Remember to: create a team and appoint lead(s). create a workplan. keep it simple - avoid unnecessary complexity. start small, then increase over time. Review and revise to suit your organizational needs: one size does not fit all!

47 CHEO Framework - select references Greenham, S.L. & Bisnaire, L. (2008). An outcome evaluation of an inpatient crisis stabilization and assessment program for youth. Residential Treatment for Children and Youth, 25(2), Bisnaire, L.M. & Greenham, S.L. (2009). Implementation of Total Clinical Outcomes Management in as acute care pediatric inpatient setting. In J.S. Lyons & D.A. Weiner (Eds.). Behavioral Health Care Strategies: Total Clinical Outcomes Management. Kingston, NJ: Civic Research Institute. Lyons, J.S. (2009). Communimetrics: A communication theory of measurement in human service settings. New York: Springer.

48 The Ontario Centre of Excellence for Child and Youth Mental Health - select references MacDavid, J.C., Huse, I., Hawthorn, L. (2013) Program Evaluation and Performance Measurement: An Introduction to Practice. Sage publication

49 Questions?

50 Questions or comments? To submit questions electronically, use the question box located in your control panel To submit questions verbally, use the raised hand icon also located in the control panel

51 Upcoming webinars Selecting indicators and measures: Tuesday, February 4, 2014 at 1 p.m. Working with data part 1 (data collection): Tuesday, February 18, 2014 at 1 p.m.

52 For more information on this webinar or topic, please contact: Jana Kocourek, Knowledge Broker Evaluation support services ext

53 Evaluation Please complete the survey at the end of this webinar. Your feedback is very important to us, so we thank you for taking the time to share your thoughts!