EENet presents the fourth webinar in the Health Canada DTFP-ON series: The Costing Project: Data Quality in OHRS Reporting

Transcription

1 EENet presents the fourth webinar in the Health Canada DTFP-ON series: The Costing Project: Data Quality in OHRS Reporting Thursday April 28, pm 2:30pm (EDT)

2 The Costing Project: Data Quality in OHRS Reporting HOUSEKEEPING The audio is being stream via your computers. For optimal sound, please use external speakers or earphones. If you are still having trouble hearing our presenters, you can dial into Participant PIN: # This webinar will be recorded and posted on the project s webpage following the presentation. Please also let us know via the chat box if someone is watching the webinar with you! Some collected data from the webinar might be used for reporting. We would appreciate having your feedback on today s knowledge exchange webinar. You will receive a link to an online survey towards the end of the webinar. Thanks in advance for the 5 minutes of your time to complete our online feedback survey.

3 The Costing Project: Data Quality in OHRS Reporting Before we begin, we would like to know a little about YOU! EENet would like to know WHICH SECTORS are participating in today s webinar. Please answer the poll: What sector do you work in? Addictions only Mental health only Addictions and mental health Other (please specify which sector in the chat box)

4 The Costing Project: Data Quality in OHRS Reporting Before we begin, we would like to know a little about YOU! EENet would like to know WHO is participating in today s webinar. Please answer the poll: What is your main role in relation to the addictions and/or mental health sectors? Agency Leadership Direct Service Provider Knowledge Broker/Implementation Staff Person with Lived Experience and/or Family Member Policy Maker/Decision Maker System Planner Researcher/Research staff Other (please specify in chat-box) Not in the addictions and/or mental health sectors

5 The Costing Project: Data Quality in OHRS Reporting Now we would like to know a little more about YOU! EENet would like to know WHERE everyone is participating from. Please answer the poll: Which Local Health Integration Network (LHIN) area are you participating from? 1. Erie St. Clair 2. South West 3. Waterloo Wellington 4. Hamilton Niagara Haldimand Brant 5. Central West 6. Mississauga Halton 7. Toronto Central 8. Central 9. Central East 10. South East 11. Champlain 12. North Simcoe Muskoka 13. North East 14. North West I am participating from outside of Ontario Not sure

6 Health Canada s Drug Treatment Funding Program (DTFP) Ontario DTFP Drug Treatment Funding Program Ontario Systems Projects Through consultations with provincial and territorial governments and non-governmental organizations, Health Canada developed the Drug Treatment Funding Program (DTFP) in 2008 as part of the Treatment Action Plan under the National Anti-Drug Strategy. The DTFP is a federal contribution program designed to support provinces and territories in their efforts to strengthen evidence-informed substance use treatment systems; and address critical gaps in substance abuse treatment services, particularly for at-risk youth. 6

7 Health Canada s Drug Treatment Funding Program (DTFP) Ontario DTFP Drug Treatment Funding Program Ontario Systems Projects The DTFP supports sustainable improvement in the quality and organization of substance use treatment systems through investments in the following areas: IMPLEMENTATION OF EVIDENCE INFORMED PRACTICE STRENGTHENING PERFORMANCE MEASUREMENT & EVALUATION LINKAGE & KNOWLEDGE EXCHANGE 7

8 DTFP-ON Projects for FY2015- December 2016 Project IMPLEMENTATION OF EVIDENCE INFORMED PRACTICE The Implementation of the Ontario Perception of Care Tool for Mental Health and Addictions (OPOC-MHA) Implementation of a new staged screening and assessment process for addictions Best practices in Peer Support Services Developing and testing Internet-based interventions (IBI) for substance use in youth (NEW INITIATIVE) Trauma-informed substance use screening and assessment tools for First Nations and Inuit peoples (NEW INITIATIVE) STRENGTHENING EVALUATION & PERFORMANCE MEASUREMENT Development and standardization of cost-based performance indicators Project Lead Kim Baker, CAMH Kim Baker, CAMH Janis Cramp, AMHO Karen Urbanoski, Joanna Henderson, Gloria Chaim, CAMH Renee Linklater, CAMH Kim Baker, CAMH Strengthening performance measurement for mental health and addictions in Ontario (NEW INITIATIVE) Improving reporting compliance and data quality among Ontario s addiction treatment agencies Evaluating Residential Support Services Evaluation Plan for Youth Care (NEW INITIATIVE) LINKAGE & EXCHANGE Karen Urbanoski, CAMH Claudio Rocca, CAMH Janis Cramp, AMHO Laura Mills, Pine River Institute Evidence Exchange Network (EENet) Nandini Saxena, CAMH 8

9 Supports Ontario s 10 year Comprehensive Mental Health and Addictions Strategy Offices across Ontario Capacity and expertise in knowledge exchange, implementation, equity & engagement, information management, & evaluation Houses EENet, a knowledge exchange network that connects people with evidence about mental health and substance use problems in Ontario

10 The Costing Project: Data Quality in OHRS Reporting Today s presenters: Dr. Garth Martin, is a private consultant working on the Costing Project. His background includes a 40 year association with the Addiction Research Foundation and CAMH in areas of clinical service, clinical administration, research and knowledge exchange. He was formerly cross-appointed to the Faculty of Social Work at the University of Toronto and is a past Director of the Collaborative Program in Addictions Studies at U of T. Priya Kumar, is an Implementation Coordinator for the Costing Project, as well SS&A and OPOC implementation in the North-West LHIN. Her education and work experience include neuropsychological research, health sector management, and design thinking, all with a particular focus on brain and mental health, and addictions.

11 The Costing Project: OHRS Reporting and Data Quality *This project is made possible through a financial contribution from Health Canada and the support of the Ministry of Health and Long-Term Care (MOHLTC). The views expressed do not necessarily represent the views of Health Canada or MOHLTC.

12 OHRS and past phase of the Costing Project Current phase: Performance indicators Current phase: Costing Training Program

13 The Costing Project The DTFP-Costing project aims to develop reliable and valid cost-based performance measures to assess and compare cost of A&MH services. Ontario Healthcare Reporting Standards = source of data for development of these cost-based performance measures

14 Ontario Healthcare Reporting Standards (OHRS) Ontario version of national standard, Management Information System (MIS) Links program s revenues/expenses to its service delivery statistics Ontario healthcare providers mandated to report to OHRS Q2, Q3, Year-end

15 Why report OHRS data? Provides objective frame of reference for planning and resource allocation Facilitates measurement of program performance Supports accurate understanding of the cost of services provided 15

16 Issues identified in past phases Significant data quality problems across A&MH sectors High variability in service reporting High variability in expense reporting Inconsistent interpretation of data standards Methodological problems in use of existing indicators as performance measures Need for standardized cost-based indicators + OHRS data quality training

17 Common issues with client data Missing data Incorrect Service Recipient statistics reported Reporting to incorrect Functional Centre Inconsistent OHRS and CSR reporting Mismatching of financial and statistical data 17

18 Common issues with financial data Incorrect reporting of ED Compensation - e.g., Clinical Management FC Plant Operations expenses reported in Residential or Bricks and Mortar FC s Centralized Administration expenses not allocated to CMH&A sector activity Inconsistencies in Type 2 revenues/expenses between OHRS and SRI reports Lumping expenses (e.g., Supplies) applicable to several FC s in only 1 FC Inconsistencies in Type 3 revenues between OHRS and SRI reports Administrative expenses in a Type 3 fund Misreporting of food expenses Collapsing FC s in M-SAA agreement

19 Issues with existing performance indicators Apply only to Type 2 Funds Combine dissimilar statistics in computing the cost-per-client contact Large variations in cost of facilities distorts administrative costs Cost-per-resident day does not take into account the scope of services to non-residents Cost-per-individual served does not account for the type or duration of service provided

20 Recommendations for development of Guidelines for standardizing use and interpretation of existing indicators Development of 5 OHRS e-learning modules

21 Recommended guidelines for assessing the cost of addictions treatment services Identified methodological issues associated with existing cost indicators Proposed additions/revisions to indicators to enhance their utility as performance indicators Suggested key indicators for cost comparisons Specified indicators to be considered in the interpretation of observed differences on key indicators Proposed algorithms for standardizing and interpreting cost comparisons

22 MIS/OHRS elearning modules OHRS training support for CMH&A agencies Reporting & interpretation of OHRS data Accessible at: (scroll to Dec.8, 2015) Collaborators: Southwest Addiction and Mental Health Coalition, and MOHLTC Healthcare Data Branch Five Modules: 1. Overview of MIS/OHRS 2. OHRS Client Statistics 3. OHRS Financial Reporting 4. OHRS Reporting-How to Use Your Data 5. Auditing and Troubleshooting Your Data

23 Current Phase of the Costing Project: Performance Indicators

24 Costing indicators SPI Pilot project exploring the use of service provider interaction (SPI) statistics as cost and productivity performance measures Working group - work from Guidelines to standardize the use and interpretation of indicators

25 SPI pilot project SPI statistics (Individual and group) are measures based on service input time - time spent by Health Service Providers in direct contact with clients Pilot project examines using SPI as basis for measuring cost and productivity within and among agencies, in contrast to the current approach of using service output measures (e.g., visits & group participant interactions) Tests potential performance indicators for accuracy and consistency based on SPI and SPGI times Six participating agencies mix of large and small agencies; rural and urban services

26 Working Group Goal: To build on the Guidelines to standardize a set of performance indicators for assessing, interpreting and benchmarking the costs of addictions treatment A Working Group has been formed, comprised of members from the Project Team, 3 LHINs, and Mental Health Sector. It will work in collaboration with the LHIN SRI Indicators Group Deliverables: Standardized performance indicators that allow effective measurement and interpretation of agency performance Reliable performance measurement data that can better assist MOHLTC and LHINs make informed decisions on policy, funding, and resource allocation at agency, regional, and provincial levels

27 Current Phase of the Costing Project: Data Quality Training Initiative

28 The Costing Training Program Stage 1: Data Quality & OHRS Requirements Stage 2: Data Quality Culture in your Agency Stage 3: Network of Data Champions

29 Data quality culture is Embedding data quality bestpractices in everyday work so there is On-going awareness of how data quality impacts program planning, funding, decisionmaking Greater accountability around documenting, collecting, or reporting data Embedding tools/practices that enable high data quality in daily work

30 Why the need for a data quality culture? System- and agency-level decisions impact quality of care. Without accurate and complete data, incorrect decisions around service provision are made, impacting quality of care for clients. Incorrectly identified gaps Ineffective evaluation Ineffective allocation, use of resources Decreased quality of care: Clients do not receive the right care, at the right time, in most effective way.

31 Stage 1: Data Quality & OHRS Requirements 3 webinars across 10 LHINs ~270 CMH&A staff registered from various roles: Finance/Data experts, Direct service providers, Agency leaders Overview of OHRS reporting requirements: Reporting structures Client statistics + reporting scenarios Data quality tools and auditing data

32 Ontario Healthcare Reporting Standards Primary Accounts Balance Sheet Accounting Centres Functional Centres Secondary Accounts THE OHRS REPORTING STRUCTURE Financial Revenues and Expenses Statistical Human Resources Service Delivery (Client) Activity

33 Ontario Healthcare Reporting Standards Primary Accounts Balance Sheet Accounting Centres Functional Centres Secondary Accounts THE MATCHING PRINCIPLE Financial Revenues and Expenses Statistical Human Resources Service Delivery (Client) Activity

34 Service Delivery (Client) statistics # Service Recipients # New Referrals # Individuals Served # Significant Others (SO) # Admission and Discharge # Attendance Days # Resident Days # Visits # Group Statistics # Service Provider Interactions # Wait-time Statistics Visit the Health Data Branch Web Portal:

35 Troubleshooting and auditing data quality issues Auditing - sampling data to measure against various quality characteristics (completeness; accuracy; consistency; duplication; correct time periods) Troubleshooting - logical approach to identifying/correcting data problems 1. Audit - check for accuracy, eliminate obvious issues: entry errors, missing data 2. Dig deeper to determine problem & source 3. Take corrective action

36 Uses for the reported data and tools Comparison time periods, between programs, between peer agencies, within LHIN, across LHINs Trending - client service needs, volumes, demographic changes, etc. Forecasting - finances and service delivery Benchmarking support best practices, performance targets and program standards Budgeting Resource allocation staff, funding, time 36

37 Stage 1: Evaluations 70% I have an increased understanding of OHRS (n=68). 70% I better understand the structure for reporting financial and statistical data (n=68). 60% 60% 50% 50% 40% 40% 30% 30% 20% 20% 10% 10% 0% Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 0% Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree

38 Stage 1: Evaluations 80% 70% I better understand my role and responsibility I have an increased understanding of to report accurate data (n=68). OHRS. 70% 70% I I have better learned understand how inaccurate the structure reporting for can reporting affect financial my agency and statistical (n=68). data. 70% 60% 60% 60% 50% 50% 50% 40% 40% 40% 30% 30% 30% 20% 20% 20% 10% 10% 10% 0% Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree 0% Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree

39 Stage 2: Data Quality Culture in your Agency 5 full-day events across 10 LHINs 197 staff attended from various roles across CMH&A orgs Connecting reporting requirements to data quality culture: Reporting requirements + Scenarios 4 Case studies 2-part Group Brainstorm: Data quality challenges + Solution Generation

40 Stage 2 brainstorm: Data quality challenges Staff engagement/buy-in (enhancing DQ culture) Competing priorities/limited time = clinical care >> data quality Most staff trained for clinical care, not complex data mgmt. Lack of clarity re: why data are being reported Education and training (new staff & refresher) Inconsistent interpretation of OHRS definitions & requirements Uncertainties in reporting complex statistics/scenarios High staff turnover Other (e.g.) Communication gaps - software vendors Data entry practices inconsistent or lacking across orgs Agencies have limited resources, esp. smaller agencies

41 Stage 2 brainstorm: Solution generation Challenge: Competing priorities = difficult to make data quality a priority = difficult to enhance data quality culture in the organization. Proposed Solution: Involve all staff in developing data-centric solutions/changes, targetsetting, data reviews, policies. Install on-going and sustainable datacentric processes into daily work. Necessary processes/conditions: Regular data discussions and audit checks with all staff Mgmt. supports with time and task management i.e., priority setting On-going feedback; incentives for good data quality performance Train all staff in reporting requirements; assign QA Leads Outline expectations of data-related tasks in job descriptions, performance evaluations, even for clinical staff

42 Stage 2: Evaluations I better understand my role & responsibility to report accurate data (n=163). I better understand how to navigate complex OHRS statistics (n=163). 70% 70% 60% 60% 50% 50% 40% 40% 30% 30% 20% 20% 10% 10% 0% Strongly Disagree Disagree Neither Disagree or Agree Agree Strongly Agree 0% Strongly Disagree Disagree Neither Disagree or Agree Agree Strongly Agree

43 Stage 2: Evaluations 70% I better understand the links between OHRS and data quality best practices (n=163). 70% I better understand how I can apply data quality best practices in my agency (n=163). 60% 60% 50% 50% 40% 40% 30% 30% 20% 20% 10% 10% 0% Strongly Disagree Disagree Neither Disagree or Agree Agree Strongly Agree 0% Strongly Disagree Disagree Neither Disagree or Agree Agree Strongly Agree

44 Stage 3: Network of Data Champions Supporting sustainable data quality + addressing needs/challenges from Stage 2 A. Resource-based: support around training/education; information on best practices B. Process-based: address/implement processes/best practices that impact data quality Two components: A. Resource-based: Online presence on Evidence Exchange Network (EENet) B. Process-based: LHIN-based Working Group/CoP

45 The Network on EENet (Resource-centric) Open-access online space OHRS and other data-centric resources Need for training, education, and related supports Means of (data-centric) communication and P2P learning Populated by Costing team and you!

46 The Network on EENet (Resource-centric) Open-access online space OHRS and other data-centric resources Need for training, education, and related supports Means of (data-centric) communication and P2P learning Populated by Costing team and you! Example features Stage 1 & 2 training materials Data tip of the week Data Success story of the month Scenario bank; Moderated forum Links to HDB tools, checklists, presentations;

47 The Network on EENet Anticipated timeline: Exploration/Installation: Now - Summer, 2016 Stakeholder engagement, production, testing Implementation: Fall-Winter, 2016 Enrollment, moderating, updating resources, on-going development

48 The Network on EENet Anticipated timeline: Exploration/Installation: Now - Summer, 2016 Stakeholder engagement, production, testing Implementation: Fall-Winter, 2016 Enrollment, moderating, updating resources, on-going development Call for EENet Data Champs! Join us in developing, leading, and sustaining this space! Advise on resources/populating Network; Moderate forum (e.g., 1 week); be a beta tester Various levels of OHRS/DQ expertise welcome Priya.Kumar@camh.ca

49 LHIN-based Network (Process-centric) LHIN-based, Working Group + CoP Identify, prioritize, and implement solutions to data quality challenges in the LHIN Pilot with 2 LHINs Process, results, best-practices to be shared via EENet Network

50 LHIN-based Network (Process-centric) Anticipated Timeline LHIN-based, Working Group + CoP Identify, prioritize, and implement solutions to data quality challenges in the LHIN Pilot with 2 LHINs Process, results, best-practices to be shared via EENet Network Exploration/Installation: Summer, 2016 Stakeholder engagement, identifying champions, challenge selection, implementation planning Implementation: Fall-Winter, 2016 Solution implementation, process and outcome sharing on EENet Network (on-going)

51

52 The Costing Project: Data Quality in OHRS Reporting Thanks to all participants for joining today s webinar. EENet would also like to give a special THANKS to Priya and Garth for today s presentation! Please take a few minutes to answer our survey on today s webinar: The recording of today s webinar will also be posted the project web page shortly.

53 Next webinar in Health Canada DTFP-ON series: SAVE THE DATE! Trauma-informed screening and assessment tools for First Nations and Inuit Peoples Thursday June 30, pm 2:30pm EDT Registration will soon be available here: For additional updates about the Ontario DTFP projects, sign up for our newsletter at: