Welcome and thank you for joining us. Bienvenue et merci de vous joindre à nous. We will begin shortly

Transcription

1 Welcome and thank you for joining us. Bienvenue et merci de vous joindre à nous. We will begin shortly 1

2 Please turn on your computer speakers to hear today s session. If you do not hear music, then please confirm your computer speakers are on and check that your volume is unmute. If you do not have computer speakers, then please dial to listen in English via teleconference. Quant aux participants francophones, veuillez composer le numéro pour vous connecter en français par téléconférence. 2

3 CFHI s Measurement for Quality Improvement Webinar Series - Session 1 Making Data Matter: Measurement Basics for Quality Improvement October 22, :00-1:00 pm (EST) cfhi-fcass.ca 3

4 Welcome and Introductions Kaye Phillips, Senior Director of Education, Evaluation and Performance Improvement, CFHI Melanie Rathgeber, CFHI QI & Measurement Faculty and Principal, Merge Consulting Trevor Strome, CFHI QI & Measurement Coach and Informatics / Process Improvement Lead, Emergency Program, Winnipeg Regional Health Authority and Assistant Professor, Department of Emergency Medicine, University of Manitoba 4

5 Welcome to All Participants Including CFHI s EXTRA Program for Healthcare Improvement Fellows CFHI s INSPIRED Approaches to COPD Collaborative CFHI s Partnering with Families and Patients for Quality Improvement Collaborative CFHI s Reducing Antipsychotic Medication use in Long Term Care Collaborative And everyone else! 5

6 Who s Registered? Who s on the Line?

7 Session Objectives Participants Will Learn: 1. The importance and challenges of measurement for quality improvement 2. The value of using run charts and control charts to analyze data over time 3. How to construct and analyze run charts 7

8 The Value of Data Measurement should be used to speed things up, not slow things down. Using data can: Help us ask the right questions Guide decision making and make solutions apparent Help us know where to focus our time and resources IHI (1996). Breakthrough Series guide: Reducing delays and waiting times. Boston: Institute for Healthcare Improvement. 8

9 Problems with Data What gets in the way of using the data we collect to guide our decision making? Use the Chat function to type in your response. 9

10 Problems with Data 1. The results are wrong. We don t have the right data. 2. The data are too old These results might not be statistically significant. We need to focus on this outlier or trend. 10

11 Problems with Data The results are wrong. We don t have the right data. Resolutions: Be very clear about definitions and methods used. Make sure the definition is right for your environment. Spend time up-front outlining the problem you are trying to improve. If it is a priority area, how can you get support for the data you need? Know the purpose of data for improvement versus accountability. Do you have some data that can be used to help you improve? 11

12 Problems with Data The data are too old. Resolutions: Collect and show some small samples of data in real time. Results can be validated from larger data sets later. Use run charts with weekly samples. Collect data as part of existing processes. Data from this week 12

13 Problems with Data These results might not be statistically significant. Resolution: Analyze data over time. Use probability based analysis (run charts) or evidence of special cause variation (control charts). 13

14 Problems with Data We need to focus on this outlier or trend. Resolution: Understand that this is human nature: there are results that jump out at us and we feel compelled to focus on them. Use probability based analysis (run charts) or evidence of special cause variation (control charts) to focus on true outliers and trends. 14

15 A Simple Run Chart Data displayed in time order Data is collected and displayed weekly or monthly. Centre line = median of the data points or = baseline value 100% 80% 60% 40% 20% 0% Percent of Patients with Self-Management Goals Documented April May June July Aug Sept Oct Nov Dec Jan 15

16 The Run Charts Tell Us: Current results (baseline) How results change over time (e.g. each week) When we have reached our target If the target has been sustained Real change vs. natural fluctuation 100% Percent of Patients with Self-Management Goals Documented 80% 60% 40% 20% 0% April May June July Aug Sept Oct Nov Dec Jan 16

17 The Value of Data in Real Time Run Charts can tell us what is happening in real time. 100% 80% 60% Percent of Patients with Self-Management Goals Documented 40% 20% 0% 17

18 The Value of Data Over Time Run Charts detect true patterns and trends over time. Not just what is happening before and after a 100% change. 80% Percent of Patients with Self-Management Goals Documented 60% 40% 20% 0% 18

19 Pre and Post Change Bar Chart: What is the Interpretation? 19

20 Scenario 1 Scenario 1. Pre-post data Scenario 1. Data displayed in a run chart over time. change made between week 7 and 8 20

21 Scenario Average Before Change = Average After Change = The t-test shows a significant difference: t(22)=7.6, p<.001 Adapted from Perla R.J., Provost L.P., & Murray S.K. (2011). The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Quality & Safety, 20(1):

22 Week1 Week2 Week3 Week4 Week5 Week6 Week7 Week8 Week9 Week10 Week11 Week12 Week13 Week14 Week15 Week16 Week17 Week18 Week19 Week20 Week21 Week22 Week23 Week Average Before Change = Average After Change = pre-change post-change change is not sustained t(22)=7.6, p<.001 Adapted from Perla R.J., Provost L.P., & Murray S.K. (2011). The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Quality & Safety, 20(1):

23 Don Berwick on Data Over Time: Evaluators and medical journals will have to recognize that, by itself, the usual O X O experimental paradigm is not up to [the] task. Many assessment techniques developed in engineering and used in quality improvement statistical process control, time series analysis, simulations, and factorial experiments have more power to inform about mechanisms and contexts than do RCTs. Berwick, D. M. (2008). The science of improvement. JAMA; 299(10):

24 Questions? Please submit your comments/questions electronically using the Chat Box on the bottom right of your webinar screen 24

25 Making a Run Chart 2 Options 1. Use Chart Functions in Excel See Worksheet How to Make a Run Chart in Excel 25

26 Making a Run Chart 2 Options 2. Run Chart Excel Template note: you need to complete a one time free registration 26

27 Days The Result The Start of a Run Chart Wait Time to See a Specialist 27

28 Freezing a Baseline Median: Calculate the Median of Only the Baseline Points 80 Percent of Providers Using COPD Screening Tools E-chart screening tools installed on week

29 Analyzing a Run Chart Simple rules are used to analyze a run chart for evidence of change/difference. The rules tell you whether the data points are distributed randomly, or whether there is a specific pattern which indicates something has actually changed. (p<.05). Four rules: Perla, R.J., Provost, L.P., & Murray, S.K. (2011). The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Quality and Safety, 20(1),

30 Analyzing a Run Chart Two Common Rules: 1. Evidence of a trend 2. Evidence of a shift Note: trends and shifts can be evidence of improvement, or evidence that things are worse than usual. 30

31 Example: The Trend Rule 50 A Trend: Five or More Consecutive Points All Going in the Same Direction

32 Is There a Trend? (5 or more points all going in same direction) 30 Number of Residents with a Care Plan Updated since Start of the Collaborative

33 Yes there is one trend:

34 80 A Shift: Six or More Consecutive Points All On One Side of the Median

35 Is There a Shift? (6 consecutive points all on the same side of the median) 80 Percent of Complex Patients Attached to an Primary Care Team

36 Yes, there is a shift. 80 Percent of Complex Patients Attached to an Primary Care Team

37 How to interpret a shift or trend, in plain language: 80 Percent of Complex Patients Attached to an Primary Care Team There is a signal of a non-random pattern There is less than 5 % chance that we would see this pattern if something wasn t going on, i.e. if there wasn t a real change 37

38 Questions? Please submit your comments/questions electronically using the Chat Box on the bottom right of your webinar screen 38

39 Using Run Charts for Improvement: An Example Aim: To reduce percent of chronic disease patients whose score on the PHQ-9 (a brief depression inventory) indicates moderate to severe depression 39

40 Outcome and Process Measures at Baseline Key Message: Value in measuring baseline even before changes are made. 40

41 Outcome Measure Over Time Key Message: No change according to run chart rules. 41

42 Process Measures Over Time Key Message: Evidence of a shift; team can be confident that improvement in this process is not due to chance. Key Message: Evidence of an early shift then no change; helps team understand why outcome might not be improving. 42

43 1/1/10 2/1/10 3/1/10 4/1/10 5/1/10 6/1/10 7/1/10 8/1/10 9/1/10 10/1/10 11/1/10 12/1/10 1/1/11 2/1/11 3/1/11 4/1/11 5/1/11 6/1/11 7/1/11 8/1/11 9/1/11 10/1/11 11/1/11 12/1/11 1/1/12 2/1/12 3/1/12 4/1/12 5/1/12 6/1/12 7/1/12 8/1/12 9/1/12 10/1/12 11/1/12 12/1/12 1/1/13 2/1/13 3/1/13 An Introduction to Control Charts: 40 Number of Resident Falls per Month 35 UCL LCL

44 Some things control charts can do, that run charts don t: 1) Can deal with different sample sizes 2) Can determine change quickly (improvement will be picked up more quickly than in a run chart) 3) Predict performance in coming weeks or months 4) Can stratify to understand reasons for differences in results (e.g. high and low performance) 44

45 e.g. highlighting differences in performance. 90% Percent of Patients Who Rated Their Experience with Discharge Planning as Very Good/Excellent 80% 70% 60% 50% UCL 40% 30% 20% LCL 10% 0% ICU Nov ICU Dec ICU Jan ICU Feb Emerg Nov Emerg Dec Emerg Jan Emerg Feb Surg Nov Surg Dec Surg Jan Surg Feb Med Nov Med Dec Med Jan Med Feb 45

46 Every system is perfectly designed to get the results it gets. - Dr. Paul Batalden 46

47 Healthcare Analytics and the Information Value Chain Performance Objectives Business Processes What DID Happen Data Analytics What IS Happening Decisions & Actions Outcomes Evaluation Improvement Approach What Will Happen Quality Goals 47

48 Going beyond reporting Healthcare organizations require better insight into their operations and accountability for their performance. Healthcare organizations must allow for creative use of available data and analytic tools to foster decision making in real time and near the point of care. To keep up with pace of change, healthcare organizations must adopt an agile approach to data and analytics that values innovation and experimentation. 48

49 Integrating analytics with quality and safety programs Improving quality and safety there is nothing else! Healthcare analytics tools work best when used in conjunction with active quality improvement initiatives and quality control process Quality improvement approaches such as Lean and Six Sigma benefit greatly from the information and insight made possible through analytics Move beyond quality reporting: analytics tools can be used to answer the tough questions and provide innovative solutions in healthcare improvement 49

50 Questions? Please submit your comments/questions electronically using the Chat Box on the bottom right of your webinar screen 50

51 CFHI Measurement Webinar Series Upcoming Sessions. November 12: Improvement Cost-Benefit Analysis with Dr. Nicole Mittman Gain an introductory understanding of resource use and costs using case examples of healthcare improvement Learn where to source data on resources and costs and ways to think about units of measurement that are applicable to your quality improvement needs. Find out ways to embed economic parameters into your data collection processes 51

52 CFHI Measurement Webinar Series Upcoming Sessions. January 28, 2014: Analyzing Data Over Time for Quality Improvement with Melanie Rathgeber and Trevor Strome The differences between a run chart and control chart How to create and analyze control charts Linking data from QI activities with organizational performance data 52

53 Available Resources for Everyone: How to Make a Run Chart in Excel Run Chart Case Example Additional Resource List CFHI Collaboratives & EXTRA Fellows: Access to coaching Instructions on how collaborative team members can access coaching with Melanie and Trevor will be made available through your respective Online Learning Platforms. 53

54 Thank You! Please complete a short survey available here: cfhi-fcass.ca 54