How will you know How Will You Know That a Change Is An Improvement? Robert Lloyd, PhD John Boulton, MD Day 2 Concurrent Breakout Session 15 September 2014 1. If the change(s) you have made signal a true improvement? If you have sustained improvement? 2. If it is the right time to implement the change(s) 3. If it is time to spread the change(s) to other areas? 4. If it is time to stop measuring? 1
How will we know that a 4 An Option: Use Common Sense? change is an improvement? There are many hazards to the use of common sense. Common sense cannot be measured. You have to be able to define and measure what is significant. Without statistical methods you don t know what the numbers means. Source: Dr. W. E. Deming as quoted by N. Mann, The Keys to Excellence: The Story of the Deming Philosophy. Mercury Books, London, England, 1989: 62. 1. By understanding the variation that lives within your data 2. By making good management decisions about this variation (i.e., don t overreact to a special cause and don t think that random movement of your data up and down is a signal of improvement). 2
Unplanned Returns to Ed w/in 72 Hours Month M A M J J A S O N D J F M A M J J A S ED/100 41.78 43.89 39.86 40.03 38.01 43.43 39.21 41.90 41.78 43.00 39.66 40.03 48.21 43.89 39.86 36.21 41.78 43.89 31.45 Returns 17 26 13 16 24 27 19 14 33 20 17 22 29 17 36 19 22 24 22 u chart 1.2 1.0 0.8 0.6 0.4 0.2 0.0 1 UCL = 0.88 Mean = 0.54 LCL = 0.19 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 9/15/2014 Understanding Variation: You have a choice! How will we know that a change is an improvement? 6 5 STATIC VIEW Descriptive Statistics Mean, Median & Mode Minimum/Maximum/Range Standard Deviation Bar graphs/pie charts Rate per 100 ED Patients DYNAMIC VIEW Run Chart Control Chart (plot data over time) Statistical Process Control (SPC) Percent of patients assessed for pressure ulcers Percent of patients assessed for pressure ulcers 2013 2013 2014 2014 Which time period is better? This is the difference between static and dynamic displays of data! Now, which time period is better? Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004. 3
How do we analyze variation for 8 The Pioneers of Statistical Thinking quality improvement? W. Edwards Deming (1900-1993) Walter Shewhart (1891 1967) Joseph Juran (1904-2008) Run and Control Charts are the best tools to determine if our improvement strategies have had the desired effect. 4
Measure Measure 9/15/2014 Elements of a Run Chart Elements of a Control Chart Pounds of Red Bag Waste 6.00 5.75 5.50 5.25 5.00 4.75 4.50 4.25 4.00 3.75 3.50 3.25 Time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Point Number Four simple run rules are used to determine if non-random data patterns are present The centerline (CL) on a Run Chart is the Median Median=4.610 ~ X (CL) Number of Complaints 50.0 45.0 40.0 35.0 30.0 25.0 20.0 15.0 10.0 5.0 An indication of a special cause Jan01 Mar01 May01 July01 Sept01 Nov01 Jan02 Mar02 May02 July02 Sept02 Nov02 Time Month UCL=44.855 A B C CL=29.250 C B A (Upper Control Limit) X (Mean) LCL=13.645 (Lower Control Limit) 5
But, you need a Roadmap for your Quality Measurement Journey 11 12 The Quality Measurement Journey AIM (How good? By when?) Concept Measure Operational Definitions Data Collection Plan Data Collection Analysis ACTION AIM (How good? By when?) Concept Measure Operational Definitions Data Collection Plan Data Collection Analysis ACTION Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004. Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004. 6
Common Cause Variation Is inherent in the design of the process Is due to regular, natural or ordinary causes Affects all the outcomes of a process Types of Variation Results in a stable process that is predictable Also known as random or unassignable variation Special Cause Variation Is due to irregular or unnatural causes that are not inherent in the design of the process Affect some, but not necessarily all aspects of the process Results in an unstable process that is not predictable Also known as non-random or assignable variation 13 How will you know 1. If the change(s) you have made signal a true improvement? 2. If you have sustained improvement? 3. If it is the right time to implement the change(s) 4. If it is time to spread the change(s) to other areas? 5. If it is time to stop measuring? SPSP 7
Run Chart Rules are used to determine if a change has occurred A Shift: 6 or more A Trend 5 or more Random Variation (common cause) 16 Too many or too few runs Use the run chart rules to determine if a change has occurred An astronomical data point A shift = 6 or more data points above or below the baseline median (centerline) A trend = 5 data points constant going up or down Source: The Data Guide by L. Provost and S. Murray, Jossey-Bass Publishers, 2011. Nothing has changed here! 8
Measure LOS (minutes) 9/15/2014 How will you know 1. If the change(s) you have made signal a true improvement? 2. If you have sustained improvement? 3. If it is the right time to implement the change(s) 4. If it is time to spread the change(s) to other areas? 5. If it is time to stop measuring? SPSP 320 300 280 260 240 220 200 180 160 Sustained Improvement First identify a shift or a trend in the data Then look to see if 3 or more data point have stayed at the new level. Time Median A downward shift in the data (6 data points below the median) 2/16/11 3/16 4/13 5/11 6/8 Week 3 more data points staying at the new level of performance 9
How will you know 1. If the change(s) you have made signal a true improvement? 2. If you have sustained improvement? 3. If it is the right time to implement the change(s) 4. If it is time to spread the change(s) to other areas? Degree of Belief When Making Changes to Improve Degree of belief that a change will result in improvement HIGH MODERATE Source: The Improvement Guide, Langley, J. et al, Jossey-Bass, 2009: 145. Change needs further tesing Successful change! 5. If it is time to stop measuring? SPSP Unsuccessful change! LOW Developing a change Testing a change - cycle 1, cycle 2, cycle 3 Implementing a Change 10
Implementing a Change Conditions for Implementing a Change Current Situation Resistant Indifferent Ready Baseline Testing Successful Testing Begin implementation on pilot unit Evidence of improvement during implementation Low Confidence that current change idea will lead to Improvement Cost Risk of of not failure succeeding large large Cost Risk of of not failure succeeding small small Very Small Scale Test Very Small Scale Test Very Small Scale Test Very Small Scale Test Very Small Scale Test Small Scale Test Note that when you move to full implementation things may actually get worse for a little bit. High Confidence that current change idea will lead to Improvement Cost Risk of of not failure succeeding large small large Cost Risk of not failure succeeding small small Very Small Scale Test Small Scale Test Small Scale Test Large Scale Test Note the conditions for Implementing a change! Large Scale Test Implement 11
How will you know 1. If the change(s) you have made signal a true improvement? 2. If you have sustained improvement? 3. If it is the right time to implement the change(s) Spreading a Change First identify a shift or a trend in the data. Then look to see if 6 or more data point have stayed at the new level. This indicates that you are holding the gains. A downward shift in the data (6 data points below the median) 6 more data points staying at the new level of performance 4. If it is time to spread the change(s) to other areas? 5. If it is time to stop measuring? SPSP Collaborative Holding the Gains John Whittington OSF Healthcare 12
How will you know 1. If the change(s) you have made signal a true improvement? 2. If you have sustained improvement? 3. If it is the right time to implement the change(s) 4. If it is time to spread the change(s) to other areas? 5. If it is time to stop measuring? SPSP Two Simple Rules for Measuring Outcome Measures always! Process Measures it depends! 13
How often do you need to measure? It is not uncommon for a team to want to stop collecting data, especially after they have been at it for a year or two! The reliability of the process and your need to know how the process is functioning should drive the frequency of data collection and analysis. A Simple Rule for Outcomes Outcome Measures Always! As long as you are concerned about the quality and safety of the care that you deliver, you should continue to track the outcomes! For example, how long should these outcomes be measured? When do you stop measuring your financial results? When should a diabetic patient stop tracking his or her blood glucose? How long should we monitor the vital signs of an ICU patient? When should airport security stop assessing passengers for weapons? How long does a local water authority need to measure the quality of the water going through its pipes? When should schools stop measuring the progress of students? 14
A Simple Rule for Processes Process Measures it depends! Process measures usually demonstrate improvement before outcome measures. Process measures may be revised during an improvement project; new data will then need to be collected and tracked. A process measure should demonstrate improvement (against the run chart rules) and then STAY at the new level of performance for at least 3 reporting periods to be considered sustained. Frequency of Process Measures Regularly (daily, weekly or monthly) Done to improve a specific measure (reduce variation or shift the centerline of process performance) Periodically (once every 2-3 months) Done when statistical improvement has been noted, sustained AND the process is highly reliable (audit approach can be used here) Once or twice a year (why bother?) Stop measuring! Done when performance is so reliable, stable and capable that it is time to move on to measure something new. 15
Minutes ED to OR per Patient 1. Make process performance visible Minutes ED to OR per Patient 1200 1000 800 600 400 200 0 1200 1000 800 600 400 200 0 Current Process Performance: Isolated Femur Fractures Baseline Data 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 Sequential Patients Process Improvement: Isolated Femur Fractures Change Introduced here 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 Sequential Patients 2. Determine if a change is an improvement Minutes ED to OR per Patient 1200 1000 800 600 400 200 0 The Steps in Statistical Thinking Holding the Gain: Isolated Femur Fractures Sustained Improvement 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 Sequential Patients 3. Determine if we are holding the gains The Statistical Thinking Framework Systems Thinking Process Philosophy of Measurement Variation Analysis Statistical Methods Data Improvement Action Statistical Thinking will be found in all aspects of organizational behavior and performance Source: American Society for Quality, Statistics Division. 16
Attributes of a Person Who Understands Variation They understand the different ways in which variation is viewed (static versus dynamic). They explain changes in terms of common causes and special causes. They use graphical methods to plot data over time, learn from data and expect others to consider variation in their decisions and actions. They understand the concept of stable and unstable processes and the potential losses due to tampering. They understand the capability of a process or system before changes are attempted. Theory and Prediction The Sequence of Improvement requires Measurement Test under a variety of conditions Developing a change Testing a change Make part of routine operations Implementing a change Sustaining improvements and Spreading changes to other locations 34 17
35 In short, statistical thinking is a way to approach all aspects of work. It is a way of thinking about numbers and how they can be used to make improvements. Statistical thinking is the primary way to know if a change has led to improvement! Statistical Thinking is a Way of Life! 18