SPC, COPQ & Waste into Dollars

Transcription

1 SPC, COPQ & Waste into Dollars Erika Sundrud, M.A. Associate Vice President Quality, Safety & Performance Improvement Objectives Describe how to use data to drive improvement and decision making Develop a data collection plan Use measures of data and variation Interpret graphical displays of data 2

2 The Concept of Knowledge Data symbols, facts, numbers Information data that has been processed Knowledge application of data and information Understanding the process by which one can take knowledge and synthesize new knowledge; "why? Wisdom evaluated understanding; the process by which we also discern, or judge; future thinking, vision and design 3 Data Exist to Answer Questions These questions exist only to help answer much larger questions: Why? What? Why did that happen? Why did that change work? What should I do now? 4

3 Making decisions that are not backed by data You barely scratch the surface of truly understanding the situation. 5 Data 6

4 The Concept of Knowledge Knowing what we Know Knowing what we Don t Know Using what we Know 7 Initial Steps in Designing a Data Collection Program Clear statements of Monitoring Goals and specific Objectives Compile/summarize available data Develop conceptual models (Logic Model) Select indicators for monitoring and determine the appropriate sampling design and protocols 8

5 Data Collection Plan 1. Purpose of Collecting Data 2. Generalizations to be made from data 3. What data will be collected 4. Measurement process to be used 5. Sampling methodology to be used 6. Form design 7. Pilot test 8. Assignment of responsibility 9. Training 10. Monitoring against the plan 11. Storage, editing 9 Data Collection Plan 1. Why collect the data? The purpose, objective, or question to be answered needs to be clearly defined. 2. What methods will be used for the analysis? The method of analyzing the data should be determined to assure that the data is collected in a way that allows the purpose/objective to be achieved or the questions to be answered. 3. What data will be collected? The purpose for collecting and method of analyzing determines what data will be collected. Operational definitions and the context of the data need to be provided. The context would include the date, time, location, person, equipment and particular item being measured. Without this information, the data would be suspect and, in many cases, would not allow detailed analysis. 4. How will the data be measured? This would include any tools needed, such as measuring devices or standards for comparison, the procedure for obtaining a measure and the place to record it, such as a check sheet. 10

6 Data Collection Plan 5. When will the data be collected? This would define the days, shifts, time, duration and sequence for collecting the data. 6. Where will the data be collected? This defines the actual location where the data will be collected. It would be collected from files or records, or at different locations (steps) within the process. 7. Who will collect the data? The persons who will collect the data should be those who work in the process and are involved in improving it. If possible, they should collect the data as part of their regular duties, but may have to be relieved if performing their duties would not allow for this. 8. What training is needed for the data collectors? The training needs to explain why the data is being collected and how the data is to be collected. It should also allow for the data collectors to review the check sheet and ask any questions. ALWAYS PILOT YOUR DATA COLLECTION PLAN! 11 CONCEPT MEASURE DEFINITION CALCULATION Mean ( X ) The arithmetic average of a set of numbers. = Sum of all X values # of X values Measures of Data Central Tendency Median ( X ) The midpoint of a set of numbers after they have been ordered. Order the data highest to lowest. For an odd # of Xs, = the middle value. For an even # of Xs, = the average of the two center values. Mode Range ( R ) The value in a set of numbers which appears most frequently. The difference between the highest and lowest numbers in a set of numbers. Example: 1,2,3,3,3,4,4,5,6 The mode = 3. R = X Max X Min Variability Variance ( S 2 ) The average of the squared deviations from the mean. S 2 = (X1 X ) 2 n 1 Standard Deviation ( S ) The square root of the variance S = S 2 12

7 Variation First Fundamental Theorem: 13 What is Variation? Statistics textbook definition*: When data values differ from each other Other terms used: diversity, uncertainty, dispersion, and spread Merriam-Webster Dictionary: 3b: a measure of the change in data, a variable, or a function *Source: Siegel, A., Practical Business Statistics, 4 th Edition. 14

8 Types of Variation Common Cause uncontrollable, natural variation that occurs within every process. Special Cause variation that is caused by external forces causing the process to become unstable or out-ofcontrol. If you don t know which variation is which: You may make unnecessary changes tampering You may miss needed changes bad process management Examples: Signature Drive to work 15 Understanding Variation When people don t understand variation: They see trends where there are no trends They try to explain natural variation as special events They blame and give credit to people for things over which they have no control 16

9 Glossary Run Chart simple chart of data points over time (I usually plot the median on the chart as well). Can be used on virtually any type of data (i.e., counts of events, percentages and dollars). Require no statistical calculations. Understood easily by everyone. Cannot determine process capability. Control Chart a run chart that includes Upper and Lower Control Limits (3 standard deviations from the mean). Used to assess the stability of a particular process. Useful Quality Management Tool. They can t make predictions or plan for the future you must do that! 17 Statistical Comparisons: LOS Variable N Mean (Avg) Median St. Dev. Min Max LOS LOS LOS What would you say if you saw this data? What does it tell you? Not a trick question. Source: Balestracci, D. Data Sanity : Statistical Thinking Applied to Everyday Data. Special Publication ASQ Statistics Division. Summer

10 Now What Does This Tell You? LOS Hospital LOS Hospital LOS Hospital Source: Balestracci, D. Data Sanity : Statistical Thinking Applied to Everyday Data. Special Publication ASQ Statistics Division. Summer Setting up a Run Chart In an Excel Spreadsheet, type in the overtime percentage for each day of the month. Calculate the average overtime percentage for each month. Using the Excel function key, calculate the overall median for all months. In a separate Excel Spreadsheet, create a database that looks like the following: OT% 6.92% 9.30% 5.58% 6.84% 6.78% 8.52% 8.84% 8.36% 10.80% 8.63% Median 7.00% 7.00% 7.00% 7.00% 7.00% 7.00% 7.00% 7.00% 7.00% 7.00% Go to the Chart Wizard tool and create a line chart. 20

11 SIMPLE Data Analysis Process For a quick look at your data: 1. PLOT data and the median in SIMPLE run chart (previous slide) 2. Then consult the data analysis guidelines 12% 11% 10% 9% 8% 7% 6% 5% 4% 3% Overtime % Months *Source: Balestracci, D. Data Sanity : Statistical Thinking Applied to Everyday Data. Special Publication ASQ Statistics Division. Summer SIMPLE Data Analysis Process Data Analysis Guidelines: 3. Count the number of Runs (runs are the number of data series above and below median) o How many Runs do you see here? 12% 11% 10% 9% 8% 7% 6% 5% 4% 3% 1 2 Overtime % Months

12 SIMPLE Data Analysis Process Data Analysis Guidelines: 4. Refer to Table 4.2 Tests for Number of Runs Above and Below the Median o Look for Range of Runs for 23 Data Points (don t count any points exactly at the Median) o What is the Range? o We only found Six Runs 8-16 Runs expected 12% 11% 10% 9% 8% 7% 6% 5% 4% 3% Overtime % Months SIMPLE Data Analysis Process Data Analysis Guidelines: 5. Since 6 Runs is not in the Range of 8-16, we can conclude that different processes may be going on here 12% 11% 10% 9% 8% 7% 6% 5% 4% 3% o Special Causes may have caused process to change after month 14. o Generally, a successful intervention will tend to create a smaller than expected number of runs. * Overtime % Months *Source: Balestracci, D. Data Sanity : Statistical Thinking Applied to Everyday Data. Special Publication ASQ Statistics Division. Summer

13 SIMPLE Data Analysis Process Data Analysis Guidelines: 6. Another way to find out if special causes have influenced your process: o 8 or more consecutive points are on one side of the median. 12% 11% 10% 9% 8% 7% 6% 5% 4% 3% Overtime % Months 25 SIMPLE Data Analysis Process Data Analysis Guidelines: 7. Diagnosing Trends o Trends in data are said to be SEVEN or more consecutive points in any one direction (SIX if you have 20 or less observations) o Omit from count any points that repeat previous value *Source: Balestracci, D. Data Sanity : Statistical Thinking Applied to Everyday Data. Special Publication ASQ Statistics Division. Summer

14 The Headlines Scream Great News! 27 Reality Projected

15 Now What? We have a run chart and determined that: A process change occurred around Month 14 We have 8 consecutive points on one side of data (before process change indication) If the process had in fact changed, start fully monitoring from Month 14 onward. Now we are set to monitor moving forward Control Chart. 10% 8% 6% Overtime % 4% Month 29 Behind the Scenes Control Charts use three sigma limits (sigma is the standard deviation) to minimize the possibility of making these errors. For a normally-distributed process output, only (0.3%) of the output falls outside + three-sigma # Standard Mean Deviations 30

16 Setting up a Control Chart Using the Excel function key, calculate the standard deviation for each month. Take the standard deviation times 3. Plus and minus the mean OT% 6.92% 9.30% 5.58% 6.84% 6.78% 8.52% 8.84% 8.36% 10.80% 8.63% Mean 7.00% 7.00% 7.00% 7.00% 7.00% 7.00% 7.00% 7.00% 7.00% 7.00% 3SL 8.68% 8.76% 8.65% 8.68% 8.66% 8.73% 8.74% 8.63% 8.69% 8.66% -3SL 5.32% 5.24% 5.35% 5.32% 5.34% 5.27% 5.25% 5.37% 5.31% 5.34% Go to the Chart Wizard tool and create a line chart. Data are plotted in time sequence with time always presented along the horizontal or X axis. The variable of interest, or key quality characteristic, is placed on the vertical or Y axis. 31 Control Charts Helps you find where the data may indicate special causes in the process. 12% 11% 10% 9% 8% 7% 6% 5% 4% Overtime % Months 3 SD Mean -3 SD 32

17 Control Charts P-chart very easy to use Use when you have a percentage you calculate 3 Standard Deviations common recommendations of statisticians (less likely to treat common causes as special causes) 12% 11% 10% 9% 8% 7% 6% 5% 4% Overtime % Months 3 SD Mean -3 SD 33 Control Chart Checks for Control Spotting Special Causes: or more data points falling outside the control limits 2. 7 or more consecutive points increasing or decreasing or more points on one side of the centerline or more points altering up and down 12% 11% 10% 9% 8% 7% 6% 5% 4% 1 1 Overtime % Months SD Mean -3 SD 34

18 Data based Decision Making Managers are routinely faced with interpreting their metrics and making a real-time decision as to whether the latest data point tells them to do something. Good graphical depiction goes a long way Seasoned managers can see signals through the noise Statistics can take the subjectivity out of such decisions One size does not fit all 35 What if it is Out of Control? Look at time frame when process is out of control Identify the major drivers within that process Identify potential outside influences to the process Develop strategy to minimize those outside influences if they are bad (if possible) You may need to implement a short-term fix, but be sure to find long-term solutions 36

19 What if it is Out of Control? Look at time frame when process is out of control Document on control chart to show causes of outof-control variation. Some Special Cause Variation may be good if it caused an improvement in the process you may need to look for a way to incorporate it in the longterm process. 37 What if it is in Control? In control and you are happy with the results of the process/data (Where it should be): Monitor the process using control charts and look for special cause variation In control and you are unhappy with the results of the process/data: Develop a strategy to improve the overall process while still monitoring to see if you make successful improvements 38

20 Improvement Strategies Improvement Strategies should be Determined By Sources of Variation SPECIAL CAUSE: Investigate. Find out what s different. Seek to stabilize the process. COMMON CAUSE: Take action on the system as a whole. Change the process if you want to improve! By reducing the amount of variation or by moving the whole process in the desired direction. 39 Be Careful Average calculations with no knowledge of the variation will not tell you anything about the process and changes made. Bar graphs really don t tell you anything about trends and differences from year to year. Be careful when setting arbitrary targets your process may never let you get there Example: flipping a coin Remember to: Simply plot the data on a run chart (if nothing else) Refer to Eight Common Traps in Balestracci publication 40

21 Possible Applications ED Wait Time IP Infection Rates EMTALA Compliance Expenses/Cost Length of Stay Overtime % Education Hours Blood Utilization % ADRs Nursing HPPD Complication Rate Mortality Rate % Residents on 9+ Meds MANY, many more 41 OTHER CHARTS 42

22 If the Question is: Which variables out of many are occurring most? Use PARETO CHART 43 Pareto Chart NUMBER OF FALLS % 100% 100% 100% 100% 100% 90% % 80% 70% % 5 33% PM 11 PM 4 PM 7 PM 8 PM 5 PM 6 PM 9 PM 10 PM Time 60% 50% 40% 30% 20% 10% 0% 44

23 Pareto Chart Help to narrow your focus to simplify problem solving. Narrows down your opportunities for improvement to what is manageable and will provide the best return on your investment. ***Directs your attention to the most frequent defects or nonconformities, but not necessarily the most important ones. 45 If the Question is: How is this one variable distributed (what is the spread of LOS, Cost, etc.)? Use HISTOGRAM 46

24 Histogram Cost per Case 3,500 3,400 3,300 3,200 3,100 3,000 2,900 2,800 Jan Feb Mar Apr May Jun DRG Histogram Bar chart for one variable. Each bar equal, distinct. Use to visualize central location, shape and spread of the data. Most often used with money, time, throughput, or a scaled measurement (i.e. dollars, weight, age, height). BE CAREFUL Does little good for interpretation if the process is not stable. Doesn t show stability or capability in and of itself! 48

25 If the Question is: Is variable A possibly related to variable B? Use SCATTER DIAGRAM 49 Scatter Diagram n=1 1.2 Calories Consumer/Weight Gain 1 Weight Gained in Pounds Weight Gained y=mx+b Calories Consumed 50

26 Scatter Diagram An indication of the relationship between independent and dependent variables Does NOT prove causation Does suggest further investigation 51 Interpreting Scatter Plots Narrow band of points extending from the lower left to the upper right suggests a positive correlation. Means that as one factor increases so does the other. Possible to predict the approximate value of one factor when you know the value of the other. 52

27 Interpreting Scatter Plots Narrow band of points extending from the upper left to the lower right corner suggests a negative correlation. Means that the factors react opposite to one another as one increases the other decreases. Possible to predict the approximate value of one factor when you know the value of the other. 53 Interpreting Scatter Plots Circular pattern suggests no correlation or relationship exists between the two factors you are studying. No way to predict reliably one factor from the other. 54

28 Lean: Cost of Poor Quality (COPQ) All activities and processes that do not meet agreed performance and/or expected outcomes Costs that would disappear if every task were always performed without deficiency. COPQ = Actual Cost Minimum Cost Three categories of COPQ: o Inspection/Appraisal o Internal Failures (Customer doesn t know about them) o External Failures (Customer knows about them) The Tip of the Iceberg: When accounting for Quality, don t forget the often hidden Cost of Poor Quality. J. A. DeFeo, May Methods for Estimating Cost Unit Cost Method Total Resource Method Gold Standard Process Flow Method 56

29 COPQ: Unit Cost Method Focuses on unit costs, requires two pieces of data: The number of times the error/deficiency occurs The average cost for correcting and recovering from that error/deficiency when it does occur Most appropriate when: Deficiencies occur rarely and may be costly Deficiencies are complex and require the participation of many departments to correct Deficiencies occur frequently and correcting them is so routine that those involved may not realize their pervasiveness Remember when calculating unit costs: Include benefits as well as wages and salaries Include allocated capital costs for major equipment and facilities While this is a minor consideration for many activities that can be safely ignored, it is vital for some activities The Tip of the Iceberg: When accounting for Quality, don t forget the often hidden Cost of Poor Quality. J. A. DeFeo, May COPQ: Total Resource Method Requires two pieces of data: Total resources consumed in a category The percentage of those resources consumed for activities associated with poor quality Example: How an operational unit calculates the human resource time to process customer complaints and the dollar value of that time The Tip of the Iceberg: When accounting for Quality, don t forget the often hidden Cost of Poor Quality. J. A. DeFeo, May

30 COPQ: Gold Standard Method Often used in hospitals Comparing hospital to the gold standard or best practice Quantifying how far your hospital is from the gold standard Source: Achieving and Sustaining Improvement in Cardiac Medication, Adrienne Elberfeld and Carolyn Pexton 59 COPQ: Process Flow Method Map the Value Stream All the steps required to complete a service beginning to end Flow chart the process, specifically looking for: Handoffs Time during and between actions, total process time Dollars used in each action Bottlenecks Lack of standardization Rework loops Potential resources used The Tip of the Iceberg: When accounting for Quality, don t forget the often hidden Cost of Poor Quality. J. A. DeFeo, May

31 What do you take away? Designing a Monitoring Program using the data collection questions Understand variation a little better How to find out if your process has significant variation Simple Run Charts Control Charts Know how to find out if processes in control or out of control and next steps to take 61 Summary of Key Points Before you initiate a data collection program, start with clear statements of goals and objectives. Decide on the measurement process and sampling methodology. Common cause variation is uncontrollable, natural variation that occurs within every process. Special cause variation is caused by external forces causing the process to become unstable or out-of-control. 62

32 Summary of Key Points If you don t know which variation is which: You may make unnecessary changes tampering You may miss needed changes bad process management A trend is identified when there are SEVEN or more consecutive points in any one direction (SIX if you have 21 or less observations) Omit from count any points that repeat previous value. 63 Resources Articles Balestracci, D. Data Sanity : Statistical Thinking Applied to Everyday Data. Special Publication ASQ Statistics Division. Summer Copies: ASQ Quality Information Center ( ). isixsigma website: Control Chart Wizard is excellent American Society of Quality: Engineering Statistics Handbook Online: Performance Resources Tools SPC Templates developed by Buddy Pelter 64

33 Resources Donald Wheeler. Understanding Variation. Knoxville: SPC Press Inc, Donald Wheeler. Making Sense of Data. SPC for the service sector. Knoxville: SPC Press Inc, WE Deming. Out of the Crisis. Massachusetts: MIT Donald M Berwick. Controlling Variation in Health Care: a consultation from Walter Shewhart. Med Care 1991; 29: Resources American Society for Quality American Hospital Database Health Grades CDC AHRQ HCUP - JCAHO - CMS QIO - CMS QnetExchange CMS Quality Assistance 66