Developing QC Plans Part 3. Performing a Risk Assessment to Identify Hazards and Target Failure Modes

Size: px

Start display at page:

Download "Developing QC Plans Part 3. Performing a Risk Assessment to Identify Hazards and Target Failure Modes"

Donna McBride
6 years ago
Views:

1 Webinar Series sponsored by Thermo Fisher Scientific Developing QC Plans Part 3. Performing a Risk Assessment to Identify Hazards and Target Failure Modes James O. Westgard Sten A. Westgard Westgard QC, Inc. Madison, WI 1

2 Webinar Series sponsored by Thermo Fisher Scientific September 17. Regulatory requirements and strategic approach October 8. Right sizing SQC procedures using Westgard Sigma Rules November 12. Performing a risk assessment to identify hazards and target failure modes December 10. Mitigating risk with an Individualized QC Plan 2

risk assessment Review CMS hazard identification guidance Identify variables ibl in risk ikmodels dl 2

3 Part 3. Performing a Risk Assessment to Identify Hazards and Target Failure Modes Review industrial process for risk assessment Review CMS hazard identification guidance Identify variables ibl in risk ikmodels dl 2 vs 3 factors, ranking scales, graphic vs calc of risk Illustrate application with qualitative scale Calculate risk to prioritize failure modes Identify controls to mitigate risks Calculate residual risks 3

4 A Plan for Developing QC Plans Validate Safety Characteristics Determine Sigma Quality Audit QC Practices Yes Analyze 2 levels of controls/day? No Develop Total QC Plan Develop Risk Based IQC Plan Monitor Quality & Patient Safety Improve Quality & Safety 4

5 Option for IQCP Organize RiskProject Team Assess Hazards, ID Failure Modes Evaluate Risk of Failure Modes Identify Controls to Reduce Residual Risks Implement IQC Plan 5

6 How does industry assess risk? (1) 1. Review the process or product 2. Brainstorm potential failure modes 3. List potential effects of failure modes 4. Assign SEVERITY ranking (SEV) 5. Assign OCCURRENCE ranking (OCC) 6. Assign DETECTION ranking (DET) 7. Calculate Risk Priority Number (RPN) 6

7 Industrial Calculation of Risk (2) 3 factor risk model Risk = SEV*OCC*DET = RPN Where SEV is severity OCC is occurrence DET is detection RPN is Risk Priority Number 7

8 How does industry assess risk? (3) 8. Prioritize the failure modes for action 9. Mitigate high risk failure modes Eliminate or reduce occurrence Improve detection Minimize severity of harm 10. Calculate RPN again to assess residual risks and determine acceptability 8

9 How does industry mitigate risk? (4) Occurrence Design for safety to eliminate failure modes Disclose safety characteristics Detection Build in control mechanisms Provide alerts to identify problems Severity Provide instructions for safe use and precautions for use errors Informlabof of limitations and recommend QC 9

10 Review CMS Guidance for IQCPs IQCP includes 3 components Risk assessment to evaluate hazards (steps 1 5) QC Plan to identify control mechanisms (steps 6 7) QA Plan to monitor quality and safety (step 8) To conduct a risk ikassessment, the laboratory must identify the sources of potential failures and errors for a testing ti process, and evaluate the frequency and impact of those failures and sources of errors 10

11 Review CMS Guidance for IQCP The Clinical and Laboratory Standards Institute (CLSI) document EP23 A Laboratory Quality ControlBased onrisk Management provides helpful guidance to laboratories on the development ofquality control plans for test systems. 11

12 CLSI EP23A Methodology (1) Identify Project, Assemble Team 1. INPUT INFORMATION Medical Requirements, Regulatory/Accred. Requirements, Manuf. Test System Info, Test Site Requirements 2. HAZARD IDENTIFICATION Review manufacturer s Risk Report, Map/chart process flow Identify failure modes (fishbone), Identify prevention/detection 3. RISK ESTIMATION 2 factor risk model, Probability of occurrence of harm, Severity of harm 4. RISK EVALUATION Risk Acceptability Matrix 12

13 CLSI EP23A Methodology (2) 5. RISK CONTROL Identify control mechanisms to optimize detection and lower risk to acceptable level 6. LABORATORY QC TOOL BOX Pre analytic controls, SQC, EQA, Trueness Controls Integrated Liquid Controls, Function Tests, Procedural Controls Electronic Checks, Calibrator Checks, Repeat Patient Testing Delta Checks, Implausible Values, Patient Population Algorithms (AoN) Correlation algorithms, Post analytic controls 7. QUALITY CONTROL PLAN Review conformance to requirements, Implement control mechanisms 8. QUALITY IMPROVEMENT Monitor performance, Reduce errors, Improve quality 13

14 Assess Hazards to Identify Failure Modes CMS prescribes specific categories and conditions that must be included din your risk ikassessment Specimen Environment Reagent Test System Testing personnel 14

15 Assess Hazards to Identify Failure Modes Hazards Checklist for Identification of Targeted Failure Modes (1) Test, Test System Project Group, Date Sigma Quality=(%TEa %Bias)/%CV y( Step/Procedure/Activity in the Total Testing Process Specimen Patient preparation Collection Labeling Storage, preservation, stability Transportation Acceptability and rejection Referral l( (to other lb) labs) What might cause a problem? Data? In house or manufacturer? Targeted Failure Mode? 15

16 Hazards Checklist for Identification of Targeted Failure Modes (2) Step/Procedure/Activity /A i in What might ihcause the Total Testing Process a problem? Environment Temperature Airflow/ventilation Light intensity Noise and vibration Humidity Altitude Dust Water Utilities (elect stability) Adequate space Reagent Shipping/receiving Storage condition requirements Expiration data Preparation Data? In house or manufacturer? Targeted Failure Mode? 16

17 Hazards Checklist for Identification of Targeted Failure Modes (3) Step/Procedure/Activity /A i in What might ihcause the Total Testing Process a problem? Test System Inadequate sampling Clot detection Interference detection Hemolysis Lipemia Icterus Turbidity Calibration Mechanical/electronic failures Optics Pipettes, pipettors Barcode readers System controls & function checks Procedural, electronic controls Liquid controls Temperature controls Software/hardware Data transmission to LIS Result reporting Data? In house or manufacturer? Targeted Failure Mode? 17

18 Hazards Checklist for Identification of Targeted Failure Modes (4) Step/Procedure/Activity in the Total Testing Process What might cause a problem? Data? In house or manufacturer? Targeted Failure Mode? Testing Personnel Training Competency Education, experience qualification Adequate staffing 18

19 Risk Assessment CLSI EP23A Model Risk combination of the probability of occurrence of harm and the severity of that harm Risk = OCC*SEV Detection not considered in EP23 risk assessment even though the purpose of a QC Plan is to provide detection of those failure modes that can not be eliminated or prevented 19

20 Variables in Risk Assessment Number of risk factors 2 factor models use OCC and SEV 3 factor models add DET Ranking scales common in industrial applications 1 5 common recommended in CLSI EP23A 1 3 useful for qualitative assessment Risk evaluation RPN calculation l or risk acceptability bl matrix 20

21 Generic Industrial Rankings RANKING SEVERITY OCCURRENCE DETECTION 10 Highly hazardous Very high Non detectable 9 Hazardous Very high Very improbable 8 Very high High Improbable 7 High High Very low 6 Moderate Moderate Low 5 Low Moderate Moderate 4 Very low Moderate Moderately high 3 Minor Low High 2 Very minor Very low Very high 1 None Remote Highly detectable Ref: Dailey. The FMEA Pocket Handbook. Pages

22 CLSI EP23A & ISO Ranking Scales and Acceptability bl Matrix Occ curren nce Ra ank Severity Rank Negligible Minor Serious Critical Catastrophic Frequent Probable U U Unacceptable Risk Occasional Remote Acceptable Risk U U Improbable U 22

23 Issues with Occurrence Ranking CLSI EP23A relates probability of occurrence to frequency of failures Rank of 1: Frequent, once per week Rank of 2: Probable, once per month Rank of 3: Occasional, once per year Rank of 4: Remote, once every few years Rank of 5: Improbable, once in life of system But, frequency doesn t reflect number of patients affected by failure patients affected by failure 23

24 Patients Potentially Harmed Depends on frequency of failure and number of patient test results affected by that failure Occurrence of hemolysis Frequency may be daily, but only be a single patient Occurrence of calibration error Frequency may be weekly, but would affect an entire run of patients, which could include the whole day or even more than one day More difficult to rank patients harmed than frequency of fil failure 24

25 Risk Assessment Westgard Six Sigma Model Quantitative approach that focuses on defect rate and estimation of number of potentially harmful patient test results produced Westgard JO. Six Sigma Risk Analysis: Designing analytic QC plans for the medical laboratory. Madison WI:Westgard QC, Recommends using TJC Proactive Risk Reduction methodology rather than CLSI EP23A Basis for simpler qualitative methodology here 25

26 Risk Assessment Example Westgard Qualitative Model 3 factors Rank on scale 0, 1, 2 Rank Severity Occurrence Detection 0 Very low (no harm) None to a few patients Very low or unknown 1 Moderate (delay) Some patients Moderate (50%) 2 Veryhigh (patient harm) Many patients Very high ( 90%) Calculate risk as SEV*OCC*(2 DET) Estimates will range from 0 to 8 Mitigate failure modes when risk is 2 and higher 26

27 Prepare Electronic Spreadsheet as Risk Assessment Table Note that CMS does not recommend nor require use of any specific tools, such as Failure Modes and Effects Analysis (FMEA), which is the standard tool in industry From Q&A in CMS Memo of 8/16/2013 Will laboratories be required to use a process map, fishbone diagram, formal risk assessmentcharts and protocols, etc., in their IQCP? No, CLIA will not require the use of these tools in the development of an IQCP. 27

28 Example Risk Assessment Table Failure Mode and Effects Analysis (FMEA) Step/Activity Targeted Effect SEV Cause OCC Controls DET Risk Failure SEV*OCC Mode *(2 DET) 28

29 Example POC Analyzer Cartridge provides multiple tests Sigma > 5 for triglycerides, glucose, creatinine Sigma > 4 for cholesterol, HDL Specimen is whole blood Small specimen size, finger sticks or small tubes Automated separation in cartridge to provide serum or plasma samples for analysis Analysis time is 5 minutes/cartridge Other cartridges available for other tests 29

30 FMEA Example: Specimen acquisition Turbidity, hemolysis, icteris Step/Activity Targeted Effect SEV Cause OCC Controls DET Risk Failure SEV*OCC* Mode (2 DET) 1 Photo. 1 Manuf. Specimen Turbidity Changes acceptability Hemolysis results Interfer. Check??? 0 1*1*(2 0) = 2 30

31 FMEA Example: Sample processing Instability when sample is on cells Step/Activity Targeted Effect SEV Cause OCC Controls DET Risk Failure SEV*OCC* Mode (2 DET) 1 Photo. 1 Manuf. Specimen Turbidity Changes acceptability hemolysis results Interfer. Check??? Sample processing Stability on cells Low glucose 2 Cell enzymes 1 SOP, Prompt analysis 0 1*1*(2 0) = 2 0 2*1*(2 0) = 4 31

32 FMEA Example: Reagent stability Affects enzymatic reactions Step/Activity Targeted Effect SEV Cause OCC Controls DET Risk Failure SEV*OCC Mode *(2 DET) 1 Photo. 1 Manuf. Specimen Turbidity Changes acceptability hemolysis results Interfer. Check??? Sample processing Analysis/ Test system Stability on cells Reagent stability Low glucose Low creat 2 Cell enzymes 2 Environ cond s 1 SOP, Prompt analysis 2 Electronic check 0 1*1*(2 0) = 2 0 2*1*(2 0) = 4 1 2*2*(2 1) = 4 32

33 FMEA Example: Operator variability Increases test variability Step/Activity Targeted Effect SEV Cause OCC Controls DET Risk Failure SEV*OCC Mode *(2 DET) Specimen Sample processing Turbidity Changes 1 Photo. 1 Manuf. hemolysis results Interfer. Check??? Stability on cells Low glucose 2 Cell enzymes 1 Prompt analysis 0 1*1*(2 0) = 2 0 2*1*(2 0) = 4 Analysis/ /Test Reagent Low 2 Environ 2 Electronic 1 2*2*(2 1) system stability creat cond s check = 4 Analysis/ Operator Variable 2 Operator 2 Training 0 2*2*(2 0) Operator technique results skills SOP = 8 33

34 FMEA Example: Turnaround Time Workload may delay results Step/Activity Targeted Effect SEV Cause OCC Controls DET Risk Failure SEV*OCC Mode *(2 DET) 1 Photo. 1 Manuf. Specimen Turbid Changes acquisition specimen results Interfer. Check??? Sample processing Analysis/Test System Time on cells Cartridge stability Low glucose Low creat 2 Cell enzymes 1 Environ cond s Analysis/ Operator Variable Operator technique results skills Result reporting 2 Operator 1 SOP. Prompt analysis 2 Electronic check TAT Delay 1 Workload 2 SOP, prompt analysis 0 1*1*(2 0) = 2 0 2*1*(2 0) = 4 1 1*2*(2 1) = 2 2 Training 0 2*2*(2 0) = 8 0 1*2*(2 0) = 4 34

35 Prioritization for Risk Mitigation Target any failure mode whose risk score 2 Prioritize failure modes with higher scores Score of 8 is maximum Many risk factor combinations yield scores of 4 and 2 Prioritize score of 2 due to SEVERITY over other scores of 2 35

36 Example of Risk Prioritization Targeted Failure Mode Risk Mitigation SEV OCC DET Residual Risk SEV*OCC*(2 DET) Operator variability (8) Samplestability (4) Turnaround time (4) Cartridge stability (2) Specimen acceptability (2) 36

37 Examples of Risk Mitigation (1) Priority 1: Operator variability (8) Monitor by Repeat Patient Test control daily Must establish expected limits of variability Repeat Patient Test daily for 20 days Calculate SD based on differences Set control limits for expected variation Priority 2: Glucose instability (4) Reduce processing time & improve storage (cold) Monitor pre analytic time to identify delays 37

38 Examples of Risk Mitigation (2) Priority 3: Turnaround Time (TAT) (4) Utilize patient log to track time of draw, time of analysis, and time of report Establish alert times to identify workload problems Identify conditions that lead to high workload Improve process to reduce TAT 38

39 Examples of Risk Mitigation (3) Priority4: Reagent/test system instability (2) For POC, analyze stable control material every other day to improve detection, or Repeat Patient Test vs a reference method/lab Qualify new reagent lots using right sized SQC Priority 5: Specimen acceptability (2) Materials, process, training for specimen acquisition Test manufacturer s control to validate capability to detect lipemia, hemolysis, icterus detect lipemia, hemolysis, icterus 39

40 Residual Risk What is risk after mitigation? Targeted Failure Mode Risk Mitigation SEV OCC DET Residual Risk SEV*OCC*(2 DET) Operator variability (8) Repeat Patient Test *2*(2 2) = 0 Sample stability (4) Improved storage & *0*(2 0) = 0 processing Turnaround time (4) Improve process flow, monitor pre analytic time on patient log Cartridge stability (2) Analyze stable control, or RPT vs Ref Method/Lab Specimen acceptability Instruct patients, (2) validate manuf. control *1*(2 1) = *2*(2 2) = *1*(2 1) = 1 40

41 REVIEW Risk Assessment Methodology Remember, CMS does not define nor recommend any standard methodology! Variables in methodology to consider 2 or 3 factor risk assessment model Ranking scales for risk factors Determination of risk Evaluation of acceptable tbl risk ik 41

42 REVIEW Process for IQCP Organize RiskProject Team Assess Hazards, ID Failure Modes Evaluate Risk of Failure Modes Identify Controls to Reduce Residual Risks Implement IQC Plan 42

43 REVIEW Qualitative Risk Assessment Process Start with CMS hazards list to identify failure modes Decide whether to use 2 or 3 factor risk model Examplehere uses SEVerity, OCCurrence, anddetection Define ranking scale (3, 5, 10 categories) Example here ranks factors as 0, 1, or 2 Define risk evaluation (Graphical vs calculation) Example here calculates risk as SEV*OCC*(2 DET) Prioritize risks Mitigate unacceptable risks 43

44 Risks of Risk Management A structured risk assessment methodology is needed to develop reliable QC Plans FMEA is a standard tool, but requires education and training for proper applications CLSI EP23A guidance neglects DETECTION and uses arbitrary risk acceptability matrix Risk assessment and evaluation is subjective 44

45 Risk Based QC has its own risks, ik particularly l residual risks ik Laboratory is depending on manufacturer to own up to limitations of performance Manufacturer passes residual risks on to lab Laboratory implements manufacturer s controls in its QC Plan Laboratory identifies additional controls on basis of its own risk assessment Still have residual risks 45

46 Who s responsible for residual risks in a medical laboratory? The director must consider the laboratory s clinical and legal responsibility for providing accurate and reliable patient test results versus the cost implications of reducing the quality control frequency testing. CMS, SOM item D

47 What s next? 4 th Webinar December 10 will focus on the implementation of Individualized QC Plans 47

48 Thank you for your kind attention! 48

49 Thank you for listening (Questions?) MUCH, MUCH MORE Can be found online At 49

Avoiding the Most Frequent Joint Commission Laboratory Survey Findings using Robust Process Improvement Tools (RPI) AVOIDING THE MOST

Avoiding the Most Frequent Joint Commission Laboratory Survey Findings using Robust Process Improvement Tools (RPI) AVOIDING THE MOST FREQUENT JOINT COMMISSION LABORATORY SURVEY FINDINGS USING ROBUST PROCESS