Control Charts and Trend Analysis for ISO 17025

Transcription

1 2015 Webinar Series Control Charts and Trend Analysis for ISO /27/2015 Speakers Cynthia Mangione, Microbiology QAO, New York State Department of Agriculture and Markets, Food Laboratory Division, Albany, NY Karen Stephani, Quality Manager, New York State Department of Agriculture and Markets, Food Laboratory Division, Albany, NY Objectives At the conclusion of this program, participants will be able to: Develop quality control charts in Excel and calculate standard deviation based on data Use quality control data to estimate uncertainty of measurement Develop Excel charts to track trending for microbiological qualitative analysis Continuing Education Credit CE credit is not available for this program, but you will receive a Certificate of Attendance. Evaluation/Certificate of Attendance Process Continuing education credit is not available for this program. Individuals who successfully complete the program and evaluation by 5/27/2015 will receive a Certificate of Attendance. 1. Go to to complete the evaluation. 2. After you complete the evaluation, you will automatically receive the Certificate of Attendance. 3. Type or print your name on the certificate. Archived Program The archived streaming video will be available within two days. Anyone from your site can register view the program. URL: Comments, opinions, and evaluations expressed in this program do not constitute endorsement by APHL. The APHL does not authorize any program faculty to express personal opinion or evaluation as the position of APHL. The use of trade names and commercial sources is for identification only and does not imply endorsement by the program sponsors. This program is copyright protected by the speaker(s) and APHL. The material is to be used for this APHL program only. It is strictly forbidden to record the program or use any part of the material without permission from the author or APHL. Any unauthorized use of the written material or broadcasting, public performance, copying or re-recording constitutes an infringement of copyright laws. The Association of Public Health Laboratories (APHL) sponsors educational programs on critical issues in laboratory science. For more information, visit

2 Control Charts and Trend Analysis for ISO Speakers: New York State Food Laboratory s Quality Assurance Team 1 ISO Requirements, 5.9.1: The laboratory shall have quality control procedures for monitoring the validity of tests and calibrations undertaken. The resulting data shall be recorded in such a way that trends are detectable and, where practicable, statistical techniques shall be applied to the reviewing of the results. This monitoring shall be planned and reviewed and may include, but not be limited to, the following: a)use of CRMs and/or RMs b)proficiency-testing; c)replicate tests; d)retesting; e)correlation of results for different characteristics. 2 ISO Requirements, The laboratory shall have quality control procedures for monitoring the validity of tests and calibrations undertaken. We address this in two places in our management system: Our Quality Manual, section 5.9 o The NYS Food laboratory monitors the quality of test results by the inclusion of quality control measures in the performance of tests and participation in proficiency testing programs. The laboratory runs Quality Control (QC) samples with each batch of samples. Trending Analysis and Control Charting procedures can be found in NYS SOP GP-5-15 Trending Analysis, Control Charting and Method Uncertainty 3 1

3 NYS Food Lab Requirements GP-5-15 Section 5.2 Control Charting: GP-5-15 Trending Analysis, Control Charting, and Calculation of Measurement Uncertainty, Section 5.2: A control chart is a graph of test results with respect to time or sequence of measurements, with limits drawn within which results are expected to lie when the analysis is in a state of statistical control. A procedure is in statistical control when results consistently fall within established control limits. Control charts can be constructed by hand or by using a commercial QC or statistical software (e.g. NWA Quality Analyst), or a spreadsheet (e.g. Excel). Each section of the laboratory shall determine the appropriate control charts to use for their specific analytical methods. Examples of control charts are the Mean Value control chart and the Range control chart. 4 3 types of Mean Value Control Charts we discuss in our SOP: Lab Control Sample charts (QC samples) Matrix Spike Control samples Process control charts this is just like the QC one, just spelled out in more detail in the SOP for our purposes. 5 6 Control Charts for Laboratory Control Samples These are charts created using a QC sample this can be a reference material, an old PT sample, etc, but it must have a known value associated with it. This may be characterized by your laboratory. We use Excel to create our control charts. We try to use a QC that is a similar matrix and close to the range of expected results. Each time the QC is run, the analyst enters the point into the control chart, and this allows them to quickly see if the run passed or failed. Mercury in Fish example 2

4 Control Chart for Matrix Spikes These are charts created using a blank matrix that has been spiked with a known concentration of analyte. Note that finding a blank matrix can often be problematic, especially if you are looking for trace level analytes. These are used when an appropriate reference material is unavailable. This is typical with food testing, as finding RMs in the right matrix is often difficult and/or cost prohibitive. We chart the percent recovery of the spike. As long as the results fall within specified criteria, the QC passes. A typical acceptance for matrix spikes is %, but for large screens with many analytes, often % is acceptable. 7 Calculating control limits: If an RM is being used that has a certified value with statistics (i.e. an acceptable range or standard deviation), we will create a control chart using those numbers as our upper and lower control limits and upper and lower warning limits After 20 points have been added to the chart, we will then calculate our own in-house statistics to determine our UCL/LCL and UWL/LWL Mercury in Fish example - recalculating We also recalculate after points to update our in-house statistics: Example - recalculating after 20 points 8 Control Limits for Mean Value Control Charts, GP-5-15, section : Since the control limits are based on probability, when a system is in statistical control o 2/3 of the values should be within the mean ± 1s. o 19/20 or 95 % of the values should be within the mean ± 2s. (Upper and Lower Warning Limits) o All or 99.7% of the values should be within the mean ± 3s. (Upper and Lower Control Limits) 9 3

5 NYS Food Lab Requirements Control Chart Interpretation: For Mean Value Control Charts, the process may be out of control if: One value or more fall outside 3s (outside the upper or lower control limits). Two or more consecutive values fall outside 2s (outside the upper or lower warning limits) on the same side of the mean. A series of seven or eight consecutive values fall all above or all below the mean. An increasing or decreasing trend is detected. 10 Range Control Charts: Control Chart for Duplicate Sample Data: These are used when using the same QC sample over time to generate an average is impossible, and spiking is not an option. (Dairy chemistry samples, for example) Two samples in a batch are analyzed in duplicate and the % difference or the absolute difference of the duplicates is plotted. After points have been collected, calculate the average range of duplicates. There are tables (Youden) for determining the percentage that should fall above and below the 50% line We are not currently using these in the lab for any accredited analyses, so we are NOT the experts seek more guidance on Range control charts elsewhere! 11 Trend Analysis: Section of the ISO standard also states that The laboratory shall have quality control procedures for monitoring the validity of tests and calibrations undertaken. The resulting data shall be recorded in such a way that trends are detectable This monitoring shall be planned and reviewed Section 5.1 of GP-5-15 addresses this requirement: All accredited laboratory analysis shall be analyzed for trending. Trending can be accomplished by periodically reviewing control charts, databases or excel spreadsheets. Each analysis shall be evaluated by the section supervisor and the QAM, or designee, to determine the best way to monitor for trending. Control charts will not be created for qualitative analyses. Qualitative analyses shall have databases or excel spreadsheets developed for the purpose of trending analysis. At a minimum the QAM, or designee, shall review all analyses for trending once a year. 12 4

6 Trend Analysis of GP-5-15 The objective of reviewing control charts is to catch problems and make corrections before the situation has become out of control. Systematic trends shall be investigated. Charts shall be monitored by analysts entering data, whenever practicable. All charts shall be reviewed on a routine basis by the Quality Assurance Officer (QAO) responsible for the specific section of the laboratory. The section QAO shall determine the time period (e.g. once a month) or a number of points collected (e.g. 10 data points) as a definition of routine basis. Documenting out of control situations shall be the responsibility of the analyst, the section supervisor and the QAO. If it is determined the process is out of control, a corrective action shall be started. 13 Measurement Uncertainty: We also use our control charts to estimate our measurement uncertainty, as required by section of the standard: Testing laboratories shall have and shall apply procedures for estimating the uncertainty of measurement. Section of our Quality Manual states: The NYS Food Laboratory maintains control charts as described in NYS SOP GP-5-15: Trending Analysis, Control Charting and Calculation of Measurement Uncertainty. Control chart limits are used to estimate measurement uncertainty, unless otherwise specified in the method procedure. 14 Measurement Uncertainty: GP-5-15 section 5.6 details how we calculate our measurement uncertainty: When the laboratory control sample (LCS) can be run through all method steps, the standard deviation (SD) from the LCS precision data shall be used as an estimate of combined measurement uncertainty. A relative SD (or CV) may also be used. It is recommended that 20 or more data points be obtained to estimate the standard deviation and/or relative standard deviation. The estimate of uncertainty shall be calculated using the formula: Measurement Uncertainty for a Defined Matrix (LCS) = k x SD Where k (the coverage factor) equals 2 (for 95% confidence) when using 20 data points. If fewer than 20 LCS data points are available, the coverage factor k should be the appropriate t statistic for 95% confidence for the associated degrees of freedom. Consult an appropriate student s t-distribution table. 15 5

7 Qualitative Tests - Microbiology Requirement: For qualitative tests appropriate controls shall be included whenever possible in order to demonstrate that the test worked. The suitability of the controls used shall be justified by the laboratory. Every SOP states what positive and negative controls shall be used in that procedure: MICRO-MTH-301 Salmonella VIDAS Method A positive culture control (Salmonella (sp) Typhimurium, MDP-014, pya3553) and negative culture control (E. coli ATCC 25922) shall be run with each day s batch of samples. Qualitative Tests - Microbiology Excel spread sheet VIDAS tracking false positive rate using Test Value PCR ABI 7500 fast tracking false positive rate using number of cycles. Microbiology - Salmonella Trending 17 Questions 18 6