Successful performance in proficiency testing (PT) is required

Transcription

1 Performance Characteristics of Several Rules for Self-interpretation of Proficiency Testing Data R. Neill Carey, PhD; George S. Cembrowski, MD; Carl C. Garber, PhD; Zohreh Zaki, MD Context. Proficiency testing (PT) participants can interpret their results to detect errors even when their performance is acceptable according to the limits set by the PT provider. Objective. To determine which rules for interpreting PT data provide optimal performance for PT with 5 samples per event. Design. We used Monte Carlo computer simulation techniques to study the performance of several rules, relating their error detection capabilities to (1) the analytic quality of the method, (2) the probability of failing PT, and (3) the ratio of the peer group SD to the mean intralaboratory SD. Analytic quality is indicated by the ratio of the PT allowable error to the intralaboratory SD. Failure of PT was defined (Clinical Laboratory Improvement Amendments of 1988) as an event when 2 or more results out of 5 exceeded acceptable limits. We investigated rules with limits based on the SD index, the mean SD index, and percentages of allowable error. Results. No single rule performs optimally across the range of method quality. Conclusions. We recommend further investigation when PT data cause rejection by any of the following 3 rules: any result exceeds 75% of allowable error, the difference between any 2 results exceeds 4 times the peer group SD, or the mean SD index of all 5 results exceeds 1.5. As method quality increases from marginal to high, false rejections range from 16% to nearly zero, and the probability of detecting a shift equal to 2 times the intralaboratory SD ranges from 94% to 69%. (Arch Pathol Lab Med. 2005;129: ) Successful performance in proficiency testing (PT) is required of all clinical laboratories licensed under the provisions of the Clinical Laboratory Improvement Amendments of Acceptable limits are specified in terms of fixed concentrations, fixed percentages, a fixed concentration and a fixed percentage, or 3 times the PT peer group SD. Proficiency testing providers express participant results in a variety of formats. Some providers provide pass-fail scoring alone, while others provide SD index (SDI) scores and graphic presentations of each result s deviation from the target value as a percentage. When passing scores are obtained, users must interpret the PT data for themselves to determine whether the data indicate the presence of significant bias or random error. Careful self-examination of Clinical Laboratory Improvement Amendments of 1988 mandated PT data can be useful because there are 5 replicates in a single PT event, more than the 2 or 3 controls in a typical analytic run. Therefore, PT data have higher sensitivity to error than control data. The performance characteristics of several rules for interpreting PT data were previously studied. 2 Accepted for publication March 31, From the Peninsula Regional Medical Center, Salisbury, Md (Drs Carey and Zaki); University of Alberta Hospital, Capital Health Authority, Edmonton, Alberta (Dr Cembrowski); and Quest Diagnostics, Incorporated, Lyndhurst, NJ (Dr Garber). Dr Zaki is now with Doctors Pathology Services, Dover, Del. The authors have no relevant financial interest in the products or companies described in this article. Reprints: R. Neill Carey, PhD, Peninsula Regional Medical Center, 100 E Carroll St, Salisbury, MD ( neill.carey@ peninsula.org). Counting rules based on SDI limits and rules with limits based on the mean SDI were investigated. It was found that rule performance depended on the ratio of the survey group SD, s g, to the internal laboratory SD (intrainstrument SD), s i. It was also found that as s g /s i increases, the sensitivities of different rules change. While no single set of rules performed optimally throughout the range of s g / s i values encountered, screening with the SDI rule was recommended (see Table 1 for a list of rules investigated in the present study and their definitions). Follow-up was recommended on data that failed the screening rule with a combination of the and R 4.0 SDI or R 3.0 SDI rules to demonstrate random error and the x 1.5 SDI or x 1.0 SDI rules to demonstrate systematic error. The choice of range and mean rules depended on the value of s g /s i. In the present work, we studied the performance characteristics of additional rules and compared the sensitivities of the rules for detecting errors with the probability of failing PT. MATERIALS AND METHODS Using Minitab (Minitab Inc, State College, Pa), we designed a computer program similar to the one used previously to study the performance characteristics of different rules for interpreting PT results 2 and simulated PT performance for a typical automated chemistry analyzer. We used the group mean (x g) and group SD (s g ) for each PT sample from the 1991 College of American Pathologists C-C survey. 3 Three analytes were selected to span the range of quality (based on the ratio of allowable error to s i ) from Six Sigma 4 to marginal (Table 2): potassium, creatine kinase (CK), and iron. The intrainstrument SD (s i ) was obtained from the College of American Pathologists quality assurance group summary report for control products XLS-71 and XPS- Arch Pathol Lab Med Vol 129, August 2005 Self-Interpretation of Proficiency Testing Data Carey et al 997

2 Table 1. Quality Control Rules Studied*, at least 1 result exceeds x g 2.0 s g limit, at least 1 result in each of 2 events exceeds x g 2.0 s g limit, at least 2 results exceed x g 2.0 s g limit, at least 1 result exceeds x g 2.25 s g limit, at least 1 result exceeds x g 3.0 s g limit, at least 2 results exceed x g 3.0 s g limit x 1.0 SDI, mean of 5 samples exceeds x g 1.0 s g limit x 1.5 SDI, mean of 5 samples exceeds x g 1.5 s g limit R 3.0 SDI, difference between any 2 results exceeds 3.0 s g R 4.0 SDI, difference between any 2 results exceeds 4.0 s g, combination rule; invoke if any rule exceeds limit, 1 result exceeds 75% of analyte-specific allowable error 5 x &1 50% EA, all results are on the same side of the mean, and 1 result exceeds 50% of analyte-specific allowable error /5 x &1 50% EA, combination rule; invoke if any rule exceeds limit x 1.5 SDI / /R 4.0 SDI, combination rule; invoke if any rule exceeds limit 2 PT limit, fail PT event * SDI indicates SD index; x g, group mean; s g, group SD; EA, allowable error; and PT, proficiency testing. Table 2. Method Quality* Method Quality Sigma QC Requirement Unacceptable 2.0 Total analytic error cannot be maintained allowable error Marginal 2.0 to 3.0 QC procedures with high probability of false rejection and replicate measurements of patient and QC samples Fair 3.0 to 4.0 Multirule with 4 6 QC measurements per run Good 4.0 to 6.0 Multirule with 2 3 QC measurements per run Six Sigma 6.0 Weak QC procedures with 2 QC measurements per run * Adapted from Garber and Carey 4 and Westgard. 6 QC indicates quality control. Allowable error divided by intrainstrument SD, assuming bias is negligible. / combination control rule. For example, rule. 153, 5 which incorporated the data from approximately 30 Ortho Vitros 400s and 700s (formerly known as Kodak Ektachem). Control products XLS-71 and XPS-153 are aliquots of the same lot and were analyzed in the 1991 College of American Pathologists C-C survey. Parameter scenarios of simulated conditions for analytes are shown in Table 3. For potassium, s g /s i averaged We set the allowable error at 0.5 meq/l (Clinical Laboratory Improvement Amendments of 1988 limits 1 ). The ratio of allowable error to s i averaged 7.1, making the analytic quality of potassium at least Six Sigma (Table 2) using the classifications of Garber and Carey 4 and Westgard. 6 For CK, s g /s i averaged We set the allowable error at 15% (Quality Management Program Laboratory Services, previously Laboratory Proficiency Testing Program, Ontario Medical Association 7 ). The ratio of allowable error to s i averaged 3.8, making the quality of CK fair. For iron, s g /s i averaged We set the allowable error at 20% (Clinical Laboratory Improvement Amendments of 1988 limits). The ratio of allowable error to s i averaged 2.9, making the quality of iron marginal. For each analyte, we simulated trial analyses of groups of 5 PT samples per PT event. We quantified the performance characteristics of each rule in terms of its probability for false rejection (P fr, probability to signal an error for an event when only the inherent error of the method is present) and its probability for error detection (P ed, probability to signal an error for an event when increased systematic or random error is present). Figure 1 shows our concept of the performance characteristics of the ideal rule for interpreting PT data. The x-axis is the percentage of allowable error. At low percentages of allowable error, the probability of error detection (probability of rejection) for the rule is negligible. At a desired transition point, the probability of rejection increases rapidly until the rule is nearly 100% certain to reject and indicates that there is a potential problem with the method. Every rule has 3 performance characteristics: P fr, P ed for each amount of increased systematic error, and P ed for each amount of increased random error. We calculated P fr as the proportion of reject signals when only the inherent error of the method was included in the simulated events. We calculated P ed as the proportion of reject signals when each stated amount of systematic error (bias) or random error (imprecision) was included in the simulated events. We studied the rules listed in Table 1. These include the rules identified previously as optimally performing, 2 with exception of the SDI rule (which had high probability of false rejection, P fr.38 for potassium, a Six Sigma method). We included 2 new rules, (invoked when 1 result exceeds 75% of analytespecific allowable error, a near miss ) and 5 x &1 50% EA (invoked Table 3. Parameter Scenarios of Simulated Conditions for Analytes Analyte Units Allowable Error Group Mean Group SD Potassium meq/l 0.5 meq/l Creatine kinase U/L 15% Iron g/dl 20% Intrainstrument SD Sigma Arch Pathol Lab Med Vol 129, August 2005 Self-Interpretation of Proficiency Testing Data Carey et al

3 Figure 1. Power function for idealized proficiency testing rule. At low percentages of allowable error, the probability of rejection is negligible. At a desired transition point where the error is a significant proportion of allowable error, the probability of rejection increases dramatically. P ed indicates the probability for error detection. Figure 3. Performance characteristics of selected proficiency testing (PT) rules for systematic error for creatine kinase (mean s g /s i, 1.70; mean E A /s i, 3.8). P ed indicates the probability for error detection. Figure 2. Performance characteristics of selected proficiency testing (PT) rules for systematic error for potassium (mean s g /s i, 1.23; mean E A /s i, 7.1). P ed indicates the probability for error detection. Figure 4. Performance characteristics of selected proficiency testing (PT) rules for systematic error for iron (mean s g /s i, 2.68; mean E A /s i, 2.9). P ed indicates the probability for error detection. when all results are on the same side of the mean and 1 result exceeds 50% of allowable error). We also studied the probability of PT failure ( 2 results exceeding allowable error) as errors increase, to ascertain which rules can detect errors before PT failure occurs. We used the model for the participating laboratories used previously. 2 RESULTS Figures 2, 3, and 4 are graphs of the performance characteristics of selected rules for systematic error for potassium, CK, and iron, respectively. The x-axes of these figures differ slightly from the x-axis of Figure 1; they are labeled in terms of systematic error, expressed as bias measured in terms of the intrainstrument SD, s i.abiasof 0.0 s i corresponds to the probability of false rejection (P fr, the probability that the rule signals an error when none is present aside from the inherent random error of the method). Biases of 0.5 and 1.0 s i are small shifts relative to allowable error for most analytes, and biases of 2.0 and 3.0 s i are shifts that could cause PT problems for low-quality analytes. There are dramatic differences in the abilities of the different rules to detect errors that must be detected to pass PT. For potassium (Six Sigma quality), any rule except the rule will have sufficient power to detect significant errors without excessive false rejections. For iron (marginal quality at 2.9 Sigma), however, the only rules that can detect systematic error before there is significant risk of failing PT are the and 5 x &1 50% EA rules. Creatine kinase (fair quality at 3.8 Sigma) is transitional; the traditional counting rules detect significant errors; however, they are less sensitive than they are for potassium. The rule has a significant probability of false rejection for CK and iron (although less than the previously recommended SDI rule 2 ). In comparison, a quality control (QC) procedure for routine internal control with the 1 3s /2 2s combination control rule (a combination of the 1 3s and 2 2s control rules), using 2 controls, 8 has P ed of roughly.50 to detect a shift of 2.0 s i and a high probability of detecting a shift of 3.0 s i. Table 4 summarizes the performance of the rules we studied for detection of systematic error. Arch Pathol Lab Med Vol 129, August 2005 Self-Interpretation of Proficiency Testing Data Carey et al 999

4 x 1.0 SDI x 1.5 SDI Table 4. 5 x &1 50% EA /5 x &1 50% EA x 1.5 SDI / /R 4.0 SDI x 1.0 SDI x 1.5 SDI 5 x &1 50% EA /5 x &1 50% EA x 1.5 SDI / /R 4.0 SDI x 1.0 SDI x 1.5 SDI 5 x &1 50% EA /5 x &1 50% EA x 1.5 SDI / /R 4.0 SDI Probability of Error Detection for Candidate Rules to Detect Systematic Error* Intrainstrument SD Units * Abbreviations are explained in the footnote to Table 1. Potassium Creatine Kinase Iron Response to Random Error Figures 5, 6, and 7 are graphs of the performance characteristics of selected rules for random error for potassium, CK, and iron, respectively. The x-axes of these figures are labeled in terms of random error: 1.0 s i, corresponding to the probability of false rejection (P fr, only the inherent random error of the method is present); 2.0 s i, a doubling of random error; and 3.0 s i, a tripling of random error. As with systematic error, rules that perform well for potassium and even for CK have low sensitivity to significant errors with the marginal iron method. Therefore, for iron, only the rule can detect significant random errors with higher probability than the probability to fail PT. In comparison, QC rules for routine internal control also have low probability to detect random error. For a doubling of random error, the 1 3s /2 2s control procedure with 2 controls 8 has P ed of roughly.30 for detecting the increased error. For a tripling of random error, the same control procedure has P ed of roughly.60 for detecting the increased error. Table 5 summarizes the performance of the rules we studied for detection of random error. Suggested Combination Rule No single rule performs optimally across the range of method quality and s g /s i ; however, we were able to combine 3 rules sensitive to systematic and random errors for analytes with different method qualities to create a combination rule with broad sensitivity to systematic and random errors. The combination rule x 1.5 SDI / /R 4.0 SDI has sensitivity to systematic and random errors across the range of method quality we studied. Its P fr is high only for marginal methods. Power functions for this rule for systematic and random errors are shown in Figures 8 and 1000 Arch Pathol Lab Med Vol 129, August 2005 Self-Interpretation of Proficiency Testing Data Carey et al

5 Figure 5. Performance characteristics of selected proficiency testing (PT) rules for random error for potassium (mean s g /s i, 1.23; mean E A /s i, 7.1). P ed indicates the probability for error detection. Figure 6. Performance characteristics of selected proficiency testing (PT) rules for random error for creatine kinase (mean s g /s i, 1.70; mean E A /s i, 3.8). P ed indicates the probability for error detection. Table 5. R 3.0 SDI R 4.0 SDI Probability of Error Detection for Candidate Rules to Detect Random Error* /5 x &1 50% EA x 1.5 SDI / /R 4.0 SDI R 3.0 SDI R 4.0 SDI /5 x &1 50% EA x 1.5 SDI / /R 4.0 SDI Multiples of Intrainstrument SD Potassium Creatine Kinase Iron R 3.0 SDI R 4.0 SDI /5 x &1 50% EA x 1.5 SDI / /R 4.0 SDI * Abbreviations are explained in the footnote to Table 1. Figure 7. Performance characteristics of selected proficiency testing (PT) rules for random error for iron (mean s g /s i, 2.68; mean E A /s i, 2.9). P ed indicates the probability for error detection. 9. As method quality increases from marginal to Six Sigma, false rejections range from 16% to nearly zero, and the probability of detecting a shift equal to 2 times the intrainstrument SD ranges from 94% for the marginal method to 69% for the Six Sigma method. COMMENT Previously, screening PT data was recommended with the 2 1s rule and confirming error with the x 1.0 SDI,R 4.0 SDI,,1 1.5 SDI,andR 3.0 SDI rules, depending on the s g /s i of the method. 2,9 That work did not consider the relationship of allowable error and the s g /s i of the method and the ultimate risk of PT failure. The present work shows that this pathway had (1) high probability of false rejections and sensitivity to small, possibly insignificant errors for high-quality methods and (2) low sensitivity for significant errors for low-quality methods. We have become aware of more optimally performing rules that are based on the PT allowable error rather than the SDI. The rule we Arch Pathol Lab Med Vol 129, August 2005 Self-Interpretation of Proficiency Testing Data Carey et al 1001

6 Figure 8. Performance characteristics of x 1.5 SDI / /R 4.0 SDI combination rule for systematic error. P ed indicates the probability for error detection; CK, creatine kinase; and K, potassium. Figure 9. Performance characteristics of x 1.5 SDI / /R 4.0 SDI combination rule for random error. P ed indicates the probability for error detection; CK, creatine kinase; and K, potassium. are suggesting as the outcome of the present study, the x 1.5 SDI / /R 4.0 SDI combination rule, is much more reliable for detecting quality failures that threaten PT failure, if not corrected, and focuses remedial effort on lowerquality methods where PT failure is much more likely to occur. Proficiency testing results provide an opportunity for participants to detect errors that have been missed by their usual internal QC procedures, because internal QC limits are inappropriately large or the errors were missed during review of internal QC data, or because of matrix differences between the QC materials and the PT materials. Proficiency testing results potentially have higher probability to detect errors than internal QC results from a single run because there are 5 samples in a PT challenge versus the customary 2 or 3 samples used in internal QC. When a participant reviews the results of a PT event to check for rule failures, the SDI results and the pictorial display of the results must be examined. The pictorial display of results on College of American Pathologists survey reports makes it easy for the participant to detect and 5 x &1 50% EA rule failures. Examples of these failures are Figure 10. Example plots of the relative distance of results from targets as percentages of allowed deviation from a College of American Pathologists survey report demonstrating 1 75%EA (survey set C-C, 2003) and 5 x &1 50% EA (survey set C-B, 2003) rule failures. shown in Figure 10 as C-C and C-B, respectively. (Note added in proof: For 2005 surveys, CAP has changed the format and reduced the size of these plots.) We recommend inspecting PT data with the x 1.5 SDI / /R 4.0 SDI combination rule. It has high sensitivity to systematic and random errors that have significant probability of causing PT failure regardless of method quality. The x 1.5 SDI and R 4.0 SDI rules are sensitive to systematic and random errors, respectively, for high-quality methods and have near zero probability of false rejection. When PT data fail either of these rules, bias or random error is almost certain to be present. For mid- to low-quality methods, the rule is sensitive to systematic and random errors. When there is a failure of the rule, follow-up is necessary to determine whether the cause of the failure is systematic or random error or a false rejection. If the PT data also fail the 5 x &1 50% EA rule, systematic error is likely because sensitivity of this rule to systematic error is as good as that of the rule, after allowance for the P fr of the latter. For the analytes we studied, P fr for the 5 x &1 50% EA rule is.034 or less. The x 1.0 SDI rule, with P fr approaching zero, is also useful for confirming systematic error in intermediate-quality methods if there is failure of the rule without failure of the x 1.5 SDI rule. We do not recommend screening with the x 1.0 SDI rule because it is sensitive to small, probably insignificant systematic errors for Six Sigma methods. For all methods, comparison of internal QC data from the date of the PT event with peer group data may help confirm the presence of bias. Investigating rule failures for intermediate- and marginal-quality methods is difficult because of its high P fr (.16 for iron) and the lack of sensitivity of other rules for random error. If the PT data also fail the R 3.0 SDI rule, random error is confirmed; however, PT failure is more likely to occur than R 3.0 SDI rule failure (and even R 2.0 SDI rule failure 2 ) as random error increases for iron. The R 3.0 SDI rule is more sensitive to random error for CK (P ed for doubling of random error is.407 vs probability of PT failure of.059). Review of QC data from the time of the PT event for indications of random error may be helpful. Sometimes it is impossible to confirm whether a rule failure represents random error or false rejection. In this case, a check of within-run precision by running 5 to 10 replicates of control material or a patient specimen, calculating the within-run SD, and comparing it with the manufacturer s claim for within-run precision may be the simplest way to decide whether increased random error is present. Fig Arch Pathol Lab Med Vol 129, August 2005 Self-Interpretation of Proficiency Testing Data Carey et al

7 Figure 11. Flowchart for interpreting proficiency testing (PT) data with the recommended combination rule. QC indicates quality control. ure 11 is a flowchart for interpreting PT data with the recommended combination rule. Ultimately, a laboratory s PT performance for a particular method is limited by the inherent quality of the method. For example, if the best precision a laboratory can achieve with a given method is at the 3.0 Sigma level, this method is a potential source of PT problems. Laboratories must consider method quality during the selection process and, whenever possible, select methods with the best analytic performance. Information is available in manufacturers package inserts and published method evaluations. Then, the method must be evaluated to demonstrate that the user can achieve the quality the manufacturer has claimed. Clinical Laboratory Standards Institute (formerly NCCLS) guideline EP15-A provides a procedure for accomplishing this with minimal experimental effort. 10 Garber and Carey 4 and Westgard 11 provide guidance for performing method evaluations and interpreting method evaluation data. To maintain precision, laboratories should review method quality as new lots of control materials are introduced, by comparing the SD used to set control limits (representing the true performance of the QC procedure) with allowable error by calculating the Process Sigma (E A / s i ). Whenever the Process Sigma is less than 4, QC ranges must be carefully set to maintain sensitivity to error, and control failures must be investigated and problems corrected. If there is a high prevalence of errors, more sensitive control procedures should be implemented. These methods should be scrutinized in daily operation and should be improved or replaced. Whenever a laboratory selects a multitest automated testing system, there is a compromise of the quality of 1 or more methods to gain the other benefits of automation. Some analytes are generally problematic because state-ofthe-art precision performance is marginal with respect to allowable error; sodium and chloride are examples. There are problems when PT samples have low concentrations of some analytes, for example, urea nitrogen and whole blood ethanol. These 4 analytes have been sources of near misses in our laboratories. We thank Ray Gambino, MD, and Peter Mallon, PhD, for creating the concept of a near miss in reviewing PT in the late 1980s at Quest Diagnostics, Incorporated (MetPath, Inc, Teterboro, NJ). A near miss was defined as a result that was in the outer 25% of the acceptable range. This situation prompted an investigation into the cause for that result being so far off target and required implementation of appropriate corrective action, to prevent further degradation and failure on the next event. References 1. Medicare, Medicaid and CLIA programs: regulations implementing the Clinical Laboratory Improvement Amendments of 1988 (CLIA), 57 Federal Register 7002 (1992). 2. Cembrowski GS, Hackney JR, Carey RN. The detection of problem analytes in a single proficiency test challenge in the absence of Health Care Financing Administration rule violations. Arch Pathol Lab Med. 1993;117: College of American Pathologists. Comprehensive Chemistry Survey: Set C- C. Northfield, Ill: College of American Pathologists; Garber CC, Carey RN. Evaluation of methods, In: Kaplan LA, Pesce AJ, eds. Clinical Chemistry: Theory, Analysis and Correlation. 4th ed. St Louis, Mo: CV Mosby Co; 2003: College of American Pathologists. Quality Assurance Services: Group Summary Report: Lot-Date Data Through June 1991, for Controls XLS-71 and XPS Northfield, Ill: College of American Pathologists; Westgard JO. Six Sigma Quality Design and Control. Madison, Wis: Westgard QC Inc; Ontario Medical Association. Cardiac Markers 0104: Survey Report. Toronto, Ontario: Laboratory Proficiency Testing Program, Ontario Medical Association; April Westgard JO, Barry PL, Hunt MR, Groth T. A multi-rule Shewhart chart for quality control in clinical chemistry. Clin Chem. 1981;27: Cembrowski GS. Thoughts on quality-control systems: a laboratorian s perspective. Clin Chem. 1997;43: Clinical Laboratory Standards Institute (formerly NCCLS). User Demonstration of Performance for Precision and Accuracy: Approved Guideline. Wayne, Pa: NCCLS; CLSI document EP15-A. 11. Westgard JO. Basic Method Validation. Madison, Wis: Westgard QC Inc; Arch Pathol Lab Med Vol 129, August 2005 Self-Interpretation of Proficiency Testing Data Carey et al 1003