Mathematical Analysis of Cervical Cancer Screening Policies (Does the Federal Requirement Make Any Sense?)

Size: px
Start display at page:

Download "Mathematical Analysis of Cervical Cancer Screening Policies (Does the Federal Requirement Make Any Sense?)"

Transcription

1 Mathematical Analysis of Cervical Cancer Screening Policies (Does the Federal Requirement Make Any Sense?) James C. Benneyan, Ph.D. Northeastern University, Boston MA ABSTRACT This paper summarizes several recent analytic studies to investigate cervical cancer screening policies commonly used in the examination of Pap smears for early detection. In particular, we investigate and compare: a) the policy mandated by the Clinical Laboratory Improvement Amendments Act (CLIA), b) recently proposed automated rescreening approaches, and c) possible improvements and alternatives to CLIA. These alternatives are motivated by a desire to reduce the high costs associated with diagnostic errors and are similar to manufacturing policies that use multiple 1% inspections to eliminate nonconforming product in the presence of inspection error. Using mathematical and expected cost models, significant improvements in early detection and total costs are to be shown possible by changing the policies currently mandated under CLIA. Additionally, the use of automated rescreening devices recently approved by the FDA in many cases can be a move in the wrong direction in terms of accuracy and cost, despite widespread marketing by the manufacturers to the contrary. INTRODUCTION Recent litigation and several large settlements due to false-negative Pap smear results have attracted considerable media attention and public concern. For example, Robb (1993) stated that 15 of every 1 pathologists currently are involved in a malpractice lawsuit, of which 10% directly concern a false-negative determination from a Pap smear. In order to minimize future liability and total costs, the re-design of clinical laboratory policies should depend on the ability to detect truly positive and truly negative patients and on all costs that are incurred as a result of these policies. Motivated by these statistics and concerns, we therefore have derived several mathematical models as part of an interdisciplinary effort to develop a thorough statistical and economic understanding of cancer screening processes, to compare alternatives, and to identify the overall optimal policy. This approach is similar philosophically to that taken in industry, in which probability and inspection cost models are used to determine whether or not to inspect production and to identify the effects of multiple inspections of accepted or reected items. Currently, as an attempt to improve the accuracy of the overall screening process, the 1976 Clinical Laboratory Improvement Act (amended 1988) requires that a pathologist randomly sample and rescreen a minimum of 10% of all Pap smear slides that initially are udged to be negative by laboratory technicians (called cytotechnologists). Additionally, 1% of slides udged to be positive by cytotechnologists are re-examined by a pathologist. An editorial by Melamed and Flehinger (1992) in the medical ournal Acta Cytologica, however, criticized the use of this mandatory 10% random rescreening rule as a means of detecting unacceptable performance in Pap smear screening. Numerous other authors also have questioned this policy since its original introduction in 1976 and proposed alternative quality assurance policies. A study by Ashton (1989), however, indicated that most cervical cytology laboratories continue to apply the 10% rescreening rule as a method of quality assurance, with some facilities rescreening negative slides at rates higher than 10% (Koss, 1993). More recently, automated rescreening systems have been advocated by their manufacturers (Koss et al 1994, Patten et al 1996), although their value relative to CLIA has been questioned and point to the need for further analysis (Hutchison et al 1996, O Leary et al 1998). The models summarized below can be used to help study and compare the effectiveness of CLIA, of rescreening rates other than 10%, multiple rescreenings, and of automated rescreening in terms of overall sensitivity, specificity, and total costs. Results to-date suggest that significant improvements in the detection of cervical cancer and in total costs are possible via alternate screening policies instead of the current policy mandated under CLIA, with one to three repeated screenings by cytotechnologists and 0% or 1% of all negative slides being rescreened by a pathologist (Benneyan, 1997). Furthermore, the proposed use of automated rescreening has been shown in many likely scenarios to actually increase cost and reduce accuracy when compared to either CLIA of the optimal non-automated policy (Benneyan 1997, Kaminsky et al 1997). 1 Proceedings of the 20 IEMS Conference

2 PROBABILITY OF DETECTING TRULY POSITIVE OR TRULY NEGATIVE PATIENTS For conventional (non-automated) screening processes, Benneyan (1997) and Benneyan and Kaminsky (1995) developed mathematical models that determine the probability of detecting truly positive and truly negative patients, based on the following assumptions and notation: 1. Each cytotechnologist i and each pathologist has a sensitivity of 1 - α ci and 1 - α p, respectively. Thus, the probability that a pathologist will correctly classify a truly positive slide as positive is 1 - α p and the probability that the i th cytotechnologist will correctly classify a truly positive slide as positive is 1 - α ci. These sensitivities are assumed to remain constant over time and to be independent of the rescreening rate. 2. Each cytotechnologist i and each pathologist has a specificity of 1 - β ci and 1 - β p, respectively. Thus, the probability that the i th cytotechnologist will correctly classify a truly negative slide as negative is 1 - β ci, the probability that a pathologist will classify a truly negative slide as negative is 1 - β p. Again, these specificities are assumed to remain constant over time and are independent of the rescreening rate. 3. Each slide may be inspected repeatedly by a specified number,, of independent cytotechnologists. A negative reading by any cytotechnologist i < causes a slide to be read by the next technologist. If any technologist determines a slide to be positive, the slide is given to a pathologist for confirmation. If all cytotechnologists determine the slide to be negative, it is randomly selected at a rescreening rate of (r x 1)% by a pathologist. With these assumptions, the probability of detecting truly positive and truly negative patients can be shown to be: ( ) Probability of detecting a truly positive patient = p = 1 αp 1 ( 1 r) αci, and 1 Probability of detecting a truly negative patient = q = 1 βp1 ( 1 r) ( 1 βc ) i. 1 Additionally, if n positive patients are screened independently, then the number of correct positive and correct negative identifications can be modelled with binomial distributions with parameters n and p and parameters n and q, respectively. As shown below, these mathematical models can be used to examine the magnitude of the expected number of correct positive and negative identifications as a function of α ci, α p, β ci, β p, and the rescreening rate r. Figure 1 illustrates the effect of the rescreening rate on the overall sensitivity for various scenarios, on the left using ust one cytotechnologist and on the right using two identical cytotechnologists (with α c1 = α c2 ). Overall Policy Sensitivity After 1 Screening per Smear Overall Policy Sensitivity After 2 Screenings per Smear 0 ab Technician ensitivit ab Technician ensitivit ound ound escreening Rate (%) escreening Rate (%) Figure 1: Number of Correct Diagnoses per 10 Positive Patients (Pathologist Sensitivity = 0.95) As these examples show, the mandated 10% rescreening policy does not significantly improve the detection of truly positive patients when compared to a 0% rescreening policy. Additionally, significant improvements in the detection of truly positive patients are possible by using more than one cytotechnologist. Finally, it can be shown that the pathologist sensitivity, 1 - α p, is a mathematical upper bound on the overall sensitivity of any sequential screening policy, regardless of the rescreening rate, whereas the overall specificity is bounded mathematically above by the pathologist specificity, 1 - β p. 2 Proceedings of the IEMS Conference

3 Figure 2 summarizes similar results for the number of correct identifications of truly negative patients. Figures 1 and 2 also nicely illustrate the inherent tradeoff that confronts risk managers - namely that repeated screenings and additional cytotechnologists decrease the rate of false-negatives, but increase the false-positive rate and screening costs, making identification of the best policy difficult. These tradeoffs therefore are incorporated into the following cost model in order to help determine the optimal minimum total cost laboratory screening policy. System Specificity with 1 Cytotechnologist Screening System Figure Specificity 7b: Rescreening with 2 with 2 Cytotechnologists Screenings Number of Correct Diagnoses per 10 Negative Patients Cytotechnologist Specificity Bound Number of Correct Diagnoses per 10 Negative Patients Cytotechnologist Specificity Bound Rescreening Rate (%) Rescreening Rate (%) Figure 2: Number of Correct Diagnoses per 10 Negative Patients (Pathologist Specificity = 0.80) TOTAL COST OF CANCER SCREENING POLICY Benneyan (1997) and Benneyan and Kaminsky (1995) also developed several cost models to help determine the total cost of any specific lab screening policy, based on the following additional assumptions and notation: 4. Slides that are udged to be negative by all cytotechnologists either are accumulated in a pool until that pool reaches a pre-determined size m, at which time a random sample of size rm is selected for rescreening by a pathologist, or are randomly selected to be rescreened immediately with probability r. 5. The following costs may be incurred a random number of times per every n processed smears: k 1 = per slide screening cost by a cytotechnologist, k 2 = per slide screening cost by a pathologist, k 3 = average cost per false positive result, k 4 = average cost per false negative result, and k 5 = per slide screening cost by a automated rescreening process. Note that all of the realities of the true system are captured in the above assumptions and that the concept of a cycle in assumption 4 defines a regenerative process. Using these costs and the previous notation, the expected value of the cost per n processed smears for any particular values of and r, denoted EC,r, can be shown to be: 1 i 1 i 1 1 EC r, = k np( c) + i c c c k n( p) c i k( i) k ck ci i k + k ( α α 1 α α 1 1 β β ) β + ( 1 βc ) k = 2 = 1 = 1 2 k = 1 k = ( ) + ( ) + ( kn 2 ( 1 p) 1 ( 1 r) p αc ( 1 p) 1 βc kn( p) c ( r) knp p) i i β 1 i α 1 α c ( 1 r) i This cost model now can be used to evaluate a variety of screening policies for any specified of values of k 1, k 2, k 3, k 4, α ci, α p, β ci, β p, p, and n. Moreover, the optimal rescreening rate r and number of cytotechnologists which minimize the total expected cost also can be found, such as by a simple half-interval search. For example, Table 1 compares the expected costs for four situations where the following values were assumed, based on discussions with pathology practitioners, and assuming identical cytotechnologists such that α c1 = α c2 = α c3 = = α c5 : k 1 = $3., k 2 = $9., k 3 = $750., n = 1, and p = Proceedings of the IEMS Conference

4 Table 1. Examples of Total Expected Cost per 1 Smears and Identification of the Optimal Policy Number of Technicians Rescreen Rate (%) Technician Sens =.95 Pathologist Sens =.95 Expected Cost per 1 Processed Smears Average Cost per False-Negative = $5,0 $50,0 Technician Sens =.60 Pathologist Sens =.60 Technician Sens =.75 Pathologist Sens =.95 Technician Sens =.75 Pathologist Sens = $1,273 * $5,337 $2,695 $22, ("CLIA") $1,673 $5,593 $2,953 $20, $2,072 $5,848 $3,210 $19, $2,472 $6,104 $3,467 $18, $2,871 $6,360 $3,725 $16, $3,271 $6,615 $3,982 $15, $3,670 $6,871 $4,239 $14, $4,070 $7,127 $4,497 $12, $4,470 $7,382 $4,754 $11, $4,869 $7,638 $5,011 $9, $5,269 $7,894 $5,269 $8, $1,434 $4,760 * $1,861 * $9, $1,845 $5,102 $2,230 $9, $2,257 $5,444 $2,599 $9, $2,669 $5,785 $2,968 $9, $3,080 $6,127 $3,337 $9, $3,492 $6,468 $3,706 $9, $3,903 $6,810 $4,075 $9, $4,315 $7,152 $4,444 $9, $4,727 $7,493 $4,813 $8, $5,138 $7,835 $5,182 $8, $5,550 $8,176 $5,551 $8, $1,890 $4,804 $2,2 $6, $2,283 $5,168 $2,383 $6, $2,676 $5,532 $2,765 $6, $3,068 $5,896 $3,146 $7, $3,461 $6,260 $3,528 $7, $3,853 $6,624 $3,910 $7, $4,246 $6,988 $4,291 $8, $4,639 $7,352 $4,673 $8, $5,031 $7,716 $5,054 $8, $5,424 $8,080 $5,436 $8, $5,816 $8,444 $5,818 $9, $2,339 $5,082 $2,368 $5,994 * : : : : : : 4 1 $6,070 $8,697 $6,071 $9, $2,766 $5,440 $2,775 $6,212 : : : : : : 5 1 $6,310 $8,938 $6,312 $9,687 Cost per single technician evaluation = $3 Technician Specificity = 95% Cost per single pathologist evaluation = $9 Pathologist Specificity = 95% Cost per false-positive = $750 Incidence Rate of Cervical Cancer = 1.5% * = Optimal policy 4 Proceedings of the IEMS Conference

5 DISCUSSION The scenario in column 3 represents fairly accurate technicians and pathologists with high sensitivities and specificities, resulting in the minimal total cost policy being a single cytotechnologist screening and a pathologist rescreening rate of 0% (not 10%). Note that a savings of approximately $4 per 1 smears would result from changing from current practice to this optimal policy, equating to roughly $208 million per year if extrapolated to the 52 million smears examined annually in the U.S. Under CLIA, however, this policy is not permitted. It also is interesting that the policy of single screening with a 10% rescreening rate, which is the minimum permitted under CLIA, has a higher expected cost than 2 cytotechnologist screenings with a 0% rescreening rate. The scenario in column 4 uses the same costs and high specificities, but now with lower sensitivities (more falsenegatives) for both the technicians and pathologists. These results indicate that the optimal policy now is to use 2 cytotechnologists, again with a rescreening rate of 0%, at a savings of over $832 per 1 smears, or roughly $418 million and 93,6 fewer false-negatives per year. In this case, the multiple sequential technician screenings counteract their low individual sensitivities, although the expected cost of the optimal policy is considerably higher than for the previous example. The overall cost of poor individual sensitivity therefore can be quite high. The example in column 5 may be the most likely in terms of sensitivities, again resulting in the optimal policy being to use 2 independent screenings and a 0% rescreening rate. In this case, changing from CLIA to this optimal policy would result in $1,092 savings per 1 smears and an increase in overall process sensitivity by over 20%, or roughly $568 million and 120,4 fewer false-negatives per year. For completeness, column 6 illustrates a more extreme false-negative cost, resulting in 4 independent evaluations and 0% rescreening being optimal by several magnitudes, with a savings over CLIA of almost $15,0 per 1 smears and an increase in overall process sensitivity by over 28%, or roughly $7.4 billion per year and 164,0 fewer false-negatives per year. This last case illustrates that scenarios theoretically can exist where something more than double rescreening is optimal. This can happen, for example, when the cost of false-negatives is extremely high relative to that of false-positives and to that of a pathologist examination, as might be the case when testing for life threatening medical conditions. Note that high false-negative costs very dramatically increase the total cost of any laboratory policy, with less rescreening resulting in higher total costs. The above results are similar to industrial experiences where multiple inspections significantly improve overall outgoing quality in the presence of inspection error, the optimal number of re-inspections often is between 1 to 3, and partial inspection of manufactured items never results in minimum costs. In fact, Benneyan and Kaminsky (1995) showed that the optimal screening policy for any combination of sensitivities, specificities, incidence rate, and number of cytotechnologists also always must employ either 0% or 1% rescreening. That is, in no case will partial rescreening ever result in the best possible laboratory policy. This is a very significant result in light of current laboratory practices and federal requirements to the contrary. An algorithm also has been developed to identify the optimal amount of rescreening in any particular scenario that minimizes all competing costs, again very similar to industrial inspection research. For example, Figure 5 compares an example of the identification of the optimal inspection policy on the left with the identification of the optimal cervical cancer screening policy on the right. $10. $8. $6. $4. $2. $0. ($2.) Inspection Cost Accepted NC Cost Reected NC Cost Reected C Cost (Sale Price) Total Expected Profit Total Exp Cost per 1 Smears $25,0 $20,0 $15,0 $10,0 $5,0 $0 Total 0% Rescreening Rate Total 1% Rescreening Rate * ($4.) Number of Inspections () Number of Cytotechnologist Screenings Figure 5: Examples of Optimal Redundant Industrial Inspection and Cancer Screening Policies 5 Proceedings of the IEMS Conference

6 CONCLUSION The models summarized in this paper can assist risk managers, clinicians, and regulatory bodies to design more effective and minimum cost screening policies and requirements. Preliminary exploration across a range of scenarios developed from the literature suggests that significant improvements are possible by revising the CLIA legislation, estimated between 90,0 to 160,0 fewer false-negatives and $4 to $850 million savings per year nationwide. Additionally, these models have been used to show that: CLIA mathematically never is optimal by any criteria and always increases total costs; Overall CLIA sensitivity never can be improved beyond certain mathematical bounds; No amount of partial rescreening is ever optimal; and In some plausible scenarios, 2-4 multiple evaluations of each smear is optimal. Although not the focus here, similar results have shown that the proposed use of automated rescreening technology recently approved by the FDA can dramatically increase overall costs without significantly improving sensitivity, despite widespread marketing by manufacturers to the contrary (Benneyan , Kaminsky et al 1997). It is significant that these theoretic results also closely agree with a recent independent large empirical study (O Leary et al 1998). REFERENCES Ashton PR: "American Society of Cytotechnology Quality Assurance Survey Data: Summary Report". Acta Cytologica 33: , Benneyan JC: Optimal Policies for Clinical Laboratory Quality Control, AHCPR Grant Summary ROSHS , 1998, Benneyan JC: Some Approaches to Quality in the Presence of Inspection Error, PhD dissertation, Industrial Engineering and Operations Research, University of Massachusetts, Amherst, Benneyan JC, Kaminsky FC: "Statistical and Economic Models for Analysis and Optimal Design of Laboratory Screening Policies for Cervical Cancer", Annals of Operations Research, 67: , "Clinical Laboratory Improvement Amendments of 1988, PL 1-578", Congressional Record. 134: , Hutchinson ML, Inhorn SL, Papillo J, Lapidus SN: A Perspective on Modern Quality Control Methods, Acta Cytologica., Kaminsky FC, Benneyan JC, Mullins DL: "Mathematical Models for Evaluating Overall Process Sensitivity, Specificity, and Cost of Automated Rescreening in Cervical Cytology", Acta Cytologica 41(1): , Kaminsky FC, Benneyan JC: "Alternative Inspection Policies in Pap Smear Testing for the Detection of Cervical Cancer", presented at the First International Applied Statistics in Medicine conference, Koss LG: "Cervical (Pap) Smear: New Directions". Cancer 71:1406:1412, Koss LG, Lin E, Schreiber K, Elgert P, Mango L: "Evaluation of the Papnet Cytologic Screening System for Quality Control of Cervical Smears", Anatomic Pathology 101(2): , Melamed MR, Flehinger BJ: "Reevaluation of Quality Assurance in the Cytology Laboratory". Acta Cytologica 36:61-65, O Leary TJ, Tellado M, Bucker S-B, Ali IS, Stevens A, Ollayos CW: PAPNET-Assisted ReScreening of Cervical Smears: Cost and Accuracy Compared with a 1% Manual Rescreening Strategy, JAMA 279: , Patten SF Jr, Lee JSJ, Nelson AC: "NeoPath, Inc.: NeoPath AutoPap 3 Automatic Pap Screener System", Acta Cytologica 40(1):45-52, Robb JD: "The Pap Smear is a Cancer Screening Test: Why Not Put the Screening Error Rate in the Report?". Diagnostic Cytopathology 9: , Proceedings of the IEMS Conference