Executive Veterinary Program University of Illinois December 11-12, 2014 Outline Diagnostics Dr. Randall Singer Professor of Epidemiology review of sensitivity, specificity, predictive value test agreement parallel and serial testing herd-level sensitivity, specificity Using Diagnostic Tests Diagnostic tests are imperfect the outcomes are not always correct For diagnostic testing to be perfect, it would require that: All individuals without the disease of interest would have one uniform value for the test, All individuals with the disease have a different but uniform value for the test, and thus All test results are consistent with the results of the diseases or those of the non-diseased. Variation exists within each of the 3 basic factors: the tests (their reproducibility), the group with the disease, and the group without the disease. Test Sensitivity and Specificity Need a reference ( gold standard ) to classify animals as infected or uninfected (these methods are usually time-consuming, expensive, complex, and/or invasive not used as routine tests) Diagnostic SE and SP are a function of the selected cut-off 1
Definitions: SE and SP Analytical SE: Minimum concentration of the analyte (pathogen/antibody) that needs to be present in an assay for a positive result to occur Diagnostic SE: Proportion of truly diseased individuals correctly identified Pr(T+ D+) Analytical SP: Ability of the test to not react to analytes other than the analyte of interest Diagnostic SP: Proportion of truly non-diseased individuals correctly identified Pr(T- D-) Test Sensitivity and Specificity eg. Trichinella spiralis infection in pigs gold standard = digestion of a diaphragmatic sample but time-consuming, requires slaughter of the pig alternative: ELISA to detect antibodies validity of ELISA results can be expressed relative to the gold standard Test Sensitivity sensitivity # diseased animals classified as diseased by the test total # diseased animals ELISA sensitivity = 40 45 = 89% (sensitivity = 100% - false negative reactor rate) ELISA false negative reactor rate = 5 45 = 11% Test Specificity specificity # uninfected animals classified as uninfected by test total # uninfected animals ELISA specificity = 45 55 = 82% (specificity = 100% - false-positive reactor rate) ELISA false-positive reactor rate = 10 55 = 18% 2
Reasons for Poor Test Sensitivity tolerance (natural, induced) eg. exposure to BVDV during early gestation: antibodynegative infected calves timing testing prior to antibody response post-parturition colostrum Ig serum Ig non-specific inhibitors eg. anticomplement blocking antibodies eg. IgG1 vs. IgG2 immunosuppression laboratory errors ELISA against Dictyocaulus viviparus lungworm of cattle false negatives sensitivity Reasons for Poor Test Specificity cross reactions eg. M. paratuberculosis and M. tuberculosis non-specific reactions exposure unrelated to disease eg. vaccination, passive immunization, previous exposure laboratory errors ELISA against Dictyocaulus viviparus lungworm of cattle Test Sensitivity and Specificity in general, sensitivity and specificity are correlated and depend on cut-point: false positives specificity sensitivity specificity sensitivity specificity 3
Test Sensitivity and Specificity highly sensitive tests: few false-negatives results early phases of pathogenesis (many aetiologic possibilities) screening test in disease control, eradication programs severe zoonotic diseases exotic diseases highly specific tests: few false-positive results confirm diagnosis follow-up test major implications of a positive test eg. slaughter of a test reactor Screening Evaluation with gold standard Training sample: sample of diseased and nondisease individuals Obtain: Pr(T+ D+) = SE Pr(T- D- ) = SP Really want is: Pr(D+ T+) = Predictive value positive (PVP) Pr(D- T- ) = Predictive value negative (PVN) Bayes theorem Predictive Values Predictive value positive positive predictive value: probability that an animal testing positive is infected negative predictive value: probability that an animal testing negative is uninfected how tests are used Sensitivity and specificity of a test based on a gold standard SE = Pr(T+ D+)=a/(a+c) SP = Pr(T- D-) = d/(b+d) PVP = Pr(D+ T+) = a/(a+b) PVN = Pr(D- T-) = d/(c+d) Overall accuracy = (a+d) / N D+ D- T+ a b a + b T- c d c + d a + c b + d N 4
Predictive Values Predictive Values positive predictive value # test-positive diseased animals total # test-positive animals negative predictive value # test-negative non-diseased animals total # test-negative animals ELISA positive predictive value = 40 50 = 80% positive-testing animal has 80% chance of being infected ELISA negative predictive value = 45 50 = 90% negative-testing animal has 90% chance of being uninfected Predictive Values ELISA +ve 8 17 25 ELISA ve 1 74 75 Total 9 91 100 sensitivity = 89%, specificity = 82% positive predictive value = 8 25 = 32% (80) negative predictive value = 74 75 = 99% (90) predictive value influenced by disease prevalence disease common NPV disease rare PPV Choosing a cut-off value It depends on the purpose of the test High SE: Zoonosis with high public health impact Introduction of highly contagious infection in disease free population Detection in early stages of infection when many potential causes High SP: Need to confirm diagnosis Cost of false positive is high 5
Choosing a cut-off value For simplicity, test results are commonly reported as positive/negative Information is lost to the user Lab may choose the cut-off that minimizes FP and FN error Other cut-offs may be more appropriate for specific situations Important!! Changing cut-off cannot be done without proper assessment Receiver Operating Characteristic (ROC) curve Obtain SE (true positive fraction) and 1-SP (false positive fraction) for each possible cutoff It does not depend on the original unit or range of the test results It assumes that mean value of D- is < than mean value of D+ subpopulations ROC curve The area under the curve (AUC) is a global statistic of diagnostic accuracy Non-informative = 0.5 Less accurate = >0.5, 0.7 Moderately accurate = >0.7, 0.9 Highly accurate = >0.9, <1 Perfect test = 1 AUC: probability that a randomly selected individual from D+ has a greater test value than one from D- AUC gives equal weighting to SE and SP Good to compare multiple tests Testing in Series all samples tested with test #1 positive samples tested with test #2 test 1 +ve test 2 +ve -ve specificity ( false-positives) Testing in Parallel all samples tested with both tests positive result = positive in either test test 1 +ve or test 2 +ve +ve sensitivity ( false-negatives) 6
Herd-Level Testing Herd-Level Testing > one animal can be tested herd status (infected, uninfected) herd-level sensitivity: truly infected herd classified infected by the test herd-level specificity: truly uninfected herd classified uninfected by test Herd-Level Testing performance of herd-level tests affected by: individual-level test sensitivity, specificity number of animals tested critical number of positives at which herd is declared test positive Herd-Level Specificity HSP: (specificity) animals tested eg. test specificity = 82% 2 cows tested: HSP = (82%) 2 67% 3 cows tested: HSP = (82%) 3 55% 10 cows tested: HSP = (82%) 10 14% if test specificity is <100%, as more animals are tested the herd-level specificity decreases Herd-Level Sensitivity HSE: 1 (1 apparent prevalence) animals tested apparent prevalence estimated from true prevalence (TP): TP * SE + (1 TP) * (1 SP) eg. apparent prevalence = 50% 2 cows tested: HSE = 1 (1 0.5) 2 75% 3 cows tested: HSE = 1 (1 0.5) 3 87.5% 10 cows tested: HSE = 1 (1 0.5) 10 99.9% as more animals tested, herd-level sensitivity increases Summary sensitivity: identification of infected animals specificity: identification of uninfected animals predictive value: based on test result parallel testing: sensitivity serial testing: specificity herd-level sensitivity, specificity: multiple testing to determine herd status 7