OIE principles and methods for validation of diagnostic tests. Ian Gardner

Transcription

1 OIE principles and methods for validation of diagnostic tests Ian Gardner

2 Overview of presentation Manual chapter on validation and guidelines Stages in the validation pathway Statistical methods guidelines (stage 2) Modifications for wildlife diseases (stages 1 and 2) Coherence between terrestrial and aquatics manuals Further guidance is needed Experimental challenge studies to estimate DSe/DSp

3 Principles & Methods of Validation of Diagnostic Assays for Infectious Diseases Updated chapter approved May 2013 Identical chapter in both manuals because principles and methods are same Chapter Terrestrial Manual Chapter Aquatics Manual Work in progress! Does not provide explicit guidance on multiplex assays or new technologies

4 Intended Purpose(s) for assay Essential Prerequisites Study Design and Protocol Assay Validation Pathway

5 Intended Purpose(s) for assay Essential Prerequisites Study Design and Protocol Analytical Specificity Analytical Sensitivity STAGE 1 Analytical characteristics Repeatability and preliminary Reproducibility Assay Validation Pathway

6 Intended Purpose(s) for assay Essential Prerequisites Study Design and Protocol Analytical Specificity Analytical Sensitivity STAGE 1 Analytical characteristics Repeatability and preliminary Reproducibility Assay Validation Pathway Diagnostic Specificity Diagnostic Sensitivity Cut-off Determination STAGE 2 Diagnostic characteristics Traditional approach assuming a perfect reference standard Latent class analysis to deal with imperfect reference standard

7 Intended Purpose(s) for assay Essential Prerequisites Study Design and Protocol Analytical Specificity Analytical Sensitivity STAGE 1 Analytical characteristics Repeatability and preliminary Reproducibility Assay Validation Pathway Diagnostic Specificity Diagnostic Sensitivity Cut-off Determination STAGE 2 Diagnostic characteristics STAGE 3 Reproducibility

8 Intended Purpose(s) for assay Essential Prerequisites Study Design and Protocol Analytical Specificity Analytical Sensitivity STAGE 1 Analytical characteristics Repeatability and preliminary Reproducibility Assay Validation Pathway Diagnostic Specificity Diagnostic Sensitivity Cut-off Determination STAGE 2 Diagnostic characteristics STAGE 3 Reproducibility Deployment to other Labs STAGE 4 Implementation

9 Intended Purpose(s) for assay Essential Prerequisites Study Design and Protocol Analytical Specificity Analytical Sensitivity STAGE 1 Analytical characteristics Repeatability and preliminary Reproducibility Adjunct tests validated Assay Validation Pathway Diagnostic Specificity Diagnostic Sensitivity Cut-off Determination STAGE 2 Diagnostic characteristics Provisional recognition STAGE 3 Reproducibility Validated for original intended purpose(s) Deployment to other Labs STAGE 4 Implementation International recognition (OIE)

10 Intended Purpose(s) for assay Essential Prerequisites Study Design and Protocol Analytical Specificity Analytical Sensitivity STAGE 1 Analytical characteristics Repeatability and preliminary Reproducibility Assay Validation Pathway Diagnostic Specificity Diagnostic Sensitivity Cut-off Determination STAGE 2 Diagnostic characteristics STAGE 3 Reproducibility Deployment to other Labs STAGE 4 Implementation Daily in-house QC Monitoring and maintenance of validation criteria Proficiency testing

11 Guidelines to complement Manual chapter (7 completed, 1 under study) Antibody detection assays Antigen detection assays Nucleic acid detection assays Method uncertainty Statistical approaches to validation Reference samples and panels Wildlife (modification of chapter ) Comparability of assays after minor changes in a validated test method (under study) Axel Colling and colleagues at AAHL, Geelong

12 Guideline on statistical approaches to validation Stage 2: diagnostic sensitivity (DSe) & specificity (DSp) Perfect reference standard Yes No Single binary candidate test Single continuous candidate test Binary candidate and reference test DSe and DSp (95% exact CI) DSe and DSp (95% exact CI) and Area under ROC curve (95% CI) DSe and DSp (95% CI or PI) by latent class methods DSe and DSp (95% exact CI) for relevant subpopulations DSe and DSp (95% exact CI) for relevant subpopulations

13 Guideline on statistical approaches to validation Stage 2: ROC analysis - cutoff independent method

14 Latent class analysis (LCA) for estimation of diagnostic sensitivity & specificity Challenges in PCR validation Traditional statistical methods can t demonstrate PCR to be more sensitive than an imperfect reference test (e.g. virus isolation [VI]) LCA solves the problem; allows calculation of probability that sensitivity of PCR > sensitivity of VI (and similarly can show specificities are comparable) Example: African Horse Sickness PCR (Horse diseases -- parallel session)

15 Guideline on validation of tests for wildlife diseases Challenges: Stage 2: difficulty in obtaining sufficient samples for estimation of DSe and DSp Experimental infections may be only source of reference samples Regulations limiting or prohibiting possession and international shipment of samples Poor sample quality Limited knowledge of pathogenesis/epidemiology of many diseases

16 Validation of tests for wildlife diseases Sample size credit for a validated test in a taxonomically-related wildlife species Provisional recognition (national authorities) at stage 2a for an intended purpose(s)

17

18 Effect of reduced sample size Note: samples must be representative of the target condition of interest (e.g. infected vs diseased) and appropriate for designated purpose

19 Disease-specific Manual chapters Questions Are the approach and terminology similar in reporting of diagnostic characteristics in aquatics and terrestrial manual? Methods Manual chapters approved May 2014 Aquatics (all 4) IHHN (shrimp), infectious salmon anemia and salmon alphavirus (fish), herpesvirus-1 microvariant (oysters) Multispecies (3/7) Paratuberculosis, Leishmanosis, Bluetongue

20 Bluetongue +++ = recommended method; ++ = suitable method; + = may be used in some situations, but cost, reliability, or other factors severely limits its application; = not appropriate for this purpose. Although not all of the tests listed as category +++ or ++ have undergone formal validation, their routine nature and the fact that they have been used widely without dubious results, makes them acceptable.

21 Infectious salmon anemia virus a = recommended for reasons of availability, utility, DSe and DSp b = standard method with good DSe and DSp c = method has application in some situations (cost, accuracy and other factors limit its application d = method not recommended for this purpose These are somewhat subjective. Although not all the tests listed as category a or b have undergone formal standardisation and validation, their routine nature and the fact that they have been used widely without dubious results makes them acceptable

22 Scenarios needing additional guidance for stage 2 validation Experimental studies to estimate DSe/DSp Aquatic animals and wildlife, in particular Key questions Is the design appropriate and does it adequately mimic natural exposure/infection? What are the key elements that should be reported in a peer-reviewed publication to allow that assessment by readers/end-users?

23 c) Experimentally infected or vaccinated reference animals Samples obtained sequentially from experimentally infected or vaccinated animals are useful for determining kinetics of antibody responses. Single time-point sampling of individual experimental animals can be acceptable [for estimation of DSe] (e.g. one sample randomly chosen from each animal). Nevertheless it should be noted that for indirect methods of analyte detection, exposure to organisms under experimental conditions, or vaccination, may elicit antibody responses that are not quantitatively and qualitatively typical of natural infection in the target population (Jacobson, 1998). The strain of organism, dose, and route of administration to experimental animals are examples of variables that may introduce error when extrapolating DSe and DSp estimates to the target population. In cases when the near-impossibility of obtaining suitable reference samples from naturally exposed animals necessitates the use of samples from experimental animals for validation studies, the resulting DSe and DSp measures should be considered as less than ideal estimates of the true DSp and DSe

24 Thank you