JGK TRAINING PROGRAMME MODULE 2: METHOD VALIDATION AND MEASUREMENT OF UNCERTAINTY IN VETERINARY LABORATORIES PRACTICAL COURSE

Transcription

1 Registered as JKG Lab AfriQA (Pty) Ltd Reg. No. 2012/154570/07 P.O Box 1581 Potchefstroom Sylvia Street `` Potchefstroom 2531 Tel: Fax: Cell: : kangumbaj@gmail.com 2: joule@jgklab.co.za Website: JGK TRAINING PROGRAMME MODULE 2: METHOD VALIDATION AND MEASUREMENT OF UNCERTAINTY IN VETERINARY LABORATORIES PRACTICAL COURSE SAVC ACCREDITATION: AC/1551/16 CPD POINTS: JULY 2017 OBJECTIVES: 1. To understand and apply the principles of method validation according to OIE requirements 2. To understand the approach to measurement of uncertainty for veterinary diagnostic assays 3. To design and carry out validation studies for veterinary diagnostic assays 4. To analyze and interpret validation data, and compile validation reports for veterinary diagnostic assays THEORY I. BACKGROUND INTENDED PURPOSE OF THE ASSAY OIE Statement of Purpose National or state programmes to be supported by the intended purpose(s) DESCRIPTION OF THE ASSAY OIE prescribed test OIE alternative test Other REFERENCES Publications Acceptability of the test in scientific, regulatory and client community TEST METHOD PROTOCOL Detailed SOP II. ASSAY DEVELOPMENT AND FEASIBILITY PRIOR TREATMENT OF SPECIMEN Collection Transport Preparation REAGENT SELECTION Controls and reagents Africa s leader in Laboratory Quality Assurance Programs & Accreditation Page 1

2 Criteria for selection ASSAY OPTIMIZATION Optimization Standardization FEASIBILITY TESTING Range of analyte in positive samples Negative samples Exotic diseases unavailability of samples REPEATABILITY Intra and inter-assay variation Between operator variation ANALYTICAL SENSITIVITY Endpoint dilution Serial testing Etc. ANALYTICAL SPECIFICITY Target and non-target analytes Cross-reactions (FP) Comparison of results with Gold standard III. DIAGNOSTIC PERFORMANCE OF ASSAYS REFERENCE TEST (GOLD STANDARD) Gold standard CONTROLS AND REFERENCE REAGENTS Criteria for use Availability SELECTION OF REFERENCE POPULATIONS Target population (infected and free of the target disease) Sampling frame Criteria for selection of reference animals Confirmation of the status of reference animals SAMPLING OF THE REFERENCE POPULATION Sample size Sampling procedure DIAGNOSTIC SENSITIVITY Proportion of infected reference animals Sensitivity Relative sensitivity 2 x 2 contingency table DIAGNOSTIC SPECIFICITY Proportion of uninfected reference animals Specificity 2 x 2 contingency table REPRODUCIBILITY Participating Labs Reproducibility panel Administration of the trial Trial results (analysis of variation) Africa s leader in Laboratory Quality Assurance Programs & Accreditation Page 2

3 PERFORMANCE OF TEST AGAINST REFERENCE TEST (GOLD STANDARD) Evaluation of candidate test and reference test ASSAY INTERPRETATION Results from low positive, high positive and negative samples Precision and accuracy Cut-offs RESULTS PRESENTATION Presentation of Se and Sp estimates together with sample sizes and confidence intervals 2 x 2 table Other appropriate/basic statistical methods Discussion of performance parameters in relation to the study design and intended use of the test Perfect/imperfect Gold standard IV. TECHNOLOGY TRANSFER AND REPRODUCIBILITY TECHNOLOGY TRANSFER Implementation of the new method in other laboratories (beyond reproducibility) Performance of the test in other laboratories in comparison to the original laboratory MONITORING VALIDITY OF ASSAY PERFORMANCE Discrepant analysis Assessment of categorical test data as it becomes available Estimation of prevalence in target population Calculation of positive and negative predictive values Assessment of precision and accuracy Assessment of reproducibility estimates through eqa (z-score, Youden plots, estimates for random vs systemic errors, basic stats, etc.) Establishing of lower and upper control limit for internal controls MAINTENANCE AND ENHANCEMENT OF VALIDATION CRITERIA I. Plans for continuous validation assessment: - additional results from target population - extending validation criteria into other populations or adjusting assumptions for original population - replacement of reagents V. MEASUREMEMT OF UNCERTAINTY UNCERTAINTY OIE principle STANDARD DEVIATION Relative Standard deviation Expanded Standard deviation Calculation of Standard deviation (Square root of variance) PROBABILITY DENSITY FUNCTION Gaussian PDF Africa s leader in Laboratory Quality Assurance Programs & Accreditation Page 3

4 TEACHING METHODOLOGY LECTURES PowerPoint Presentations Flip chart Handouts (Flash discs) Content: (1) PowerPoint presentations: all presentations covered during the course will be loaded (2) Course materials: 18 appendixes covering the entire course will be loaded (3) Notepads: supplied to participants for taking notes DEMONSTRATION WORKED EXAMPLES Several examples of validation performed in and outside South Africa covering most veterinary diagnostic fields will be used to teach the module: Veterinary microbiology Serology Veterinary Public Health PRACTICAL GUIDED PRACTICAL (1) Setting of repeatability and reproducibility conditions and calculations of derived data a. Intra-plate and inter-plate variations b. Between analysts variations c. Between days variations d. Mean e. Standard variations f. Coefficient of variations (2) Quantitative data a. Conversion of data into logarithmic values b. Z-score calculation c. 95% Confidence interval calculations (3) Performance characteristics of a test a. Diagnostic sensitivity calculations 2 x 2 contingency table b. Diagnostic specificity calculations 2 x 2 contingency table (4) Sample size calculations a. Table for sample size calculation b. Random and systematic sampling frames (5) Diagnostic Sensitivity (DSe) and Specificity (DSp) estimates a. Allowable errors in the DSe and DSp estimates b. Confidence levels (6) Interpretation of test results a. Positive predictive value calculations 2 x 2 contingency table b. Negative predictive value calculations 2 x 2 contingency table Africa s leader in Laboratory Quality Assurance Programs & Accreditation Page 4

5 (7) Data and templates with instructions will be provided Appendixes 1, 3 and 11 (handouts). Participants will follow the instructions and will be guided to: a. Develop validation protocols b. Analyze data c. Compile simple validation reports Data for the practical will be provided (handouts): Appendix 2, Appendix 3 (8) Own data secondary validation of existing assays a. Participants will pick own assay(s) among those currently used at the laboratory. These can be already validated assays, or candidate tests for validation b. Using the supplied template with instructions, participants will review one validation study, which they have done in their Labs or design a new validation study for the assay (if assay not validated yet). Appendixes 1, 6, 7, 8 and 11 (handouts) (9) Evaluation of the validation status of an assay (i.e. to determine if the validation study was carried out efficiently) Templates will be provided and used to evaluate the validation status of assay(s) a. Template for extension of existing assays (adapted from SCAHLS) Appendix 6 (handout) b. Template for serological assays (adapted from SCAHLS) Appendix 7 (handout) (10) Measurement of uncertainty Appendix 17 (handout) a. Mean b. Standard deviation c. Relative standard deviation and coefficient of variation d. Uncertainty and expanded uncertainty e. 95% Confidence interval and range GROUP DISCUSSIONS Team 1: Diagnostic bacteriology Team 2: Microbiology (incl. participants working in PCR, virology, etc.) and VPH Team 3: Diagnostic serology PARTICIPANTS EVALUATION Written tests Assignments Group discussions: breakaway session and report back COURSE EVALUATION Questionnaire WORKING TOOLS FOR PARTICIPANTS SUPPLIED BY JGK a. Notepads b. Pens Africa s leader in Laboratory Quality Assurance Programs & Accreditation Page 5

6 c. Pencils d. Highlighters e. Erasers f. Flash discs g. JGK self-calculated sheets (JGK CALCULATORS): Include, but not limited to calculations of: Mean Standard deviation Relative standard deviation Coefficient of variation Uncertainty Expanded uncertainty 95% Confidence interval and range Logarithmic values (base 10 only) Diagnostic sensitivity 2 x 2 contingency table Diagnostic specificity 2 x 2 contingency table Sample size Prevalence Positive predictive value 2 x 2 contingency table Negative predictive value 2 x 2 contingency table Z-score and outliers NOT SUPPLIED, BUT ESSENTIAL FOR THE COURSE Electronic calculator Laptop with Excel, Word and PowerPoint Africa s leader in Laboratory Quality Assurance Programs & Accreditation Page 6