Roundtable Discussion Method Validation & Verification

Size: px

Start display at page:

Download "Roundtable Discussion Method Validation & Verification"

Brittany McLaughlin
6 years ago
Views:

2 WHAT is Validation? Reference Codex CAC/GL 74 1 ISO USDA FSIS 3 US FDA 4 Validation Definition Process to establish the performance characteristics and limitations of an analytical method: which analytes, in what kind of matrices, in the presence of which interference. Result = precision and trueness values of a certain analytical method under the examined conditions. 1 [pg 2] Establishment of the performance characteristics of a method and provision of objective evidence that the performance requirements for a specified intended use are fulfilled. 2 [pg 2] Process to measure performance characteristics of a particular test, with the goal of determining whether the test is equivalent to the reference test for the intended conditions of use. Equivalent = the performance characteristics are statistically indistinguishable. 3 [pg 2] Demonstration that adequate confidence is provided when the results obtained by the alternative method i.e. the commercially available kit, are comparable to or exceed those obtained using the reference method using the statistical criteria contained in the approved validation protocol. 4 [pg 16] ISO The confirmation by examination and the provision of objective evidence that the 17025: particular requirements for a specific intended use are fulfilled. 6 3M All Rights Reserved 2

3 Summary of ALL Validation reference definitions: Validation: 1. Performance characteristics are defined, then 2. compared to a reference method, and then 3. statistically evaluated to determine equivalence A method is validated for the matrices [& sample sizes] included in the validation 3M All Rights Reserved 3

4 Categories of Microbiological Test Methods Performance Characteristics Included in a Validation Study 1 Performance Characteristic Quantitative Qualitative (presence/absence) Relative Accuracy (% Recovery) Yes No Matrix Effects Yes Yes Precision Yes No Sensitivity Yes Yes Specificity Yes Yes Inclusivity Yes Yes Exclusivity Yes Yes False Positive Rate Yes Yes False Negative Rate Yes Yes LOD [POD or LOD50] Yes Yes LOQ Yes No Ruggedness Yes Yes Linearity/Range Yes No Dr. Michael Brodsky AOAC International 2013 Annual Meeting 3M All Rights Reserved

5 Methods requiring validation: New or original methods Modifications to validated methods Extension of scope to include additional analytes, matrices, or changes in intended use Changes involving new technology or automation Significant parameter changes: reagents/apparatus, time /temperature incubation periods, enrichment media, etc. It is possible that small changes/matrix extensions to a related product etc would not need a full validation but may need some level of testing to assess for unintended differences. 3M All Rights Reserved

6 WHY are Validated methods important? Validated methods are required for conducting Official Testing Official = adjective, relating to an authority or public body and its activities and responsibilities [Oxford online dictionary] European Union regulations specifically stipulate validated methods that have been collaboratively studied Internationally, regulatory agencies require, or at least prefer, to use methods that have been validated and collaboratively studied. Some global regulatory agencies require AOAC Official Methods of Analysis to receive Final Action, before they may be used as an Official method 3M All Rights Reserved

7 REQUIREMENTS of methods for Official use Agency Codex CAC/GL 21 8 (EC) No Method Validation Requirements Whenever possible, only methods for which the reliability (accuracy, reproducibility, inter- and intra-laboratory variation) has been statistically established in comparative or collaborative studies in several laboratories should be used. The use of alternative analytical methods is acceptable when the methods are validated 2073/ against the reference method in Annex I and if a proprietary method, certified by a third party in accordance with the protocol set out in EN/ISO standard or other internationally accepted similar protocols, is used. Pg 9 USDA FSIS 11 US FDA 4 All analytical methods described in FSIS-MLG have been scientifically validated and are considered fit for their intended purpose. Laboratories may also use other validated testing methods that differ from the methods described in FSIS-MLG. For the intended conditions of use, the performance characteristics of the new method and the well characterized method should be statistically indistinguishable Pg 13 FDA-BAM presents the agency's preferred laboratory procedures for microbiological analyses of foods and cosmetics. Methods fully validated in a collaborative study monitored and evaluated by an independent accrediting body e.g. AOAC-OMA, AFNOR, etc. require in-house verification for the first use 3M All Rights Reserved 7

8 What is considered an OFFICIAL method? Official Methods = Methods used or accepted for use by a regulatory authority Traditional, cultural methods [may be jurisdiction specific] ISO Method ISO Validated Methods AOAC Official Methods FDA Bacteriological Analytical Methods USDA FSIS Microbiology Laboratory Methods HC Compendium of Analytical Methods, Volumes 1 to 5 Any of the above methods MAY be accepted for use by regulatory agencies, IF: They have been compared to an acceptable reference method They have been validated as fit for purpose for the matrices analyzed 3M All Rights Reserved 8

9 Validation Schemes ISO AFNOR: French National Organization for Standardization MicroVal: European certification body for microbiology methods NordVal: Nordic certification body: Denmark, Finland, Iceland, Norway and Sweden AOAC Research Institute [RI]: International Standards Organization Performance Tested Method (PTM) Official Method of Analysis (OMA) 3M All Rights Reserved 9

AFNOR and MicroVal Certification schemes based on EN/ISO 16140: Comparison Study to reference method 1. Independent expert laboratory co-ordinates validation 2.

10 AFNOR and MicroVal Certification schemes based on EN/ISO 16140: Comparison Study to reference method 1. Independent expert laboratory co-ordinates validation 2. Inter-laboratory study [collaborative study] 3. All Foods Claim Regular audit of the quality system of the manufacturer [based on EN standard (ISO 9002)] Regular verification of the certified methods post certification [every 4 years] 3M All Rights Reserved

AOAC Research Institute AOAC Performance Tested Method [PTM] Pre-collaborative + 1 Independent lab: I/E; Ruggedness testing, paper review of quality system.

11 AOAC Research Institute AOAC Performance Tested Method [PTM] Pre-collaborative + 1 Independent lab: I/E; Ruggedness testing, paper review of quality system. Yearly renewal paper process only NO all foods claim only for matrices evaluated AOAC PTM / Official Method of Analysis [OMA] Harmonization Complete AOAC PTM program / Initiate collaborative study to achieve OMA First Action status; Final Action status after 2 yrs use in the industry without issue AOAC PTM / OMA/ MicroVal Harmonization Both programs run simultaneously 1 collaborative study 3M All Rights Reserved

12 Method Scheme Websites list validated methods including claimed matrices, sample sizes AFNOR MicroVal AOAC 5a-1f6d-4d78-a498-19b9763d11b4&hkey=33b744f6-f71e-456a &CCO=5 NordVal 3M All Rights Reserved 12

13 Comparison of VERIFICATION definitions Reference Codex ISO USDA FSIS US FDA ISO 17025:2005 Verification Definition Could not find a method verification definition Codex documents [only process verification] Demonstration that a validated method functions in the user's hands according to the method's specifications determined in the validation study and is fit for purpose Laboratory should demonstrate on-going competence in performing the method at their facility, which would include participating in proficiency test programs 11 Pg 13 The confirmation by examination and the provision of objective evidence that specified requirements have been fulfilled. 4 Pg 16 The Laboratory shall confirm that it can properly operate standard methods before introducing the tests or calibrations 3M All Rights Reserved 13

14 Verification for USE Proof that the laboratory can perform the [validated] method as intended This applies only to matrices and sample sizes that were included in the validation study FDA has detailed Guidance on in-house verification of a validated method before use 4 Pg 16 3M All Rights Reserved 14

15 Distinguishing Validation and Verification Validation Demonstrate the METHOD is equivalent to the reference method [for the matrices validated] based on defined method criteria Verification Demonstrate in the hands of the USER that the method can be performed to meet the defined method criteria 3M All Rights Reserved 15

16 Categories of Microbiological Test Methods Performance Characteristics Included in the Verification of a Validated Method 1 Performance Characteristic VALIDATION Quantitative VALIDATION Qualitative (P/A) VERIFICATION (Where Applicable) Relative Accuracy (% Recovery) Yes No Yes Matrix Effects Yes Yes Yes Precision Yes No Yes Sensitivity Yes Yes Yes Specificity Yes Yes Yes Inclusivity Yes Yes No Exclusivity Yes Yes No False Positive Rate Yes Yes Yes False Negative Rate Yes Yes Yes LOD [POD or LOD50] Yes Yes No LOQ Yes No No Ruggedness Yes Yes No Linearity/Range Yes No No Dr. Michael Brodsky AOAC International 2013 Annual Meeting 3M All Rights Reserved

17 CONTINOUS Verification - ISO requirement Proficiency Testing: Should be done several times per year Should be handled as a routine sample not specially handled Can help determine the performance of individual laboratories for specific tests Is used to monitor laboratories continuing performance Can demonstrate an analyst s ability to properly perform the test Test samples: Certified proficiency test providers Internal Ring tests 3M All Rights Reserved 17

19 ISO 16140: Six Parts Part 1: Definitions Part 2: Protocol to compare REFERENCE vs ALTERNATIVE method Part 3: Verification Part 4: Single Lab Validation Part 5: Use of Factorial Design for Collaborative Study Part 6: Confirmation 3M All Rights Reserved

20 ISO Verification Still Drafting so do NOT share this information! Current Timeline: Drafting Group (DG) meeting held in UK, first week of April Co-chairs will meet in July to summarize and work on FINAL DRAFT Send to Working Group (WG) 3 for review & comments OCT-NOV WG 3 Meeting 3M in US for final discussion Sent for SC9 Review prior to ISO SC9 Plenary Meeting in May/June M All Rights Reserved 20

21 ISO IMPLEMENTATION Verification Demonstrate competence to perform the validated method Applies only to methods with published validation data (reference or alternative methods). Test 1 sample Type used in the validation, applicable to the user lab Sample TYPE Verification Demonstrate competence to perform the method with sample TYPES routinely tested in the user lab Select sample TYPES from the CATEGORIES included in the scope of the validated method, but not tested in the validation study 3M All Rights Reserved 21

22 ISO IMPLEMENTATION Verification Sample TYPE Method ITEM Reference method w/published data Validated alternative method 1 relevant ITEM from the TYPES tested in the validation study Reference method with or without data Validated alternative method ITEMS from the TYPES included in the scope of the method, not tested in the validation and relevant to the user laboratory 3M All Rights Reserved 22

23 ISO Table A. 1: Classification of sample types & suggested target combinations for validation studies CATEGORIES Raw Milk & Dairy Products Heat Processed Milk & Dairy Products Raw meat & Ready-to-cook meat products (except poultry) Ready-to-eat, ready-to-reheat meat products Raw Poultry & ready-to-cook poultry products Ready-to-eat, ready-to-reheat meat poultry products Eggs & egg products (derivatives) Raw & ready-tocook fish & seafoods (unprocessed Ready-to-eat, ready-to-reheat fishery products Fresh produce & fruits Processed fruits & vegetables Dried cereals, fruits, nuts, seeds and vegetables Infant formula & infant cereals Chocolate, bakery products & confectionary Multicomponent foods or meal components Primary production samples Pet food & animal feed Environmental samples (food or feed production There are 18 CATEGORIES recognized and harmonized between ISO and AOAC 3M All Rights Reserved 23

24 ISO Verification (TYPE) CATEGORIES TYPES ITEMS Raw Meat & Readyto-Cook meat products (except poultry) Ready-to-eat, Ready-to-reheat meat products Fresh meats (unprocessed) Read-to-cook (processed) Cooked meat products Fermented or dried meat products Raw cured (smoked Aw >0.92) Raw cured (smoked Aw <0.92) Canned meat (ambient stable) Carcasses, meat cuts, Carpaccio s Minced meat, meat preparation, Carpaccio s Carcasses, swabs, rinsates Frozen burger patties, marinated beef shish-kababs Cooked ham, pate Salami Filet de sax, lard Cobourg ham, dry cured ham Corned beef There are 82 sample TYPES listed under the 18 categories 3M All Rights Reserved 24

25 ISO Data Analysis DG 3 met with DG 2 (statistics) in April to work on study design Still drafting, but will be based on Bias Natural contamination preferred to inoculated Determine L, M, H levels Compare results to published data to determine acceptibility 3M All Rights Reserved 25

27 References 1. CODEX - CAC/GL Guidelines on performance criteria and validation of methods for detection, identification and quantification of specific dna sequences and specific proteins in foods 2. ISO/DIS Microbiology of food and animal feed Method validation Part 1: Vocabulary Currently under revision 3. USDA FSIS: Guidance for Test Kit Manufacturers, Laboratories: Evaluating the Performance of Pathogen Test Kit Methods 4. US FDA: Guidelines for the Validation of Analytical Methods for the Detection of Microbial Pathogens in Foods 5. ISO 17025:2005 General requirements for the competence of testing and calibration laboratories NO LINK, this is copyrighted material AOAC International Annual Meeting presentation, with permission from: Michael Brodsky, President, Brodsky Consultants, Thornhill, Ontario, Canada mhbrodsky@rogers.com 7. CODEX CAC/GL Principles for the Establishment and Application of Microbiological Criteria for Foods 8. COMMISSION REGULATION (EC) No 2073/2005 of 15 November 2005 on microbiological criteria for foodstuffs : 9. Health Products and Food Branch (HPFB) Standards and Guidelines for Microbiological Safety of Food - An Interpretive Summary 6. Equivalent Methods USDA FSIS: Establishment Guidance for the Selection of a Commercial or Private Microbiological Testing Laboratory: AOAC: AOAC INTERNATIONAL Methods Committee Guidelines for Validation of Microbiological Methods for Food and Environmental Surfaces Official Methods for the Microbiological Analysis of Foods 3M All Rights Reserved 27

A2LA. R231 Specific Requirements: Threat Agent Testing Laboratory Accreditation Program. December 6, 2017

A2LA. R231 Specific Requirements: Threat Agent Testing Laboratory Accreditation Program. December 6, 2017 Laboratory Page 1 of 17 Laboratory December 6, 2017 2017 by A2LA All rights reserved. No part of this document may be reproduced in any form or by any means without the prior written permission of A2LA.