THE AOAC RESEARCH INSTITUTE PERFORMANCE TESTED METHODS SM PROGRAM. Validation of Rapid Microbiological Methods

Transcription

1 THE AOAC RESEARCH INSTITUTE PERFORMANCE TESTED METHODS SM PROGRAM Validation of Rapid Microbiological Methods

2 Outline of Workshop About AOAC Validation Overview PTM Validation Components Qualitative Study Design Quantitative Study Design Environmental Surface Testing Report and Review Certification and Publication Renewal and Modification

3 About AOAC

4 AOAC Vision and Mission That there be worldwide confidence in analytical results To serve the communities of analytical science by providing fit-for-purpose methods and services for assuring quality measurements

5 AOAC Solves problems by providing science based solutions Globally recognized Independent third party Establishes standards Voluntary consensus through stakeholder panels and working groups Official and regulatory methods Conducts conformity assessment Official Methods of Analysis SM Performance Tested Methods SM

6 AOAC AOAC does not: Regulate products Buy or sell food, beverage products, or proprietary technologies Promote specific food and beverage products Do risk assessments Set tolerance levels Own a laboratory or provide laboratory services

7 AOAC INTERNATIONAL Founded in 1884 in Philadelphia Founded the Official Methods of Analysis SM Program Originally Association of Official Agricultural Chemists In 1965 became Association of Official Analytical Chemists In 1991 became AOAC INTERNATIONAL AOAC represents all of its method communities Chemistry and microbiology methods ~ 3500 members worldwide 1/3 of members overseas (mostly Europe) ~2700 Official Methods

8 AOAC Research Institute Incorporated in 1991 Administers the AOAC Performance Tested Methods SM program Commercial and proprietary methods Chemistry and microbiology methods Harmonized with AOAC Official Methods SM program >100 Current certificates International membership PTM Certification Mark has international recognition

9 AOAC RI Performance Tested Methods SM (PTM) Program Experience validating a wide variety of method techniques Cultural methods Immunoassay Lateral flow Polymerase chain reaction (PCR), RT-PCR Enzyme methods Bioluminescence Chemiluminescence Electrical resistance ID methods

10 AOAC RI Performance Tested Methods SM (PTM) Program Experience validating methods for a wide variety of microorganisms Listeria monocytogenes & Listeria spp. Salmonella E. coli, E. coli O157:H7, & STEC Coliforms Total plate counts Campylobacter Vibrio Staphlyococcus aureus & Staph enterotoxin Yeasts and molds Bacillus anthracis

11 Up to Date List of PTM Approved Methods

12 Validation Overview

13 Programs for Rapid Methods Performance Tested Methods SM Administered by AOAC Research Institute Official Methods of Analysis SM Administered by AOAC INTERNATIONAL

14 Method Validation Guidelines AOAC Guidelines For Validation Of Qualitative And Quantitative Food Microbiological Official Methods Of Analysis J. Assoc. Off. Anal. Chem. 85, (2002) Available at

15 PTM Program Technical Requirements Method Developer Responsibilities: Inclusivity Exclusivity Matrix Study Robustness Stability Lot-to-Lot Variation Independent Laboratory Responsibilities: Matrix Study 1 Laboratory 1-4 Foods or surfaces (claim dependent)

16 OMA Program Technical Requirements Pre-Collaborative Study Inclusivity Exclusivity Matrix Study Collaborative Study Matrix Study At least 10 labs for qualitative methods At least 8 labs for quantitative methods 1-6 Foods AOAC, USDA, FDA, ISO, Health Canada reference methods Reproducibility

17 PTM Program Validation Procedure Consulting Required Waiver can be granted upon approval of managing director Application New methods and modifications Data Collection Method developer and independent studies Review Performance Tested Methods status granted or declined Certification and Publication

18 Consulting Method Developer Submits: Consulting application and fee Claim Target analyte(s) Matrices Method instructions (package insert, user guide, SOP)

19 Consulting AOAC RI Provides: Overview of current technical requirements AOAC Guidelines Appropriate reference methods Specific validation protocols Reviewed and approved by General Referee Validation flowcharts

20 Consulting Benefits Understanding for both sides AOAC has better understanding of the method Method developer has better understanding of AOAC requirements Can significantly reduce method developer time and cost Most current technical updates and requirements

21 PTM Application Method Developer Submits: Application Package insert, user manual, directions for use Product labels QA/QC synopsis Application fee

22 PTM Study Method Developer Study Inclusivity and exclusivity Matrix Study Lot-to-lot Stability Robustness

23 PTM Study Independent Laboratory Study Contracted by AOAC RI Method developer provides method materials and training Performs matrix study Report

24 Review Method developer submits complete study report and package insert (PI) Reviewed by general referee and 2 expert reviewers Method developer addresses reviewers comments/questions as necessary Report and PI revisions submitted as necessary Decision to grant or decline approval, based on reviewer consensus

25 Certification License Agreement and Use of the Mark List on PTM Approved Methods website Annual Review & Certification Renewal

26 Publication Certificate issued Certification Mark issued Certification Report published in Inside Laboratory Management Study report published in Journal of AOAC INTERNATIONAL

27 Harmonization with Official Methods of Analysis SM Method developer can use PTM study to satisfy the OMA pre-collaborative requirement Both PTM and OMA requirements must be met Allows for company to perform one large study rather than two separate ones

28 Harmonization Collaborative Study Collaborative Study Qualitative 10 labs minimum with valid data 1-6 matrices, depending on claim 3 levels (uninoculated, low, and high)/matrix 6 test portions/level/matrix/method Quantitative 8 labs minimum with valid data 1-6 foods, depending on claim 4 levels (uninoculated, low, medium, and high)/matrix 2 test portions/level/matrix/method

29 Harmonization Flowchart Application for harmonized study RI prepares PTM & OMA protocols Sponsor collects internal data, IL performs study Sponsor writes report Certification report published in ILM PTM manuscript published in JAOAC Approved study = PTM certification Report reviewed by GR, Stat, 2 ER & 1 Comm. H OMA Final Action Published in AOAC OMA and JAOACI Approved study = OMA First Action Collaborative study (OMA) & review

30 Harmonization with MicroVal MicroVal is a European organization for the validation and certification of alternative methods for the microbiological analysis of food and beverages. EN-ISO Protocol for the validation of the alternative methods forms the basis for the European certification of alternative methods, thus fulfilling European legislation.

31 Harmonization with MicroVal Method developer can use MicroVal Expert Laboratory study to satisfy the PTM inclusivity, exclusivity and matrix study requirements Both PTM and MicroVal requirements must be met Since all matrix study work is done by the Expert Lab, an additional independent lab study is not required Additional PTM studies (lot-to-lot, stability, robustness) are performed by the method developer

32 Harmonization with MicroVal Allows for company to perform one large study rather than two separate ones Can use common expert reviewers, but still include GR for AOAC Each validating organization retains its own acceptance criteria

33 Validation Complete Questions? For additional information, go to

34 PTM Validation Components

35 PTM Validation Components Inclusivity Exclusivity Matrix Study Stability Lot-to-lot Robustness

36 Responsibilities Method Developer Inclusivity Exclusivity Robustness Lot-to-lot/stability Matrix Study AOAC-RI Coordinates independent laboratory testing Matrix Study

37 Inclusivity To ensure the test kit can detect a variety of target organisms Serovar, species, genus or class At least 50% should be food or environmental isolates Source: ATCC, NTCC, etc. Origin: chicken, cheese, etc. Cultures grown in test kit media Test at approximately ten fold higher than the LOD Inclusivity

38 Exclusivity To determine the test kit s ability to discriminate target organisms from non-target organisms Report source and origin of all isolates Cultures grown in non-selective media that supports growth Does not need to be the same for all isolates If an isolate generates a positive result Re-test isolate in test method selective enrichment Report both results Exclusivity

39 Robustness Introduce minor variations to the method and observe effects Critical steps in the method Steps that an end user could likely vary Incubation temperature Incubation time Reagent volume Sample volume Robustness

40 Lot-to-lot and Stability To ensure the manufacturing is consistent between lots and to support the shelf life of the test kit Can combine lot-to-lot and stability Test 3 lots Newly manufactured Middle of term Near expiration date Stability Lot-to-lot

41 Matrix Study Candidate method Reference method Matrix Study Claim Allocation of matrix testing Choosing Foods Sample preparation Analysis Paired or unpaired samples

42 Matrix Study Candidate Method Method of analysis submitted for validation. The method can be proprietary or noncommercial. Reference Method The appropriate official method that is applicable to the analyte and sample type that the candidate method is intended to detect. Candidate methods are compared to reference methods when available.

43 Matrix Study AOAC Official Methods of Analysis USDA FSIS Microbiological Laboratory Guidebook US FDA Bacteriological Analytical Manual ISO Methods Health Canada Other (Compendium, SMEDP, etc.)

44 Matrix Study Claim Claim Number of Matrices Number of Categories Variety At least 10 At least 5 Selected At least 5 At least 2 Food Category At least 5 1

45 Matrix Study Allocation of Matrix Testing Option 1 Method developers required to validate all claimed matrices Independent lab evaluates 20% of total claimed matrices Same strain/serovar for each matrix as was tested in MD lab (if artificially contaminated) Option 2 AOAC RI coordinates all matrix study testing Performed in an independent lab

46 Matrix Study Allocation of Matrix Testing Number of Claimed Matrices Number of Matrices Evaluated by Independent Lab

47 Matrix Study Choosing Foods Likely to contain target analyte Associated with outbreaks Where analyte can survive ph, water activity, spices, etc. Target market for test method Trial run of all foods before beginning validation study

48 Matrix Study Choosing Foods Use naturally contaminated samples whenever possible Salmonella in poultry products Campylobacter in carcass rinses Artificially contaminated samples can be prepared if necessary A different isolate should be used for each matrix tested Details in Annex A of the Micro Guidelines

49 Matrix Study Competitive Flora At least one food type must be tested with competing flora (natural or artificial) Naturally present in raw matrices Always done in environmental surface studies Helps simulate conditions found in nature If artificially contaminating, competing organism should be at least ten times higher than target

50 Matrix Study Cultures for artificial contamination Matrix study should use foodborne isolates Single strain per matrix, not a pool of strains Internationally recognized source ATCC or equivalent Laboratory isolate Confirmation documentation required Biochemical Serological

51 Matrix Study Cultures for artificial contamination Lyophilized cultures Unless known, recommended to do aging study of organism in matrix Preparation Historical counts Turbidity Known counts

52 Matrix Study Food Preparation Purchase Sufficient amount to cover a paired or unpaired study, plus MPN analysis and aerobic plate counts Screening Recommended to use both the test method and reference method If negative for target, prepare food for inoculation Chop, cut, grind, melt, etc. Use same lot of material for inoculated samples and uninoculated control samples

53 Matrix Study Artificial Contamination Liquid Add diluted organism to large quantity and mix, stir, etc. Dry Add lyophilized organism to food and mix: roll, shake, etc.

54 Matrix Study Artificial Contamination Solid/moist Add diluted organism drop-wise over bulk quantity of food, homogenize as appropriate Spray

55 Matrix Study Artificial Contamination Bulk inoculation required Make sure to maintain integrity of matrix when inoculating with prepared culture Be careful not to dilute matrix with the inoculum

56 Matrix Study Food Stabilization Perishable foods Store 2-4 days at 2-6 C Quantitative Listeria store for 1 day at 2-6 C Room temperature, shelf stable foods Store minimum of 2 weeks at room temp Test periodically to ensure level is stable Frozen foods Store minimum of 2 weeks at -20 to -30 C

57 Matrix Study Analysis Perform test method according to method instructions Perform reference method according to most current published method Perform MPNs according to reference method Determine aerobic plate count (APC) All test portions, MPNs, and APC are to be initiated the same day

58 Matrix Study Paired common enrichment 25g test portion enrichment Candidate Method Presumptive Result Reference Method Isolate and confirm CONFIRMED RESULT

59 Matrix Study Unpaired separate test portions Candidate Method 25g test portion enrichment Reference Method 25g test portion enrichment Transfer step, If necessary Run Assay Presumptive Result Isolate and confirm Isolate and confirm Candidate Method CONFIRMED RESULT Reference Method CONFIRMED RESULT

60 Matrix Study Confirmations All samples, regardless of presumptive result, must be subjected to isolation procedures for confirmation of target colonies Confirm according to the reference method protocol, or as directed by the test method instructions if included Confirm the number of colonies as outlined in reference method

61 Qualitative Method Study Design

62 Qualitative Method Method of analysis whose response is either presence or absence of the analyte detected either directly or indirectly in a certain amount of sample. AOAC Guidelines

63 Validation Study Components Inclusivity Exclusivity Matrix Study Stability Robustness Lot-to-lot

64 Inclusivity Qualitative To ensure the test method can detect a variety of target organisms Serovar, species, genus or class At least 50% of food origin Minimum of 50 isolates 100 for Salmonella Report source and origin of all isolates Inclusivity

65 Exclusivity Qualitative To determine the test method s ability to discriminate target organisms from non-target organisms Minimum of 30 isolates Report source and origin of all isolates Exclusivity

66 Robustness Qualitative Introduce minor variations to the method and observe effects Critical steps in the method Steps that an end user could likely vary Incubation temperature Incubation time Reagent volume Sample volume Robustness

67 Robustness Qualitative Qualitative Use 2 target isolates and 1 non-target 5 replicates of each for a total of 15 results Culture organisms Target - in test method enrichment media Non-target - in non-selective enrichment media Analyze with test method

68 Lot-to-lot and Stability Qualitative To ensure the manufacturing is consistent between lots and to support the shelf life of the test method components Can combine lot-to-lot and stability Test 3 lots Newly manufactured Middle of term Near expiration date 2 target isolates and 1 non-target 5 replicates of each for a total of 15 results Stability Lot-to-lot

69 Matrix Study Qualitative Target Contamination Levels Natural or artificial contamination to achieve appropriate levels Matrix Study Important to test within the range of the limit of detection of the test method Goal is to achieve fractional results for at least one level for at least one method (test or reference or both) 5-15 positive out of 20 replicates

70 Matrix Study Qualitative Target contamination level Low: cfu/25g High: 2-5 cfu/25g Uninoculated: 0 cfu/25g Adjust levels as necessary to achieve fractional results Data from one level is acceptable, as long as results are fractional Often do 2 levels just in case 20 replicate test portions each for high and low levels, 5 for uninoculated Per level per method if unpaired samples

71 Matrix Study Qualitative If natural contamination is found: 2 different lots, 20 replicate test portions per lot 20 test portions per lot per method if unpaired Fractional recovery required for at least one lot Food may be temperature abused or diluted with uncontaminated product to achieve appropriate levels

72 Matrix Study Qualitative Use a large scale Most Probable Number (MPN) to confirm contamination levels 3 x 100g, 3 x 10g, 3 x 1g, 3 x 0.1g MPN = statistical estimate of contamination level Determine using reference method Use same lot as test portions Initiate on same day as test portions

73 MPN 100 g 100 g 100 g 10 g 10 g 10 g 100g sample + 900ml media 10g sample + 90ml media 1 g 1 g 1 g 0.1 g 0.1 g 0.1 g 1g sample + 9ml media 0.1g sample + 9.9ml media

74 MPN

75 Matrix Study Confirmation of qualitative levels - MPN Record number of positive flasks/tubes Level 100g 10g 1g 0.1g High 3/3 3/3 0/3 0/3 Low 2/3 1/3 1/3 0/3 Calculate MPN Index using 3 values Level 100g 10g 1g 0.1g High 3/3 3/3 0/3 0/3 Low 2/3 1/3 1/3 0/3

76 MPN Calculations Number of positive tubes Using MPN table, determine MPN Index 100g 10g 1g MPN/25g

77 Statistical Analysis Qualitative: Paired and Unpaired Test for significant difference - Chi-square (χ 2 ) χ 2 results greater than 3.84 indicate a significant difference between two methods at a 95% confidence level Sensitivity Specificity

78 Qualitative Statistics - Paired Contingency Table Test Method Result Positive Negative Total Reference Method Result Positive a b a + b Negative c d c + d Total a + c b + d McNemar s χ 2 = ( b c 1) 2 / b + c Sensitivity rate = a / a + b Specificity rate = d / c + d

79 Qualitative Statistics-Paired Raw Data Sample Test Method Presumptive Test Method Confirmed Reference Method Result Total pos Uninoc

80 Qualitative Statistics-Paired Contingency Table Test Method Result Positive Negative Total Reference Method Result Positive Negative Total 11 9 McNemar s χ 2 = ( 1 0 1) 2 / 1 = 0 Sensitivity = 11 / 12 = 0.92 or 92% Specificity = 8 / 8 = 1 or 100%

81 Qualitative Statistics-Unpaired Contingency Table Conf. pos Conf. neg Alternative Method Pres. pos A B Pres. neg C D Reference Method E F Mantel-Haenszel χ 2 = (n-1)(af (B+C+D)E) 2 where n = A + B + C + D + E + F (A+B+C+D)(A+E)(B+C+D+F)(E+F) Relative sensitivity = A/E

82 Qualitative Statistics-Unpaired Performance Parameters Relative sensitivity = A/E Proportion of presumptive positive results that confirmed positive for the alternative method relative to the proportion positive results for the reference method

83 Qualitative Statistics-Unpaired Raw Data Sample Test Method Presumptive Test Method Confirmed Reference Method Result Total pos Uninoc

84 Qualitative Statistics-Unpaired Contingency Table Conf. pos Conf. neg Alternative Method Pres. pos 11 2 Pres. neg 1 6 Reference Method 12 8 Mantel-Haenszel χ 2 = (40-1)(88-108) 2 = 0.1 (20)(23)(17)(20) Relative sensitivity = 11/12 = 0.92 or 92%

85 Data Presentation Matrix Inoculating strain Level of contamination MPN (cfu/25g) Number of test portions Reference method results Alternative method results Presumptive Confirmed Chi square Sensitivity rate Specificity rate

86 Quantitative Method Study Design

87 Quantitative Method Method of analysis whose response is the amount of the analyte measured either directly (e.g. enumeration in a mass or volume) or indirectly (e.g. color absorbance, impedance, etc.) in a certain amount of sample. AOAC Guidelines

88 Validation Study Components Inclusivity Exclusivity Matrix Study Stability Robustness Lot-to-lot

89 Inclusivity Quantitative To ensure the test method can detect a variety of target analytes Serovar, species, genus or class Minimum of 30 isolates Inclusivity Report source and origin of all isolates Test cultures in non-selective broth No inclusivity for total count methods

90 Exclusivity Quantitative To determine the test method s ability to discriminate target organisms from non-target organisms Minimum of 20 isolates Report source and origin Exclusivity of all isolates No exclusivity for total count methods

91 Robustness Introduce minor variations to the test method and observe effects Critical steps in the method Steps that an end user could likely vary Incubation temperature Incubation time Reagent volume Robustness Sample volume

92 Robustness Quantitative Use 1 target isolate and 1 non-target 1 target isolate at high and low level 5 replicates of each for a total of 15 results Culture organisms in non-selective broth Analyze using test method

93 Lot-to-lot and Stability Quantitative To ensure the manufacturing is consistent between lots and to support the shelf life of the test kit Can combine lot-to-lot and stability Test 3 lots Stability Newly manufactured Middle of term Lot-to-lot Near expiration date Use 1 target isolate and 1 non-target 5 replicates of target at a high level and 5 replicates at a low level

94 Matrix Study Quantitative Target inoculum levels Natural or artificial contamination at appropriate levels If inoculating, important to have 10 fold between levels Uninoculated: 0 cfu/g Low: cfu/g Medium: cfu/g High: ,000 cfu/g 5 replicate test portions per level 5 per level per method if unpaired samples Matrix Study

95 Matrix Study Quantitative If natural contamination is found: Quantitative method 3 different lots 5 replicate test portions per lot 3 different contamination levels Food may be temperature abused or diluted with uncontaminated product to achieve appropriate levels

96 Statistical Analysis Quantitative Perform log 10 transformation of microbial counts Perform outlier tests Cochran and Grubbs or equivalent Plot data Test method on y-axis and reference method on x-axis Calculate repeatability, s r Calculate relative standard deviation for repeatability, RSD r Perform significance tests ANOVA, paired t-test or independent t-test

97 Quantitative Statistics Medium Inoculation Level Raw Data Sample CFU/g RM Log 10 RM CFU/g TM Log 10 TM

98 Quantitative Statistics Method Agreement Plot data Test method on y-axis and reference method on x-axis Use log 10 values Plot all levels on one graph Provide regression line, line formula and correlation coefficient Do not plot data if there is a missing value, i.e. >1100 or TNTC or <10 Missing data and corresponding value are removed from data analysis

99 Quantitative Data 3 levels, ten fold apart

100 Quantitative Statistics Repeatability (Standard deviation) Sample Log 10 RM Log 10 TM Mean Calculate s r and RSD r (RSD r = S r /mean) for each method using the log 10 values Ref Method s r = 0.22 RSD r = 0.71 Test Method s r = 0.03 RSD r = 0.010

101 Quantitative Statistics t-test or Analysis of Variance (ANOVA) To determine if a significant difference between the alternative method mean and the reference method mean can be detected Performed for each contamination level per matrix Typically a 95% confidence interval If p > 0.05, no significant difference detected

102 Quantitative Statistics Resulting data from t-test in Excel

103 Data Presentation Matrix Inoculating organism Contamination level Organisms/g Log 10 organisms/g Mean Repeatability, s r Relative standard deviation, RSD r P-values from t-test or ANOVA comparison of means

104 Environmental Surface Testing Study Design

105 Environmental Surface Testing Test kits used in the analysis of environmental surfaces Typically qualitative methods Same parameters used for food testing are required for surface testing: Inclusivity Exclusivity Exclusivity Robustness Lot to lot/stability Matrix study Matrix Study Inclusivity Stability Lot-to-lot Robustness

106 Environmental Surface Testing Reference methods Listeria USDA FSIS MLG Salmonella FDA BAM Guideline Study Protocol for Validation of Environmental Surfaces Claims for Target Organisms. Draft document

107 Matrices Stainless steel Rubber Wood Food grade painted surfaces Plastic polyethylene, polypropylene, polycarbonate Ceramic, glazed earthen material or glass Sealed concrete Cast iron (coated to prevent rusting) Air filter material

108 Claim Environmental Surfaces All 9 surfaces required 5 using sponge 4 x 4 surface area 4 using swab 1 x 1 surface area Selected Surfaces 2-8 Surfaces Each surface listed Individual Surface type

109 Surface Inoculation Can add stabilizer to inoculum Dilute in non-nutritive buffer Inoculate surface At least one surface must be contaminated with a competitor at a level 10 times greater than the target Dry overnight Recovery levels can vary Strain/surface combinations Laboratory conditions (humidity, temperature) Preliminary testing is critical

110 Surface Sampling Sample surface Swab or sponge Sponge/swab entire sampling area in both a horizontal and vertical motion Swab/sponge held in neutralizing broth (DE broth) at room temperature for 2 hrs prior to testing Add enrichment media to collection device

111 Performing Study If unpaired samples, side by side surface areas are tested, one for each method Perform remainder of test method and reference method Estimate initial level by determining plate count of inoculum (MPNs not applicable) At least one level by at least one of the methods must achieve fractional recovery 5-15 positive test portions out of 20

112 Performing Study All samples, regardless of test method results must continue through confirmation process Chi-square statistics and performance parameters are determined as per food studies based on whether test portions are paired or unpaired Present data as per food studies omitting MPN results

113 Report and Review

114 Report Data collection is complete Independent lab has submitted report Internal lab testing is complete Analyze data See qualitative and quantitative sections for specific statistical information Compile into single report

115 Report Format Template is provided with the validation outline Format is in place to facilitate subsequent publication in JAOAC Indicate if method is PTM only or if harmonized with OMA

116 Report Format Method / Kit Title: AOAC Performance Tested Method SM XXXXXX Abstract Method Authors Submitting Company Independent Laboratory Reviewers 1 Scope of method 1.1Target analyte 1.2 Matrices 1.3 Summary of validated performance claims 2 Definitions 2.1 Limit of Detection 2.2 Accuracy 2.3 Precision 3 Principle General description of the scientific principle of the test method. 4 General Information General information about the target analyte, matrix, occurrence of the analyte in nature and its effect.

117 Report Format 5 Test Kit Information 5.1 Kit Name 5.2 Catalog Number 5.3 Ordering Information USA. address with telephone, fax, , and website addresses Europe. other offices if applicable. 5.4 Test Kit Reagents Reagent one. brief description Reagent two Etc. as needed. 6 Additional supplies and reagents Cite supplies and reagents as needed. brief description, specifications, and source. 7 Apparatus 7.1 Heating block. brief description, specifications, source. 7.2 Temperature-controlled waterbath. brief description, specifications, source. 8 Standard Reference Materials Example: ECRC (available from the E.coli Reference Center, Pennsylvania State University, University Park, PA., USA) 9 Standard Solutions Example: Tris-borate EDTA electrophoresis buffer (TBE). Dissolve 108 g Tris, 9.3 g NA 2 EDTA, 55 g boric acid in ca 800 ml H20. Adjust ph to 8.2 with concentrated HCl and bring final volume to 1 L with H20.

118 Report Format 10 Safety Precautions Example: Electrophoresis gels contain ethidium bromide which is a potential mutagen. Observe all of the manufacturer s precautions for handling and disposal of these gels. 11 General Preparation 11.1 Assemble the gel electrophoresis apparatus according to the manufacturer s instructions Turn on the heating blocks and set temperatures to 37 o and 95 o C. 12 Sample Preparation Describe how samples are processed. 13 Analysis 13.1Title of first major step in analysis such as Extraction General description of process if needed Step-by-step description of process Etc as needed Title of second major step in analysis such as Sample Incubation General description of process if needed Step-by-step description of process Etc as needed. 14 Interpretation and Test Result Report Description of interpretation procedure and how results are reported.

119 Report Format 15 Method Developer Validation Studies Validation studies in the conducted in the Method Developer laboratory 15.1 Title of first study such as Robustness Testing General description of the study such as: The effects of perturbations in four method parameters were investigated: 1) temperature [69.5 C / 70.5 C]; 2) reduced sample volume (45 µl / 50 µl); 3) time delay (0 min / 15 min) in cooling rack; and 4) sample volume (3 µl / 5 µl) Methodology Results 15.2 Title of the second study such as Matrix Study Methodology Results 16 Independent Validation Study Validation study conducted by an independent laboratory under the direction of the AOAC Research Institute Title of study such as Matrix Study Methodology Results 17 Discussion General discussion of the all of the studies with particular emphasis on unexpected results, new knowledge about the analyte, matrix, or analytical technique. 18 Conclusion 19 References

120 Report Format Abstract Method Authors Submitting Company Independent Lab Reviewers Introduction Materials and methods Summary of results Discussion Conclusion References Acknowledgements

121 Review Method Developer submits Study report Package insert (directions for use) QA/QC description Package labels

122 Review Report, Package Insert, and review form sent to: PTM: GR and 2 expert reviewers Harmonized PTM/OMA: GR, 2 expert reviewers, 1 Committee H member, and statistical advisor 2 weeks allowed for initial review Reviewer comments are sent to author Technical consultant available to answer questions and assist with responses

123 Review Report and/or package insert revised as necessary and resubmitted to reviewers One week allowed for review Decision or additional comments sent to author Same time schedule as above for any additional reviews, if needed Review complete once consensus (for or against) is achieved

124 Potential Problems Fractional recovery requirement not met Methods not followed exactly Did not use correct or current reference method Did not follow current validation policies Data not acceptable Test method did not perform as well or better than the reference method

125 Potential Resolutions Can retest if: Fractional recovery is not achieved Method(s) not followed exactly Out of date or incorrect methods or policies were followed Matrices can be removed from claim if test method is unsuccessful To be determined by expert reviewers

126 Certification and Publication

127 Approval Criteria Applicant s performance data supports and confirms all claims made in the method s descriptive insert Independent laboratory s performance data corroborates the applicant s performance data within the statistical limits specified in the testing protocol Test method must perform as well or better than the reference method

128 Approved Upon approval, the method author will receive the following Approval letter Certification mark Certificate Web entry License agreement Request for ILM article/certification report

129 Use of Certification Mark License agreement is executed covering the Instituteowned PTM certification mark Authorizes the method developer to use the mark on the approved test method, packaging, insert, informational and promotional materials Denotes that the method was evaluated by the AOAC RI and confirms claims Mark must be discontinued if certificate is not renewed

130 Certificate Information Certificate of Performance Tested SM Status Certificate No. (number) The AOAC Research Institute hereby certifies that the performance of the test method designated as: Full Test Method Name manufactured by Company Name Company Address Country of Company has been reviewed under the AOAC Research Institute's Performance Tested Methods SM Program, and found to perform as stated by the manufacturer. This certificate authorizes the manufacturer to display the AOAC Performance Tested SM certification mark along with the statement - "THIS TEST KIT'S PERFORMANCE WAS REVIEWED BY AOAC RESEARCH INSTITUTE AND WAS FOUND TO PERFORM TO THE MANUFACTURER'S SPECIFICATIONS" - on the above mentioned test kit until December 31, (Current Calendar Year). Signed for AOAC Research Institute: Date: Scott G. Coates Managing Director

131 Sample Web Entry

132 Publications Inside Laboratory Management Summary of study Serves as certification report Journal of AOAC INTERNATIONAL Complete study Web site

133 Renewal and Modification

134 Renewal Applicant certifies that no changes have been made to the test method since the original PTM approval and the method performs as originally evaluated Submit Method Modification Review Form if changes have been made Granted for 1 year

135 Renewal Changes in test method Notification - responsibility of the licensee to notify AOAC-RI of any test method changes Failure to appropriately notify the AOAC RI of changes may result in the cancellation of PTM certificate

136 Modification Changes made in test method which effect: Test method instructions Labeling changes Deletion or restatement of validated claims or procedures Increase/decrease stability claims Test method performance Changes to method components Changes to enrichments/procedure Changes to equipment Changes to claim Changes to manufacturing process

137 Modification Review Levels Level 1 Internal AOAC RI review Change does not alter the validated performance of the test method Level 2 Licensee data required Reviewed by GR Level 3 Licensee and independent data required Reviewed by GR and 2 Expert Reviewers

138 Modification Testing requirements Level 1 Document review, no additional testing required Level 2 Matrix study required, at least 3 matrices Modified method compared to reference method Data collected by licensee

139 Modification Testing requirements Level 3 Licensee and independent lab data required Same as for new PTM study Matrix extension Matrix study required Independent lab data required if total claimed matrices for the method exceeds 4

140 Emergency Response Validation (ERV) Program Purpose is to respond immediately to emerging food contamination crisis Salmonella contamination in peanut butter Melamine in infant formula Provides a rapid one-matrix, one-analyte validation of candidate method Candidate methods evaluated using blind-coded samples prepared by independent lab Special Certificate of Validation granted upon approval PTM certified and OMA validated methods eligible (where applicable)

141 Cancellation of PTM Certificate If a licensee has: Not complied with original agreement relative to the use of the mark Not responded adequately to complaints of poor performance reported by users Modified the test method without notifying the RI Failed to submit a renewal application Requested PTM status to be discontinued

142 Complaints Formal complaints from applicant to be directed to the AOAC RI Managing Director in writing Formal complaints from test method user discussed with the method licensee Failure to address user complaints will result in the potential cancellation of PTM certificate

143 Potential Problems Incomplete method instructions Too vague Important parameters not described Method not fully adapted to new matrices Clinical method often applied to foods and other matrices that vary greatly in protein, fat, and other components Manufacturing and quality control problems

144 Summary Background Validation Overview PTM Validation Components Qualitative Study Design Quantitative Study Design Environmental Surface Testing Report and Review Certification and Publication Renewal and Modification

145 Proposed Revisions to the AOAC Guidelines for Microbiological Method Validation

146 AOAC RI Disclaimer The proposed revisions contained in this presentation are being reviewed but have not been approved by the AOAC Methods Committee on Microbiology nor the Committee on Statistics This presentation is applicable to the Performance Tested Methods Program and the Official Methods of Analysis Program

147 Proposed Revisions New terms and concepts for Qualitative Methods POD Probability of Detection Example analysis of qualitative SLV data LPOD Cross-laboratory Probability of Detection Repeatability Reproducibility Example analysis of qualitative collab data Study design changes MPN estimation 147

148 POD: Probability of Detection The proportion of positive analytical outcomes for a qualitative method for a given matrix at a given bacterial level or concentration. POD = x/n For example, if 17 out of 20 replicates at a given concentration are positive by the candidate method, then the candidate method POD is 17/20 = 0.85 or 85% at that concentration. Report POD values with 95% confidence intervals

149 POD: Probability of Detection POD C = POD of candidate method POD R = POD of reference method POD CP = POD of candidate method presumptive result POD CC = POD of candidate method confirmed result Replaces sensitivity and specificity No distinction between matched and unmatched analyses Does not require a reference method

150 Graph POD Values vs. Concentration Listeria Detection 100% Proportion Detected (POD) 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Concentration (MPN/25g) CP CC C R

151 dpod: Difference of Two POD Values dpod values are differences between two POD values: dpod C = POD C -POD R dpod CP = POD CP -POD CC Calculate 95% confidence intervals on dpod values If the confidence interval on dpod does not contain zero, then the two POD values are statistically different.

152 LPOD: Cross Laboratory POD Calculated as the mean POD across laboratories in a collaborative study: LPOD = x / N LPOD C, LPOD R, LPOD CP, LPOD CC Calculate 95% confidence intervals of LPOD values Graph LPOD vs. Concentration

153 dlpod: Difference of Two LPOD Values dlpod values are differences between two LPOD values: dlpod C = LPOD C LPOD R dlpod CP = LPOD CP LPOD CC Calculate standard deviation of dlpod, s dpod Calculate 95% confidence intervals of dlpod values If the confidence interval on dlpod does not contain zero, then the two LPOD values are statistically different.

154 Repeatability Calculate repeatability standard deviation, s r s r = LPOD(1- LPOD) Calculate repeatability standard error, SE r SE r = s r / n

155 Reproducibility Calculate the reproducibility standard deviation, s R s R = (POD l LPOD) 2 / L-1 Calculate 95% confidence interval on s R

156 Comparison of Repeatability and Reproducibility Compare SE r to s R If SE r falls within the 95% confidence interval of s R, then the laboratory effect is not significant. The contribution of the laboratory effect to the variation in POD values across laboratories can be estimated by s lab = (s R2 (s r2 /n)) If s R < Se r, then report s lab as 0%.

157 Study Design Changes Matrix Study Qualitative Methods Single Lab At least 3 levels: POD = 0, POD = , POD = replicates each level Collab Minimum of 8 laboratories At least 3 levels: LPOD = 0, LPOD = , LPOD = replicates each level

158 MPN Estimation SLV 5-tube 3-level MPN 2- or 3-fold serial dilutions Can use reference method replicates in calculation of MPN formula provided Collab Use reference method replicates from all labs

159 Reminder The proposed revisions contained in this presentation are being reviewed and have not been approved by the AOAC Methods Committee on Microbiology nor the Committee on Statistics Questions?

160 Special Thanks to our AOAC Statistical Advisors: Paul Wehling, Robert LaBudde, and Daryl Paulson