AOAC Official Methods of Analysis SM. Darryl Sullivan, Covance Laboratories and Past President, AOAC INTERNATIONAL

Transcription

1 AOAC Official Methods of Analysis SM Darryl Sullivan, Covance Laboratories and Past President, AOAC INTERNATIONAL

2 On March 28, 2011, the AOAC INTERNATIONAL Board of Directors approved an alternative path to achieve an Official Method (Official First Action status) for methods selected and reviewed using the AOAC volunteer consensus standards development processes. How this change came about

3 Rationale for Change AOAC s ability to validate fully collaboratively studied methods has been steadily on the decline, approving only three in AOAC was repeatedly disappointing customers and communities who needed methods to solve problems. AOAC already had a reputation of being slow and old with cumbersome processes was potentially facing decline in the confidence of our brand.

4 Rationale for Change AOAC has evolved and now acts as a problem solver through a broader process of consensus building and standards development. AOAC is trying to meet community/customer needs by gathering the world s authorities to articulate and evaluate methods needs through expertise and judgment.

5 Rationale for Change AOAC decided to find a way to give proper weight to the confidence we have in the judgment and collective knowledge of our experts. AOAC decided to align our brand and method output closer to our proven standards development processes The new or alternate path

6 How it Works At a Glance Funded Stakeholder Panel Working Groups to establish Standard Method Performance Requirements (SMPRs) Expert Review Panels to adopt methods as Official First Action based upon performance against SMPRs

7 How it Works: The Details Expert Review Panels Must be supported by relevant stakeholder body Membership is carefully managed and properly vetted by the AOAC Official Methods Board Holds transparent public meetings only Remains in force to monitor methods as long as method is in First Action Status.

8 How it Works: The Details Official First Action Status Decision Method adopted by ERP must perform adequately against the SMPR set forth by the stakeholders Method becomes Official First Action on date when ERP decision is made.

9 How it Works: The Details Official First Action Status Decision Methods to be drafted into AOAC format by a knowledgeable AOAC staff member or designee in collaboration with the ERP and method author. Report of decision complete with ERP report regarding decision including scientific background (references etc) to be published concurrently with method in traditional AOAC publication venues

10 How it Works: The Details Transition to Final Action Status ERP will monitor performance and data submitted for two years Further data indicative of adequate method reproducibility performance to be collected. Data may be collected via a collaborative study or by proficiency or other testing data of similar magnitude

11 How it Works: The Details Transition to Final Action Status Removed from Official First Action and OMA if no evidence of method use or no data indicative of adequate method reproducibility is forthcoming ERP makes recommendations to the Official Methods Board (OMB) OMB renders decision on transition to Final Action Status

12 Expected Benefits More Official Methods of Analysis generated We can provide solutions faster and take full advantage of collective expertise of AOAC members Methods can be put into regular use right away generating more useable data to evaluate performance

13 Expected Benefits OMA can be more flexible if a method is not performing up to ERP expectations it is removed The transition from first to final action becomes more meaningful and dynamic, more credence is given to final action methods.

14 Time for Change AOAC will convene its first Expert Review Panel charged with adopting Official First Action Methods of Analysis this afternoon. Many more ERPs will follow, developing out of the many stakeholder activities going on at AOAC

15 Questions and Comments? Thank you!

16 Standard Method Performance Requirements [SMPR] Guideline Gaithersburg, Maryland, USA Thursday June 30, 2011

17 Background First written in First used for the Endocrine Disruptor Compounds (EDC) project in Reviewed and revised by Official Methods Board in Still in review, but also in use.

18 Background Resulted when AOAC staff started a project create a standard SMPR format. It was realized that: If the SMPR format required certain parameters (i.e. recovery ) then a definition was needed. If we defined parameters then we needed to offer guidance on how to collect data.

19 Background If we defined parameters then we needed to offer guidance on how to collect data. If we offered guidance on how to collect data then we needed to provide guidance on acceptance criteria. If we offer guidance on acceptance criteria then we needed to explain the concept. So 2 pages turned into...

20 Background... into 13 pages with 14 pages of appendices. But... it a single, comprehensive document with good information all in one place for many different kinds of methods.

21 Philosophical Direction An attempt to bring to together several different AOAC technical documents. OMB Guidelines Microbiology Methods Guideline (Appd. X) AOAC Single Laboratory Guideline BTAM Guideline Best Practices for Microbiological Method (BPMM) Validation

22 Philosophical Direction An attempt to bring to together several different types of methods: Chemistry Microbiology Qualitative Quantitative Identity

23 Components of the Guideline 1. SMPR Format 2. Recommended Performance Requirement Parameters 3. Definitions 4. Recommendations for Evaluations 5. Explanations 6. Appendices

24 Architecture of Performance Requirements in SMPR Guideline Classification of methods Quantitative / Qualitative Main component / trace (contaminant) Identification method. Type of data single laboratory independent collaborative study

25 Big Notes! No distinction made between microbiology and chemistry! Not intended to require SLV independent lab collaborative Identification methods separated from qualitative methods,

26 Classifications of Methods 9 Quantitative Method ( main component 1 ) Quantitative Method (trace or contaminant 2 ) Qualitative Method (main component 1 ) Qualitative Method (trace or contaminant 2 ) Identification Method Collaborative Study Parameters Independent Single laboratory validation

27 Classifications of Methods 9 Qualitative Method (main component 1 ) Qualitative Method (trace or contaminant 2 ) Identification Method Parameters Single Laboratory validation Independent Reference Method Comparison Inclusivity/Selectivity Exclusivity/Specificity Environmental Interference Laboratory Variance Bias Probability of Detection TBD 5 Reference Method Comparison Inclusivity/Selectivity Exclusivity/Specificity Environmental Interference Laboratory Variance Bias Probability of Detection (POD) at the AMDL Probability of Detection (POD) at the AMD Reference Method Comparison Inclusivity /Selectivity Exclusivity/Specificity Precision Environmental Interference Bias Bias Collaborative Study POD (0) POD (c) Laboratory Probability of Detection 8 POD (0) POD (c) Laboratory Probability of Detection POD (0) POD (c) Laboratory Probability of Detection

28 Parameters Reference Method Comparison Inclusivity/Selectivity Exclusivity/Specificity Environmental Interference Laboratory Variance Bias Probability of Detection (POD)

29 Inclusivity/Selectivity Definition: Strains or isolates or variants of the target agent(s) that the methodcan detect. Recommendation: Analyze one test portion containing a specified concentration of one inclusivity panel member. More replicates can be used.

30 Exclusivity/Specificity Definition: Strains or isolates or variants of the target agent(s) that the method must not detect. Recommendation: Analyze one test portion containing a specified concentration of one exclusivity panel member. More

31 Bias Definition: The difference between the expectation of the test results and an accepted reference value. Bias is the total systematic error as contrasted to random error. There may be one or more systematic error components contributing to the bias. No recommendations.

32 Probability of Detection (POD) Definition: The proportion of positive analytical outcomes for a qualitative method for a given matrix at a given analyte level or concentration. Already discussed.

33 Probability of Detection (POD) Recommendations: Determine the desired Probability of Detection at a critical concentration. Consult with table 7 to determine the number of test portions required to demonstrate the desired Probability of Detection.

34 No definitions or recommendations Reference Method Comparison Environmental Interference Laboratory Variance

35 Summary SMPR summarized a variety of AOAC guidelines. SMPR is comprehensive, but not detailed. SMPR includes chemistry and microbiology; quantitative and qualitative. Work in progress.

36 Chi-Square Statistics in Method Validation ISPAM Microbiology and Chemistry Working Groups 30 June, 2011 Dan Tholen, M.S.

37 Chi-Square and Related Issues Different designs Statistical estimators vs. statistical tests McNemar Chi-Square test in ISO Related estimators and tests Relationship to POD and dpod

38 Estimators vs. Hypothesis Tests Estimators provide best estimates of parameters of interest, based on design Accuracy, Sensitivity, Specificity, POD, etc. Hypothesis tests provide advice on whether differences in estimators could have occurred by chance Assumes a statistical distribution Requires consideration of Type 1 and 2 error Requires decision level

39 Estimators vs. Hypothesis Tests Estimators should be accompanied by confidence intervals that show a range of values that could be the correct value Similar to measurement uncertainty Hypothesis tests usually come with a reject or do not reject decision, and perhaps with a p value which is a likelihood for the evidence if H 0 is not true Not as informative as C.I. or MU

40 Chi-Square Analysis Recommened in ISO 16140, and in proposed CD 16140:2011 And in protocols influenced by ISO Used for testing equivalence of methods by looking at discordant results Very powerful technique, often based on only a few discordances out of hundreds of agreements (that is, small differences between methods can be significant)

41 Comparative Accuracy Separate study for several categories of food (up to 5 categories) Select at least 3 types of food from each category Select at least 20 samples representative of each type Independent samples, not replicates Ideally 10 negative, 10 positive Test each sample with both methods

42 Two Way Designs Unpaired: from the same sample, but separate test portions Paired: from the same sample and shared first step in the enrichment procedure From same enrichment medium (microbiology) From the same extraction (chemistry)

43 2 x 2 Layout Paired (and ISO unpaired) Method B (Reference) Total Method A (Alternati ve) Present Absent Present a b N A+ Absent c d N A- Total N B+ N B- N

44 Estimators - ISO Paired Relative accuracy: AC = (a+d)/n Relative specificity: SP = (d)/n B- Relative sensitivity: SE = (a)/n B+ Sensitivity (altern): SE alt = (a+b)/(a+b+c) Sensitivity (refrnc): SE ref = (a+c)/(a+b+c) Alternative method results confirmed for reference method negatives (cells b & d)

45 Estimators Unpaired Relative accuracy: AC = (a+d)/n Relative specificity: SP = (d)/n B- Relative sensitivity: SE = (a)/n B+ These estimators are same as for paired All alternative method results are confirmed, estimators are listed separately for confirmed and unconfirmed results

46 Chi-Square Test ISO Only McNemar test is discussed 2 = b-c 2 / (b+c) (1 degree of freedom) Considers only discordant results Other estimators not tested, McNemar is considered the most sensitive test to rule out differences due to random error Requires minimum size, b and c (b+c>22) Often the exact test (Binomial) is used for small size samples, or in all cases

47 Chi-Square Test ISO Test for significant difference in proportion of positives P+ A = (a+b)/n P+ B = (a+c)/n Since P+ A and P+ B both use a, the proportions are correlated Most sensitive test is for whether b and c are statistically different Binomial, p=0.5 n=b+c

48 Equivalence with POD Concept P+ A : POD A = (a+b)/n P+ B : POD B = (a+c)/n P+ A - P+ B = dpod Test of dpod using the Binomial (for H 0 : dpod=0) is the same as the McNemar Large and small numbers of tests Both paired and unpaired For single lab or multi-laboratory studies

49 Note on Nordval Nordval (May 2010), for Qual. Chemistry Uses same 2x2 layout, for paired data Does not use Chi-Square. Uses Kappa, a measure of agreement that corrects for random agreement The extent to which agreement exceeds chance agreement is a measure of concordance Nordval recommends agreement > 80%

50 Note on Nordval Uses Kappa, a measure of agreement that corrects for random agreement That is, if methods A and B are totally unrelated, there is a likelihood that they will agree on a lot of results, just by chance e.g., if A reports 80% positive and B reports 70% positive, then we expect them to agree 56% of the time, just by chance (0.8x0.7 = 0.56) So the extent to which the methods agree in excess of 56%, is a measure of concordance

51 Classic Unpaired Design When two methods are used on unpaired samples For example, drug studies on treatment and placebo groups This is a classic design, not done in method validation studies Mentioned in ISO 16140, but not used

52 2 x 2 Layout Unpaired (not used in ISO 16140) Result Total Method Present Absent Method A e f N A Method B g h N B Total N + N - N

53 Estimators Assume random samples from same population, randomly assigned to A or B Proportion Positive A = (e)/n A (= POD A ) Proportion Negative A = (f)/n A Proportion Positive B = (g)/n B (= POD B ) Proportion Negative B = (h)/n B Accuracy, sensitivity, specificity not defined unless all results are confirmed

54 Chi-Square and POD P+ A - P+ B = dpod Chi-Square test is the same as a Binomial test of null hypothesis: H 0 : dpod = 0

55 Chi-Square Test Checks only for differences between observed and expected numbers of results in each cell Expected based on random assignment of subjects to A or B, so expect same proportion of positives in A and B (and same proportion of negatives) Expected calculated from marginal frequencies

56 Estimators POD Concept P+ A : POD A = (e)/n A P+ B : POD B = (g)/n B P+ A - P+ B = dpod Chi-Square test is the same as a Binomial test of null hypothesis: H 0 : dpod = 0

57 Thank you

58 For the Validation of Qualitative Methods Paul Wehling June 30,

59 Qualitative (Binary) Methods Methods that are restricted to 2 possible outcomes: Positive or Negative Pass or Fail Heads or Tails 1 or 0 Yes or No Presence or Absence Identified or Not Identified 2

60 POD Parameter Probability of Detection General Designed to be used by any Qualitative (Binary) Method Microbiological Chemical Bio Threat Agent Methods Botanical Identification Allergens 3

61 POD Parameter Method Parameter that describes and predicts method behavior Probability of Detection or POD The probability of getting a positive result at a given concentration of analyte. POD is a function of concentration 4

62 POD Curve 5

63 POD Curve 6

64 POD A simple descriptive statistic that describes the method performance at a given concentration. It is a calculation of proportion of observed positive outcomes per total trials. This simple statistic is inherent in all other systems, such as Chi-Square, LOD, RLOD. The POD Concept is only new in that it recognizes the POD as a key parameter and plots a graph of POD vs concentration. 7

65 WHY PLOT POD? Plot of POD Curves are intended to assist method users To assist users in selecting best method for intended use. Understanding POD Curve is crucial for interpretation of results. The POD curve can be an indicator of the usefulness of the method. If POD were constant across all concentrations, the method would not be useful. 8

66 VALIDATION The task of validating a qualitative (binary) method is characterizing the POD curve at critical concentration points. 1. Make up a series of test materials at concentrations of interest. 2. Analyze with replication 3. Calculate the proportion of positive responses at the concentrations. 4. Plot observed proportions as POD curve by concentration. 9

67 10

68 Example POD Response Curve 11

69 A BIT ABOUT POD POD is a combination of sensitivity, specificity, false positives, false negatives. Where did they all go? 12

70 POD Where s my False Negative? POD Response vs Concentration False Negative at 1 ppm 1-POD(1 ppm) Specificity 1-POD(0) Sensitivity at 1 ppm POD(1 ppm) False Positive POD(0) Concentration (ppm) 13

71 Some Statistics To do Classical Collab Statistics, Code Results Positive = 1 Negative = 0 Use AOAC Calculations from Quantitative Stats to estimate Mean = POD Reproducibility Standard Deviation Repeatability Standard Deviation Laboratory Standard Deviation 14

72 POD = Mean LAB1 LAB2 LAB3 Trial Trial Trial Trial Trial Trial Trial Trial Trial Trial Mean LPOD =

73 Analogous Parameters Method Attribute General Mean or Expectation Repeatability Variance Reproducibility Variance Laboratory Variance Expected difference between Two Methods* Quantitative Parameter Quantitative Estimate Qualitative Parameter Mean, μ Mean, POD 2 r 2 R 2 L Bias, B x 2 s r 2 s R 2 s L x x r 2 R 2 L dpod Qualitative Estimate POD or LPOD POD 2 s r 2 s R 2 s L POD

74 Difference Between Methods - dpod Compare any two methods by comparing POD values at a given concentration. Difference by subtraction dpod = PODc PODr dpod is always dependent on concentration 17

75 dpod(3.5) = dpod (2) = dpod (c = 0.5) =

76 dpod Curve vs Concentration 19

77 Big Ideas Combine sensitivity, specificity, false positive, false negative into 1 parameter Probability of Detection or POD Graph POD vs. Concentration with Confidence Intervals Compare methods by difference of POD at same concentration Use the classic statistical model and descriptive stats for quantitative methods as the tool for calculating qualitative stats. 20

78 POD Concept Works for single lab and Multilab experiments. Works for paired and unpaired designs. Provides harmonization across qualitative/quantitative methodologies. Does comparisons and hypothesis tests via confidence interval analysis equivalent to chi-squared tests. POD Curve plots mean response and uncertainty on the same graph. 21

79 Qualitative Method Validation Studies for Quantal Data: LOD, dpod, PRE = RLOD and ω Robert A LaBudde, BS, MS, PhD, ChDipl ACAFS, PAS AOAC Statistical Advisor Least Cost Formulations, Ltd. Old Dominion University Copyright 2011 by Robert A LaBudde 1

80 Summary Example POD vs. Concentration curves Ideal POD vs. Concentration curve Transition range models Method performance requirements I Method performance requirements II Limit of Detection ( LOD ) The Concentration Fallacy for micro Copyright 2011 by Robert A LaBudde 2

81 Summary (cont d) Method Performance Parameters III Difference in POD: dpod(c,r) Odds ratio ω Poisson Efficiency Ratio ( PRE ) Relative LOD ( RLOD ) Examples: Micro Copyright 2011 by Robert A LaBudde 3

82 Summary (cont d.) Non-micro methods Choice of metamer Warning for micro studies Conclusions & Recommendations Copyright 2011 by Robert A LaBudde 4

83 Example POD vs. Concentration curves Copyright 2011 by Robert A LaBudde 5

84 Ideal Response vs. Concentration curve POD = Probability of Detection = # Positive / # Trials = mean of 0 or 1 data The ideal test method gives POD = 0 at Concentration = 0, and POD = 1 for all concentrations > 0. For real methods, there is a transition from POD = 0 to POD = 1 over a range of Concentration. Copyright 2011 by Robert A LaBudde 6

85 Transition range models True shape of transition curve depends upon underlying model of what happens in test method. Symmetric distribution threshold crossing: Probit and Logit (historically these have been most commonly used). Asymmetric distribution threshold crossing: can be concave or convex shape. Hormesis : drop-off at high concentrations. Copyright 2011 by Robert A LaBudde 7

86 Transition range models (cont d) There are a dozen or more possible model forms in common use. Choice of a model form is subjective and subject to controversy. Some curves convex, some curves concave, some symmetric. Logit and probit are traditionally used as middle ground when true shape is unknown. Copyright 2011 by Robert A LaBudde 8

87 Transition range models (cont d) Advantage over individual POD values may be improved precision by pooling across concentrations. If model form is incorrect, may have worse precision than individual POD values. Generally requires Concentration be known accurately. Copyright 2011 by Robert A LaBudde 9

88 Method Performance Requirements I: Confirmation At the most basic level, a qualitative method is meant to discriminate between the presence and absence of an analyte. At zero concentration, POD < POD max with 95% confidence. (Control false positive fraction.) At moderate concentration, POD > POD min with 95% confidence. (Control false negative fraction.) Attainment of these two requirements validates the method as a confirmation or identification method in testing. Copyright 2011 by Robert A LaBudde 10

89 Method Performance Requirements I (cont d) No real method, despite claims, has POD = 0 at zero concentration or POD = 1.0 even at high concentration, due to various error sources, including human-in-the loop. One method is better than another if it has lower POD (FPF) at zero concentration and higher POD (lower FNF) at moderate concentration. Copyright 2011 by Robert A LaBudde 11

90 Method Performance Requirements II: Transition region The I set of requirements does not speak to the transition range of the POD vs. Concentration curve. A method which satisfies the POD min performance requirement at lower Concentration is better than another method does so at higher Concentration. A method which has POD < 1 may still be useful in repeated testing if no better method is available (e.g., outbreak investigations for micro). Copyright 2011 by Robert A LaBudde 12

91 Limit of Detection LOD One way commonly used to characterize a method in the transitional range is to estimate the concentration at which a particular POD is attained. LOD50 : Concentration for which POD = 0.50 LOD90 : Concentration for which POD = 0.90 Various techniques for estimation, including nonparametric ones, such as linear interpolation and Spearman-Kaerber (POD-based), or assumed models. Requires several points (at least 2, preferably more) in the transitional or fractional range. Requires accurately known concentrations! Copyright 2011 by Robert A LaBudde 13

92 LOD50 Copyright 2011 by Robert A LaBudde 14

93 The Concentration Fallacy for Micro Methods The transition region for qualitative methods for micro testing typically occurs below 10 CFU/test portion and so cannot be quantified by plate count methods, particularly with other flora present. Instead a MPN method is used, based on a reference qualitative method. So Concentration is determined from POD (not v.v.), and typically has large error limits (e.g., + 60% or much worse). POD is known more accurately than Concentration. Models fitting POD using Concentration as a predictor are invalid. LOD50 will be imprecise and unknown to a multiplicative factor (bias) due to clumping of cells. Copyright 2011 by Robert A LaBudde 15

94 Micro: 1-Hit-Poisson Model Copyright 2011 by Robert A LaBudde 16

95 Method Performance Requirements III: Comparison of Methods Two methods which both satisfy the I requirements equally can only be discriminated if one or the other has data in its transition range. There are a number of measures of effect in common use to compare a candidate method C to a reference method R (or any two methods) to each other, based on measured POD values at different concentrations in the transition range (i.e., fractional POD range). Copyright 2011 by Robert A LaBudde 17

96 Difference in POD: dpod(c,r) The most basic comparison between a reference method R and a candidate method C is the difference in their POD values at a fixed concentration dpod(c,r) = POD(C) POD(R) Non-constant for difference concentrations. Expected difference in # positives easily estimated as n x dpod(c,r). Requires no assumptions, applicable in all cases. Copyright 2011 by Robert A LaBudde 18

97 Odds ratio ω The most common measure of effect used to compare two binary methods in scientific research is the odds ratio or ω POD(R)/[1-POD(R)] ω = POD(C)/[1-POD(C)] If a Logit model is appropriate, the odds ratio is a constant across concentrations. Copyright 2011 by Robert A LaBudde 19

98 Poisson Relative Efficiency PRE or R LaBudde, R.A. (2006). Statistical analysis of interlaboratory validation studies. X. Poisson-plot and Poisson relative efficiency to compare the dose-response curves of two presence-absence methods. TR239. Least Cost Formulations, Ltd., Virginia Beach, VA. ln [ 1 POD(C) ] R = = ---- ln [ 1 POD(R) ] γ R γ C where the one-hit Poisson model 1HPM is assumed to hold. Copyright 2011 by Robert A LaBudde 20

99 PRE (cont d) If both the reference and candidate methods obey the 1HPM model with cluster sizes γ R and γ C, resp., the R = γ R / γ C is the ratio of the two cluster sizes needed. If the reference method is better (has a lower γ or LOD50), then R < 1.0. If the 1HPM is valid, R will be constant across different concentrations. Generally applicable to micro studies only. Copyright 2011 by Robert A LaBudde 21

100 Relative LOD RLOD Anon. (2008). ISO If a 1HPM model assumption is made for the mathematical form of POD, and log(concentration) is used as the metamer in the model, a complementary-log-log model results. For the complementary-log-log model, RLOD = R, and γ R and γ C are the factor coefficients for the Method term in the regression model. RLOD is the same value as PRE = R. Copyright 2011 by Robert A LaBudde 22

101 R vs. ω vs. POD If POD(C) and POD(R) are both small, R ~ ω ~ POD(C) / POD(R) If POD(C) and POD(R) are both large, R ~ ω ~ POD(C) / POD(R) If POD(C) ~ POD(R), R ~ ω ~ POD(C) / POD(R) ~ 1 Differ more otherwise. Copyright 2011 by Robert A LaBudde 23

102 Examples: Micro Analyte Matrix Study Level MPN(R) dpod(c,r) w(c,r) R(C,R) Salmonella Raw ground turkey Unpaired H Salmonella Raw ground beef #1 Unpaired H H Salmonella Raw ground beef #2 Unpaired H L Salmonella Dried whole egg Paired H L Salmonella Milk chocolate #2 Paired H L Salmonella Dry dog food Paired H L E. coli O157:H7 Raw ground beef Unpaired H L Copyright 2011 by Robert A LaBudde 24

103 Method Performance Requirements III Possible performance requirements: dpod(c,r) < dpod max with 95% confidence ω(r,c) > ω 0 with 95% confidence R(C,R) > R min with 95% confidence Copyright 2011 by Robert A LaBudde 25

104 Non-Micro Methods Toxins, residues, chemicals, allergens, botanicals. There is a large literature associated with POD vs. Concentration modeling and fitting for toxicology. Complementary-log-log is not typically a good match for non-micro methods, typically logit and probit have been used successfully. None of the standard regression models work for botanical identification methods where complex thresholding occurs. Copyright 2011 by Robert A LaBudde 26

105 Choice of metamer Most models use either Concentration directly or log(concentration) as a predictor. The transform of Concentration to a new independent variable is called the metamer of Concentration. It should be noted that linear models using Concentration as metamer and linear models using log(concentration) as metamer cannot both be correct. Copyright 2011 by Robert A LaBudde 27

106 Warning for micro studies In the case of micro methods, cells are indivisible, and CFUs finitely divisible, so sampling error dominates at low concentrations. Method differences are obscured, if they have low LOD50 s. The 1HPM or complementary-log-log model will appear to fit the data very well, and this is fallacious because Concentration is determined assuming 1HPM in the MPN method. Micro study modeling should not use numerical values of Concentration! Copyright 2011 by Robert A LaBudde 28

107 Conclusions Models involving Concentration are flawed at inception for micro methods. (This applies to Method Performance Requirements III in the transition region and to LOD50.) Serial dilution should be considered as a possible method to achieve a POD-independent Concentration estimate. Many alternative models exist, each with their own history and literature. PRE (aka RLOD or R) depends on the assumption that the 1HPM is correct, which it often is not. Copyright 2011 by Robert A LaBudde 29

108 Conclusions (cont d) LOD50 requires Concentration be known reasonably accurately, problematic for micro. LOD50 can be nonparametrically estimated from POD, with no model assumptions. The choice of statistic used to characterize the transition region of POD vs. Concentration should be made based on the scientific validity of the model assumptions and the ease and usefulness of interpretation of the result. Chemical-based methods have accurate Concentrations; Micro studies do not. Copyright 2011 by Robert A LaBudde 30

109 Conclusions (cont d) Use of the wrong model form will may give poorer results than using the POD vs. Concentration curve directly, and comparing methods by POD difference ( dpod ). Model forms that work for one analyte or matrix may not be appropriate for another, even in the same scientific method area. Nonparametric methods (POD included) that are distribution and model assumption-free are preferred to unvalidated model assumptions. Copyright 2011 by Robert A LaBudde 31

110 Working Group on Statistics Provide advisory guidance to Micro and Chem Working groups on aspects of statistical methodologies. Advise on strengths, weaknesses and applicability of various models. Advise on power of various validation experiment designs. Look for potential areas of agreement and encourage flow of ideas across Chem/Micro working groups.

111 Working Group on Statistics Develop scientific consensus on the best statistical techniques to use for validating qualitative methods.

112 Microbiological Harmonization June 29/30, 2011 Comparison of Method Validation Schemes

113 Worldwide Validation Schemes ISO 16140: internationally accepted standard for microbiological method validation AOAC Microbiology Guidelines Health Canada Part 4 NordVal (essentially w/o collaborative) FDA Draft Guidelines USDA/FSIS Draft Guidelines

114 Comparison of Elements Comprehensive table constructed Six schemes compared Qualitative: 30 topics Quantitative: 19 topics Initial effort on Qualitative 5 of 6 schemes are either new or under revision

115 Areas of Divergence Microbiology Working Group (WG) had 2 teleconferences Identified the 5 most significant areas of divergence among the 6 schemes. Nominated a Project Group (PG) with representative from each organization ISO/NordVal both use so only 1 representative

116 Significant Topics From 30 topics 5 were chosen as most critical: Reference method choice Food/Sample Matrix Applicability Table. Selection of Food/Category # of levels/ # of samples/sample size/ # of laboratories: Method Comparison & Collaborative Definition of fractional positive recovery Data analysis (Chi square, RLOD, LOD, POD) & performance parameters reported

117 Today 8 page comparative summary table prepared for the 5 significant topics PG will hold inaugural meeting to share ideas on harmonization

118 QUALITATIVE methods ISO AOAC OMA Health Canada NordVal FDA Draft USDA/FSIS ISO Doc N 1199 (ISO CD )PIV C Pending revision of Part 2 AOAC OMA Draft revision document dated 3/24/11 Health Canada Draft Part 4 dated March, 2011 NordVal Protocol for the validation of alternative microbiological methods March 2009 FDA Guidelines FDA s Qualitative Microbiology Methods Validation (ORA-LAB.7 version 1.2), pending revision (proposed revision marked in red). Draft USDA/FSIS Guidelines Disclaimer: The use of the term validation is not intended to have any application to the implementation of 9 CFR 417.4(a)(1) on initial validation of HACCP plans. The Draft FSIS Guidelines deals exclusively with the evaluation of pathogen test kit methods. Pre- Collaborative Phase(s) -Reference Method -Defined in ISO st priority is ISO method, 2 nd priority is CEN method, if neither exists, then 3 rd priority is other recognized methods Note: definition still under discussion at ISO level to open up for non ISO/CEN methods (PIV) -Can be various preexisting recognized analytical methods e.g. AOAC OMA, ISO, FDA BAM, FSIS MLG and Health Canada -If no appropriate Ref can indicate NA in summary tables for POD -Acceptable Ref published by HC (Part 1) -May include any methods from methods organizations, such as AOAC, BAM, APHA, ICMSF, IDF, ISO etc. -Where no Ref exists, MMC assess on case by case basis ISO, CEN, NMKL, BAM, etc. It is up to the applicant; however, as the EU regulation in EC 2073/2005 Microbiological criteria states EN ISO methods, these are most frequently used. -Must be BAM, unless there is no BAM reference method. -If these is no BAM reference method, but if there is a nationally/internationally recognized reference method, then FSIS MLG, AOAC, ISO, and Health Canada are all potential reference methods. APHA, ICMSF, and IDF methods also may be used as reference methods. For FSIS regulated products, the current FSIS method, which is found in the Microbiology Laboratory Guidebook (MLG), is the most appropriate reference cultural method for validating methods used by FSISregulated establishments. FDA BAM, or methods referenced by ISO or Codex Alimentarius may be appropriate. Non-cultural methods applicable in some circumstances. Page 1 of 8

119 -Selection of food ISO AOAC OMA Health Canada NordVal FDA Draft USDA/FSIS RA -5 categories for all foods applications, 3 food types per category (see below) -Feed, environmental samples and primary production samples (PIV)are additional categories RLOD Same, except 1 food type per category (if possible) a different food type SLV -All claimed matrices must be included in the study, in other words, no defined categories, and all foods claim not applicable -Environmental surfaces claim require 3-7 different surfaces (# required is under review (RF) IV At least 1 matrix that was tested in the SLV. For every 5 foods claimed, 1 food matrix must be included -5 categories for all foods applications, 3 food types per category (Table 1). -Environmental samples is additional category RA, SE, SP, Kappa -5 categories for all foods applications, 3 food types per category (see below) -Feed, environmental samples are additional categories LOD Same, except 1 food type per category (if possible) a different food type The selection of foods is determined by FDA s regulatory needs. Matrices commonly sampled in FSIS regulated establishments: meat, poultry, and egg products, and environmental samples (sponges, swabs, brines) All claimed matrices must be included in the study. Contains proposal to create matrix categories based on intrinsic properties. All Foods claim not applicable Page 2 of 8

120 - Food category/type/ item ISO AOAC OMA Health Canada NordVal FDA Draft USDA/FSIS Each Food type can be made of various relevant food items. Annex B provides guidance ( not mandatory) These are then grouped together to meet the sample number requirement of a food type, i.e. 20 samples. This is allow for the use of naturally contaminated samples (BL) Only one single food item is accepted to meet the sample size requirement of a food type, i.e. 20 replicates. Each food type can be made of various relevant food items. Table 1. CLARIFICATION NEEDED Can these be grouped together to meet the sample number requirement of a food type, in this case 20 samples? Yes, they can be group together to meet the sample number requirement. This notion has been introduce to allow for heterogeneity with in a food type. Products in a type may vary greatly in origin, composition, preparation processes, natural background; all those small variabilities could have an influence on the detectability of the target organism. (I.I.) Each Food type can be made of various relevant food items. At NordVal s homepage ( provides a list of food categories These are then grouped together to meet the sample number requirement of a food type, i.e. 20 samples. Currently, foods are validated individually and there are no category claims. There are no All Foods claims. Page 3 of 8

121 -No. of levels/samples ISO AOAC OMA Health Canada NordVal FDA Draft USDA/FSIS RA 20 samples per food type or 60 samples per category RLOD 3 levels -negative controls =5 samples -1 level ( theoretical LOD, with fractional positive results (BL)) = 20 samples -Another level = at least 5 samples SLV and IV 3 levels: -negative controls =5 samples -1 level with fractional positive results = 20 samples -Another (high) level = 5 or 20 samples (under review (RF) 3 levels: -negative controls =5 samples -1 level with fractional positive results = 20 samples -Another level up to 1 log higher= 20 samples RA 20 samples per food type or 60 samples per category LOD 3 levels -negative controls =5 samples -1 level ( theoretical LOD) = 20 samples -Another level = at least 5 samples Level 1, 6 replicates/level, single level Level 2, 6 replicates/level, 1 inoculated level + 1 uninoculated level (5 replicates) Level 3, 10 replicates/level, 1 inoculated level + 1 uninoculated level (5 replicates) Level 4, 20 replicates/level, 1 inoculated level + 1 uninoculated level (5 replicates) It is proposed that each of the 4 levels use 20 replicate test portions and that all levels have a negative control. For each matrix and analyte: 1) minimum 60 samples inoculated at fractional recovery level per alternative and reference method 2) 5-10 uninoculated samples per alternative and reference method -Sample size Undefined MicroVal: Is specified in the reference method, other (larger) samples size is allowed but specified in the certificate. (PIV) Standard is 25 g or 25 ml, unless Ref method specified larger sample size -25 g, but larger sample sizes are permitted -Sample size must be the same for alternate and Ref methods, consult MMC if testing composite samples Undefined 25 g unless otherwise specified. Application dependent. Portions should not be made larger without validation. Validation study conclusions from larger portions applicable to smaller portions. -Fractional positive Can be achieved by either alternate or Ref. - All samples should not be all positive or all negative. -Ideal is 10 positive and 10 negative (50%) but any fractional results is acceptable Can be achieved by either alternate or Ref. -proportion of positives 25% to 75%, ideal is approx 50% (10% to 90% is under review (RF) Can be achieved by either alternate or Ref. -proportion of positives 25% to 75%, Can be achieved by either alternate or Ref. - All samples should not be all positive or all negative. -Ideal is 10 positive and 10 negative (50%) but any fractional results is acceptable Yes, one or both methods must give 40 90% positive results. It is proposed that the percentage positive results be changed to 25 75%. defined as a range of 20-80% confirmed positive results using reference method Page 4 of 8

122 -Results analysis and criteria ISO AOAC OMA Health Canada NordVal FDA Draft USDA/FSIS RA -By type and by category -Relative accuracy AC, relative specificity SP, relative sensitivity SE -First by unconfirmed results, again by confirmed results -McNemar test as criteria, (for paired and unpaired) with caveats i.e. really not suitable for unpaired and never be interpreted by only the McNemar test RLOD -by category -LOD of alternate method divided by LOD of Ref For paired, no lower limit, but LOD alternate might not be > 2 times the LOD Ref For unpaired samples, no lower limit, the LOD alternate might not be >3 times the LOD Ref (In the ISO/CD version, I don t find any acceptability limit settled for unpaired samples, only specified for paired samples BL) The values of 2 (paired) and 3 (unpaired) are still tentative values!!!!! (PIV) - by level and by matrix - by POD Probability of Detection 95% confidence interval for the alternate the Ref and presumptive and confirmed results -then by difference between POD alternate and POD Ref, confidence level must contain zero for method to be considered not different at 95% confidence - Chi Square is not required but interesting Method Equivalence: -POD -one-tailed POD 95% confidence interval (I.I.) Performance parameters: - by level and by food, but only calculated for those that passed POD successfully For Unpaired : -Performance parameters is the comparison of presumptive vs. confirmed results of the alternate method ( not the Ref method results) -Specificity is based on presumptive results -Sensitivity is based on final ( confirmed) results - Equivalence of alternate method and Ref can only be determined by the number of true positives in both sets, done by POD method For Paired: Use absolute results where Ref can have FN Criteria: Sensitivity 98% Specificity 90.4% False negative rate < 2% False positive rate 9.6% Efficacy 94% LOD must be comparable or exceed the lower LOD of the Ref RA -By type and by category -Relative accuracy AC, -Relative specificity SP, -Relative sensitivity SE - Kappa -First by unconfirmed results, again by confirmed results Criteria: SE 95% Kappa 0.80 LOD: fit for purpose By level/individual experiment for each matrix. Per AOAC Microbiology guidelines, McNemar Chi Square statistics are used. Performed for each matrix. Unpaired study: One sided chi-square test with alpha = Criterion: indistinguishable or better performance than reference method. Paired study: Evaluate sensitivity with minimum 29 confirmed positive results. Zero false negative results from 29 confirmed positives would be consistent with a test having a sensitivity that met or exceeded 90% and zero negative results from 50 confirmed positives would be consistent with a test with a sensitivity that met or exceeded 94%. Criterion: none proposed Page 5 of 8

123 Interlaboratory Study ISO AOAC OMA Health Canada NordVal FDA Draft USDA/FSIS Applicable to alternative methods with a major modification, defined as any significant change in the design or the component reagents for a screening test, for example, the introduction of a new antibody or oligonucleotide primer. - minimum no. of valid data sets/collaborators -10; defined as individuals working independently using different sets of samples; from a min. of 5 different organizations, including organizing lab and different locations from same company -10 valid lab data sets required - Specifies that 12 labs should start Minimum of 8 labs reporting valid data, labs should be accredited per or demonstrate is functioning under equivalent quality system -10; defined as individuals working independently using different sets of samples; from a min. of 5 different organizations, including organizing lab and different locations from same company 2 for a Level 2 study, 3 for a Level 3 study, and 10 for a level 4 study. Follow guidance provided by the AOAC International Official Methods of Analysis Program -Sample size NA Is defined by the protocol of the reference method (PIV) Standard is 25 g or 25 ml, unless Ref method specified larger sample size CLARIFICATION NEEDED Consistent with Precollaborative? Sample size is 25g unless otherwise specified by the method or need for larger size ( to achieve enhance detectability, regulatory purpose or compositing) (I.I.) NA 25 g unless otherwise specified. Page 6 of 8

124 - number of foods ISO AOAC OMA Health Canada NordVal FDA Draft USDA/FSIS 1; relevant food item, inoculated with target, using a challenging enrichment protocol 1 At least 1 1; relevant food item, inoculated with target, using a challenging enrichment protocol One or more. - number of levels 3; negative control, one level which produce fractional positive and another level 3; negative control, one level which produce fractional positive and another level 3; negative control, one level which produce fractional positive and another level about 10 times greater than the detection level 3; negative control, one level which produce fractional positive and another level 2 for a Level 2 study (1 inoculated and 1 uninoculated. 3 for Levels 3 & 4 (high, low, and uninoculated. - number of replicates 8; per level of contamination -minimum of 48 results per collaborator = 8 replicates x 3 levels x 2 methods -minimum of 480 results (48 from each collaborator) = ( 240 per method) for statistical analysis 12 per level of contamination - 72 results per collaborator = 12 replicates x 3 levels x 2 methods = 72 - minimum of 720 results ( 360 per method) for statistical analysis 8 per level - min of 24 results per collaborator ( 8 x3 levels ) per method 8 laboratories; - 3 levels in duplicates 8 labs x 3 levels x 2 replicates x 2 methods 6 -Confirmation for Paired, only confirm the + Alt/- Ref, for Unpaired, confirm all enrichments Matched or unmatched, confirm all samples Confirm all samples for Paired, only confirm the + Alt/- Ref, for Unpaired, confirm all enrichments Yes. Page 7 of 8

125 - Comparisons Analyzed two ways: 1.Unconfirmed Alternate method results vs. confirmed Ref 2. Confirmed Alternate method results vs. confirmed Ref ISO AOAC OMA Health Canada NordVal FDA Draft USDA/FSIS By level and by matrix analyzed and reported separately CLARIFICATION NEEDED Consistent with Precollaborative? By level and by matrix, all result confirmed Confirmed alternate method results vs reference (I.I.) Alternative to reference method (if available). -Parameters Calculated - Interpretation Specificity ( only for Neg controls) Sensitivity ( only for inoculated levels ) Relative Accuracy (%of agreements ) RLOD of the different participants (BL) McNemar test (chi square) RLOD is for information only : analysis of deviance test to assess the laboratory effect on RLOD then acceptability of RLOD global value (BL) Cross Lab Probability of Detection (LPOD) Difference between Alternate LPOD and Ref LPOD If confidence interval of dlpod does not contain zero, then the diff is statistically significant CLARIFICATION NEEDED Consistent with Precollaborative? Yes, POD, dpod determined for each matrix-level. All dpod data is then used to assess the comparative performance of both methods All 5 method parameter(specificity, selectivity, FP, FN and method efficacy) calculated in one of two ways,, depending if sample is paired or un paired. (I.I.) CLARIFICATION NEEDED Consistent with Precollaborative? Yes, dpod one-tailed and method parameter requirement must be met. (I.I.) Rel Specificity Rel Sensitivity Rel Accuracy Kappa Criteria: SE 95% Kappa 0.80 [LOD: fit for its purpose] Per AOAC guidelines, Sensitivity, Specificity, False Negative, and False Positive Rates. Per AOAC guidelines, McNemar Chi Square statistics. Page 8 of 8

126 International Stakeholder Panel on Alternative Methods Microbiology Working Group for Harmonized Matrix Comparison Thursday, June 30, 2011 at 1:00pm 3:00pm Twinbrook HILTON WASHINGTON D.C./ROCKVILLE EXECUTIVE MEETING CENTER DRAFT MATRIX TABLES FROM: EN ISO 1614:2008 NORDVAL - AOAC BPMM AOAC BPMM WORKING GROUP MATRIX EXTENTION DRAFT ISPAM 1) EN ISO 16140:2008 E 2) NORDVAL MATRIX TABLES a. Salmonella b. Listeria c. Campylobacter d. E.coli O157 3) Annex A AOAC OMA Microbiological Guidelines 4) Appendix B BPMM AOAC Microbiological Working Group for Matrix Extension AOAC ISPAM Small Group Micro Working Groups Meeting Agenda nlm PRE-DECISIONAL Page 1

127 EN ISO 16140:2008 (E) nlm

128 EN ISO 16140:2008 (E) nlm

129

130 Matrix for Salmonella (NORDVAL) Matrix group Matrix Food examples 1. Meat 1.1 Raw red meat Minced meat, offal 1.2 Raw white meat Chicken, turkey, duck 1.3 Raw smoked salted products Bacon 1.4 Heat treated products Sliced meat and poultry products 1.5 Fermented products Salami 2. Fish 2.1 Raw fish and shelfish Raw two-shelled mollusc, raw shrimps 2.3 Heat treated fish products Heat treated shrimps and shelfish 3. Milk 3.1 Milk Raw milk 3.5 Desserts, ice-cream Ice-cream 3.6 Dry milk products Milk powder 4. Eggs 4.1 Raw egg Whole egg 4.2 Egg products Manufactured egg 4.3 Dried products Dried whole eggs 5. Vegetable products 5.1 Raw vegetables Sprouts 5.2 Dried products Spices 5.4 Fatty products Chocolate, mayonnaisesalads 7. Environment tests 7.1 Environment tests Swab tests 8. Feed 8.1 Animal feed Meat bone meal, fish meal, fish food 9. Animal faeces 10. Miscellaneous NORDVALnlm

131 Matrix for Listeria (NORDVAL) Matrix group Matrix Food Examples 1. Meat 1.1 Raw red meat Minced meat, (tatar type) 1.3 Raw smoked salted Bacon, smoked filet meat-products 1.4 Heat treated products Sliced meat and poultry products 1.5 Fermented products Salami 2. Fish 2.1 Raw fish, shelfish and Cold smoked salmon Fish products 2.3 Heat treated fish products Heat treated shrimps 3. Milk 3.1 Milk Raw milk Firm cheese Yellow cheese Soft cheese Mould cheese 3.5 Desserts, ice-cream Ice-cream 4. Eggs 4.1 Raw egg Whole egg 5. Vegetable products 5.1 Raw vegetables Cut salads, sprouts 7. Environment tests 7.1 Environment tests Swab tests, Cleaning water 10. Miscellaneous

132 Matrix for Campylobacter (NORDVAL) Matrix group Matrix Food Examples 1. Meat 1.1 Raw red meat Minced meat, offal 1.2 Raw white meat Chicken, turkey, duck 1.3 Raw smoked salted products Sliced smoked turkey meat 1.4 Heat treated products Sliced poultry meat 2. Fish 2.1 Raw fish and shelfish Raw two-shelled mollusc, raw shrimps 3. Milk 3.1 Milk Raw milk 4. Eggs 4.1 Raw egg Whole egg 4.2 Egg products Manufactured eggs 5. Vegetable products 5.1 Raw vegetables Sprouts 7. Environment tests 7.1 Environment tests Swab tests 9. Animal faeces 10. Miscellaneous Matrix for E. coli O 157 (NORDVAL) Matrix group Matrix Food Examples 1. Meat 1.1 Raw red meat Minced meat, cut meat, offal 1.3 Raw smoked salted Bacon, smoked filet meat-products 1.4 Heat treated products, ready to eat smoked products Sliced meat and poultry products, smoked turkey filet 1.5 Fermented products Salami 3. Milk 3.1 Milk Raw milk 3.2 Sour milk products Yoghurt with fruit 3.4 Cheese Mould cheese 3.5 Desserts/ice-cream Ice cream 5. Vegetable products 5.1 Raw vegetables Cut salads, sprouts 5.4 Fatty products Mayonaise-salads 7. Environment tests 7.1 Environment tests Swab tests 9. Animal feces 10. Miscellaneous