Assessment Criteria and Approaches for Rapid Detection Methods To Be Used in the Food Industry

Transcription

1 670 Journal of Food Protection, Vol. 77, No. 4, 2014, Pages doi: / x.jfp Copyright G, International Association for Food Protection Review Assessment Criteria and Approaches for Rapid Detection Methods To Be Used in the Food Industry MARTIN WIEDMANN, 1 * SIYUN WANG, 1,2 LAURIE POST, 3 AND KENDRA NIGHTINGALE 4 1 Department of Food Science, College of Agriculture and Life Sciences, Cornell University, Ithaca, New York 14853, USA; 2 Food, Nutrition and Health, Faculty of Land and Food Systems, University of British Columbia, Vancouver, British Columbia, Canada V6T 1Z4; 3 Mars, Inc., McLean, Virginia 22101, USA; and 4 Department of Animal and Food Sciences, Texas Tech University, Lubbock, Texas 79409, USA MS : Received 5 April 2013/Accepted 18 November 2013 ABSTRACT The number of commercially available kits and methods for rapid detection of foodborne pathogens continues to increase at a considerable pace, and the diversity of methods and assay formats is reaching a point where it is very difficult even for experts to weigh the advantages and disadvantages of different methods and to decide which methods to choose for a certain testing need. Although a number of documents outline quantitative criteria that can be used to evaluate different detection methods (e.g., exclusivity and inclusivity), a diversity of criteria is typically used by industry to select specific methods that are used for pathogen detection. This article is intended to provide an overall outline of criteria that the food industry can use to evaluate new rapid detection methods, with a specific focus on nucleic acid based detection methods. Rapid detection methods for foodborne pathogens and spoilage organisms have been widely adopted by the food industry. The first phase of development and implementation of rapid methods in the food industry focused on antibody-based approaches (e.g., enzyme-linked immunosorbent assays and lateral flow immunoassays) facilitated by rapid development of antibody-based technologies, including monoclonal antibody techniques, in the 1970s and 1980s. Although hybridization-based assays had been developed for detection of foodborne pathogens before the advent of PCR techniques, the invention and development of PCR and other amplification techniques (including isothermal amplification approaches) have paved the road for more widespread application of detection methods that target nucleic acids (typically DNA). Although this article focuses on nucleic acid based detection approaches, we briefly cover key issues to be addressed when considering various approaches (e.g., antibody based or nucleic acid based) for detection of foodborne pathogens. ANTIBODY-BASED ASSAYS Antibody-based commercial detection systems are still widely used in the food industry. These systems can be highly automated, allowing for high-throughput testing, or can be easy to use with limited instrumentation (e.g., lateral flow assays). The quality of antibodies is key to a highly specific and sensitive antibody-based detection system. However, this requirement represents a challenge because it is often difficult to identify an antigen that has key * Author for correspondence. Tel: ; Fax: ; mw16@cornell.edu. characteristics required for a highly specific antibody-based detection system. In a perfect world, the target antigen for an antibody-based assay needs to be (i) present on the target organism (e.g., Listeria monocytogenes) but not on any other organisms (in particular, closely related nonpathogens) and (ii) reliably expressed (ideally at high concentrations) under various culture conditions. Most antigens that would be highly specific to a pathogen facilitate host-pathogen interactions and thus are often expressed reliably only in the host (or at a specific site in the host, e.g., the intestine or inside a host cell). However, most antigens that are reliably expressed in culture medium (including in the presence of various foods) are also found in closely related nonpathogens. Typically, the structure and amino acid sequence of these antigens differ enough between the target pathogens and a closely related nontarget organism to allow for the development of antibodies that react significantly more strongly with the antigen in the target organisms than with a similar antigen found in a nontarget organism. Hence, these antibodies typically show some cross-reactivity with closely related organisms. Even antibodies that target a truly pathogen-specific protein (rather than a pathogen-specific allelic variant) will likely have at least some cross-reactivity with some other proteins. Consequently, in many antibodybased detection methods, the signal observed with a low level of the target organism (i.e., levels close to the lower limit of detection) can be similar to the signal observed with an extremely high level of a closely related nontarget organism. For example, some L. monocytogenes specific antibody-based assays may yield false-positive results when high levels of Listeria innocua are present (53). Users of antibody-based detection systems must be aware of these

2 J. Food Prot., Vol. 77, No. 4 RAPID DETECTION METHODS EVALUATION 671 issues and should have plans for appropriate secondary tests that would determine whether a positive assay result is falsely positive due to the presence of high levels of closely related organisms. Good comprehensive reviews of antibody-based detection systems in general (42) and the use of antibody-based detection systems in the food industry (9) are available. NUCLEIC ACID BASED ASSAYS Nucleic acid based detection methods can be differentiated based on the detection approach used: (i) hybridization-based assays, (ii) PCR-based assays, (iii) other thermocycling-based assays (e.g., ligase chain reaction), and (iv) isothermal amplification methods. These classifications are useful as a conceptual framework. However, considerable variation exists within any given category, and some PCR assays use PCR for target amplification and hybridization-based probes for target detection. PCR-based assays are often classified into real time (RT) and non-rt PCR assays. RT PCR assays allow for review of the PCR amplification during the PCR run(which typically involves around 40 thermocycles) and thus allow classification of samples as positive even before the assay has run for the full number of cycles. RT PCR assays typically use probes with fluorescent labels for detection of PCR products or for monitoring target amplification. In contrast, endpoint PCR assays require completion of the full number of cycles before a PCR result can be classified as positive or negative. These assays often rely on an intercalating dye such as SYBR Green for nonspecific detection of PCR products and thus require a melting curve after completion of all PCR cycles to identify whether the PCR product has the expected size. In some commercial assays, the melting curve also is required to differentiate between the target amplicon and the internal positive control amplicon, which typically is larger than the target amplicon. Although the vast majority of commercially available detection systems are PCR based, some hybridization-based systems for detection of foodborne pathogens also are available commercially. Isothermal detection methods have been used for at least 5 to 10 years in clinical applications (19), and new isothermal systems for detection of foodborne pathogens have recently (i.e., 2011 to 2012) been introduced (e.g., the 3M Molecular Detection System, 3M, St. Paul, MN; the ATLAS system, Roka Bioscience, Warren, NJ; and the ANSR system, Neogen, Lansing, MI). A key advantage of isothermal methods is that they do not require thermocycling equipment, including the associated need for regular temperature calibration of equipment. One of the potential challenges associated with isothermal methods is that they do not have the same flexibility as PCR-based methods, which allow for adjustment of primer and/or probe hybridization temperatures to assure highly specific binding to the target molecule. Evaluation of isothermal assays thus requires critical attention to the level of specificity. Typically, advanced DNA probe design is used to provide high specificity with these assays. Nucleic acid based detection methods also can be differentiated by the target molecule, including DNA, rrna, and mrna. Most commercial assays utilize DNA as a target molecule, but some assays target rrna. The advantage of using rrna as a target is that a bacterial cell typically contains thousands of copies of each rrna molecule (5S, 16S, or 23S); around 3,000 rrna copies per cell appears to be typical for bacteria growing with typical doubling times of around 30 min. In contrast, a bacterial cell typically contains between 1 and 10 copies of its chromosome. Although many researchers may assume that a bacterial cell contains only a single copy of its chromosome, bacterial cells can contain multiple copies of their chromosome because they often replicate their chromosome more rapidly than they divide. The fact that thousands of rrna molecules usually are found in one bacterial cell means that any detection method that targets this multiple copy molecule typically has at least a 1,000- fold higher analytical sensitivity (in CFUs) than does an assay that targets DNA. This difference has been utilized for example in the BAX system reverse transcriptase PCR assay for Listeria species, which targets rrna and thus does not require an enrichment step prior to the PCR but does require a 4-h resuscitation step. In addition, PCR assays can target the rrna gene (sometimes refer to as rdna), which is a DNA target. The differentiation between assays that target the rdna and assays that target rrna (i.e., the RNA that is created through transcription of the rdna) can be confusing but is important for the end user because assays targeting rrna and rdna are both available. Assays targeting rrna are more sensitive but, because of the shortened enrichment time and high sensitivity, also have a higher risk of falsepositive results due to the presence of dead bacterial cells (rrna is fairly stable and degrades slowly after the death of the bacterial cell). A common challenge with assays targeting both rrna and rdna is that the sequence of these target molecules can be highly conserved among organisms. For example, the approximately 1,500-nucleotide sequence of the 16S rrna gene in L. monocytogenes and that in L. innocua differ by only 1 or 2 nucleotides, making differentiation between these two organisms based on 16S rdna or rrna very challenging (50); this is one reason why very few 16S rrna- or rdna-based assays provide specific detection of L. monocytogenes. With the advancement of PCR-based methods, a number of multiplex PCR assays have been developed for the detection of foodborne pathogens. Multiplex assays allow for the detection of multiple gene targets in a single reaction. Some multiplex assays also may detect multiple targets in different reactions that are run in parallel; however, these assays may not be considered truly multiplex assays. PCR assays that include an internal control (which typically can be considered a second gene target) should technically also be considered multiplex assays because these assays detect two distinct genes targets (i.e., the pathogen-specific target gene and the internal control target). Multiplex assays typically are used either to detect multiple target species, serotypes, or subtypes in a single reaction (e.g., Salmonella, Escherichia coli O157:H7, and L. monocytogenes (23, 25, 43)) or to detect multiple target

3 672 WIEDMANN ET AL. J. Food Prot., Vol. 77, No. 4 genes, which allows for specific detection of a certain target pathogen. For example, most assays used to specifically detect E. coli O157:H7 target multiple genes, e.g., stx 1, stx 2, rfbe, eaea, hlya, and flic (10, 13). The practical value of multiplex assays that detect multiple pathogens is probably rather limited, particularly for testing foods and environmental samples. Even when an assay can detect multiple target pathogens in a single tube, most likely different enrichments will still be needed to allow for appropriate recovery and growth of the target pathogens. Hence, multiple enrichments would have to be pooled into a single PCR, which is not practical (e.g., because enrichment times typically differ among target organisms). Although some socalled universal enrichments have been described (1, 16, 24, 52), the performance of these enrichment methods typically does not match that of specific enrichment media and protocols that have been optimized for a single target pathogen. For example, in one study (52), the universal preenrichment broth used lacked inhibitory agents to provide selectivity for target pathogens and thus may not be suitable for samples with high levels of background microflora. In another study (24), a selective enrichment broth promoted better growth of Salmonella enterica, E. coli O157:H7, and L. monocytogenes and inhibited greater numbers of nontarget organisms, but the performance of this enrichment broth was not evaluated on naturally or artificially contaminated food samples. Samples collected by industry often are tested for only a specific target pathogen, and detection of additional pathogens is undesirable for various reasons. However, multiplex assays are useful for appropriate detection of some target pathogens such as non-o157 Shiga toxin producing E. coli (STEC) serotypes of concern (sometimes referred to as the big six serotypes (2)). A challenge with the use of multiplex PCR assays to detect a specific target organism based on multiple gene targets is that application of these assays to screen enrichment media for the target organism does not allow determination of whether positive PCR results represent gene targets in a single organism or multiple distinct organisms carrying these genes. This issue with multiplex assays is particularly important for detection of non-o157 STEC. Culture confirmation of samples that are positive by a multiplex PCR screen often requires considerable additional investment (in reagents and time) because confirmation may involve immunomagnetic separation, plating on selective and differential media (with and without an acidification step to reduce background microflora), and confirmation of suspect colonies by PCR and biochemical tests (46). The ability of nucleic acid detection platforms to support multiplexing may differ considerably. Some platforms allow detection of only a single target gene per tube, including systems that allow for detection of only a single target gene (without inclusion of an internal control) and systems that allow for detection of a single target gene plus an internal positive control. However, some PCR platforms may allow detection of six or more different targets in a single tube. More complex DNA-based detection systems (e.g., microarrays) are typically needed if more than six targets are to be detected simultaneously. An important issue with multiplex PCR assays is that they typically reduce the analytical sensitivity below that achieved when a single primer set is used (even inclusion of an internal control can reduce analytical sensitivity in some cases). Hence, stringent evaluation of the analytical sensitivity of multiplex PCR assays is critical. NEW DEVELOPMENTS In addition to antibody-based and nucleic acid based detection systems, novel approaches for detection of foodborne pathogens are frequently reported in the popular and specialty press, on the Internet, and in peer-reviewed publications. This section includes a brief discussion of some novel approaches for detection of foodborne pathogens and how these new methods can be evaluated by end users. Phage-based assays. Phages are viruses that infect bacteria. Phages have a potential for wide applications in biotechnology, including development of new detection methods (5). Phages can be used to display various ligands and through subsequent rounds of selection and counterselection may allow for identification (and subsequent production) of highly specific and high-affinity ligands that can be used in rapid detection methods in applications similar to those of antibodies (30). Ligands from naturally occurring phages also can be used to develop new ligand molecules for diagnostic applications. Phages also are being used to develop detection methods that incorporate complete phages, rather than just ligands that were developed using phages (14, 17). In these applications, phages are typically modified to carry a gene that encodes a protein that allows for rapid or easy detection of host infection by these modified phages. Examples of genes that have been used for these purposes are genes that encode luciferase (hence the phage infection event will create luminescence) or a gene that encodes an ice nucleation protein, which was used in the bacterial ice nucleation detection assay (20). Phage-based detection systems may involve a single phage or a cocktail of phages; such a cocktail is often necessary to detect a wide range of target organisms (e.g., all Salmonella serotypes). Because the targets for phage-based assays are often surface molecules that may not have been selected rationally (e.g., because they are known to be linked with a given species or because they are known to be responsible for virulence) but rather are molecules that are naturally targeted by phages, validation of these assays and particularly inclusivity testing may require use of a larger strain set. Mass spectrometry based detection methods. Mass spectrometry (MS) based methods also are increasingly being developed for rapid detection of foodborne pathogens. Conceptually, most applications of MS for the detection of foodborne pathogens fall into two categories: (i) rapid detection and characterization of proteins or other molecules that are specific for a given target organism (see Fox (12) and Sauer and Kliem (39) for reviews) and (ii) rapid

4 J. Food Prot., Vol. 77, No. 4 RAPID DETECTION METHODS EVALUATION 673 identification and characterization of PCR products. Although detection of target-specific proteins or other molecules by MS would provide the advantage of allowing direct detection and characterization (e.g., from a bacterial colony), one major challenge with this approach is that the variability in expression patterns of the target molecules (e.g., due to differences in growth phase, media composition, temperatures, oxygen tension, etc.) will complicate reliable identification. This issue must be addressed with any method that would utilize this approach. However, PCR followed by MS to identify PCR products has considerable potential for rapid detection of various target organisms (34, 35, 38) with some promising commercial applications currently available (e.g., the Abbott ID-Plex system (34)). These systems have considerable multiplexing capabilities and can be an intermediate solution between traditional PCR (limited at best to four or five targets per reaction) and full genome sequencing (which is usually more complex due to DNA preparation required, back-end analysis needs, etc.). The rapidly increasing amount of full genome sequencing data for foodborne pathogens will provide considerable opportunity for identification of targets for these types of MS-based detection systems. A potential challenge with highly multiplexed PCR-MS detection systems may be that detection of a rare target in a sample with a high level of background of other organisms may be challenging. Experimental data validating the ability of these methods to detect low levels of targets in samples with high levels of background flora (particularly when the background organisms are targeted by some of the primers in the multiplex assay) may be of importance to end users that are interested in these types of assays. Integration of detection and subtyping methods. Subtyping of foodborne pathogens (and spoilage organisms) can provide important data that can help identify contamination sources and support foodborne disease outbreak investigations (49). Traditionally,detectionand subtyping have been distinct steps, which required isolation of pure bacterial cultures for subtyping. Integration of DNA amplification with subsequent DNA sequence analysis (as implemented in the Abbott PLEX-ID) can allow detection and subtyping without the need for isolation of the target organism. PCR amplification with subsequent microarray analysis can also be used to detect and subtype target organisms without the need for isolation of the target organism. More commercial assays integrating detection and subtyping probably will be developed. With the rapid development of next-generation sequencing methods (15, 27), assays that are based on these technologies and provide both detection and subtyping are likely to become a reality soon, even though commercialization may still take a while. Evaluation of these assays must include an assessment of their ability both to detect organisms and to provide sensitive strain discrimination. For many of these systems, a particular challenge may be the ability to detect multiple strains or subtypes of the same target species in a single sample and in particular the ability to also detect a strain that may be present in low numbers relative to a second strain of the same target species that is present in high numbers. EVALUATION OF NUCLEIC ACID BASED DETECTION SYSTEMS This section will provide some guidance to end users on how to evaluate nucleic acid based assays for detection of foodborne pathogens. These guidelines could be used in an informal process or a formal process to identify current technologies that may be appropriate for more in-depth evaluation. These guidelines should not be presented or used as rules. End users differ in their needs, and these needs will play a critical role in any evaluation of a detection system. The information provide here should not conflict with existing evaluations and validations (e.g., by AOAC International or the Association Française de Normalisation [AFNOR]), and the criteria described should not be used instead of a validation but rather as guidance for how to select a detection system using a wide range of data and criteria, including information from validation studies. Although the evaluation criteria described (Tables 1 and 2) are intended to be comprehensive, some users may want or need additional criteria and/or may not use some of the criteria detailed here in their evaluations. The six-point scoring system detailed in Tables 1 and 2 allows individuals or organizations to quantitatively evaluate assays and calculate overall scores for each; the scoring also could be converted into simpler scales such as a three-point scale with scores of unacceptable, acceptable, and outstanding. Conceptually, evaluation of a new assay system may involve both quantitative analytical data (e.g., inclusivity, exclusivity, specificity, and sensitivity) and logical reasoning and deduction to determine whether a given assay is likely to perform as needed. For example, knowledge that a given gene target used is conserved in nontarget organisms would indicate that a given assay is likely to have poor exclusivity. The guidance provided here should allow end users to integrate both approaches to choose an appropriate detection system for a given application. Because standard quantitative criteria such as exclusivity and inclusivity have been discussed previously (e.g., Feldsine et al. (11), the U.S. Department of Agriculture, Food Safety and Inspection Service (45), and the OIE (33)), a detailed discussion of these criteria will not be included here. The evaluation criteria detailed here are categorized into two types: (i) assay-specific criteria, which should be used to evaluate assays for various pathogens (e.g., Salmonella and L. monocytogenes) even when they are part of the same line of assays produced and distributed by a given company and (ii) product line specific criteria, which will apply to a specific back-end detection system manufactured by a given supplier (back-end detection system is defined here as the final system that analyzes samples after enrichment). Criteria to evaluate pathogen-specific assays: inclusivity. Inclusivity (also sometimes referred to as analytical specificity) is defined as the ability to detect

5 674 WIEDMANN ET AL. J. Food Prot., Vol. 77, No. 4 TABLE 1. Pathogen-specific evaluation criteria for rapid molecular assays to be validated for routine use a Key criteria for evaluation Target Key factors affecting performance Quantitative evaluation (scale of 0 5) b Inclusivity: ability to detect different strains and/or subtypes of the target organism, also sometimes referred to as analytical specificity Exclusivity: ability to yield negative results with nontarget organisms Diagnostic sensitivity (sensitivity) Should be validated to detect various strains and/or subtypes of the target organism Should be validated to not detect nontarget organisms Should be validated with naturally contaminated samples to function with high sensitivity (e.g.,.98%) and few false-negative results; validation on spiked samples can be misleading (e.g., organisms sometimes grow faster when spiked) Target gene must be highly conserved and present in all strains of the target species or serotype; target gene should be required for virulence so that strains that lack the target gene are unlikely to cause disease (bona fide virulence genes may not have been identified for all target pathogens) Target gene must be absent from nontarget organisms; if target gene is present in other organisms, gene must be highly divergent from variants present in target organism and primers must target gene regions specific to target organism False-negative results could be caused by (i) slow growth in enrichment culture with target organism not yielding appropriate levels for positive result, (ii) target organism outcompeted by nontarget organisms during enrichment, (iii) organisms that cause false-positive results with the gold standard method 0 tested on a small no. of standard laboratory or collection strains and/or no strong scientific support for choice of target gene 3 tested on a diverse strain collection, but diversity does not represent a variety of geographical regions or food matrices 5 (i) tested on isolates from different sources (human, food, environmental), different countries and continents, and a wide variety of subtypes and/ or (ii) strong scientific support that target is unique to target organism and linked to virulence 0 tested on a small no. of nontarget organisms and/or against only nontargets highly divergent from target organism 3 tested on a diverse strain collection (.30 isolates) but diversity set does not include some key organism likely to cross-react and/or does not represent a variety of geographical regions or food matrices 5 (i) tested on a large no. of closely related nontarget organisms, ideally with organisms isolated from target foods and/or (ii) strong scientific support that target is absent from nontarget organisms or would confer virulence if present in nontarget organisms 0 no data on false-negative rate for naturally contaminated samples or false negative rate above level determined to be acceptable by end user 3 data on diagnostic sensitivity available for some matrices of interest and false-negative rate at level determined to be acceptable by end user 5 data on diagnostic sensitivity available for all matrices of interest and false-negative rate below level determined to be acceptable by end user

6 J. Food Prot., Vol. 77, No. 4 RAPID DETECTION METHODS EVALUATION 675 TABLE 1. Continued Key criteria for evaluation Target Key factors affecting performance Quantitative evaluation (scale of 0 5) b Diagnostic specificity (specificity) Analytical sensitivity (detection limit) Reproducibility: ability to perform reproducibly in different laboratories and with different personnel Repeatability: ability to produce the same results in the same laboratory with the same equipment and personnel Should be validated with naturally contaminated samples to function with high specificity (e.g.,.98%) and few false-positive results; validation on spiked samples can be misleading Detection limit can be reported as limit in the food matrix before enrichment (e.g., 1 CFU/25 g) or limit for the molecular assay (i.e., CFU/ml required for a positive result) Results should be highly reproducible; this should be validated with spiked samples because it is virtually impossible to reliably obtain naturally contaminated replicate samples Results should be highly repeatable; this should be validated with spiked samples because it is virtually impossible to reliably obtain naturally contaminated replicate samples False-positive results could be caused by (i) detection of nonviable organisms, (ii) nontarget organisms carrying the target gene(s), (iii) cross-contamination, (iv) false-negative results with the gold standard method Analytical sensitivity can be highly affected by matrix, physiological state of the target organism, growth of the target organism in enrichment culture, and the presence of competing microflora Poor reproducibility could be the result of (i) technically difficult assay (leading to technical errors by personnel, e.g., crosscontamination), (ii) reagents not standardized sufficiently, (iii) equipment not performing reproducibly, (iv) poorly optimized detection system, (v) sensitivity of equipment or assay system to environmental factors (e.g., humidity, temperature) Poor repeatability could be the result of (i) technically difficult assay (leading to technical errors by personnel, e.g., crosscontamination, (ii) reagents not standardized sufficiently, (iii) equipment that does not perform repeatably 0 no data on false-positive rate for naturally contaminated samples or false-positive rate above level determined to be acceptable by end user 3 data on diagnostic specificity available for some matrices of interest and false-positive rate at level determined to be acceptable by end user 5 data on diagnostic specificity available for all matrices of interest and false-positive rate below level determined to be acceptable by end user 1 no data on analytical sensitivity available except for theoretical calculation assuming that a single cell can grow in an enrichment culture 3 experimental data indicate sensitivity below that for traditional culture (e.g.,,10 CFU/ml of enrichment culture) for some matrices of interest to end user 5 experimental data indicate sensitivity below that for traditional culture (e.g.,,10 CFU/ml of enrichment culture) for all matrices of interest to end user 0 extremely poor reproducibility (e.g.,,80%) 1 no evaluation of reproducibility performed 3 some limited data indicate,99% reproducibility for selected assays or evaluation indicates issues with reproducibility (e.g., reproducibility between 90 and 99%) 5.99% reproducibility based on an evaluation by at least five laboratories 0 no evaluation of repeatability performed 3 repeatability evaluated with small no. of samples (,100) and/or limited no. of matrices and/or performed with samples spiked with large inoculum (e.g.,.100 CFU/g) 5.99% repeatability based on evaluation with $200 samples spiked at low levels and food matrices relevant to user

7 676 WIEDMANN ET AL. J. Food Prot., Vol. 77, No. 4 TABLE 1. Continued Key criteria for evaluation Target Key factors affecting performance Quantitative evaluation (scale of 0 5) b Compatibility with existing and standard enrichment media Availability of positive control strains for enrichment cultures Certification Back-end detection methodology should be compatible with currently used and/or standard enrichment media Should have a specific positive control strain available, which can be easily detected and differentiated from wild-type strains to allow for rapid identification of contamination with the positive control strain Preferably certified by ISO, AFNOR, or AOAC International Some back-end assays may not be compatible with currently used and/or standard enrichment media, e.g., carryover of certain compounds in the enrichment culture interferes with the assay Uniquely marked control strains are desirable but not typically available for many assays User must set minimum requirement and specify the organization from which they require certification 0 no data on compatibility of assay with standard enrichment media available or back-end assay not compatible with enrichment media that must be used by end user 3 assay is compatible with enrichment media that must be used by end user although with some limited negative effects on performance 5 assay is compatible with all enrichment media with no negative effects on assay performance 1 no uniquely marked control strains available 3 uniquely marked control strains available for some or all targets, but differentiation of wild-type and control strains requires specialized tools or assays 5 uniquely marked control strains available for all targets and a specific assay to detect and differentiate control strains from wild-type strains is available 0 not certified by user-required entity c 1 not certified, but no certification required by user 2 certification with limited requirements by one organization (e.g., AOAC Research Institute) 3 certification with stringent requirements by one organization 5 certification with stringent requirements by more than one organization a Pathogen-specific criteria typically should be evaluated separately for each assay. b Guidance for characteristics associated with selected low, medium, and high scores; full range of scores (0 to 5) should be used for each criterion. A score of 0 typically implies that a given test does not meet minimum requirements; for many criteria, minimum requirements are user specified. For some criteria a score of 0 is not available when assays not meeting minimum criteria are still appropriate for use. Scores can be assigned by multiple people or through informal or formal approaches to achieve a consensus (e.g., the Delphi method). c When a user requires certification by a given organization, an assay will be assigned a score of 0 even when the assay has been certified by a different organization. different strains of the target organism. Mechanistically, a nucleic acid based assay s inclusivity depends on the gene targeted by the assay. When a gene is highly conserved and found in all strains of a given target organism (e.g., L. monocytogenes), then the assay should have close to 100% inclusivity. Statistically, inclusivity is defined as the proportion of tested isolates of the target organism that is detected with the assay. For example, if 100 Salmonella isolates were tested with a given assay and 99 of these isolates were identified as Salmonella, then the assay would have an inclusivity of 99%. The AOAC International OMA Manual Appendix X (11) requires the use of at least 50 pure strains of the specific microorganism being studied for inclusivity testing. Although 50 strains may be sufficient for some pathogens, given that an appropriately diverse strain set is used, larger

8 J. Food Prot., Vol. 77, No. 4 RAPID DETECTION METHODS EVALUATION 677 strain sets are likely needed for inclusivity testing for some assays, e.g., those targeting Salmonella. Consequently, AOAC International (11) recommends that the number of strains for inclusivity testing of Salmonella be increased to at least 100 strains that are selected to represent the majority of known serovars of Salmonella. Inclusivity data for a given assay must be judged based on both the percent inclusivity reported and the strain set used for inclusivity testing. In practice, it is fairly easy to assemble a strain set that is likely to yield 100% (or close to 100%) inclusivity, for example by selecting well-characterized strains or serotypes that are known a priori to be easily detectable by a given assay (e.g., because gene or genome sequence data are available to indicate that a pair of primers will amplify a group of strains). However, it is also possible in many cases to include strains so that the percent inclusivity is,100%. For example, strains with unusual phenotypes or genotypes have been described for many pathogens. Some of these strains either are missing genes that are used as targets for DNA-based assays or are carrying variants of the target genes that are so divergent that they typically yield negative results. For example, most assays that detect STEC will include primers that detect stx 1 and stx 2 genes (in addition to primers detecting other genes). However, a rare variant of the stx 2 gene (stx 2f ) has a sequence so divergent from the other stx 2 gene variants that it is typically not detected by stx 2 assays. Hence, an testing set that includes strains that carry stx 2f will typically yield inclusivities of,100%. As part of the selection of isolate sets for inclusivity testing, the end user should consider the target organism diversity expected. For example, a food company that evaluates a Salmonella assay for adoption by units located throughout the world, including Asia, may require use of a strain set for inclusivity testing that includes serotypes that are common in Asian countries but not in the United States, such as Salmonella serotypes Virchow and Stanley (51). Criteria to evaluate pathogen-specific assays: exclusivity. Exclusivity (sometimes also referred to as specificity) was defined by Feldsine et al. (11) as the lack of interference... from a relevant range of non-target strains, which are potentially cross-reactive. Statistically, exclusivity is defined as the percentage of nontarget strains that give a negative result with a given assay. AOAC International (11) recommends that at least 30 strains representing potentially competitive organisms be tested as pure culture preparations. Similar to strain selection for inclusivity testing, selection of strains for exclusivity testing will play an important role in how exclusivity testing data are interpreted. For experts with a knowledge of a given assay and the biology of the target organism, it is in most cases relatively easy to select 30 strains that will yield 100% exclusivity. However, some strains may cross-react in a given assay, yielding an exclusivity of,100%. For example, many molecular assays for L. monocytogenes use the gene hly as a target; these assays may cross-react (and yield false-positive results) with hemolytic L. innocua strains, which are avirulent but carry a hly gene that is highly homologous to the L. monocytogenes hly gene (4, 21). Inclusion of hemolytic L. innocua strains in an exclusivity panel for an assay that targets the hly gene thus will likely yield false-positive results. Another issue surrounding exclusivity testing concerns the number of bacterial cells used in the exclusivity testing. Often inclusivity and exclusivity testing will use pure cultures as input into the assay (without prior incubation in an enrichment medium), but some inclusivity and exclusivity tests may start from inoculation of the enrichment medium with pure cultures. Typically, a standard number of bacterial cells (e.g., 1,000 to 10,000 CFU per assay for PCR-based methods) is used as input into an assay for both the inclusivity and exclusivity panels. Although many assays are able to yield correct positive and negative results with target and nontarget strains when both types are present at the same level, some assays (in particular some antibody-based assays but also some 16S rrna- or rdna-based assays) may perform poorly when challenged with a large number of bacterial cells from nontarget strains. Hence, a practically relevant design for exclusivity testing should use low, medium, and extremely high levels for the nontarget bacterial strains and determine whether a given assay can differentiate low levels of the target strains from high levels of the nontarget strains. For example, for a PCR assay the end user may request data that (i) use strains in the inclusivity panel at a low level (e.g., 10 CFU) as an assay input (meaning the PCR assay receives 10 CFU as an input) and (ii) use strains in the exclusivity panel at low, medium, and high levels (e.g., 10, 10 3, and 10 6 CFU) as an assay input. For immunoassays, which typically require much higher numbers of cells as inputs into the assay, these tests will require much higher bacterial levels but should still include the use of 100- to 10,000-fold higher levels of the nontarget organism. Criteria to evaluate pathogen-specific assays: diagnostic sensitivity. Diagnostic sensitivity, also sometimes referred to as the sensitivity rate (11) or just sensitivity (45), is a measure of the probability that a test will correctly classify a positive test sample as positive; guidance on how to calculate sensitivity has been previously published (11, 45). Diagnostic sensitivity is sometimes confused with inclusivity or even, incorrectly, used interchangeably with inclusivity. The confusion between diagnostic sensitivity and inclusivity probably occurs because these two criteria are related; an assay with a low inclusivity likely will also yield a considerable number of false-negative results. As a hypothetical example, a Salmonella assay that has low inclusivity (e.g., detects only 80% of all Salmonella serotypes) would also typically have a low sensitivity rate when tested with naturally contaminated samples because it will yield negative results with all samples that contain one of the Salmonella serotypes this assay does not detect. However, an assay that has a high level of inclusivity (e.g., 98%) can still have low diagnostic sensitivity, for example when an assay is easily inhibited or does not detect low levels of the target organism (e.g., one to five cells per 25-g sample) that are detected by the gold standard method. Although diagnostic sensitivity can be determined with spiked samples, diagnostic sensitivity data are more mean-

9 678 WIEDMANN ET AL. J. Food Prot., Vol. 77, No. 4 ingful when generated using naturally contaminated samples. One key challenge for determination of diagnostic sensitivity is that it may differ for different matrices in a given assay; hence, a given assay typically will not have a single diagnostic sensitivity value, but rather end users may need to generate diagnostic sensitivity values for various matrices, such as raw meat, raw eggs, cereal, and environmental samples. Considerations when determining diagnostic sensitivity of a given assay using spiked samples include the choice of challenge strain(s), the choice of product to be spiked, the spiking level, and the physiological state of the target cells spiked into a given sample. For example, use of a target strain that is easily detected by the method to be tested and that grows rapidly and easily in the enrichment medium chosen will facilitate high diagnostic sensitivity. In contrast, use of a strain that is poorly detected by a given assay (e.g., due to mismatches between the primers used in an assay and the target gene in a given strain) or a strain that grows poorly in the enrichment medium used (or at the enrichment temperature recommended) will typically result in low diagnostic sensitivity. Similarly, inoculation with injured cells or the use of matrices that contain high levels of competitive microflora (in particular organisms that may outcompete the target organism during enrichment) is likely produce lower diagnostic sensitivity than is a scenario where the food matrix is inoculated with an actively growing strain and contains limited or no competitive microflora. Many challenges also are associated with determining the diagnostic sensitivity of an assay with naturally contaminated samples, including the fact that typically it is difficult to gain access to a sufficient number of samples naturally contaminated with the target organism. Some target organism matrix combinations may be more challenging than others; for example, ground poultry samples positive for Salmonella may be easy to find, but environmental samples naturally contaminated with Salmonella may be more difficult to find in dry processing plant environments. Availability of appropriate naturally contaminated samples often will be a deciding factor for determining whether the diagnostic sensitivity of a specific matrix target organism combination should be determined using spiked or naturally contaminated samples. Another challenge for determining the diagnostic sensitivity using naturally contaminated samples is that a new method must be compared with a gold standard method, which is often a traditional standard method (e.g., from the U.S. Food and Drug Administration Bacteriological Analytical Manual or the International Organization for Standardization [ISO] in Geneva). Although this comparison is more often a challenge for determining the diagnostic specificity of an assay, it can also present a challenge for determining the diagnostic sensitivity. For example, an imperfect gold standard method can be problematic when it produces falsepositive results, because by definition a gold standard method provides the correct results and hence a falsepositive result obtained with the gold standard method would be interpreted as a false-negative result with the new method being evaluated. For most gold standard methods for detection of foodborne pathogens, false-positive results are rare and can be detected by further characterization of bacterial isolates that are obtained with the gold standard method from a sample that is negative with the new assay. For example, the gold standard method for the detection of L. monocytogenes will typically yield a positive result for samples that contain hemolytic L. innocua, but these samples should yield negative results with appropriately designed molecular methods. The discrepancy would typically be resolved when the isolate in question is further characterized, which would identify the hemolytic Listeria isolates as L. innocua and hence indicate that the result obtained with the gold standard method was falsely positive. In summary, interpretation of diagnostic sensitivity data for a new assay must take into consideration the study design used to determine the diagnostic sensitivity. In general, likely reasons for low diagnostic sensitivity include (i) slow growth in the enrichment medium with the target organism thus not yielding appropriate levels for a positive result, (ii) the target organism being outcompeted by nontarget organisms during enrichment, (iii) organisms that cause false-positive results with the gold standard method, and (iv) an assay with low inclusivity. Full or partial inhibition of the back-end detection system by carryover of compounds from the sample and enrichment media could also contribute to low diagnostic sensitivity but should be detected by failures of the internal or matrix control to yield amplification. Criteria to evaluate pathogen-specific assays: diagnostic specificity. Diagnostic specificity, also sometimes referred to as the specificity rate (11) or specificity (45), is a measure of the probability that a test will correctly classify a negative test sample as negative; guidance on how to calculate specificity has been previously published (11, 45). Diagnostic specificity is sometimes confused with exclusivity or even, incorrectly, used interchangeably with exclusivity. The confusion between diagnostic specificity and exclusivity probably occurs because these two criteria are related; an assay with a low exclusivity likely will also yield a considerable number of false-positive results. As a hypothetical example, an E. coli O157:H7 assay that has a low exclusivity (e.g., yields positive results with a number of non-o157:h7 isolates) would also typically have low diagnostic specificity when tested with naturally contaminated samples because it will yield positive results with samples that contain non-o157:h7 strains that produce a positive reaction in this assay. However, an assay that has a high level of exclusivity when tested with a set of closely related nontarget organisms can still have a low level of diagnostic specificity, for example when the assay produces positive results for nontarget organisms that were not included in the exclusivity panel. Although a single study is often used to determine both the diagnostic sensitivity and the diagnostic specificity of a given new assay, some differences exist between diagnostic sensitivity and diagnostic specificity determinations. Obviously, testing of samples spiked with selected nontarget

10 J. Food Prot., Vol. 77, No. 4 RAPID DETECTION METHODS EVALUATION 679 organisms is not a viable approach for determining the diagnostic specificity of a new assay; rather, diagnostic specificity should be determined with naturally contaminated samples. Procurement of such samples for determining diagnostic specificity is less challenging than procurement of the same type of samples for diagnostic sensitivity testing; diagnostic specificity testing does not require samples that are contaminated with the target organism but rather relies on samples that are negative for the target organism. Selection of samples plays a critical role for determining the diagnostic specificity of a given assay; samples selected for testing should represent a variety of sources (e.g., different food matrices, raw materials from different geographical regions, and environmental samples from different plants) to assure a high likelihood of including samples that may contain organisms that crossreact and produce false-positive results in the assay being evaluated. Samples that contain high levels of closely related nontarget organisms may also be more likely to yield a false-positive result. In general, false-positive results can be caused by (i) detection of nonviable organisms, (ii) detection of nontarget organisms carrying the target gene, (iii) cross contamination, and (iv) false-negative results with the standard method. Multiplex PCR assays that detect multiple gene targets (e.g., assays that detect O157:H7 or non-o157 STEC) to identify a given organism can be a particular challenge with regard to false-positive results. Specifically, a multiplex PCR assay used to detect multiple genes associated with non-o157 STEC strains (e.g., stx 1, stx 2, and eaea) may yield a putative positive STEC result (e.g., presence of stx 2 and eaea) when used to screen an enrichment culture if these two genes are found in distinct isolates (e.g., a Shigella strain that carries stx 2 and an enteropathogenic E. coli strain that carries eaea but not stx 2 ) even though the enrichment culture contains no target organism that contains both genes. Although one may debate whether this type of result is truly a false-positive result for the assay (because both target genes were indeed found in the enrichment culture), this example illustrates how these types of multiplex assays can yield false-positive results for STEC. Limited data are available regarding whether this issue is practically relevant or largely theoretical (40, 44). False-negative results based on standard methods can be an important issue when determining the diagnostic specificity of a new assay because a false-negative result obtained with the gold standard method will be interpreted as a falsepositive with a new assay. For example, standard methods for L. monocytogenes detection may specify that 5 to 10 Listeria-like colonies on a selective medium must be characterized by additional tests to determine whether they are L. monocytogenes. Often, all isolated colonies tested as part of the standard procedure are confirmed to not be L. monocytogenes, while a PCR assay performed on the same enrichment culture may produce a positive L. monocytogenes result; this is typically the case when L. monocytogenes is present in low numbers in a mixed population with other Listeria species. Although some of these issues have been resolved with the inclusion, in many standard methods, of colorimetric media that allow differentiation of L. monocytogenes, similar issues can still arise. This is likely to be a particular problem with validation of molecular methods for the detection of E. coli O157:H7 and particularly non-o157 STEC, for which simple and sensitive standard methods are almost completely lacking. In general, the target organism can be identified by performing extensive follow-up work on samples that are positive with a new method and negative with the gold standard method. Criteria to evaluate pathogen-specific assays: analytical sensitivity. Analytical sensitivity is often also referred to as detection limit. The way analytical sensitivity of a given assay is reported can be inconsistent; it can be reported as the detection limit in the food matrix before enrichment (e.g., 1 CFU/25 g) or as the detection limit for the molecular assay (i.e., CFU per milliliter required after enrichment for a positive result in the assay). When the analytical sensitivity is reported as the detection limit in the food matrix before enrichment, the value has often not been experimentally validated but rather represents a theoretical detection limit (which assumes that a single cell present in the food sample can grow sufficiently during enrichment to yield a positive result). Validation of a detection limit for one target cell in the starting amount of sample (e.g., 25 g) would typically require incubating a large number of samples with an inoculum calculated to be 1 CFU and an evaluation of the fractional number of positive results. The physiological state of the target organism, the presence of competitive microflora, and the food matrix also can have a considerable impact on the ability of a single cell to grow to a level that can be reproducibly detected with a given backend detection system. The analytical sensitivity of the actual back-end detection system (e.g., a given PCR assay) is important because highly sensitive back-end detection systems will allow shorter enrichment times and hence more rapid detection times in food or environmental samples. The data provided for the sensitivity of the back-end detection system sometimes are reported as cells or particles (e.g., most PCR assays claim a sensitivity of a single cell) or as cells per milliliter of enrichment culture; in the latter case, the detection limit is determined by both the analytical sensitivity of the detection system and the amount of enrichment culture that is tested in a single assay (often PCR assays utilize the equivalent of only 1 to 10 ml of enrichment culture in a single reaction). Overall, end users will typically not be too concerned with the actual analytical sensitivity per se but with the overall assay time for a system that has a given analytical sensitivity. Knowledge of the analytical sensitivity of a system will allow the end user to assess the feasibility and plausibility of a given enrichment time for a certain target pathogen. Criteria to evaluate pathogen-specific assays: reproducibility. Reproducibility is defined as the ability to be accurately reproduced in different laboratories with different equipment and personnel. Although statistical methods for evaluation of reproducibility have been developed and