Simulation-based assessment of Microbiological Criteria on Salmonella in poultry meat 1

Transcription

1 SCIENTIFIC REPORT OF EFSA Simulation-based assessment of Microbiological Criteria on Salmonella in poultry meat 1 ABSTRACT European Food Safety Authority 2, 3 European Food Safety Authority (EFSA), Parma, Italy A simulation exercise was performed to contribute to the assessment of the impact of Salmonella spp. on public health. The aim of this simulation approach was to investigate the effect of the prevalence of contaminated broiler carcasses at slaughterhouse level, after chilling but before further processing, on the probability of meeting the Microbiological Process Hygiene Criteria (MPHC) as defined by Regulation (EC) No 2073/2005 and its amendments. Using different scenarios it was possible to simulate the implementation of the monitoring procedures at slaughterhouse level and to relate carcass prevalence to the probability of meeting the criteria and vice versa. As an example, considering a true prevalence of contaminated carcasses at the slaughterhouse equal to 5%, assuming a test with sensitivity 90% and specificity 100% and a constant prevalence over time, the probability of meeting the criteria is equal to 68.45%. Furthermore, it is also possible for an observed frequency of meeting the microbiological criteria at a given slaughterhouse, e.g. equal to 95.54%, to estimate the underlying true prevalence of contaminated carcasses, which is equal to 3% in the considered example. The results from these simulations will contribute to the analysis of the outcome from the EU baseline survey at slaughterhouse level (EFSA-Q ). KEY WORDS Salmonella, Poultry, Broilers, Microbiological, Process Hygiene Criteria, Simulations, Sensitivity, Specificity 1 On request of EFSA, Question No EFSA-Q , issued on 20 December Correspondence: amu@efsa.europa.eu 3 Acknowledgement: EFSA wishes to thank the members of the Working Group on Model-based assessment of Microbiological Criteria on Salmonella in poultry meat for the preparation of this EFSA scientific output: Alessandro Mannelli, Eric Parent, Federica Barrucci, Matthias Greiner, Jukka Ranta, Kostas Koutsoumanis, Mary Howell, Natalie Commeau; Lieven De Zutter for the peer-review and EFSA s staff members Gabriele Zancanaro, Saghir Bashir, Didier Verloo, Maria Teresa Da Silva Felicio, Renata Leushner, Elena Mazzolini, Luis Vivas-Alegre for the support provided to this EFSA scientific output. Suggested citation: European Food Safety Authority; Simulation-based assessment of Microbiological Criteria on Salmonella in poultry meat. [58 pp.]. doi: /j.efsa Available online: European Food Safety Authority,

2 SUMMARY The evaluation of risks associated with certain hazards along the food chain contributes to improving food safety. Risk managers increasingly need quantitative decision support in order to lay down coherent and fair rules to ensure safe food to the European consumers. In this context, the definition of microbiological criteria along the whole production chain is one of the most important issues. The Assessment Methodology Unit was asked to contribute to the analysis of the results of the EU baseline survey on the prevalence of Salmonella spp. in broiler carcasses 4 at slaughterhouse level, after chilling but before further processing, by building, developing and validating a model to assess the impact of Salmonella spp. on the probability of meeting the Microbiological Process Hygiene Criteria. The pattern of the prevalence of infection/contamination along the whole food chain can be described by a stochastic model. Such modelling requires extensive data to obtain reliable and robust results. Data are required for the whole of the food chain, from production to retail, to make inferences for each Member State. Suitable data to build such a complex model were not available for this report. For this reason it was decided to adopt a simulation-based approach. The aim of this simulation exercise was to investigate the effect of the prevalence of contaminated broiler carcasses at slaughterhouse level, after chilling but before further processing, on the probability of meeting the Microbiological Process Hygiene Criteria (MPHC). Using different scenarios, it was possible to simulate the implementation of the monitoring procedures at slaughter house level as laid down in Regulation (EC) No 2073/2005 and its amendments, and to relate carcass prevalence to the probability of meeting the criteria and vice versa. As an example, considering a true prevalence of contaminated carcasses at slaughterhouse level, after chilling but before further processing, equal to 5%, assuming a test with sensitivity 90% and specificity 100% and constant prevalence over time, the probability of meeting the criteria is equal to 68.45%. Furthermore, it is also possible for an observed frequency of meeting the microbiological criteria at a given slaughterhouse, e.g. equal to 95.54%, to estimate the underlying true prevalence of contaminated carcasses, which is equal to 3% in the considered example. The simulation allowed a set of output tables to be produced that can be used to assess the probability of meeting the microbiological criteria, i.e. not exceeding the MPHC limit of 7 out of 50 neck-skin pooled samples, for a range of observed carcass prevalence values. Output tables were produced that can be used to estimate the true carcass prevalence when the criteria are met at a given frequency. The results from these simulations will contribute to the analysis of the outcome from the EU baseline survey at production. The observed prevalence values from the survey can be used as input data in the model to estimate what is the probability of meeting the microbiological process hygiene criteria at Member State level. 4 EFSA-Q : Analysis of the baseline survey on the prevalence of Campylobacter in broiler batches and of Campylobacter and Salmonella on broiler carcasses, in the EU, 2008, Part B: Analysis of factors associated with Salmonella contamination of broiler carcasses 2

3 TABLE OF CONTENTS Abstract... 1 Summary... 2 Table of contents... 3 Background as provided by EFSA... 4 Terms of reference as provided by EFSA Introduction Data Availability Overview of the adopted approach Methods Prevalence Conversion Tables Key Assumptions underlying the calculation of conversion tables Simulation for the Microbiological Process Hygiene Criterion (MPHC) MPHC Simulation steps Key Assumptions underlying the simulation model Software Results Outputs Use of the Output Tables with MPHC in force Assessing the MPHC from Observed prevalence Assessing observed prevalence when P meet is, e.g., around 95% Analysis of the impact of the microbiological process hygiene procedures and rules on the probability (P meet ) of not exceeding the MPHC limit Number of individual neck skin samples per pool Number of pooled samples per week MPHC value of the acceptability limit Sensitivity and Specificity of the Neck Skin Test (Se NST & Sp NST ) Discussion Simulation approach and cumulative binomial distribution function Constant prevalence Baseline Survey and Slaughterhouse (NST) test Sensitivity analysis Slaughterhouse effect An alternative way to address the interpretation of the probability of meeting the MPHC Previous related EFSA work Conclusions and recommendations References Appendices A Prevalence Conversation Tables B Simulation results C Graphical representation of the moving window functioning D Application of the simulation model to the Zoonoses Baseline results E Model linking poultry infection to carcass contamination after chilling in the slaughterhouse F R programs Abbreviations

4 BACKGROUND AS PROVIDED BY EFSA Simulation-based assessment on Microbiological Process Hygiene Criteria Risk assessments relating to food safety over more than one step along the food chain are frequently hampered by the lack of quantitative data. However, risk managers increasingly need quantitative decision support in order to define microbiological criteria, along the whole production chain, to be consistent with target prevalence at retail. In 2008, several mandates on this topic have been sent to the EFSA BIOHAZ Panel and to EFSA Task Force on Zoonoses Data Collection (EFSA-Q , EFSA-Q , EFSA-Q , EFSA-Q , EFSA-Q , EFSA-Q , EFSA-Q , EFSA-Q A, EFSA-Q B see Appendix IV 5 for more details). All these mandates require the quantification of the link between control measures at production and/or at retailing and the resulting prevalence of microorganisms in food at retail. In the absence of extensive longitudinal data from the whole food chain, statistical models can be used to describe such a link (Ranta J. et al, 2010). Based on these models, simulations can be made, under certain scenarios or assumptions, to provide risk managers with useful information. More specifically, the EFSA Task Force on Zoonoses Data Collection recently adopted technical specifications for a coordinated monitoring programme for Salmonella and Campylobacter in broiler meat at retail in the EU. One of the main aims of this programme is to assess the effectiveness of implementation of Community Salmonella criteria for broiler meats (see Report of Task Force on Zoonoses Data Collection on proposed technical specifications for a co-ordinated monitoring programme for Salmonella and Campylobacter in broiler meats at retail in the EU Adopted on 29th August 2008 ). Once the coordinated programme has been carried out and the EU survey results are available, the purpose of the statistical inference will be to assess whether the observed Salmonella prevalence estimates at retail are compatible or not with compliance of food business operators (FBOs) to the Salmonella criteria laid down in Regulation (EC) No 2073/2005. This requires an understanding of the link between observed prevalence at retail and observed prevalence at production. A stochastic model is a mathematical simulation based tool to formalise this link and can be used to meet the objective. The modelling approach is described in Appendix I 6. The main sources of variability and uncertainty, such as inter-batch variability at production, sampling uncertainty and test sensitivity, are represented by probability distributions. Then, given the sampling characteristics (e.g. sampling design, batch sizes, total number of batches) and assuming that the Salmonella criteria are met for all produced batches, the prevalence at retail can be simulated for each Member State and at EU level. This simulated distribution is then compared to the observed prevalence and its 95% confidence interval (CI) estimated from the survey and corrected for the import. In this way it will be possible to have a quantitative estimate about the overlap between simulated (under assumption that all Salmonella criteria are met) and the observed prevalence at Member State level (see Figure 1). The same modelling approach can be used to simulate what could be the Salmonella prevalence results in fresh broiler meat at retail using alternative values for c and n in order to provide information for the setting of Salmonella compliance criteria for fresh poultry meat laid down by Regulation (EC) No 2160/2003. Appendix II 7 gives more details on the methodological approach of this assessment. The Assessment Methodology Unit was asked to contribute to the analysis of the EU baseline survey results at production and to the analysis of the EU baseline survey at retail based on the proposed technical specifications by building, developing and validating a model to assess the likelihood of 5 The cited appendix is in the mandate M Internal Mandate proposed by EFSA to the Assessment Methodology Unit on Production-To-Retail Microbiological Modelling 6 See footnote 4 7 See footnote 4 4

5 compliance with the upper level of Salmonella prevalence at retail given the Process Hygiene and/or Food Safety Criteria 8 in force. Predicted distribution of prevalence at retail under compliance Observed prevalence at retail (outcome of the EU baseline survey according to the technical specifications EFSA Q ) Confidence Interval around the observed prevalence Figure 1 Example of assessment where the Salmonella criterion is met and not met: given the sampling characteristics (e.g. sampling design, batch sizes, total number of batches) and assuming that the Salmonella criteria are met for all produced batches, the prevalence at retail can be simulated for each Member State and at EU level. This simulated distribution is then compared to the observed prevalence and its 95% confidence interval (CI) estimated from the survey and corrected for the import. This way it will be possible to have a quantitative estimate about the overlap between simulated (under assumption that all Salmonella criteria are met) and the observed prevalence at Member State level. In the first scenario (above), the observed prevalence and the CI is within the range outlined by the simulated (predicted) distribution. In the second scenario (below), the observed prevalence and the CI is outside the range outlined by the simulated (predicted) distribution and the overlap between the two ranges is very limited. This latter scenario, in this example, could be indicative that Salmonella criteria are not met. This modelling approach will also contribute to the analysis of the outcome from the EU baseline survey at production (EFSA-Q ), focusing in particular on the impact of possible risk factors identified by the experts (EFSA-Q ) and on the impact of alternative microbiological criteria via simulations. More generally, this probabilistic model approach can be developed, optimized and adapted to different foodstuffs where microbiological criteria are available. The outcome of this exercise will be a flexible and adaptive tool able to provide information each time it is needed to investigate one link or more along the food chain, according to the characteristics of the concerned foodstuff and the involved bacteria. 8 Due to be in force in

6 TERMS OF REFERENCE AS PROVIDED BY EFSA The Assessment Methodology Unit is asked to: Simulation-based assessment on Microbiological Process Hygiene Criteria Build a model linking process hygiene criteria at production level and/or food safety criteria at retail to the baseline prevalence estimates at production or at retail in foodstuffs. The resulting models should allow for: o o testing the compliance of Member States to the microbiological criteria in place, and investigating the impact of alternative microbiological criteria via simulations. The adopted approach investigates the link between the Microbiological Process Hygiene Criteria (MPHC) and the baseline prevalence at slaughterhouse. The tables (see Annexes A and B ) allow to estimate the true carcass prevalence starting from a given observed carcass prevalence and to estimate the probability of meeting the MPHC given a true carcass prevalence value. Therefore, it was possible: o o to use the observed carcass prevalence values from the baseline survey to estimate the compliance of Member States to the MPHC, i.e. the probability, at Member State level, of meeting the MPHC given the observed prevalence (see Annex D ); to investigate the impact of alternative MPHC, e.g. number of carcasses to be pooled, samples to be taken per week, maximum allowed number of positive samples, (see Section 5.3) Validate the statistical models (model fit, model comparison); The adopted approach (see Section 1) is based on a deterministic simulation model that does not foresee any validation process. The whole simulation process mechanistically reproduces step by step the sampling and testing procedures as laid down in the relevant Regulation. Write short but comprehensive technical guidelines on how to use the model in the upcoming analysis. This should include a clear list of assumptions to be made and/or checked as well a list of input and output data; The assumptions underpinning the simulation model are exhaustively discussed in this report and must be taken into account when interpreting the results (see Sections and 4.2.3). The simulation model can be used in two different ways, either using the available outcome tables (see Annexes A and B ) or running the simulation starting from a csv file (see Annex F ). Implement a simulation-based example to illustrate the use of such model. The simulation model was run using the outcome of the baseline survey study as an input. The probability of meeting the MPHC given the observed prevalence was calculated for each Member State (see Appendix D ). The deliverables should include a workable and documented program-code to use the model. 6

7 1. Introduction Simulation-based assessment on Microbiological Process Hygiene Criteria In order to assess the impact of Salmonella spp. on public health, the pattern of the prevalence of infection/contamination along the whole food chain, from production to retail, could be described by a quantitative model. One of the most common tools in risk analysis, among quantitative models, is the stochastic model which can incorporate the main sources of variability and uncertainty impacting the variable of interest. For example, Salmonella prevalence at each stage, from production to retail, could be described by probability distributions which take into account all the identified and parameterised risk factors. Nevertheless, such modelling requires extensive data to obtain reliable and robust results. In this specific case, detailed data are required for the whole of the food chain, from production to retail, to make inferences for each Member State. At this point in time, suitable data to build such a complex model were not available (see Section 2). For this reason it was decided to adopt a simplified deterministic approach, based on the simulations of and the comparison between different scenarios. This model focussed on the carcass prevalence at slaughterhouse level and its corresponding probability of compliance with the microbiological process hygiene criterion (MPHC). This new approach was found to address the key issues in the terms of reference. The results from the simulation based approach will contribute to the analysis of the outcome from the EU baseline survey on the prevalence of Salmonella spp. on broiler carcasses. The key question of interest to address is: What is the relationship between the true prevalence of contaminated carcasses after chilling (TCP) and the probability of meeting the MPHC? This relationship can be considered in two directions: From cause (true carcass prevalence) to possible effect (probability of meeting the criteria); an observer can start from a known (observed) carcass prevalence and wants to know the probability of meeting the criteria; or From effect (probability of meeting the criteria) to possible cause (true carcass prevalence); an observer can start from an observed frequency of meeting the criteria (e.g. a given slaughterhouse or MS reported that 95% of the sampling windows tested less than 8 positive pooled samples out of 50) and wants to know what is the underlying prevalence of contaminated carcasses. Once the outcomes of the implementation of the monitoring procedures are simulated under different scenarios, it is possible to relate any carcass prevalence value to its probability of meeting the criteria and vice versa. The details of the simulation based approach are given Section 4. The results are given in Section 5. The discussion covers the issues faced and suggestions are given for similar projects in the future. 2. Data Availability As discussed in the introduction, to build a model as described in the background (see page 4), data are needed from each Member States on the whole of the food chain, from production to retail. Although a strong attempt was made to obtain such data, they were not available to EFSA during the course of this project. One of the most important pieces of information needed to implement the model, as outlined in the first approach, was the within-flock prevalence (i.e. number of birds infected by Salmonella at flock level). The flock prevalence (i.e. number of Salmonella positive flocks at Member State level) was 7

8 available from an earlier Baseline Survey 9, but no official publication or other sources of information allowed for any inference at the individual bird level. From the above mentioned Baseline Survey, the level of detail available was on the number of positive boot-swabs for each tested house, but this was not sufficient to make any reasonable prediction on the proportion of infected birds (within-flock prevalence). An alternative way to estimate the degree of infection at individual bird level (within-flock prevalence of infected birds) could be to start from the prevalence of infected birds entering the slaughterhouse: this information can be retrieved, e.g. performing a caecum test / cloacal swabbing at the beginning of the slaughter chain. Nevertheless, this strategy could not be followed as that test was not performed in occasion of the above mentioned Baseline Survey for Salmonella (pooled samples were just tested for Campylobacter). Following the flow from the farm to the chilled carcass, another striking point was the estimation of the impact of the slaughterhouse on the final prevalence of contaminated carcasses: the aim was to find a link between the prevalence of infected birds and the prevalence of contaminated carcasses after processing. The final report of the work developed on the procurement project CT/EFSA/BIOHAZ/2008/01 10 (Question No -EFSA-Q ) shows the results of an experiment carried out in a few slaughterhouses set in two European Member States. Appendix E depicts a probabilistic model linking poultry infection to carcass contamination after chilling in the slaughterhouse. Three stages (infection before entering the slaughterhouse, cross-contamination during the evisceration step and contamination during further transformation in the slaughterhouse) can be observed through various tests and measurements. Unfortunately, the reported database does not contain sufficient information to produce reliable results as far as the following issues are concerned: The variability between countries cannot be characterized since only two (contrasted) countries are present in the sample; The variability between slaughterhouses cannot be fully specified either. In addition, the sample at hand is not representative: a selection bias arises from the fact that the slaughter houses have been chosen so as to exhibit the greatest number of positive results; Finally, although experts agree to stress the role of sanitary working conditions when the Salmonella criterion is not met, no quantitative information per slaughterhouse was available to evaluate how much more severe management rules would lower the Salmonella prevalence. A literature search was carried out however insufficient data to be considered as representative of the average European situation was identified. The data identified were even less adequate if the aim was to make inference at single Member State level. A simple version of the simulation-based approach, consisting in a deterministic approach based on solution via simulation, was thus considered as the best option to address the terms of reference. Section 3 gives the outline of the project and of the outcome. All the details on the methodology underpinning the approach are given in Section Overview of the adopted approach The results of the model approach consist of two different outcomes or steps: the first allows inferring about true prevalence of Salmonella contamination at carcass level (TCP) from apparent (observed) carcass prevalence (ACP) and vice versa, using the formula 9 Report of the Task Force on Zoonoses Data Collection on the Analysis of the baseline survey on the prevalence of Salmonella in broiler flocks of Gallus gallus, in the EU, [1] - Part A: Salmonella prevalence estimates 10 Fate of Salmonella spp. on broiler carcasses before and after cutting and/or deboning 8

9 of Rogan and Gladen which takes into account the sensitivity and the specificity of the test used (Rogan and Gladen, 1978); the second allows estimating the probability of meeting the microbiological process hygiene criterion (P meet ) starting from the true carcass prevalence (TCP, inferred at the first step) and vice versa. The second step is based on a simulation that mechanistically reproduces the implementation, at slaughterhouse level, of the sampling procedures and criteria laid down in Reg. 2073/2005 and its amendments. Different scenarios concerning different combinations of n (number of pooled samples in the assessment period, i.e. 10 consecutive sampling session), c (maximum allowed number of positive pooled samples in an assessment period), among other parameters, were investigated and the outcome reported in tables (see Appendixes A and B ) Section 4 gives the details on both steps. 9

10 4. Methods 4.1. Prevalence Conversion Tables Conversion tables to estimate the True Carcass Prevalence (TCP) from the Apparent (observed) Carcass Prevalence (ACP), and vice versa, accounting for test sensitivity and specificity are presented in Appendix A (Table 9 and Table 10). The following formulae were used (Rogan and Gladen, 1978): and solving for TCP will give: where: ACP=Apparent (observed) Carcass Prevalence; TCP=True (inferred) Carcass Prevalence; Se BS and Sp BS =Sensitivity and specificity of the test used at individual level, as in the Zoonoses Baseline Survey (see also Acronyms). As already mentioned above, for the simulation-model purposes, different fixed values for Se BS and Sp BS were used to investigate their impact on the results. The calculations were repeated for a range of values (for all feasible combinations) as follows: TCP (True Carcass Prevalence): From 0.5% to 30% in steps of 0.5% Se BS : 50% to 100% in steps of 10% Sp BS : 80% to 100% in steps of 10% Key Assumptions underlying the calculation of conversion tables The following key assumptions were made for the use of the conversion table from the apparent prevalence to the true prevalence: The ACP is greater than 1 specificity ; If Salmonella is in/on the carcass, then it is assumed that the neck skin is contaminated by Salmonella. In the same way, if Salmonella is not present on the neck skin, it is assumed that the carcass is not contaminated. This is a definition about the true state of a sample and an assumption on its relation to the carcass contamination. The technical specifications of the test method are not considered here; Se BS and the Sp BS are known values, fixed at different levels. Se BS + Sp BS > Simulation for the Microbiological Process Hygiene Criterion (MPHC) MPHC The MPHC is based on the Neck Skin Test (NST) that is conducted on the carcasses of slaughtered birds in European slaughterhouses according to Regulation (EC) No 2073/2005 and its amendments. The NST involves taking a 25 gram sample of neck skin from 3 carcasses after chilling (i.e. a pool size of 3) and testing for the presence of Salmonella. The outcome for each pooled NST can only be positive or negative (presence/absence test). 10

11 Five pooled samples are taken per week and the results grouped into assessment periods of ten consecutive sampling sessions (i.e. 50 samples should be collected over a ten-week window). Assessment is undertaken weekly using a moving window of the 10 week assessment periods. Hence when testing has been undertaken in the previous year there are 52 ten-week windows in a period of one year (with some windows containing samples from the previous year). An assessment is therefore a weekly check in which if more than 7 out of 50 (pooled) samples from the previous 10 weeks test positive for Salmonella, then the limit for the MPHC has not been met (and corrective action should be instigated). In Appendix C a graphical representation of the moving window can be found. In the scheme, a period of 22 weeks was considered, i.e. 13 moving windows of ten weeks each. For each moving window, the sum of the positive samples out of 50 (total number of samples in 10 weeks) can be calculated (MW Sum): if the sum is below 8, then the microbiological criterion is met (1), otherwise it is not met (0). The sum of the number of times a moving window met the microbiological criterion divided by the total number of moving windows in this period (13 in the example), gives the frequency at which the criterion itself is met (P meet, 0.54 in the example) Simulation steps The following parameters have to be set taking into account EC Regulation 2073/2005 and its amendments: True Carcass Prevalence (TCP) = a range from 0% to 100% Number of individual neck-skin samples to be included in the pool = 3 Number of pooled samples per week = 5 Number of weeks per test period (sampling/moving window) = 10, i.e. 50 pooled samples Sensitivity (Se NST ) and Specificity (Sp NST ) of the test (at pooled sample level) reflecting the capacity of the Neck Skin Test (NST) to detect Salmonella in the pooled sample when Salmonella is present and the probability not to detect Salmonella in the pooled sample when Salmonella is not present; MPHC limit, i.e. maximum number of positive samples allowed out of 50 pooled samples = 7. The monitoring programme and MPHC are simulated using the following steps: 1. The sum of the probability of the truly positive pooled samples and the false positive pooled samples (APP, Apparent Pooled Prevalence) is calculated according to: o o o The true carcass prevalence of contamination (TCP); The number of neck-skin samples to be included in the pool (k); The sensitivity and the specificity of the test used by the laboratory to which the samples are sent by the Food Business Operator, i.e. the slaughterhouse in this case (Se NST and Sp NST ); as follows: APP = (1-(1-TCP) k )*Se NST + (1-TCP) k *(1-Sp NST ) 2. A number of pooled samples testing positive out of 50 (i.e. out of the total number of samples collected in a single assessment period or moving window) is sampled from a binomial distribution with prevalence equal to the calculated APP. As TCP is assumed to be constant in 11

12 an assessment period, APP also is constant for the same period. This step is iterated times. Positive Test Results (out of 50 pooled samples) ~ Binomial (50, APP) 3. If the number of positive pooled test results is greater than the limit (7 out of 50), record a zero (0); else record a one (1). 4. The proportion of successes over all the iterations is calculated. Since the pooled samples are here modelled independently and identically distributed, this proportion can be interpreted as the probability of meeting the microbiological process hygiene criterion, given an average TCP that is constant over time. Repeat the simulations for a range of values (for all feasible combinations) as follows: TCP (True Carcass Prevalence): From 0.5% to 30% in steps of 0.5% NST pool size: 2, 3 and 4. Pooled Samples per week: 4, 5, 6 and 7. Weeks per test period (sampling/moving window): 10. MPHC cut-off values: 1 to 20. Se NST : 50% to 100% in steps of 10% Sp NST : 80% to 100% in steps of 10% Key Assumptions underlying the simulation model The following key assumptions were made for the simulation model: A constant average prevalence of carcass contamination over time (i.e. at minimum over the 10 week window). This does not necessarily mean that there is no between-batch variability; Interventions or corrective actions by the FBO that could affect the prevalence of future batches have not been taken into account. More precisely, the effect of these interventions or corrective actions has not been quantified. On the other hand, it must be noted that, assuming an average constant prevalence, actually some external factor keeping the prevalence itself within certain limits is considered; If Salmonella is in/on the carcass, then it is assumed that the neck skin also is contaminated by Salmonella. This is a definition about the true state of a sample, and assumption of its relation to carcass contamination, irrespective of testing; The pool is truly negative if the three neck skin samples composing the pool are truly noncontaminated. If at least one of the three neck-skin samples, or more, is contaminated, then the pool is truly positive, assuming no dilution effect (Assumption about the performance of the testing method). According to the high performance of the new analytical techniques, this is more a description of what happens in reality, rather than an assumption; Se NST and the Sp NST are known values, fixed at different levels. 12

13 Software Simulation-based assessment on Microbiological Process Hygiene Criteria The R 11 (version ) statistical software package was used to run all the analysis (simulations) and produce the outputs which are presented in Appendix A and B. 11 R Development Core Team (2009). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN , URL 13

14 5. Results Simulation-based assessment on Microbiological Process Hygiene Criteria This section describes the results and describes how to use those results Outputs The simulations were run over a range of values (for all feasible combinations) resulting in a total of 259,200 combinations each simulated 10,000. A full version of the main outputs from the simulation-based model is presented in the Appendices: Table 9 in Appendix A presents the conversion table from the ACP to the TCP according to different fixed values of Se BS and Sp BS ; Table 10 in Appendix A presents the conversion table from the TCP to the ACP according to different fixed values of Se BS and Sp BS ; Table 11 in Appendix B presents the probability of meeting the Microbiological Process Hygiene Criterion in force (limit=7/50, neck skins per pool=3; pooled samples per week=5; weeks per moving window=10) for different TCP values and different fixed values of Se NST and Sp NST Use of the Output Tables with MPHC in force The conversion tables and the simulation results can be used to quantitatively link ACP, TCP and P meet in different ways. The 2 main possible uses of the results are described in the following sections: Section details how to assess the probability of meeting the microbiological criteria (P meet ) at, e.g., 95% given an ACP value (observed prevalence); Section details how to assess the underlying TCP given some probability (frequency) of meeting the microbiological criteria in force and then how to estimate the corresponding ACP from the underlying assessed TCP; Section 5.3 details how to use the simulation model as a risk assessment tool, i.e. how to investigate the impact of any of the involved covariates (i.e. microbiological limit, number of individual neck skin samples to be pooled, number of pooled samples per week, etc.) on the probability of meeting the criteria Assessing the MPHC from Observed prevalence It is possible to estimate the probability of meeting the MPHC in force, i.e. of not exceeding the limit of 7 out of 50, starting from an observed prevalence of Salmonella contaminated carcasses (ACP) as shown in Figure 2. As an example, we can assume of having observed an ACP equal to 4.5%. From Table 9 in Appendix A, assuming Se BS =0.9 and Sp BS =1, we can estimate a TCP equal to 5% (see Table 1). Due to the sensitivity of the test (90%), some truly positive samples are not detected. Once the true carcass prevalence is calculated, it s possible to estimate the corresponding probability of meeting the microbiological process hygiene criteria (P meet ), i.e. the probability of having a number of positive pooled samples equal or below the MPHC cut-off. With the criterion in force and assuming a Neck Skin Test with Se NST =0.9 and Sp NST =1, a TCP of 5% allows meeting the MPHC criteria 68.45% of the time (see Table 2). The same steps can be followed, as described above, using the lower and the upper values of the 95% confidence interval of the observed prevalence to obtain a corresponding range of probability of meeting the criteria: the lower value will correspond to the highest P meet and the upper value will correspond to the lower P meet. 14

15 It has to be pointed out that the range of the probability of meeting the microbiological criteria obtained in this way does not represent the 95% confidence interval for that probability: more precisely, it describes the uncertainty about the probability of meeting the microbiological criteria according to the 95% confidence interval of the observed prevalence of contaminated carcasses. ACP Conversion Table (Rogan & Gladen) Se BS & Sp BS TCP Simulation table Se NST & Sp NST MPHC in force P meet Legend: ACP: Apparent Carcass Prevalence; TCP: True Carcass Prevalence; Pmeet: probability (frequency) of meeting the MPHC in force; BS: Baseline Survey; NST: Neck Skin Test; Se BS & Sp BS : sensitivity and specificity of the individual test used in the Baseline Survey; Se NST & Sp NST : sensitivity and specificity of the Neck Skin Test used at the slaughterhouse; MPHC: Microbiological Process Hygiene Criteria Figure 2 Overview on the steps from some observed prevalence value to the correspondent probability of meeting the microbiological PHC. Table 1 Extract from Table 9 in Appendix A. This conversion table allows inferring about true carcass prevalence (TCP) using Rogan and Gladen s formula, starting from some observed (apparent) carcass prevalence (ACP). The sensitivity and the specificity of the test used in the Baseline Survey are taken into account. ACP Sp BS = 100 Se BS

16 Table 2 Extract from Table 11 in Appendix B. This table allows estimating the probability (frequency) of meeting the microbiological process hygiene criteria (P meet ) simulating the implementation at slaughterhouse level of the monitoring MPHC laid down in Reg. 2073/2005 and its amendments. TCP Weeks in an assessment period Pooled Samples per week Individual Neck Skin samples per pool MPHC limit (CutOff) Se NST Sp NST P meet The method described in this section has been applied to the Zoonoses Baseline Survey results to estimate the probability of meeting the MPHC in force given the observed carcass prevalence. The results are given in detail in Appendix D Assessing observed prevalence when P meet is, e.g., around 95% It is possible to estimate what could be the observed prevalence (ACP) if the probability of meeting the criteria in force is around 95%, as shown in Figure 3. ACP Conversion Table (Rogan & Gladen) Se BS & Sp BS TCP Simulation table Se NST & Sp NST MPHC in force P meet Legend: ACP: Apparent Carcass Prevalence; TCP: True Carcass Prevalence; Pmeet: probability (frequency) of meeting the MPHC in force; BS: Baseline Survey; NST: Neck Skin Test; Se BS & Sp BS : sensitivity and specificity of the individual test used in the Baseline Survey; Se NST & Sp NST : sensitivity and specificity of the Neck Skin Test used at the slaughterhouse; MPHC: Microbiological Process Hygiene Criteria Figure 3 Overview on the steps from some probability value of meeting the criteria to the correspondent underlying apparent carcass (individual) prevalence. As an example, we can assume of having observed a P meet equal to 95.54%. From Table 11 in Appendix B, assuming Se NST =0.9 and Sp NST =1, we can estimate an underlying TCP equal to 3% (see Table 3). Once the true carcass prevalence is calculated, it s possible to estimate the corresponding apparent carcass prevalence (ACP), i.e. the prevalence of positive results from a test performed at individual level, like for the Zoonoses Baseline Survey, with Se BS =0.9 and Sp BS =1 (see Table 4). In this case, if P meet is 95.54%, the underlying ACP is estimated to be around 2.7%. 16

17 Table 3 Extract from Table 11 in Appendix B. This table also allows estimating the underlying TCP from a given (observed/desired) probability (frequency) of meeting the MPHC (P meet ) by simulating the implementation at slaughterhouse level of the MPHC as laid down in Reg. 2073/2005 and its amendments. TCP Weeks in an assessment period Pooled Samples per week Individual Neck Skin samples per pool MPHC limit (CutOff) Se NST Sp NST P meet Table 4 Extract from Table 10 in Appendix A. This conversion table allows calculating the apparent carcass prevalence (ACP) using Rogan and Gladen s formula, starting from some estimated (true) carcass prevalence (ACP). The sensitivity and the specificity of the test used in the Baseline Survey are taken into account. The test is assumed to be used at individual level. Sp BS = 100 % Se BS TCP Analysis of the impact of the microbiological process hygiene procedures and rules on the probability (P meet ) of not exceeding the MPHC limit The repetition for all ranges of values, for all feasible combinations, resulted in a total of 259,200 combinations, each simulated 10,000 times (see Section 5.1). These results allow for building and comparing different scenarios, according to: NST pool size Pooled Samples per week 17

18 MPHC limit (cut-off) values Simulation-based assessment on Microbiological Process Hygiene Criteria Se NST Sp NST More details on the possible different combinations are given in Section Number of individual neck skin samples per pool Figure 4 shows the probability of meeting the MPHC in force for different TCP values. The probability changes according to the number of individual neck skin samples to be pooled for testing. The higher the number of individual samples to be pooled, the higher the sensitivity of the monitoring system, i.e. the probability of meeting the criteria is lower. Table 5 shows the same results by considering a fixed TCP value and comparing the P meet for three different values of number of individual neck skin samples to be pooled (2, 3 and 4). The pace between the P meet values increases with the increasing of the number of individual neck skin samples. 100 Pooled samples / week = 5; Weeks = 10; Se = 1; Sp = 1; Cut Off = 7 Samples / pool = 2 : 4 Probability of Meeting the criteria (%) True Carcass Prevalence Figure 4 Probability of meeting the microbiological process hygiene criteria in force for different prevalence values (TCP). Sp NST = 1. Se NST = 1. The number of individual neck skin samples to be pooled has different fixed values, from 2 (dotted line) to 4 (dashed line). The bold line represents the distribution of the probability of meeting the criteria with the rules in force. Table 5 Estimation of P meet by increasing of one unit the number of neck skin samples to be pooled. The pace between the P meet values increases with the increasing number of pooled neck skin samples. TCP(%) Se NST Sp NST Individual neck skin samples per pool Pooled samples per week Cut-off P meet Pace

19 Number of pooled samples per week Simulation-based assessment on Microbiological Process Hygiene Criteria Figure 5 shows the probability of meeting the MPHC in force for different True Carcass Prevalence values. The probability changes according to the number of pooled samples to be collected per week: the higher the number of pooled samples to be collected per week, the higher the sensitivity of the monitoring system, i.e. the probability of meeting the criteria is lower. Table 6 shows the same results by considering a fixed TCP value and comparing the P meet for four different values of number of pooled samples to be tested (from 4 to 7). The P meet decreases by up to 17.6 percentage points. Table 6 Estimation of P meet by increasing by one unit the number of pooled samples to be tested. The decreasing pace between the P meet values does not increases proportionally to the number of pooled samples per week. TCP(%) Se NST Sp NST Individual neck skin samples per pool Pooled samples per week Cut-off P meet Pace Samples / pool = 3; Weeks = 10; Se = 1; Sp = 1; Cut Off = 7 Pooled samples / week = 4 : 7 Probability of Meeting the MPHC (%) True Carcass Prevalence Figure 5 Probability of meeting the microbiological process hygiene criterion in force with increasing prevalence values (TCP); Sp NST = 1. Se NST = 1 and different fixed numbers - from 4 (dotted line) to 7 (dashed line) - of pooled samples to be collected per week. The bold line represents the distribution of the probability of meeting the criteria with the rules in force MPHC value of the acceptability limit Figure 6 shows the probability of meeting the MPHC in force for different TCP values. The probability changes according to the set acceptability limit (cut-off), i.e. the maximum allowed 19

20 number of positive pooled samples in a moving window: the higher the number, the lower the sensitivity of the monitoring system (the probability of meeting the criteria is higher). Table 7 shows the same results by considering a fixed TCP value and comparing the P meet for different values of MPHC acceptability limit (cut-off). Table 7 Estimation of P meet by increasing of one unit the maximum allowed number of positive pooled samples out of 50 (from 7 to 8). TCP(%) Se NST Sp NST Individual neck skin samples per pool Pooled samples per week Cut-off P meet Pace Samples / pool = 3; Pooled samples / week = 5; Weeks = 10; Se = 1; Sp = 1 Cut Off = 1 : Probability of Meeting the criteria (%) True Carcass Prevalence Figure 6 Probability of meeting the microbiological criteria in force according to different limit (cut-off) values (maximum allowed number of positive pooled samples out of 50). The bold line represents the distribution of the probability of meeting the criteria with the limit in force. Figure 7 shows an important feature of the impact on the probability of meeting the MPHC: shifting the limit (cut-off) to higher values, the P meet increases faster if the underlying TCP is low. 20

21 100 Samples / pool = 3; Pooled samples / week = 5; Weeks = 10; Se = 1; Sp = 1 TCP = 2 % and 10% Probability of meeting the MPHC (%) MPHC cut off Figure 7 Probability of meeting the MPHC for different limits (from 1/50 to 20/50) for two different values of TCP (2% and 10%). Shifting the cut-off to higher values, the P meet increases faster if the underlying TCP is low Sensitivity and Specificity of the Neck Skin Test (Se NST & Sp NST ) Figure 8 to Figure 10 show the impact of the performance of the Neck Skin Test on the probability of meeting the criteria in force. The biggest impact is given by the specificity: when this is equal to 100% (see Figure 8), low TCP values have a probability of meeting the MPHC around 100%; when the specificity is set to 90% (see Figure 9), the same TCP values have a probability of meeting the MPHC below 85%, even if the test has low sensitivity values (0.5). The probability falls dramatically if the specificity is set to 80% (see Figure 10). Table 8 shows the same results by considering a fixed TCP value and comparing the P meet by decreasing by 10% the sensitivity and the specificity of the Neck Skin Test (from 1 to 0.9). The P meet respectively increases by 6.5 percentage points and decreases by

22 100 Samples / pool = 3; Pooled samples / week = 5; Weeks = 10; Cut Off = 7; Sp = 1 Se = 0.5 : 1 90 Probability of Meeting the criteria (%) True Carcass Prevalence Figure 8 Probability of meeting the microbiological process hygiene criterion in force for different prevalence values (TCP). Sp NST = 1. Se NST has different fixed values, from 0.5 (dotted line) to 1 (dashed line). 100 Samples / pool = 3; Pooled samples / week = 5; Weeks = 10; Cut Off = 7; Sp = 0.9 Se = 0.5 : 1 90 Probability of Meeting the criteria (%) True Carcass Prevalence Figure 9 Probability of meeting the microbiological criteria in force for different prevalence values. Sp NST = 0.9. The Se NST has different fixed values, from 0.5 (dotted line) to 1 (dashed line). 22