EURL-AP PCR Proficiency Test Final version. Authors: O. Fumière, A. Marien and G. Berben

Transcription

1 EURL-AP PCR Proficiency Test 2013 Final version Authors: O. Fumière, A. Marien and G. Berben May 2013

2 Editor : Centre wallon de Recherches agronomiques Service Communication Rue de Liroux, Gembloux (Belgique)

3

4 Table of contents Summary 1 1. Foreword Introduction Material and methods Study organization Material Description of the samples Material used in the preparation of the samples Homogeneity study Expression of results Results Gross results Qualitative analyses from the NRLs Overview of results and performance of the method Individual performances of NRLs in qualitative analysis Assessment of the usefulness of the cut-off quality control Conclusions Acknowledgements References. 11 Annex 1: Official announcement. Annex 2: List of participants..... Annex 3: Excel result report form... Annex 4: Composition of sample sets.. Annex 5: Gross results of participants (in numerical order of lab ID) I IV V VIII IX

5 Summary The European Union Reference Laboratory for animal proteins in feedingstuffs (EURL-AP) organised the present proficiency test for assessing the ability of the NRL network with respect to the detection of ruminant proteins in feed using PCR according the Commission Regulation n 51/2013 and the version 1.0 of the EURL-AP SOPs DNA extraction using the Wizard Magnetic DNA purification system for Food kit and Detection of ruminant DNA in feed using real-time PCR. Total number of participants was 27 NRLs (Croatian NRL participated to the study). The study was based on a set of 5 blind samples consisting of feed samples (blanks or feed matrices fortified with terrestrial processed animal proteins). In order to be in line with the forthcoming reintroduction of non-ruminant PAPs in aquafeed, four out of the 5 samples were aquafeed. All NRLs provided results in due time (deadline: 05 th of April 2013). The results indicate an excellent global performance of the network. Only four labs reported one false result out of 5 analyses to be carried out per lab (2 false positive results and 1 false negative one) and 1 lab had 2 false results (1 false positive result and 1 false negative one) on their five analyses. Corrective actions will be taken with the underperforming participants. Keywords : Processed animal proteins Aquafeed Ruminant PCR Polymerase Chain Reaction Proficiency test Qualitative analysis Page 1

6 1. Foreword European Union Reference Laboratories (EURL) formerly referred to as Community Reference Laboratories (CRL) were created in order to ensure a high level of quality and a uniformity of the results provided by European control laboratories. On 29 April 2004, the European Parliament and the Council adopted the Regulation EC/882/2004 [1], improving the effectiveness of the official food and feed controls while redefining the obligations of the relevant authorities and their obligations in the organization of these controls. On March 2011, the Commission Regulation EC/208/2011 [2], renewed the Walloon Agricultural Research Centre as European Union Reference Laboratory for animal proteins in feedingstuffs (EURL-AP, It has to develop the following priority axes: (i) (ii) (iii) (iv) (v) To provide National Reference Laboratories (NRLs) with detailed analytical methods, including reference methods for the network of Member State NRLs; To coordinate application by NRLs of the methods by organizing interlaboratory studies; To develop new analytical methods for the detection of animal proteins in feedingstuffs (light microscopy, near infrared microscopy, PCR, immunology ); To conduct training courses for the benefit of NRL staffs from Member States and future Member States; To provide scientific and technical assistance to the European Commission, especially in cases of disputed results between Member States. In this framework, the EURL-AP organised this PCR interlaboratory study for the assessment of the NRLs proficiency for the detection of ruminant proteins in feed using the PCR method as indicated in the new Commission Regulation n 51/2013 [3]. 2. Introduction According to the TSE Roadmap II, alternative analytical methods to the classical microscopy able to detect and identify the species of processed animal proteins (PAPs) in animal feed are the main condition for a possible lifting of the extended feed ban [4]. Commission Regulations n 51/2013 and 56/2013 [5] give to PCR the status of official method for the detection of PAP in feed. The objective of the present proficiency test is to evaluate performances of the network of 27 NRLs to detect the presence of ruminant processed animal proteins in feed using the ruminant PCR method [6]. Due to the next coming reintroduction of the non-ruminant PAPs in aquafeed (1 st of June 2013), the study focussed on the analyses of aquafeed samples. Page 2

7 3. Material and methods 3.1. Study organisation Official announcement of the study was made on the 08 th of February 2013 through a letter sent to the 27 NRLs of the EURL-AP network (Annex 1). A detailed list of the 27 participating labs is included in Annex 2. On the 25 th and 26 th of February 2013, the material for the test (sets of blind samples and of calibrants) was provided to the participants by express shipment. The Excel result report file containing the instructions, a recording sheet and a report summary (Annex 3) as well as an updated version of the Excel file for the cut-off determination were posted on the EURL-AP intranet the 26 th and the 28 th of February 2013 respectively. Some general recommendations were delivered to the participants: Results had to be encoded by way of an Excel result report file (Annex 3). Participants were asked to carefully read the instructions on how to fill in the result form and to testify they did it prior to encoding their results. No other support for communicating the results was accepted. A summarized results sheet was automatically generated. Participants were asked to sign the summarized results sheet and to return it by fax and to the EURL-AP. The results were taken into consideration only when both the Excel file and the fax were received by EURL-AP. The results had to be sent in both forms concomitantly to the EURL-AP by the 5 th of April The 27 participants delivered their results in due time Material Description of the samples Two samples of aquafeed containing or not processed animal proteins (PAPs) from ruminant (cattle) origin at a concentration level ~ 0.1 % in mass fraction have been prepared as shown in Table 1 and provided in duplicate to the participants. A fifth sample was chosen randomly among three series of samples prepared for the PCR implementation test Each participating lab received about 10 g of the five feed samples to extract their DNA according to the protocol imposed by the EURL-AP. A unique random number was assigned to each sample (Annex 4). Details of the samples are indicated in Table 1. Page 3

8 Table 1: Composition of the blind sample set used in the EURL-AP PCR Proficiency Test Sample Material Quantity/lab Remark % w/w cattle PAP in aquafeed % w/w cattle PAP in aquafeed 1 3 Aquafeed 1 free of ruminant PAP 4 Aquafeed 1 free of ruminant PAP 0.1 % w/w sheep PAP in feed % w/w cattle PAP in feed 1 % w/w pig PAP in feed Total 5 2 replicates of the same sample 2 replicates of the same sample 1 1 sample among the three samples Materials used in the preparation of the samples - Two aquafeed were selected among the EURL-AP sample bank. They had not to be too fatty to allow a grinding and a good homogenization. They also had to be free of any traces of ruminant DNA (this parameter was checked by PCR and microscopy). - PAP used to spike the blank aquafeed material was a cattle PAP heat treated at 137 C. A mix (100 g) at 0.1 % of ruminant PAP in mass fraction was prepared with each of the 2 aquafeed and PCR analyses were performed to control their homogeneity. Based on the results obtained, one of the 2 aquafeed was selected to prepare 1.5 kg of sample spiked at 0.1 % of ruminant PAP in mass fraction Homogeneity study Ten replicates of the sample containing ruminant PAP (sample #1 and #2 of Table 1) and fifteen replicates for the blank sample (sample #3 and #4 of Table 1) were chosen randomly and were analysed using the ruminant PCR target. Per sample replicate, 2 DNA extracts were realised according the EURL-AP Standard Operating Procedure DNA extraction using the Wizard Magnetic DNA purification system for Food kit version 1.0 ( In final, 20 and 30 Promega extracts respectively were obtained per sample type to be analyzed. The fifth sample (according to Table 1) available in each laboratory was chosen among three samples prepared for the PCR implementation test Page 4

9 Table 2 : PCR results obtained with aquafeed samples replicates Sample type Material Nr of samples analysed Nr of extraction replicates Detection with Ruminant target 1 and % w/w cattle PAP in aquafeed x positive 3 and 4 Aquafeed 1 free of ruminant PAP x negative* * However, 2 extraction replicates of a same sample resulted in twice ambiguous results which correspond to a final negative result. The homogeneity study showed that positive samples are continuously positive when analysed. Similarly the negative sample all led to negative result per sample even if in some one of the test portion replicates a positive result was once obtained Expression of results Qualitative analysis concerned the presence or absence of ruminant PAP material. These binary results were analysed by classical statistics: accuracy, sensitivity and specificity. All those statistics were expressed as fractions. Accuracy (AC) is the fraction of correct positive and negative results; it was calculated by the following equation: With : Accuracy PA NA AC PA ND PD NA PA : positive agreement (i.e. number of times detection was done when expected) NA : negative agreement (i.e. number of times there was no detection when expected) PD : positive deviation (i.e. number of times detection was done even though detection was not expected) ND : negative deviation (i.e. number of times there was no detection even though detection was expected) Sensitivity (SE) is the ability of classifying positive results as positive, it was calculated as follows: Sensitivity PA SE PA ND Specificity (SP) is the ability of classifying negative results as negative, it was calculated as follows: Specificity NA SP PD NA Page 5

10 The AC, SE and SP were calculated separately for each laboratory for the estimation of its proficiency. A consolidated AC over both parameters was used to rank each participant. Finally a global AC was also calculated for each material in order to estimate the performance of the method. 4. Results 4.1. Gross results Gross results from all participants are to be found in Annex Qualitative analyses from the NRLs Overview of results and global performance of the test Table 3 summarizes the results provided by the 27 NRLs for the five sample types submitted to qualitative analysis. Table 3: Global results expressed as accuracy (AC) for the five sample types Sample Material Nr of results AC % w/w cattle PAP in aquafeed (1) Aquafeed 1 free of ruminant PAP (1) % w/w sheep PAP in feed % w/w cattle PAP in feed (1) 7 1 % w/w pig PAP in feed (2) Accuracy means sensitivity in case of ND and specificity in case of PD. In brackets the number of false results. The overall results, expressed in terms of global accuracy (AC), reveal a very good global performance. Page 6

11 Individual performances of NRLs in qualitative analysis Individual performances were assessed for each participant by calculating the accuracy, sensitivity and specificity over the blind samples. A ranking of the labs was prepared based on the accuracy. Results are to be found in Tables 4 and 5. Table 4 summarizes the results obtained by the participants for the four aquafeed samples to all participants. Table 5 gathers the results obtained with all the samples including the fifth sample that can differ from one participant to another one. Table 4: NRL proficiencies regarding the detection of ruminant material starting from the four aquafeed samples. Ranking follows AC values. Lab code AC SE SP Concerning the ability to detect ruminant material in aquafeed, only 2 labs out of the 27 (7.5 %) provided 1 incorrect result: PD : lab 2; ND : lab 27. Page 7

12 A general ranking of the NRLs was performed on a consolidated evaluation including their proficiency in detecting ruminant material in feed samples in general (Table 5). Table 5: NRL proficiencies regarding the detection of ruminant material starting from the five feed samples. Ranking follows AC values. Lab code AC SE SP Concerning the ability to detect ruminant material in feed in general, 4 labs provided 1 incorrect result and 1 lab had 2 false results: Lab 2, 11 and 19 had 1 PD; Lab 22 had 1 ND; Lab 27 had 1 PD to add to the ND obtained with one of the aquafeed samples. Table 5 illustrates nevertheless the very good level of global performance for 22 labs out of 27 NRLs (81.5 % of the NRLs) having no false result. Page 8

13 Assessment of the usefulness of the cut-off quality control A quality control consisting of the number of copies of the ruminant target corresponding to the Ct value of the cut-off, was developed to minimize the risk of false positive result. A minimum of 9.00 copies at the cut-off was required. Indeed, depending on the variability of the lab (PCR platform + operator), the cut-off value can correspond to a too low number of copies. In Table 6, the participants are ranked by decreasing number of copies at the cut-off. Table 6: Number of copies at the cut-off value, cut-off value in cycles and false results Lab code Number of copies at the cut-off Cut-off (cycles) False results No No No No No No No No No No No* No No No No No No No No No* Yes No* No No No Yes No *False result obtained with sample 5, 6 or 7 (feed sample not common to all participants) Two participants (labs 25 and 27) out of the 27 were unable to reach the minimum criterion of 9.00 copies. Looking at the results, it appears that a majority of the false results (5 out of 6) are obtained by participants having < 9.50 copies at the cut-off value. The introduction of this quality control on the cut-off value improves the standardisation of the method. Page 9

14 5. Conclusions This study is the second assessment of the proficiency level in PCR methods for the detection of the ruminant PAP of the NRL network and the last one before the reintroduction of non-ruminant PAP in aquafeed planned for the 1 st of June The 27 participants submitted results before the deadline. Looking globally at these results sent to the EURL-AP, 92.5 % of the participating NRLs (25 labs out of 27) were able to detect correctly the presence of ruminant PAPs in aquafeed and had no false result. One NRL (Lab 27) had one false negative result and another NRL (Lab 2) had one false positive result. It must be stressed that the blank aquafeed used was a real-world sample with a rather high background noise for the ruminant target. It showed some risks of having false positive results during the homogeneity study. These conditions are however comparable to what a lab may be faced to in routine and the number of false positive results obtained remained nevertheless low (one lab, lab 2). On the other hand, detection of a positive sample is therefore also much easier (because of the high background noise) and the lab (lab 27) that failed to detect it has to be considered as under-performing. As the fifth sample was not identical for all labs, these results cannot be considered for defining individually which labs are performing or underperforming however these results give a global view of the performance of the network. The detection of ruminant PAPs in feeds in general by PCR is well implemented in the NRLs: 22 labs out of 27 NRLs (81.5 % of the NRLs) having no false result. The occurrence of false negative result is of 0.7 % (1 result out of 135 analyses) and demonstrates again the fitness of the method for the detection of ruminant PAPs. The number of false results with the fifth sample of the sets is somewhat higher. Concerning the sample with 0.1 % w/w cattle PAP in feed, if we take into account the number of observations performed during the PCR implementation test 2012, the rate of false results decreases to 1.06 %. The sample with 1 % w/w pig PAP in feed seems more problematic as the occurrence of false results remains at 13.2 % when the results of the 2 interlaboratory studies are gathered. The reason of this relatively high level of false positive is not clear as all the checks realised at the EURL-AP always gave the expected negative results. The lack of experience of the network and the influence of a detection of animal particles by a microscopic analysis carried out in parallel can be a possible explanation. Different problems were still present in some labs: 1. Some participants did not stick perfectly with the Standard Operating Procedures recommended by the EURL-AP: lab 2 did not grind the samples and lab 16 did not set the cut-off value according the protocol. 2. Lab 12 had to analyse the samples on a different thermocycler than the one used for the cut-off determination. Fortunately, this deviation had no consequence on the results. 3. Lab 22 encountered problems of mastermix mainly due to the design of the thermocycler using glass capillaries. 4. Lab 25 and 27 did not meet the requested quality criterion for the obtained cut-off. Acknowledgments We are grateful to the EURL-AP staff and the participants for their fruitful collaboration. Page 10

15 References [ 1 ] EU Commission Regulation (EC) No 882/2004 of the European Parliament and of the Council of 29 April 2004 on official controls performed to ensure the verification of compliance with feed and food law, animal health and animal welfare rules. Official Journal of the European Union L 165, 30/04/2004: [ 2 ] EU Commission Regulation (EU) No 208/2011 of 2 March 2011 amending Annex VII to Regulation (EC) No 882/2004 of the European Parliament and of the Council and Commission Regulations (EC) No 180/2008 and (EC) No 737/2008 as regards lists and names of EU reference laboratories. Official Journal of the European Union L 58, 3/3/2011: [ 3 ] EU Commission Regulation (EU) No 51/2013 of 16 January 2013 amending Regulation (EC) No 152/2009 as regards the methods of analysis for the determination of constituents of animal origin for the official control of feed. Official Journal of the European Union L 20, 23/1/2013: [ 4 ] The TSE Roadmap 2 - A Strategy paper on Transmissible Spongiform Encephalopathies for Communication from the Commission to the European parliament and the Council. Brussels, 16/07/2010, COM(2010)384 final. _Legislation/2010/07_jul2010/EU_Communication_TSE.pdf [ 5 ] EU Commission Regulation (EU) No 56/2013 of 16 January 2013 amending Annexes I and IV to Regulation (EC) No 999/2001 of the European Parliament and of the Council laying down rules for the prevention, control and eradication of certain transmissible spongiform encephalopathies. Official Journal of the European Union L 21, 24/1/2013: [ 6 ] Validation study of a real-time PCR method developed by TNO Triskelion bv for the detection of ruminant DNA in feedingstuffs. Final report. June Olivier Fumière, Aline Marien, Gilbert Berben. Page 11

16 Annex 1 Official announcement Page I

17 Page II

18 Page III

19 Annex 2 List of participants Country Austria Belgium Bulgaria Croatia Cyprus Czech Republic Denmark Estonia Finland France Germany Greece Hungary Ireland Italy Latvia Lithuania Luxemburg Netherlands Poland Portugal Romania Slovakia Slovenia Spain Sweden United Kingdom Institute Name Austrian Agency for Health and Food Safety Federal Agency for the Safety of the Food Chain National Diagnostic Research Veterinary Medical Institute Croatian Veterinary Institute Cyprus Veterinary Services Central Institute of sampling and testing in Agriculture Danish Veterinary and Food Administration Veterinary and Food Laboratory Finnish Food Safety Authority DG for Fair Trading, Consumer Affairs and Fraud Control-Laboratory Directorate Rennes Federal Institute for Risk Assessment Feedstuffs Control Laboratory Central Agricultural Office-Directorate Food and Feed Safety-Central Feed Investigation Lab. Department of Agriculture and Food Microscopy Laboratory - Seed Testing Station National Reference Centre for the Surveillance and Monitoring of Animal Feed Institute of Food Safety, Animal Health and Environment "BIOR" National Veterinary Laboratory Agroscope Liebefeld-Posieux Research Station (Switzerland) RIKILT Institute of Food Safety, Wageningen UR National Veterinary Research Institute Laboratorio Nacional de Investigaçao Veterinaria Hygiene Institute of Veterinary Health State Veterinary and Food Institute Veterinary Faculty-National Veterinary Institute-Unit for pathology of animal nutrition and environmental hygiene Laboratorio Arbitral Agroalimentario National Veterinary Institute, Department of Animal Feed Animal Health and Veterinary Laboratories Agency Page IV

20 Annex 3 a. Instruction sheet Excel result report file Page V

21 b. Recording sheet Page VI

22 c. Report summary sheet Page VII

23 Annex 4 Composition of sample sets Lab number % w/w cattle PAP in aquafeed % w/w cattle PAP in aquafeed Aquafeed Aquafeed % w/w sheep PAP in feed % w/w cattle PAP in feed % w/w pig PAP in feed Page VIII

24 Annex 5 Gross results of participants (in numerical order of lab ID) Page IX

25 Page X

26 Page XI

27 Page XII

28 Page XIII

29 Page XIV