Proficiency Testing. Protocol. Organisation and Analysis of Data

Transcription

1 Proficiency Testing Protocol Organisation and Analysis of Data Third Edition, September 2002

2 FOOD EXAMINATION PERFORMANCE ASSESSMENT SCHEME (FEPAS ) PROTOCOL FOR THE ORGANISATION AND ANALYSIS OF DATA THIRD EDITION, 2002 FEPAS CENTRAL SCIENCE LABORATORY Sand Hutton, York YO41 1LZ UK Tel: +44 (0) Fax: +44 (0) fepas@csl.gov.uk web: Page 1 of 28

3 CONTENTS 1. INTRODUCTION What is Proficiency Testing? Improving Data Quality The International Harmonised Protocol for the Proficiency Testing of (Chemical) Analytical Laboratories Accreditation Issues 5 THE ORGANISATION OF FEPAS Management Confidentiality of Participant Information 6 3. THE STRUCTURE OF FEPAS Type of Test Material Used in the Scheme Organisation of Proficiency Tests Typical Timetable 6 4. PARTICIPATION IN FEPAS Fees Reporting of Results and Methods 8 5. STATISTICAL PROCEDURES USED IN FEPAS Assessing a Test Material for Sufficient Homogeneity Establishing the Assigned Value, Xˆ Setting the Target Standard Deviation, σ p Calculation of z-scores Interpretation and Classification of z-scores Assessment of Qualitative Data REFERENCES 13 APPENDICES APPENDIX I: Glossary 14 APPENDIX II: Advisory Committee to the Food Examination Performance Assessment Scheme (FEPAS ) 17 APPENDIX III: Preparation and Homogeneity Testing of Test Materials 18 APPENDIX IV: Robust Statistics: a Method of Coping with Outliers 20 APPENDIX V: Outline FEPAS Reports 22 Page 2 of 28

4 THE FOOD EXAMINATION PERFORMANCE ASSESSMENT SCHEME (FEPAS ) 1. INTRODUCTION It is recognised that food-related legislation requires data on both the detection and/or enumeration of several microbiological contaminants, some of which present particular difficulties. Lack of an independent assessment of the quality of data being produced in these consumer safety related areas would hamper the work of enforcement authorities and would prejudice the mutual recognition of results and certificates. It would also limit the scope and reliability of microbiological surveillance work on the food supply. In particular, in the UK, reliable data are required for consumer protection purposes: to enforce the provisions of the Food Safety Act and of Regulations under the Act, the European Communities Act and other consumer protection legislation; to contribute to the study of the epidemiology of food-borne disease; for the official certification of microbial contaminants in foods for export; for use in self-certification, particularly in relation to documentation for food exports to EU countries. In addition the United Kingdom Food Standards Agency is obliged, as a result of the adoption of the Council Directive on the Official Control of Foodstuffs [1], to forward to the EC Commission the names of competent laboratories authorised to carry out analyses and examination in connection with food control. The Community has agreed quality standards for such laboratories. These are specified in the Additional Measures Food Control Directive [2]; proficiency testing is to be an integral part of such laboratory standards. The Food Examination Performance Assessment Scheme (FEPAS ) was launched in July 1997 as a commercial proficiency testing scheme to assess the performance of any laboratory carrying out microbiological examinations in the food area. This scheme supports the effective implementation of the UK Food Safety Act and European Community legislation. The acronym FEPAS is registered as a Registered Trade Mark of CSL, Department for Environment, Food and Rural Affairs. FEPAS has a website at This Protocol reviews the objectives and describes the scheme organisation and statistical treatments applied to the results submitted by participants in FEPAS, so that data obtained in the scheme will be valid and mutually recognised. A glossary of terms and acronyms used in this Protocol can be found in Appendix I What is Proficiency Testing? The ISO definition of laboratory proficiency testing is determination of laboratory testing performance by means of interlaboratory comparisons [3]. Proficiency testing is distinct from other inter-laboratory tests, such as collaborative trials (used to validate a standard method) or certification trials (used for establishing the true value of an analyte concentration in a reference material). It is a comparison of a laboratory s reported result for the analyte in question with the best estimate of the true value of the analyte. In a FEPAS proficiency Page 3 of 28

5 test this comparison, a laboratory's performance assessment, is expressed in the standardised form of a z-score. A z-score relates the error in a result to the designated standard deviation of the results for the analysis in question. This standard deviation, the so-called target standard deviation (σ p ), is set ahead of the test and reflects best practice or fitness for purpose, typically the observed standard deviation of reproducibility from a relevant collaborative trial. In a normal distribution only about 1 in 20 results will be outside two standard deviations from the mean hence FEPAS z-scores of z 2 are considered satisfactory. Put another way, if a laboratory receives a z-score outside the range z 2 it is much more likely that this is due to poor performance rather than it being a good result that just happens to be at the extremes of the distribution. A z-score within the satisfactory range does not in itself classify participants as "competent" or otherwise for the analysis in question, see Section 5.5 Interpretation and Classification of z-scores. The results from proficiency testing are one of several tools that enable participants to make that judgement themselves. Accreditation bodies will also ask laboratories for details of their performance Improving Data Quality Proficiency testing must be seen in the general context of accreditation and quality assurance Steps to Improvement Laboratories can strive to improve the quality of the data produced by: using methods which have been properly validated by inter-laboratory trials; testing and gaining experience in the use of those validated methods; setting up and using adequate and proper internal quality control procedures; seeking accreditation with the United Kingdom Accreditation Service (UKAS) or equivalent organisations. Accreditation requires the use of well documented methods, adequately trained staff and proper sample identification procedures, and participating in proficiency testing schemes, in which one or more test materials are examined by each participating laboratory and the results are statistically analysed. Of the types of action identified above, it is the participation in proficiency testing schemes which is crucial to the independent assessment of laboratory performance; the others are essential steps towards achieving a good performance but do not give independent confirmation that it is being achieved. For food analysis/examination laboratories some of the procedures referred to above are already formally in place, and in particular: methods of analysis are now available which have been validated by collaborative trial which conform to the "Protocol for the Design, Conduct and Interpretation of Collaborative Studies" [4]. In the food sector that includes the programme of collaborative trials on analytical/examination methods arranged under the auspices of CSL. This programme has led to a series of properly validated methods for food analysis/examination produced in booklet form. Page 4 of 28

6 laboratories accredited by UKAS, having demonstrated their compliance with the requirements of ISO/IEC [5]. In some countries this is government policy: UKAS accreditation has been actively promoted by the UK government [6] The International Harmonised Protocol for the Proficiency Testing of (Chemical) Analytical Laboratories No international agreement on the proficiency testing of laboratories carrying out microbiological examinations has yet been achieved. However, the aspects on analytical quality data referred to above are considered in depth on an international basis in the IUPAC/ISO/AOAC International Harmonised Protocol for the Proficiency Testing of (Chemical) Analytical Laboratories [7, 8]. Many of the prescribed principles also apply to examination and, where appropriate, the FEPAS Protocol adheres to the recommendations of the International Harmonised Protocol Accreditation Issues Proficiency Testing and Laboratory Accreditation The United Kingdom Accreditation Service information sheets on the accreditation of laboratories for food analysis strongly recommend laboratories to participate in an appropriate proficiency testing scheme as part of the accreditation process. This is because accreditation agencies working to ISO/IEC [5], WELAC criteria [9], and the EN Series of Standards [10, 11, 12] (to be replaced by ISO/IEC 17025) insist that laboratories take part in proficiency testing, provided a suitable scheme is available. It must be appreciated therefore that the purpose of participation in FEPAS is to complement the use of accredited methods within a laboratory. Participation in FEPAS does not confer accredited status on the participants; that is the responsibility of the accrediting body Accreditation of Proficiency Testing Schemes The Proficiency Testing Group (PTG) was a participant in the UKAS pilot scheme for the accreditation of proficiency testing schemes in accordance with the requirements of ILAC G13:2000. FEPAS received formal status as an accredited proficiency testing scheme in July THE ORGANISATION OF FEPAS 2.1. Management FEPAS is a proficiency testing scheme administered by the Proficiency Testing Group (PTG) at the Central Science Laboratory, Sand Hutton, York, UK. Scientific advice is provided by an Advisory Committee. Terms of reference for the Advisory Committee are given in Appendix II, and a current membership list is available from FEPAS. Day to day running of the scheme is the responsibility of PTG. This includes the administration of participants details, commissioning of appropriate test materials from specialist food testing laboratories and preparing reports on each proficiency test. Page 5 of 28

7 2.2. Confidentiality of Participant Information All information held by PTG about FEPAS participants, including their z-scores, is confidential and will not be disclosed to anyone unless explicitly agreed by the participant for a particular purpose. To preserve this confidentiality, FEPAS participants receive reports giving all the results for that assessment but without identifying individual laboratories. The laboratory code numbers used in reports are assigned in order of receipt of results from participants. Participants will be assigned the same code number in different rounds only by chance. The code number assigned to a participant in a round is given at the front of their copy of the report. Once reports are issued to participants they are regarded as being in the public domain. Although PTG is an integral part of CSL, PTG administers FEPAS in such a way that if any CSL laboratories take part in FEPAS proficiency tests they are treated in the same confidential manner as any other participants. CSL laboratories do not have access to FEPAS participants details. 3. THE STRUCTURE OF FEPAS 3.1. Type of Test Material Used in the Scheme The main objective of FEPAS is to assess the competence of laboratories carrying out microbiological examinations in the food sector. FEPAS aims to provide test materials, which simulate, as far as possible, food samples. This requires the production of test materials containing appropriate levels of target organisms (a range of levels for qualitative methods and levels which may range from 10 2 to 10 8 cfu/g for quantitative methods) in the presence of an autochthonous flora. To this end, FEPAS currently distributes stabilised food-based test materials for the detection of Listeria spp. and L. monocytogenes, Campylobacter spp., Escherichia coli O157, Vibrio spp. and Salmonella spp. and also for the enumeration of L. monocytogenes, coagulase positive staphylococci, Enterobacteriaceae, Escherichia coli, coliforms, Bacillus cereus, pseudomonads, enterococci (faecal streptococci), yeast and moulds, Clostridium perfringens, lactic acid bacteria, aerobic plate count and spore counts Organisation of Proficiency Tests FEPAS test materials are distributed at regular intervals. Participants are required to commence their examination by a given date and have 3 weeks from the dispatch date in which to return results. Within 4 weeks of the specified date for the return of data, a full assessment report will be sent to participants detailing, in an anonymous manner, their performance as compared to others in FEPAS Typical Timetable The annual programme of proficiency tests is compiled by PTG in conjunction with the FEPAS Advisory Committee. This programme of rounds is generally published in December in anticipation of the following April-March. Where short date formats are published, the UK convention of DD/MM/YY will be employed. Page 6 of 28

8 A typical FEPAS round is as follows: a) Preparation/homogeneity testing of test materials. b) Dispatch of test materials on a named and notified date from PTG, York, UK. c) Participants examine test materials and report results by the specified date (usually within 3 weeks of receipt of test materials). d) Results subjected to statistical analysis where appropriate; performance of laboratories assessed. e) Test report sent to participants approximately 4 weeks after specified date for the return of data. Participants may contact PTG at any time to seek advice on any aspect of FEPAS. 4. PARTICIPATION IN FEPAS Details about participation in FEPAS rounds are available from PTG at the address shown at the front of this document or from the FEPAS website ( Test materials are dispatched according to the published timetable. Results and methods forms and a covering letter giving full details of what is required accompany the test material. If for any reason a round is delayed or cancelled, participants are kept informed. Approximately one month after the closing date participants will receive a confidential report identifying only their laboratory code number. This report will describe the test material composition, give details of the homogeneity testing, all the results reported, the statistical analysis used, and outline details of the examination methods reported by participants. The first page of each report will carry the signature of the PTG staff member responsible for the administration of that FEPAS assessment. Whenever possible, guidance will be available on request to participants encountering difficulties Fees The fee for taking part in FEPAS is proportional to the number of rounds entered, with increasing discounts for increased participation. A list of fees, including the liability for VAT on the services provided, and CSL Standard Terms and Conditions for Proficiency Testing Schemes are included with the order forms sent to prospective participants. Once the order form has been completed and returned to PTG, participants receive confirmation of their order and are invoiced accordingly. The following specific points should be noted: 1. Participation in FEPAS is on the basis of acceptance of the CSL Standard Terms and Conditions for Proficiency Testing Schemes, available from PTG at the address shown at the front of this document or from the FEPAS website ( 2. If FEPAS is not able to issue any particular round due to circumstances beyond its control, then participants will have the choice of obtaining a credit of one round to be used within eighteen months or FEPAS can issue a refund of the fee paid for that round. FEPAS endeavours to keep to the published time scale and the test materials/organisms Page 7 of 28

9 quoted, but FEPAS reserves the right to occasionally delay the issue of a round or use an appropriate substitute test material if the planned original test material is not available at the stated time. Prior warning is given to participants. 3. No refunds are made to participants who withdraw from a round once the test materials are distributed. 4. All laboratories registered for a FEPAS round are sent the final report for that round even if they do not submit results. Participants must return data by the specified date if these are to be included in the report Reporting of Results and Methods Results The timely reporting of results in the manner requested is an important element of laboratory performance. Consequently, the submission of results in the specified units and by the closing date are considered to be part of a FEPAS proficiency test. Results are to be reported on the results form dispatched with the test materials. For qualitative methods, i.e. detection methods, a single result (present or absent) is to be reported for each test material. For quantitative methods, i.e. enumeration, participants are asked to report their raw data i.e. colony counts per dilution (as appropriate) and a final calculated result. The reported results are collated and every participating laboratory producing results will be assigned an assessment. In the case of quantitative data a laboratory will be assigned a z- score. Where participants report a less than or greater than value for an enumeration it will not be possible to assign a z-score and participants will be assessed as satisfactory or not satisfactory as appropriate. It should be noted that PTG cannot be held responsible for the misinterpretation of results submitted for a FEPAS round, e.g. the use of a comma can be ambiguous and could interpreted as either a decimal point or a thousand separator Methods Participants are informed that they should treat the test material as if it was a sample for routine examination, i.e. they are able to use the method of their choice. Participants will be asked to briefly describe their experimental procedures on a methods form. This information is presented in tabular format in the assessment report. The methods used by participants with performance outside the satisfactory range z <2 will be highlighted, but as indicated above this does not designate participants "competent" or otherwise for the analysis in question. That judgement should be make by the participants themselves. For further details, see Section 5.5 Interpretation and Classification of z-scores. 5. STATISTICAL PROCEDURES USED IN FEPAS The object of the statistical procedures employed is to obtain a simple and transparent result, which the participant and other interested parties can readily appreciate. The procedures used Page 8 of 28

10 by PTG for FEPAS rounds follow, broadly, those recommended in the IUPAC/ISO/AOAC International Harmonised Protocol for the Proficiency Testing of (Chemical) Analytical Laboratories [7, 8]. The outcome of the procedures is a z-score for each reported quantitative result, this score reflecting the deviation incurred in the result from the true value, with note being taken of the predicted or actual overall spread of results Assessing a Test Material for Sufficient Homogeneity It is recognised that the quality of the scheme rests on the assumption that the variability of the test materials received by the participants is small. PTG will ensure that any subcontracted work to prepare and/or analyse FEPAS test materials for sufficient homogeneity is carried out by a competent laboratory. PTG will only use subcontractors who hold accreditation for those methods and/or measurements used in homogeneity or stability testing of FEPAS test materials and/or who can demonstrate their competence for this type of analysis. The test materials distributed will have the required stability. Stability will take account of the total life of the test materials from preparation, through transport, to analysis by participants and of the environmental conditions encountered. Test materials are tested for homogeneity immediately after they are prepared by the procedure described in Appendix III. Homogeneity is normally assessed for each target micro-organism in a test material. Verification studies are undertaken for qualitative test materials to be used for testing detection methods, and data are presented in each round report Establishing the Assigned Value, Xˆ A consensus value is produced for each organism for examination, and is based on the results obtained by all participants. The consensus is calculated after converting all results to logarithms, and excluding any results that are clearly spurious, that is, unambiguously subject to reporting errors. Then for data sets that, outliers apart, are roughly symmetrical, the consensus is calculated as the robust mean by Huber's H15 method, using the AMC algorithm [13, 14] If the data set is markedly skewed, unusually disperse or possibly bimodal, other estimators such as the median or a mode may be used. There are no hard and fast statistical tests to determine whether such recourse is necessary. A decision to proceed in this manner is taken by PTG if necessary in consultation with one or more expert members of the Advisory Committee. In rare instances no satisfactory consensus can be obtained and consequently reliable z-scores cannot be obtained. This happens usually when the number of data is small or the data are very disperse Setting the Target Standard Deviation, σ p FEPAS specifically avoids the practise encountered in some proficiency tests of using a robustified standard deviation of the participants results as a value for σ p. Such a practise ensures that about 95% of participants receive a z-score within the range z 2 which is, by implication, satisfactory. However, scores produced in this way provide no information Page 9 of 28

11 about whether the results are fit for purpose and therefore are likely to mislead participants who are seeking useful action limits for investigations following a round. The value of σ p should prescribe not describe the participants results. Given this, the value of σ p for a FEPAS round is set at a value that reflects best practice for the analyses in question. σ p has therefore been estimated from previous FEPAS data and the FEPAS Advisory Committee has agreed that an appropriate value is It is acknowledged that perceptions change with time, and laboratory performance may improve with advances in methodology. The value of σ p therefore may need to be changed occasionally, disturbing the continuity of the scoring scheme. There is much evidence that laboratory performance responds favourably to a stepwise increase in performance standards Calculation of z-scores In order to use standard statistical methods, i.e. those that assume a normal distribution, any raw colony counts reported by participants are transformed by taking log 10 of the participant s reported result. Calculations are only performed on such log data Principle A z-score compares an estimate of the error of a result with a target value for standard deviation. The calculation procedure takes place in two stages The Error The error is defined as: x - X where x = log 10 of the participant s reported result X = the assigned value The value of X is obtained by the procedures outlined above and illustrated in Appendix V Comparison of Error with Target Standard Deviation The ratio of the error and a target value for standard deviation, σ p, that forms the criterion of performance derives the z-score: ( x X ) z = σ p where x = log 10 of the participant s reported result X = the assigned value and σ p = the target value for standard deviation The value of σ p is obtained by the procedure outlined in Section 5.3 and illustrated in Appendix V. Every participating laboratory producing results will be given one or more z-scores. However, as indicated in Section 4.2 Reporting of Results and Methods, results reported in a semi-quantitative manner as less than or more than a value, together with any qualitative results, will be collated and listed in the FEPAS round report; they cannot be used to derive a z-score but may be reported as satisfactory or not satisfactory. Page 10 of 28

12 5.5. Interpretation and Classification of z-scores It was noted in Section 1.1 that a z-score within the range z 2 is considered satisfactory but that this does not designate a participant "competent" or otherwise for the analysis in question. Inevitably, the term unsatisfactory is applied to z-scores that lie outside the range z 2. For a participant, or others, to judge the competence of a laboratory it is important therefore to understand the statistical limitations of proficiency testing Interpretation Consider first a group of laboratories that are unbiased and performing with an uncertainty defined by σ p. Each z-score is then a random selection from a normal distribution with a mean of zero and unit standard deviation. On average, about 5% of the z-scores will be outside the range ±2 and about 0.3% outside the range ±3. Now consider a mixture of laboratories where most are performing according to the specification but a small proportion are not. Since there is only a 5% probability of obtaining a result of z >2, we regard such an outcome as an indication that the participant is probably not performing according to the specification. A value of z >3 makes the same implication but more emphatically, because there is only a 0.3% probability of obtaining such an observation. The higher we make the limit that defines satisfactory the more likely we are to be correct in our designation of unsatisfactory but at the same time the more likely we are to accept a result from a laboratory that is performing worse than the specification. In the majority of instances, the results obtained from mature proficiency tests resemble what we expect from the above model, namely we have a sample of data that is approximately normal but with heavy tails and a few outliers. That is why, in most instances, we can use robust statistics to estimate the mean and standard deviation of what is regarded as an underlying normal sample. The real situation is not usually as simple as the above model because some participants results might always be drawn from a narrower distribution, so the laboratories in question have a negligible probability of getting a z-score outside z 2. Others, drawn from a wider distribution or a biased mean, may have a probability higher than 5%. Nevertheless, it is still appropriate to use boundaries of ±2 to indicate that investigative and/or remedial action is called for. Performance in a FEPAS proficiency test therefore is referred to as being satisfactory if a participant s z-score lies within the range z 2 and the outline method information supplied by participants with z-scores outside this range is highlighted in the Methods Appendix of the FEPAS report Classification The terminology satisfactory for z-scores z 2 and unsatisfactory for z >2, is widely used in proficiency testing. However, we should be aware of interpreting these terms literally. There are several reasons for saying this. Classification of any kind reduces the information content that is available in the original z-scores. A z-score of 10 demands a different action from a score of 2.1 but that fact may be obscured if both scores are simply reduced to unsatisfactory. Page 11 of 28

13 A result of z >2 is not specially rare. A laboratory that is performing in a fit-for-purpose manner could expect to encounter such a result as frequently as one result in 20 on average. An isolated result of 3> z > 2 should therefore alert us to a possible problem but probably demands no action at all, just as in a Shewhart control chart. The words satisfactory and unsatisfactory are pejorative. It may be preferable to replace them with within action limits and outside action limits. Some participants in FEPAS rounds may have different fitness-for-purpose criteria from the general σ p value used in the proficiency test and, indeed, different criteria for different customers. In these circumstances the words satisfactory and unsatisfactory are context-dependant and may mislead Assessment of Qualitative Data As yet there are no internationally agreed methods for the statistical analysis of qualitative data. Assessments will be made on the basis of the return of false positive, false negative or correct results. Page 12 of 28

14 6. REFERENCES 1 Council Directive on the Official Control of Foodstuffs, 89/397/EEC, O.J. No. L186, Council Directive on the Subject of Additional Measures Concerning the Official Control of Foodstuffs, 93/99/EEC, O.J. No. L290, "Protocol for the Design, Conduct and Interpretation of Collaborative Studies", Edited W Horwitz, Pure & Appl. Chem. 1988, 60 (6), "Collaboratively Tested Non-Statutory Methods of Food Analysis", DEFRA News Release FSD 32/92, BS EN ISO/IEC 17025:2000 (ISO/IEC 17025:1999), General Requirements for the Competence of Testing and Calibration Laboratories, BSI. 6 Measuring up to the Competition, Cm 728, HMSO, The International Harmonised Protocol for the Proficiency Testing of (Chemical) Analytical Laboratories, J. Pure & Applied Chemistry, 1993, 65, (9), The International Harmonised Protocol for the Proficiency Testing of (Chemical) Analytical Laboratories, J. of AOAC International 1993, 76, (4), WELAC Criteria for Proficiency Testing in Accreditation, 1993, Guidance Document WGD4 (EAL). 10 European Standard, EN 45001, General Criteria for the Operation of Testing Laboratories, 1989, CEN/CENELAC. 11 European Standard, EN 45002, General Criteria for the Assessment of Testing Laboratories, 1989, CEN/CENELAC. 12 European Standard, EN 45003, General Criteria for Laboratory Accreditation Bodies, 1989, CEN/CENELAC. 13 Analytical Methods Committee, Robust Statistics How not to reject outliers Part 1. Basic Concepts, Analyst, 1989, 114, Analytical Methods Committee, Robust Statistics: a method of coping with outliers, Technical Brief No 6, April Page 13 of 28

15 APPENDIX I: Glossary Proficiency Testing Scheme (Performance Assessment Scheme) The system for objectively checking laboratory results by means of an external agency (e.g. FEPAS ). It includes comparison of a laboratory's results at intervals with those of other laboratories, the main object being the establishment of trueness. Proficiency testing is designed to assess the accuracy of a laboratory's results. Proficiency testing is sometimes referred to as "external quality assessment" (EQA). Internal Quality Control (IQC) The set of procedures undertaken by the laboratory staff for continuous monitoring of operations and results in order to decide whether the results are reliable enough to be released; IQC primarily monitors the batchwise trueness of results on quality control materials, and precision on replicate analysis of test materials. Quality Assurance System/Programme (QAS) The sum total of a laboratory's activities aimed at achieving the required standard of analysis. While IQC and proficiency testing are very important components of a quality assurance programme it must also include staff training, administrative procedures, management structure, auditing, etc. Accreditation bodies judge laboratories on the basis of their quality assurance programme plus peer review of technical competence for a specific technical capability. Testing laboratory A laboratory that measures, examines, tests, calibrates or otherwise determines the characteristics or performance of materials or products. Test method A defined technical procedure to determine one or more specified characteristics of a material or product. Reference Material (RM) A material or substance one or more properties of which are sufficiently homogeneous and well-established to be used for the calibration of an apparatus, the assessment of a measurement method, or for assigning values to other materials. Certified Reference Material (CRM) A reference material, one or more of whose property values are certified by a technically valid procedure, accompanied by or traceable to a certificate or other documentation which is issued by a certifying body. True value The actual concentration of the analyte in the matrix. Page 14 of 28

16 Assigned value The value to be used as the true value by FEPAS in the statistical treatment of results. It is the best available estimate of the true value of the analyte in the matrix. Target value for standard deviation A numerical value for the standard deviation of a measurement result, which has been designated as a goal for measurement quality. Interlaboratory test comparisons Organisation, performance and evaluation of tests on the same or similar items or materials by two or more different laboratories in accordance with pre-determined conditions. Error The difference between a reported result and the assigned value. Fitness for Purpose The precision and accuracy of analytical data must be sufficient to enable the end-user of the data to make sound decisions as to whether the results/samples analysed are fit for the intended purpose. Accuracy The closeness of agreement between a test result and the accepted reference value. NOTE The term accuracy, when applied to a set of test results, describes a combination of random components and a common systematic error or bias component. Trueness The closeness of agreement between the average value obtained from a large series of test results and an accepted reference value. NOTE The measure of trueness is usually expressed in terms of bias. Bias The difference between the expectation of the test results and an accepted reference value. NOTE Bias is due to systematic error, not random error. There may be one or more systematic error components contributing to the bias. A larger systematic difference from the accepted reference value is reflected by a larger bias value. Laboratory bias The difference between the expectation of the test results from a particular laboratory and an accepted reference value. Page 15 of 28

17 Bias of the measurement method The difference between the expectation of test results obtained from all laboratories using that method and an accepted reference value. Laboratory component of bias The difference between the laboratory bias and the bias of the measurement method. NOTES (1) The laboratory component of bias is specific to a given laboratory and the conditions of measurement within the laboratory, and it may be different at different levels of microorganisms. (2) The laboratory component of bias is relative to the overall average result, not the true or reference value. Precision The closeness of agreement between independent test results obtained under prescribed conditions. NOTES (1) Precision depends only on the distribution of random errors and does not relate to the accepted reference value. (2) The measure of precision is usually expressed in terms of imprecision and computed as a standard deviation of the test results. Higher imprecision is reflected by a larger standard deviation. (3) Independent test results are defined as results obtained in a manner not influenced by any previous result for the same or similar material. Relative Standard Deviation (RSD) / (Coefficient of Variance) The standard deviation expressed as a percentage of the mean: RSD = σ 100 x where σ is the standard deviation and x is the arithmetic mean Page 16 of 28

18 APPENDIX II: Advisory Committee to the Food Examination Performance Assessment Scheme (FEPAS ) Background FEPAS maintains an Advisory Committee that includes technical specialists with detailed experience in relevant fields of testing. Members of the Committee are appointed for a maximum term of three years, with the option of reappointment for a further term thereafter. Committee members are appointed in their personal capacity as scientists with experience in a wide range of analytical sectors, but do not represent specific organisations. A list of current Advisory Committee members is available from FEPAS. Terms of Reference for FEPAS Advisory Committee The Committee will meet annually: to keep under review the scientific aspects of the Food Examination Performance Assessment Scheme and to give advice concerning its maintenance and further development, particularly in relation to laboratory accreditation systems. to advise the CSL Proficiency Testing Group (Sand Hutton, York) on the proficiency testing of laboratories carrying out microbiological examination, and on any modifications to the FEPAS Protocol and its operation, including the production, adequacy, stability and distribution of test materials and the evaluation of interlaboratory comparisons with other bodies as appropriate. Members will undertake to maintain the confidentiality of participants and their performance. Duties of the Committee Members of the Advisory Committee are to advise FEPAS on: a) the most appropriate tests required to be undertaken on test materials b) the nature of the test material(s) and microorganism(s) to be selected and, where appropriate, the test materials to be used c) the range of values to be expected for the microorganisms in the test materials d) any difficulties that may be anticipated to be encountered in the preparation and maintenance of homogeneous test materials or in providing a stable reference value for the microorganism e) the number of significant figures to which results are to be reported f) any technical difficulties raised by participants g) assist with assessment of the technical competence of laboratories undertaking test material preparation and testing h) the performance both of individual participants and of participants as a whole in specific assessments i) the technical commentary on the test reports j) evaluation of the comments from participants requesting feedback Page 17 of 28

19 APPENDIX III: Preparation and Homogeneity Testing of Test Materials It is recognised that the quality of the scheme relies on the assumption that the test materials received by participants are homogeneous and stable. The stability of the target organisms and autochthonous flora in the base matrices has been pre-determined and verified by FEPAS prior to their commercial release. Stability tests on the test materials take account of their total shelf life from time of preparation, through distribution to examination by participants and of the environmental conditions anticipated. Test materials are examined immediately after production (Day 0). Verification of the test materials is carried out using reference methods wherever possible. For verification of a batch of test materials for enumeration, a minimum of 10 samples of a batch are tested and for qualitative tests 10% of a batch may be tested. This level of quality control provides participants with a high degree of confidence that the test materials are fit for purpose. Because the test materials simulate as closely as possible real food samples, problems with homogeneity of a production batch at the time of testing by participants may occur from time to time. Where this occurs participants will be informed and no assessments will be issued. Procedure to be followed by laboratory preparing FEPAS test materials 1. Prepare the test material in a form that is thought to be homogeneous, by an appropriate method. 2. Place the test materials into the test vials that will be used for dispatch to the participants. Procedure to be followed by the laboratory carrying out homogeneity testing of FEPAS test materials Select from the production batch of test vials a number (usually n 10, ) of the containers at random. Homogenise separately the contents of each of the 10 selected vials and from each take two test portions. Analyse the 2n test portions under repeatability conditions as a single examination, by an appropriate method. Form estimates of the Poisson heterogeneity test (T1 and T2 tests) and the F-test from analysis of variance, without the exclusion of outliers. To show sufficient homogeneity, at least both the T1 and the F test should be passed. Statistical Analysis of Homogeneity Data The F-test The F-test compares the between sample variance with the variance within samples (i.e. from duplicate platings of the same samples). A high F-statistic indicates that there is more variability between samples than would be expected from the variability between duplicates. Page 18 of 28

20 Microbiological data follow a Poisson distribution, so if the data fails the F-test, which does not assume a Poisson distribution, two other statistical tests, below, may be used to assess homogeneity. T1 and T2 tests The T1 test measures whether pairs of replicates differ within an expected range, and the T2 test assesses the variability between samples. If a series of identical samples are taken from a homogeneous mix containing an unknown number of cfus, the numbers of cfus in each sample should follow a Poisson distribution. The Poisson is a skew distribution (i.e. it tends to contain a small proportion of values much higher than the rest), but as the mean number increases it gets less skew and more like a normal distribution. The first statistic that can be calculated is the T1 statistic, which measures whether the pairs of replicates differ by the amount that would be expected from a Poisson distribution. If T1 is significantly small it indicates that the data are significantly less variable than would be expected, perhaps due to the technician subconsciously getting the pairs of replicates to be similar. If it is significantly large it might indicate contamination or other problems with the enumeration. Thus, the T1 statistic does not itself test for lack of homogeneity but checks for problems with the data, which might need to be solved before a reliable assessment of homogeneity, could be made. The T2 statistic measures the variability between samples and a significantly high value would indicate a lack of homogeneity between the samples. In practice, in many fields of biology, exact Poisson distributions are not achieved and some degree of overdispersion (variance greater than the mean) is common. For this reason a significant result should not necessarily indicate that the samples are unusable; if the level of overdispersion is small it may not be important. The degree of overdispersion can be judged by dividing T2 by its degrees of freedom, with values greater than 2.0 often taken to indicate a serious lack of homogeneity. The major advantage of these tests over the F-test is that the T1 statistic will identify any excess variation between replicates that might result from poor experimental procedure; when such excess variation is present it makes the F-test insensitive to any lack of homogeneity. Page 19 of 28

21 APPENDIX IV: Robust Statistics: a Method of Coping with Outliers Page 20 of 28

22 Page 21 of 28

23 APPENDIX V: Outline FEPAS Reports This is intended to be an example of how assigned values and the target standard deviation are derived, and how z-scores may be calculated and used according to this Protocol. Numerical details have been specified for the purposes of illustration only; actual rounds will take account of factors specific to each area. The examples given are : 1. enumeration of pseudomonads in beef 2. aerobic plate count in chicken 3. detection of Salmonella spp. in chocolate powder RESULTS Participants results are tabulated according to the information supplied on the results form submitted for the relevant FEPAS round. Participants are required to report their data in either cfu/g and/or log 10 cfu/g, or detected/not detected as appropriate. Each participant is given a laboratory number, assigned in order of receipt of results. The reported results are tabulated (see Tables 1 and 2 below). STATISTICAL EVALUATION OF RESULTS The object of the statistical procedure employed is to obtain a simple and transparent result, which the participant and other interested parties can readily appreciate. Further details, including worked examples, are given in the FEPAS Protocol. The procedure follows that recommended in the IUPAC/ISO/AOAC International Harmonised Protocol for the Proficiency Testing of (Chemical) Analytical Laboratories Calculation of the Assigned Value, Xˆ The best estimate of the true number of each organism, Xˆ, was derived from the results submitted by participants. When a participant submits multiple results for an examination, only one result is included in the calculation of the assigned value, although z-scores are issued for them all. The assigned values were calculated as the robust mean of the remaining valid data after conversion to logarithms. Target Standard Deviation for the Round, σ p The value of σ p determines the standard of data quality appropriate for the microbiological examination. It has been estimated from previous FEPAS data and an appropriate value is The values for the parameters Xˆ and σ p used to calculate z-scores from the reported results of this assessment are given in Table 6. Page 22 of 28

24 Individual z-scores Participants z-scores were calculated as: ( x X ) z = σ p where x = log 10 of the participant s reported result X = the assigned value and σ p = the target value for standard deviation Participants z-scores for the enumeration examination are given in Table 1 and shown as histograms in Figures 1 and 2. Assessments for the detection examination are given in Table 2. The number and percentage of z-scores and assessments in the satisfactory range, z 2, for each examination are given in Tables 3 and 4. Page 23 of 28

25 Table 1: Results and z-scores for Pseudomonads in Beef and Aerobic Plate Count in Chicken laboratory number organism pseudomonads aerobic plate count assigned value 3.39 log 10 cfu/g assigned value 4.96 log 10 cfu/g result z-score result z-score log 10 cfu/g log 10 cfu/g a b a b a <100 cfu/g NS z-scores outside the satisfactory range are shown in bold NS = Not Satisfactory Page 24 of 28

26 Table 2: Assessments for Salmonella spp. in Chocolate Powder laboratory number organism Salmonella spp. test 1 test 2 assessment present present 005 not detected not detected NS 013 detected detected S 017 detected detected S 018 detected detected S 025 not detected no result NS 029 detected detected S 030 detected detected S 032 not detected not detected NS 038 not detected not detected NS 039 detected detected S 041 detected detected S 043 detected detected S 044 detected detected S 047 detected detected S 049 detected detected S 050 detected detected S 052 detected detected S 053 detected detected S 058 detected detected S 059 detected not detected * NS 062 not detected not detected NS 068 not detected not detected NS 072 detected detected S S = Satisfactory NS = Not Satisfactory * = Participant detected an abnormal odour in this test material Page 25 of 28

27 Table 3: Assigned values and target standard deviations test Xˆ robust mean log 10 cfu/g n data points σ p from retrospective FEPAS data Pseudomonads (beef) aerobic plate count in chicken Table 4: Number and percentage of satisfactory z-scores and assessments organism number of satisfactory scores total satisfactory z 2 and assessments number % of results Pseudomonads (beef) aerobic plate count in chicken Salmonella spp Page 26 of 28

28 b a z-score Laboratory Number Figure 1: z-scores for Pseudomonads (3.39 log10cfu/g) in Beef Test Material Page 27 of 28

29 z-score b 019a b 009a 051a b c Laboratory Number Figure 2: z-scores for Aerobic Plate Count (4.96 log10cfu/g) in Chicken Test Material Page 28 of 28

30 CENTRAL SCIENCE LABORATORY Sand Hutton, York, YO41 1LZ, UK Tel:+44 (0) Fax:+44 (0) Web: FEPAS is a Registered Trademark