Measurement of uncertainty for Elisa Tests. University of Hasselt, Center for Statistics, Hasselt, Belgium

Transcription

1 Appendix 58 Measurement of uncertainty for Elisa Tests Toussaint, J.F. 1*, Assam, P. 2, Caij, B. 1, Dekeyser, F. 1, Imberechts, H. 1, Knapen, K. 1, Goris N. 1, Molenberghs, G. 2, Mintiens, K. 1, De Clercq, K. 1 1 Veterinary and Agrochemical Research Center, Brussels, Belgium 2 University of Hasselt, Center for Statistics, Hasselt, Belgium Abstract: Accreditation according to ISO requires an evaluation of measurement uncertainty and it has to be expressed in a way that can be transmitted to the client upon request. The present paper describes and illustrates with FMD-specific ELISA a new method for the simultaneous estimation of precision and uncertainty. Dilutions of a strong positive sera / virus suspension, were tested 10 times, in duplicates, with a competitive or an indirect ELISA that are respectively used for the detection of FMD-specific antibodies and FMD antigens. The precision was determined by the analysis of variance. The normality of the residues was checked with Shapiro-Wilk test and the predicted standard errors were transformed in confidence intervals around the observed values. These confidence intervals were further transformed into probabilities of being above / below the cut-off. Logistic regression models were finally used to interpolate probability values for the whole range of possible values. The coefficients of variation (CV) for repeatability and intermediate precision respectively reached 2.9 % and 6.6 % for the competitive ELISA and 13.4 % and 21.7% for the indirect ELISA. The uncertainty about a positive (negative) test result was defined as the probability of not observing the same positive (negative) test result when performing a second test on the same sample. With the competitive ELISA, the newly developed method shows that any sample showing 4% percent of inhibition above the cut-off has an uncertainty level below 5%. In the indirect ELISA, the uncertainty was below 5% for any samples with an OD value above As a conclusion, we can affirm that the present method is capable of estimating the uncertainty for competitive and indirect ELISA. The precision and uncertainty values observed with the 2 examples immediately illustrate the difference between a normalized test (competitive ELISA) and a test without (or minor) normalization (indirect ELISA). Introduction: Diagnostic laboratories are more and more encouraged to execute their tests within a quality control system. Accreditation according to ISO requires the set-up of a validation file containing information about the intrinsic characteristics of the test (sensitivity, specificity, relative trueness, limit of detection/quantification), the precision (repeatability and reproducibility) and an evaluation of measurement uncertainty (EA-04/10, 2002). According to ILAC-G17 (2002), the uncertainty of measurement might be defined as a parameter associated with the result of a measurement that characterizes the dispersion of the values that could reasonably be attributed to the measurand, the measurand being a particular quantity subjected to measurement. In practice, the uncertainty of measurement has to be studied following two approaches (ISO/IEC 17025). The first approach consists in identifying all the components that influence the uncertainty. Once identified, their influence on the test results has to be determined and some measures have to be taken to keep the critical parameters into the acceptable limits (e.g. calibration of the pipettes to ensure correct volumes). The components of uncertainty that need to be studied during this first approach generally include: the laboratory equipment, the environment (e.g. temperature), the sample and the consumables, the staff and the procedure itself. The second approach aims at quantifying the uncertainty. It must conduct to a figure that can be transmitted to the client upon request. So far, the guidelines for the estimation of uncertainty of measurement only focused on truly quantitative tests, which are common in the field of chemistry. These guidelines are not really appropriate for diagnostic tests, where the final results are most often binary (positive or negative - with an optional category for doubtful results). Therefore, the goal of the present study was to develop a methodology for the estimation of measurement uncertainty for ELISA tests, which are widely used for the diagnosis of infectious diseases including foot and mouth disease (FMD). The 363

2 methodology was developed through a collaboration of the Belgian Veterinary and Agrochemical Research Center with the Center for Statistics of the Hasselt University. The method was developed and validated using several data sets generated by an indirect and a competitive ELISA for the detection of FMD virus and FMD antibodies, respectively. Materials and Methods: Diagnostic tests evaluated: A competitive (C-Ab-FMDV; MacKay et al. 2001) and an indirect ELISA (I-Ag-FMD; OIE, 2000) are respectively used for the detection of FMD-specific antibodies and FMD antigens. These tests are both in-house manufactured by combining self-produced reagents (virus, buffers) with reagents (rabbit and guinea pigs sera) provided by the FAO world reference laboratory for FMD (Institute for Animal Health, Pirbright, UK). The C-Ab-FMD ELISA contains a grey zone delimited by a lower and an upper cut-off, which are both deduced from the value observed for a cut-off sample that is placed on each plate (variable cut-off). The samples with a percentage inhibition (%inh.) below the lower cut-off are negative whereas samples with %inh. above the upper cut-off are considered positive (Goris and De Clercq, 2005). The I-Ag-FMD ELISA doesn t contain any grey zone. The samples are considered positive when the difference between their optical densities (O.D.) and the blank is higher than 0.1. Samples used for evaluation: The uncertainty associated with the C-Ab-ELISA was assessed using serial two-fold dilutions of a strong positive sera directed against the serotype O (Goris and De Clercq, 2005) whereas the I-Ag-FMD ELISA was evaluated on serial two-fold dilutions of FMD virus (C1 Noville) grown on SK6 cells. The dilutions were chosen to cover the whole range of potential values in the two tests, ranging from the upper plateau to the lower plateau. The dilutions were prepared and aliquoted on a single day to ensure homogeneity of the samples and to discard any error associated with preparation of the dilutions. The samples were then tested in duplicates on ten separate days, by at least three different technicians. Statistical methods: The precision (repeatability and intra-laboratory reproducibility) was determined by the analysis of variance using the procedure NESTED in SAS. The normality of the residues was checked with Shapiro-Wilk test. Then the predicted standard errors were transformed in confidence intervals around the observed values and into probabilities of being above or below the cut-off using the normal law (Figure 1). Logistic regression models were finally used to interpolate a probability value for any possible result into the whole range of the test. The uncertainty about a positive (negative) test result was defined as the probability of not observing the same positive (negative) test result when performing a second test on the same sample. Since the C-Ab-FMD ELISA uses a moving cut-off, the analysis and the modelling were performed on the differences between the %inh. observed for each sample and the %inhib. observed for the cut-off sample (CO). For the indirect ELISA, the analysis was done on normalized OD values obtained by substracting the background from the OD value of each sample. 364

3 Probability Negative Positive Cut-off Normalised OD Figure 1: Transformation of standard errors in probabilities using the normal law. For each observed value (yij), the standard error was transformed in a probability using the normal distribution N(yij,(d^ij)2) and the cut-off. The certainty of a decision (positive/ negative) was then estimated as the proportion of the confidence distribution falling above/below the cut-off. The arrow represents one individual value yij that was taken as an example. Results: Two datasets were generated by analysing the serum and the virus dilutions by C-Ab-FMD and I- Ag-FMD in duplicates on ten separate days. The serum dilutions completely covered the higher plateau and the linear phase of the ELISA but they only covered partly the lower plateau (figure 2A). On the opposite, the dilutions of the viral preparation did not fully cover the higher plateau but they adequately covered the linear phase and the lower plateau (figure 2B). The precision was estimated for the two data sets. The coefficients of variation (CV) for the repeatability and the intermediate precision (which reflects the highest level of variation within a single laboratory) respectively reached 2.9 % and 6.6 % for the competitive ELISA test and 13.4 % and 21.7% for the indirect ELISA. For the indirect ELISA, the standard errors were normally distributed in all the levels (each level of analyte corresponding to one dilution). On the contrary, the 5 levels corresponding to the higher concentrations of antibodies showed a non-normal distribution in the competitive ELISA (table 1). Individual probabilities of being positive/negative were deduced from the individual standard errors and the logistic model was used to interpolate the probabilities from the data set to the whole range of possible test results. Figure 3 illustrates the fit of the logistic model to the data of the I- Ag-FMD ELISA. The estimations generated for both tests were summarized in table 2, which contains the OD values (I-Ag-FMD) and the %inh. (C-Ab-FMD) that are associated with reference probabilities of positive/negative test results. For example, every samples with a %inhib. higher than 3.42 % above the upper cut-off have a probability of positive test result of at least 95%. By the same way, a sample with an observed %inhib. 5.83% below the lower cut-off has a 95% certainty of negative test result and, thus, an uncertainty of less than 5%. For the indirect ELISA, every sample with a normalized OD value of 0.19 have a 90 % probability of positive test result (certainty) and a 10 % probability of negative test result (uncertainty). 365

4 A Percentage inhibition upper cut-off = CO sample + 15% lower cut-off = CO sample 4% log2 dilution 3 B 2,5 Normalized OD 2 1,5 1 0,5 cut-off = 0.1 OD unit log2 dilution Figure 2: Datasets used for the development and the validation of the model. A) Competitive ELISA for the detection of FMD-specific antibodies; B)Indirect ELISA for the detection of the FMD virus. 366

5 Table 1: Normality test on the residues of the analysis of variance log 2 dilution* C-Ab-FMD I-Ag-FMD p value conclusion p value conclusion Non Normal Normal Non Normal Normal Non Normal Normal Non Normal Normal Non Normal Normal Normal Normal Normal Normal Normal Normal Normal Normal Normal Normal Normal Normal Normal Normal Normal * each dilution corresponds to a level of analyte (antibodies or virus) Positive Probability RESULTS Figure 3 : Fit of the logistic model with the individual positive predictive values calculated for the dataset generated with the indirect Elisa. 367

6 Table 2: Results associated with probabilities of positive/ negative test results p value C-Ab-FMD I-Ag-FMD positive result* negative result positive result $ negative result * In the C-Ab-Elisa, we modelized the difference between the sample and the upper (lower) cut-off. This means that a sample showing 5.35 % inhib. above the upper cut-off has a 0.99 probability of positive test result. $ in the indirect Elisa, we modelized the normalized OD values (ODsample OD blank). A sample that shows a normalized OD above 0.28 has a 0.99 probability of positive test result. Discussion: The estimation of measurement uncertainty is of a growing concern for the authorities that are in charge of accrediting microbiological laboratories. However, most laboratories are not prepared to do it since no specific guidelines currently exist for the quantification of measurement uncertainty for microbiological tests. The present paper partly solves that problem by describing a method that is suitable for one of the most popular tests, namely the ELISA. A first component of the evaluation of the uncertainty relies on i) the identification of the parameters that are susceptible to affect the uncertainty; ii) the identification of the critical ones by testing small deviations deliberately introduced; iii) the adoption of appropriate measures to keep the critical parameters into the acceptable range. The major factors to take into account have been listed in the introduction. However, the critical factors will highly differ depending on the method to be validated and their identification mainly relies on testing. This first component of the evaluation of uncertainty is straightforward and it is not the topic of the present paper, which mainly focus on uncertainty quantification. Before proceeding with the analysis and the quantification of uncertainty it is necessary to generate a dataset containing the test values associated with some keys probabilities (5% or 1% uncertainty). For ELISA tests, we recommend to use several two-fold dilutions of a strong positive samples that include: i) three dilutions covering the higher plateau; ii) the dilutions covering the linear phase, and iii) three dilutions covering the lower plateau. The present examples suggest that it is not always easy to chose the appropriate dilutions (the high positive serum was not diluted enough and the lower plateau was not covered by three dilutions) or to get a sufficiently concentrated reagent (the titers of the pure virus preparation was too low and the second dilution was no more in the higher plateau). Even though they do not strictly follow the above mentioned rule, both datasets conducted to a reliable result since the 95% and even the 99% certainty (5 and 1% uncertainty) could be estimated by the model and they corresponded to test results that were well covered by the dataset. When possible, we advice to cover the whole range of data because the validity of a dataset can only be assessed after completion of the analysis and the failure to validate a dataset implies that a new panel of samples has to be prepared and analysed. The precision values observed for both tests immediately illustrate the difference between a test that is highly normalized (competitive ELISA) and a test with minor to no normalization (indirect ELISA). The results of the C-Ab-FMD results are normalized twice by the calculation of percentage of inhibition and by the use of a variable cut-off. For that test, the repeatability and the intermediate precision CV were both below the 10 percent. On the opposite, the I-Ag-FMD results are only normalized by subtracting the O.D. background from the sample O.D. and that test was associated with higher CV of repeatability and intermediate precision. Nevertheless, the global CV of that test remained below the 25 percent, which is considered as the upper acceptable value for 368

7 ELISA (Findlay et al. 2000). The uncertainty of measurement also largely differed between both tests. A serum sample with a few percents above the upper threshold might be associated with a high probability of positive test result and a low uncertainty in the competitive ELISA, whereas a virus sample with a normalized OD value of 0.19 (nearly twice the cut-off) is only associated with a 90% certainty. Despite his high level of uncertainty, the I-Ag-FMD might still be assumed to be fit for purpose since its use is restricted to two particular situations: i) the confirmation of a positive cytopathogen effect in cell culture. In that case, the O.D. values usually exceed 0.6 even when a single small plaque ii) an emergency diagnosis on samples collected in an animal with clinical signs of FMD. Vesicles fluids from an acutely infected animal are also susceptible to contain high virus loads that produce high OD values largely exceeding the uncertainty of the test. Moreover, this emergency diagnostic test tends to be replaced by more sensitive methods like RT-PCR. In the current method, the calculation of the probabilities is based on the normal distribution of the residues of the ANOVA. For that reason, the normality has to be checked level by level and any deviation could theoretically impair the validity of the model. In practice, the scientist must particularly focus on the values around the cut-off and slight deviations might be accepted for low positive or low negative test results. The imprecision related to lack of normality might indeed be neglected when looking at samples with a probability /certainty of more than 99.9% like the highest concentrations of antibodies in the first serum dilutions analysed by C-Ab-FMD. The uncertainty of measurement as defined in the present article has the big advantage of being easy to understand by the client. Moreover the method provides the scientist with a probability of observing the same result if the test is conducted a second time in its. Such information can help him to decide to do the test again, confirm the result by another test or ask another lab to do the confirmation. The probability of observing the same test result when doing again the same test in similar conditions (same laboratory) could be interpreted as a kind of analytical predictive value. Nevertheless we do not recommend using that term (or the expression probability of positive test result ) since it might be confusing for the client. Indeed we must avoid that the client confuses the uncertainty calculated by the current method and the true (or diagnostic) predictive value, which is calculated from the sensitivity and the specificity of the test and from the prevalence of the disease. The current method is intended to work identically with any kind of ELISA. The suitability of the method was indeed demonstrated for a competitive and an indirect ELISA that differ largely among each other. The method was also conducted on many other ELISA tests conducted at VAR and it was suitable for any assay investigated. The present results also suggest that the method is suitable for tests with either fixed or variable cut-offs. Conclusions: We propose a method for the quantification of measurement of uncertainty that is suitable for ELISA tests The method conducts to a result that helps the scientist to take the decision about a test result The result is expressed in a way that is easy to understand by the client Recommendations: To ensure the validity of the analysis, it is advised to use samples that cover all the range of possible values for the test References: ISO/IEC (International Organization for Standardization /International Electrotechnical Commission) General requirements for the competence of testing and calibration laboratories. Mackay, D.K., Bulut, A.N., Rendle T., Davidson F. & Ferris N.P A solid-phase competition ELISA for measuring antibody to foot-and-mouth disease virus. J. Virol. Methods, 97: Goris, N. & De Clercq, K Quality assurance/quality control of foot and mouth disease solid phase competition enzyme-linked immunosorbent assay--part I. Quality assurance: development of secondary and working standards. Rev. Sci. Techn. OIE, 24:

8 OIE Foot and Mouth disease In Office international des Epizooties (OIE) / World Organisation for animal Healt, ed. Manual of Standards for Diagnostic Tests and Vaccines, pp EA-04/10 (European co-operation for accreditation) Accreditation of microbiological laboratories. ILAC-G Introducing the concept of uncertainty of measurement in testing in association with the application of the standard ISO/IEC Findlay, J.W., Smith, W.C., Lee, J.W., Nordblom, G.D., Das, I., DeSilva, B.S., Khan, M.N. & Bowsher, R.R Validation of immunoassays for bioanalysis: a pharmaceutical industry perspective. J. Pharm. Biomed. Anal.,21(6):