Title: Case studies on statistical methods for assessing compliance. Version no.: Final Draft Date: 11 June 2012

Transcription

1 EUROPEAN COMMISSION DIRECTORATE-GENERAL ENVIRONMENT Directorate D - Water, Chemicals & Biotechnology ENV.D.1 - Water Status box Title: Case studies on statistical methods for assessing compliance Version no.: Final Draft Date: 11 June 2012 Circulation and received comments: Chemical monitoring and emerging pollutants (CMEP- sub-group of WG E): Věra Očenášková (CZ), Lars Sonesten (SE), Jon Gulson (UK), Eric De Wulf (BE), Gaston Charleir (BE), Jonas Rodhe (SE), Ruta Rimsa (LV), Lea Mrafkova (SK), Branislav Vrana (SK), Paula Viana (PT), Nijole Striupkuviene (LT), Karin Deutsch (AT), Lis Morthorst Munk (DK), Liubka Chepanova (BG), Borislava Nenkova (BG), P. Karachorska (BG) Activity Leaders: John Batty, Hannah Green (UK) Markus Lehmann, Anja Duffek (DE) 1

2 1. Purpose of paper This paper illustrates some key considerations for assessing compliance with MAC-EQS using intermittent sampling data and presents statistical methods to assess the level of confidence in failure with case study examples of their application. It summarises and concludes the work of task 3.1B of the Chemical Monitoring and Emerging Pollutants (CMEP) group. The aim of this task was to identify approaches to compliance checking with MAC-EQS that Member States can apply in accordance with the second point of Annex I, Part B of EQS Directive 2008/105/EC Member States may introduce statistical methods, such as a percentile calculation, to ensure an acceptable level of confidence and precision for determining compliance with the MAC-EQS. 2. Background The WFD aims to achieve a good chemical status for European surface waters, which will be achieved by protecting the populations of water organisms from chemical stress. In order to ensure that the aquatic environment and human health are adequately protected the precautionary principle is adopted when deriving any Environmental Quality Standard (EQS) values. EQS expressed as an annual average value (AA-EQS) protect against chronic long-term exposure effects of pollutants, while maximum allowable concentrations (MAC-EQS) safeguard against acute exposure. The MAC-EQSs have been derived in order to be protective against short-term pollution peaks where the AA-EQS may not provide sufficient protection. Exposure to short-term pollution may result from, e.g. intermittent releases of the chemical in question, varying concentrations due to seasonal differences in the use of a substance (e.g. plant protection products) or in the flow regime of the water body concerned. For the derivation of EQS predicted-no-effect concentrations (PNECs) are used, in which extrapolations from single-species toxicity tests to ecosystem effects are made by applying assessment factors. Generally the AF is 100, but for substances with a known mechanism of action and/or a ratio between chronic and acute effects smaller than 1, the AF was lowered to 10 (or in some cases even to 2) on the basis of mesocosm studies. In any discussion about how we judge compliance against water quality standards it is important to remember: (a) What type of monitoring regime we will be using and (b) What level of protection is implied by the standard. When making an assessment of MAC-EQS compliance from intermittent monitoring data the compliance is determined from a limited number of observations. This can only provide an approximate estimate of true compliance. The confidence of the compliance assessment depends not only on the measured results, but also on the monitoring frequency. The more data we have the more confident we will be in judging compliance. Continuous monitoring would increase the confidence of the compliance assessment but in most cases it is impractical and/or unaffordable. Environmental quality standards for surface waters are assessed for compliance usually by taking monthly spot samples according to the requirements of the WFD. In this respect the chemical monitoring is always a compromise between a sufficient coverage of samples in time and space to generate significant results with proper confidence and limiting the monitoring costs. The probability of any sample exceeding a particular concentration level depends on the sampling frequency (see Table 1 in Appendix I). Regimes which adopt a higher sampling frequency will always detect more fails if compliance is 2

3 judged as a straight MAC, and will appear to have poorer compliance performance than places with more infrequent sampling. This is important when compliance is used to make comparative judgements about status in different places. The illusion of improved performance can be generated by taking fewer samples. Recognising how confident we are about true compliance performance is also important when considering the response to exceedence to ensure decisions on legal action, and investment of time and money to secure compliance are appropriately directed. In view of the limitations to assess the chemical status of surface water UK made a proposal for the application of statistical methods for determining compliance with the MAC-EQS 1 and presented it at the first plenary meeting of the CMEP group in Bratislava in vember As laid down in the preamble (14) and Annex I, Part B, 2nd point of Directive 2008/105/EC statistical methods, such as a percentile calculation may be introduced to ensure an acceptable level of confidence and precision for determining compliance with the MAC-EQS. The proposal by UK suggests the adoption of an approach set out in ISO where a percentile approach coupled with an appropriate level of confidence of failure is used so that, depending on the number of measurements and the desired statistical confidence of failure, a certain number of monitoring results above the MAC-EQS would be allowed before judging a significant compliance failure. Details are described in Annex I. In this context the relation between ecological protection goals and different assessment procedures to evaluate the risk of pollution has been discussed within the WG CMEP. This societal debate regarding which protection goals to adopt and what level of risk to consider as acceptable led to the idea to perform A questionnaire inquiry regarding current practices of compliance checking a case study to compare different compliance approaches to real monitoring data, and a case study of comparative compliance across sites using different compliance approaches 3. Questionnaire In 2011 a questionnaire was completed by the members of the CMEP to provide an overview of current practices of compliance checking of monitoring results with MAC-EQS. Fourteen responses were returned. The detailed responses have each been uploaded to CIRCA, plus a tabulated summary of answers to the key questions (Annex III). All countries that responded have transposed the EQS Directive. At present 3 out of 14 Member States that replied use statistical methods (percentiles with an assigned confidence of failure) with 2 other countries planning to adopt this approach when determining compliance against MAC-EQS. The majority of the countries and remain unsure about the introduction of statistical methods in future. Two countries do not propose to use statistical methods. The recent inquiry amongst the Member States undertaken by CMEP showed that generally monitoring data on Priority Substances are collected on a monthly basis, or sometimes 6 times a year. More frequent sampling may be undertaken e.g. to estimate pollution loads and to achieve acceptable levels of confidence and precision especially for seasonally variable substances (in practice predominantly plant protection products). The level of knowledge about which substances tend to exceed the MAC-EQS will be improved through monitoring. This includes the magnitude, frequency and the true level of failure Maximum Allowable Concentrati 3

4 In general there are two main philosophies regarding the compliance checking against the MAC-EQS: high confidence of failure to the compliance in order to reach a small risk of false accusations followed by expensive programmes of action ( benefit of doubt ) high confidence to assess exceedences of MAC-EQS in order to protect the environment ( fail safe ) which may result in an increased monitoring frequency 2 4. Case study In a second step available monitoring data were used to illustrate the different assessment procedures and to better understand what the outcome of the application of different approaches to assess compliance against the MAC-EQS would be. All are given in Annex II. The case study with the example data set on Isoproturon reveals that assessment approaches with different levels of confidence will result in different compliance judgements from the same dataset. The preferred approach will depend on the desired protection goal and level of justification for different subsequent decisions and actions. 5. Discussion points In addition to the ecotoxicological considerations with regards to the level of protection the case study raises some general points: The choice of percentile and confidence levels depends on expert judgement. Applying the agreed statistical methods would bring consistency to the assessment of compliance which recognises sampling frequency and sampling error. What are the consequences (measures, controls) when the MAC-EQS is not complied? o more frequent sampling in order to verify and control the exceedence within the chemical status assessment and /or o cost intensive actions in order to improve the chemical status What percentile and level of confidence (see Annex I) are acceptable in order to assess the chemical status? Expert judgement (e.g. hydrology, pollutant inputs) is required to refine estimates of exposure and to adjust the assessment of environmental risk. Peak concentrations above the MAC-EQS are indications for intolerable short-term pressures on the surface bodies. If we apply percentiles which allow pollution peaks to be excluded from judging significant failure, o How shall the magnitude and impact of relevant significant pressures be estimated? o How shall the chemical status of water be improved? Exceedence is not only a question of when and to which magnitude pollution is taken place but also does it remain (how long) or repeat (how often) over a distinct period, does it depend on season etc. Such an event should be investigated in order to ascertain the causes. The monitoring design has to be adapted when indicated, e.g. by more frequent sampling. The risk of repeated short-term exposure should be assessed e.g. by additional monitoring. The total time window of potential effects during the exposure period (e.g. multiple application scenario of plant protection products) should be considered in concert with the time needed for recovery. 2 Kommentar_ MAC and Percentiles_UBA_ 4

5 When is a monitoring efficient and effective in order to assess the chemical status of a surface body as a reliable basis for appropriate measures and controls? 6. Recommendation to WG E Member States may introduce a percentile calculation to ensure an acceptable level of confidence and precision for determining compliance with the MAC-EQS. If they do so, approaches laid down in this paper should be considered. The appropriate percentile and level of confidence should be chosen to balance the risk of false accusations (benefit of doubt) and the risk of not recognising failure (fail-safe). In accordance with the rules regarding monitoring frequency set out in section of Annex V to Directive 2000/60/EC the choice and interpretation of a percentile calculation should be justified on the basis of expert judgement. Appendices I II III Maximum Allowable Concentrations and Percentiles Comparison of different methods to assess compliance against the MAC-EQS to a case study dataset Questionnaire + Summary of questionnaire returns 5

6 Appendix I: Maximum Allowable Concentrations and Percentiles For many substances, the statistical distribution of concentrations can be fairly uniform and the annual mean is well correlated with the probability that high concentrations occur within the year. Also, for many types of risk the actions to ensure compliance with an annual mean will also address risks across the full range of concentrations. In the example below the 95%ile is ~2 x the 50%ile, the 99%ile is ~ 3 x the 50%ile, and the 99.9%ile is ~ 4.5 x the 50%ile. Figure 1: Example distribution of concentrations Probability 95 %ile 99 %ile 99.9 %ile %ile Concentration (ug/l) The differing assessment factors used in the derivation of the MAC-EQS limits will mean that there is considerable variability in how far the MAC-EQS is above the annual average across different substances. This has important implications for how likely it is for the MAC level to be exceeded. If the short-term standards are expressed as a summary statistic for a high percentile this allows the calculation of the confidence of failure. This recognises the interaction between the sampling regime, the distribution of likely concentrations, and the level of protection within the standard. Applying statistical methods allows us to make better site by site comparative judgements by estimating how confident we are in such assessments. When we detect failure we want to be able to compare the significance of this across failing sites to know which are the most significant for targeting action to secure compliance. However, any face value failure would always be investigated locally to ensure that any intermittent polluting actions are being managed. The extent of face value failure across a number of sites in a region or country is also important to indicate whether better widespread source control is needed. Local targeting of more expensive and resource intensive interventions could be justified if we are confident from the compliance assessment that this is warranted. This would be based on a fair comparison between sites which recognises that the chance of detecting failure is linked to how frequently we sample. Expensive action should require that we can demonstrate with at least 95 per cent confidence that a standard is failed, particularly as the cost of extra monitoring to confirm such confidence will be trivial compared to the cost of the action to secure compliance. This section illustrates the options for the establishment of Look-up Tables in accordance with the provisions of the ISO The look-up table allows a judgement about the confidence of failure over one or more complete years. The look-up table below provides the proposed number of permitted exceedences in a given set of samples for a given level of confidence (e.g. 95% confidence). When judging compliance against a MAC-EQS any particular sample could pass or fail the limit. The statistical behaviour of the number of values above a fixed limit follows the binomial distribution (whatever the shape of the underlying distribution). The binomial applies whenever chance events have just 2 outcomes (e.g. Pass or Fail), and therefore, using the binomial distribution, we can calculate bounds on the expected number of fails depending on: a) the total number of samples taken and 6

7 b) the underlying acceptable probability of failure (e.g. 5 per cent of samples, or 10 per cent of samples, or maybe 1 per cent of samples). We are comparing the observed time spent in failure with that expected if our set of results were exactly complying with the required percentile which defines the expected probability that either a pass or a fail might occur. Choosing appropriate levels of confidence When making judgements about the true underlying quality from intermittent sampled data there is always a risk of drawing a wrong conclusion. This takes two forms:! Type I - wrongly concluding a site fails when really it passes! Type II - wrongly concluding a site passes when really it fails. By the application of different levels of confidence to our compliance judgements we can minimise either Type I or Type II errors. Using 95 % confidence that a site is truly failing, gives a 5% chance of wrongly concluding failure when it is really compliant. This is a 'BENEFIT OF DOUBT' approach and is appropriate if we only want a small risk of false accusations. For example, if rectifying action is very expensive compared with the implied cost of damage due to failure ( in many cases for environmental standards the cost of damage is only an elevated risk of damage not actual damage because of the way standards are set). Using 50 % confidence that a site is truly failing gives a 50:50 chance of wrongly concluding site fails when really it is compliant - this approach takes no account of statistics of 'sampling error', it is straight 'FACE VALUE' compliance. Using 5 % confidence that a site is truly failing gives only a 5% chance of incorrectly passing a site when it has really failed. This is a 'FAIL-SAFE' approach and would be appropriate if violation of the standard would have serious consequences. Statistical methods allow sensible comparative judgements by taking account of sampling frequency. The chances of detecting failure are intrinsically linked to sampling frequency. The probability of failure being detected or of escaping detection depends strongly on the number of samples (see table 1 below). The impression of failure depends not only on quality but also on the number of samples. We could manufacture the illusion of improved performance by taking fewer samples. The table below shows the probability of there being at least one sample failing the absolute standard across different sampling frequencies if we assume that actual quality is meeting different percentile levels. Table 1: Effect of sampling rate on probability of reporting failure of a MAC sampling frequency If quality is meeting the standard for this per cent of the time Probability of at least 1 failing sample quarterly % 18.5% 3.9% 2.0% 0.4% monthly % 46.0% 11.4% 5.8% 1.2% 2 x a month % 70.8% 21.4% 11.3% 2.4% weekly % 93.1% 40.7% 22.9% 5.1% 2 x per week % 99.5% 64.8% 40.6% 9.9% 4 x per week % 100.0% 87.6% 64.7% 18.8% daily % 100.0% 97.4% 84.0% 30.6% hourly % 100.0% 100.0% 100.0% 100.0% When we do detect failure we want to be able to compare the significance of this across failing sites to know which are the most significant, and therefore where to focus efforts through site specific investigation and protective action. The table below shows how this comparison might look if we had different numbers of failing samples for different sampling frequencies. Results are presented for a range of different percentiles to describe the allowable time that quality can exceed the standard. 7

8 Table 2: Probability of failure for varying percentiles and sampling frequencies Yellow highlights 50% confidence of failing, Red highlights 95% confidence of failing Number Samples Number Fails If quality is meeting the standard for this per cent of the time % 81.5% 96.1% 98.0% % 98.6% 99.9% 100.0% % 100.0% 100% 100% % 73.5% 94.1% 97.0% % 96.7% 99.9% 100.0% % 99.8% 100% 100% % 100.0% 100% 100% % 100% 100% 100% % 54.0% 88.6% 94.2% % 88.2% 99.4% 99.8% % 98.0% 100% 100% % 99.8% 100% 100% % 100% 100% 100% % 29.2% 78.6% 88.7% % 66.1% 97.6% 99.4% % 88.4% 100% 100% % 97.0% 100% 100% % 99.4% 100% 100% % 100% 100% 100% % 6.9% 59.3% 77.1% % 25.9% 90.4% 97.2% % 51.5% 98.5% 99.8% % 73.8% 100% 100% % 88.3% 100% 100% % 95.5% 100% 100% % 98.6% 100% 100% % 99.6% 100% 100% % 99.9% 100% 100% % 100.0% 100% 100% The tables on the following pages provide the simplified look-up table summary of how many failing samples would be needed for different sampling frequencies and different confidence levels to judge a site to have failed a percentile standard. 8

9 Look-up tables for a 90 percentile standard If a site was truly complying for 90 per cent of the time the probability of any one sample passing would be 0.90 and the probability of failing would be % confident of failure 'BENEFIT OF DOUBT'. Allowed samples fails 50 % confident of failure 'FACE VALUE'. Allowed samples fails 5 % confident of failure 'FAIL-SAFE'. samples Allowed fails Look-up tables for a 95 percentile standard If a site was truly complying for 95 per cent of the time the probability of any one sample passing would be 0.95 and the probability of failing would be % confident of failure 'BENEFIT OF DOUBT'. Allowed samples fails 50 % confident of failure 'FACE VALUE'. Allowed samples fails 5 % confident of failure 'FAIL-SAFE'. samples Allowed fails 9

10 Look-up tables for a 99 percentile standard If a site was truly complying for 99 per cent of the time the probability of any one sample passing would be 0.99 and the probability of failing would be % confident of failure 'BENEFIT OF DOUBT'. Allowed samples fails 50 % confident of failure 'FACE VALUE'. Allowed samples fails 5 % confident of failure 'FAIL-SAFE'. samples Allowed fails Look-up tables for a 99.5 percentile standard If a site was truly complying for 99.5 per cent of the time the probability of any one sample passing would be and the probability of failing would be % confident of failure 'BENEFIT OF DOUBT'. Allowed samples fails 50 % confident of failure 'FACE VALUE'. Allowed samples fails 5 % confident of failure 'FAIL-SAFE'. samples Allowed fails Look-up tables for a 99.9 percentile standard If a site was truly complying for 99.9 per cent of the time the probability of any one sample passing would be and the probability of failing would be % confident of failure 'BENEFIT OF DOUBT'. Allowed samples fails 50 % confident of failure 'FACE VALUE' 5 % confident of failure 'FAIL-SAFE'. samples Allowed fails. samples Allowed fails 10

11 Appendix II: Comparison of different methods to assess compliance against the MAC-EQS to a case study dataset The application of a percentile with an assigned confidence of failure is in particular relevant for those cases, where the AA-EQS is complied and the exclusion of one or more fails to the MAC-EQS in the given dataset of at least 12 samples leads to an assessment where the sites than pass the MAC-EQS. Isoproturon is an example of a seasonally variable substance, which shows peak concentration within short time periods from its use in agriculture as a cereal herbicide. Pollution by Isoproturon is caused by both diffuse and point sources. Emissions to water come mostly from surface runoff and farm point sources (runoff from farmyards, storage facilities), and a lesser extent from field drainflow and spray drift during field application. Here we present monitoring data of Isoproturon in water collected from a surface body with a catchment area of approx km². Additional intensive measurements by daily taken mixed samples showed that observed short term pollution peaks occur over days and correlate with the hydrological regime (these data are not shown in figure 1). Figure 2: Monitoring data on Isoproturon in surface water with a frequency of once per four weeks from 2005 to 2010 During the study period every four weeks surface water samples were taken resulting in 13 samples per year (see figure 1). For the majority of water samples (67 %) the concentration of Isoproturon was smaller than the limit of quantification of 0.05 µg/l. The calculated annual average concentrations comply with the AA-EQS and 2007, one measurement result per year exceeds the MAC-EQS, which corresponds to a value of 1 µg/l. These observations resulted in appropriate measures and source controls in order to reduce the short-term emissions. As a consequence the peak concentrations decreased in the following years and the chemical status of the water regarding Isoproturon improved. In the following summary the three approaches to check compliance against the MAC-EQS are compared to the already mentioned case study dataset. Approach I compares every monitoring result against the MAC-EQS. This means for the case study dataset two measured concentrations exceeds the standard the monitoring site fails the quality standard in these two years. The compliant results of approach II and III conclude that there is no failure or even a risk of failure in particular in case of approach III. 11

12 The case study with the example data set illustrates that assessment approaches with different levels of confidence result in deviating compliance assessments of the same dataset. Table 2: Comparison of different methods to check compliance against the MAC-EQS to real Isoproturon monitoring data (presented in Fig. 1) Year. samples. Samples < LOQ (0.05 µg/l) Annual Average (EQS 0.3 µg/l) Max. Conc. (EQS 1 µg/l). Samples > MAC-EQS Compliance Approach I II III IV no percentile 0 fails allowed per year 90%ile 50% confidence of failure 1 fail allowed per year 95% confidence of failure 2 fails allowed per year 95%ile 50% confidence of failure 0 fails allowed per year Fail Comply Comply Fail Fail Comply Comply Fail All years comply AA-EQS 2 years fail All years comply All years comply 2 years fail Comment by Sweden: Statistically speaking, the reasoning is probably OK, but we lack any kind of discussion on a strategy for sampling/monitoring. The statistical method is of lesser importance if you do not control what to monitor. A pesticide used only for a limited time of the year seems pointless to search for throughout the year, but rather the time around its use in the fields. If you, instead, insist on taking samples on a monthly basis as in the example you "dilute the maximum concentrations" and will therefore have a skewed picture of the exposure to the organisms exposed to the substance. If you take away the "outliers" by making use of percentiles, "then you get rid of the problem", i.e. no serious exposure!? We assume that this example comes from some form of regular monitoring of multiple substances, but it highlights that you have to know what you want to monitor, i.e. if the exposure is more in the form of short pulses so that you can expect acute effects, or if the exposure is chronic. Also, the length of the acute exposure is of interest to estimate the dose. It could also be of interest if the monitoring object is a lake or a stream (i.e. if the turnover time is short or long). 12

13 Comparison of different methods to assess compliance against the MAC-EQS across a number of sites Table 3: Number of sites with reported failures preliminary assessment across England & Wales (UK) List Chemical Waters MAC- EQS (µg/l) MAC / AA. of Sites MAC (100%ile) 13 % of sites failing 95 percentile 99 percentile 50% conf 95% conf 50% conf 95% conf UK SP Cypermethrin All % 100% 100% 100% 100% EC PHS Tributyltin Fresh % 22% 5.0% 22% 8.9% UK SP Diazinon Fresh % 21% 7.6% 24% 9.1% UK SP Chromium Fresh % 2.8% 0% 2.8% 0.7% EC PHS Mercury Fresh % 5.1% 0% 5.1% 0.5% EC PHS Tributyltin Saline % 19% 5.6% 19% 15% EC PHS Cadmium Fresh % 3.3% 2.1% 3.3% 2.1% UK SP Cyanide (Free) Fresh % 24% 10% 24% 17% EC PHS Hexachlorocyclohexane Fresh % 3.9% 1.1% 3.9% 1.7% EC PHS Benzo(a)pyrene Fresh % 1.1% 0.0% 1.1% 0.0% EC PHS Hexachlorocyclohexane Saline % 2.5% 1.2% 2.5% 2.5% EC PS Isoproturon Fresh % 0.0% 0.0% 3.2% 0.0% UK SP Total Available Chlorine Fresh % 100% 100% 100% 100% UK SP 2-4-D Fresh % 1.5% 0.0% 1.5% 1.5% EC PHS Hexachlorobenzene Fresh % 0.6% 0.0% 0.6% 0.0% EC PHS Mercury Saline % 0.7% 0.0% 0.7% 0.0% The yellow highlighted results show where the compliance judgement changes when using a percentile approach compared to an absolute MAC allowing zero fails. The orange highlighting shows where the compliance judgement changes when different percentile and confidence levels are applied. The pink highlights substances where many of the fails are due to the analytical limit of detection not being sensitive enough. The UK approach and advice for applying statistical methods is that the 95 percentile is the most appropriate percentile with compliance assessed for both the 50 and 95 percent levels of confidence. This is appropriate given the typical, at best, monthly monitoring frequencies. Use of the twin levels of confidence ensures that single exceedences do not get ignored as these are identified as low confidence failures prompting further investigation and can result in actions to address peak concentrations (of the sort referred to in the previous example). It is also helpful to have an understanding of the extent of low confidence failures across a country or region to identify general compliance problems and the potential need for widely applied actions. The 95 per cent confidence level clearly differentiates the highest priority sites where action is needed to improve status.

14 Appendix III: QUESTIONNAIRE CHEMICAL MONITORING AND EMERGING POLLUTANTS (CMEP) TASK 3.1B: CASE STUDIES ON STATISTICAL METHODS FOR ASSESSING COMPLIANCE QUESTIONNAIRE Current practices of compliance checking of monitoring results with Environmental Quality Standards (MAC-EQS) Please answer to the following questionnaire and send the response no later than 20th May 2011 to Country: Contact person: 1. Has the Directive 2008/105/EC already been transposed into national law in your country? Yes If, please can you advise when transposition will be completed? 2. Do you currently use statistical methods for checking compliance of the MAC-EQS? Yes If Yes - Please refer to all relevant paragraphs in your national law regarding compliance checking of the MAC-EQS and application of statistical methods for compliance checking as the case may be! (Exact wording or indication of paragraphs if providing the legal act as an additional file): 3. If - Do you intend to introduce statistical methods for checking compliance of measurement results with MAC-EQS in the future? Yes (Every measurement value should comply with the MAC-EQS.) We are presently not sure about the introduction of statistical methods. If Yes Do you intend to introduce?: A 95-percentile with 95% confidence of failure (as suggested by UK at CMEP plenary meeting in Bratislava 3 in vember 2010, based on ISO ) Maximum Allowable Concentrati 14

15 Another percentile and/or another confidence level (based on ISO ), please specify: A percentile without confidence level, please specify: Other, please specify: 4. Please describe succinctly the monitoring practice (regarding frequency of monitoring, time of monitoring, additional monitoring during different seasons of the same year) in order to check compliance with the MAC-EQS! Please indicate if practices differ for a specific substance or substance group (e.g. plant protection products) where there are seasonal applications/uses! 5. Please describe succinctly your experiences in checking compliance with the MAC- EQS, particularly with regard to the application of statistical methods! 6. Do you have seen any exceedances of the existing MAC-EQS for priority substances in your country? Please specify which Priority Substances tend to fail the MAC-EQS and if possible indicate the magnitude/frequency of exceedences observed! 7. Please can you advise whether your laboratory (ies) is (are) currently able to meet the minimum requirements for QA/QC for priority list substances as set out in 2009/90/EC? Yes If no please can you indicate the substances where your methods are not compliant together with an indication of timeframe to achieve the minimum requirements of 2009/90/EC? 8. Do you have any other observations that you wish to make regarding this issue? 15

16 EUROPEAN COMMISSION DIRECTORATE-GENERAL ENVIRONMENT Directorate D - Water, Chemicals & Biotechnology ENV.D.1 - Water Brussels, 22 June 2011 ENV.D.1 2ND CHEMICAL MONITORING EMERGING POLLUTANTS PLENARY MEETING PRAGUE 30 JUNE 2011 About the document: AGENDA ITEM 3: STATISTICAL METHODS FOR COMPLIANCE CHECKING OF ENVIRONMENTAL QUALITY STANDARDS MAXIMUM ALLOWABLE CONCENTRATIONS AND PERCENTILES. The attached document was prepared following the exercise to seek Member State views on their experiences with the use of statistical methods for the purposes of compliance checking against maximum allowable concentration environmental quality standards (MAC-EQS). Members of the Chemical Monitoring and Emerging Pollutants Group are requested to take note of this proposal and prepare a recommendation for Working Group E. BACKGROUND At the first plenary meeting of the Chemical Monitoring and Emerging Pollutants Group, in Bratislava in vember 2010, the UK presented a paper setting out a proposed solution for the question of determining compliance against MAC-EQS values. This document advocated the adoption of an approach that is set out in the International Standard ISO Water quality Sampling Part 20. Compliance is assessed using a statistically - based methodology. In such an approach a look-up table is established as the basis of compliance so that countries taking many monitoring samples are not penalised against countries that take fewer samples. A final decision on the suitability of this approach was not made at that meeting but several Member States subsequently undertook further discussions on the subject in an attempt to find an acceptable outcome. THE QUESTIONNAIRE Earlier this year a questionnaire was developed to enable Member States to take the opportunity to set out their experiences of compliance checking. The questionnaire was circulated to members of the CMEP group and twelve completed forms were returned. The detailed responses have each been mounted on CIRCA but a tabulated summary is attached that records the answers to key questions. It is recommended that members of the CMEP group review the individual responses as it is not possible to fully summarise their content 16

17 ANALYSIS It is clear from the returns received that: i) Three countries already favour the adoption of a statistical approach while one further country intent to use it in future. However several different percentile values were suggested. ii) All but one country has transposed the EQS Directive iii) Most countries that responded do not have experience of the use of statistical methods when determining compliance against WFD MAC-EQS standards. iv) Most countries that responded have not yet formed a judgement over the use of statistical methods. v) There are a range of monitoring practices and compliance regimes in place that are set out in the individual returns, and the level of recorded failure is very variable vi) However the most startling conclusion is that all but one of the countries that responded could not meet all the requirements of the QA/QC Directive. While this is difficult to summarise it is clear that most countries cannot identify the true levels of failure that they carry for a number of listed substances. This also means that the number of failures reported is therefore artificially low, and that higher levels of failure can be expected as countries move to secure QA/QC compliant methods of analysis. Furthermore the perception that current levels of failure are representative and sufficiently indicative of chemical quality should be a source of concern. 17

18 SUMMARY OF QUESTIONNAIRE RETURNS 2011 Country Question 1 Question 2 Question 3 Question 3.1 Question 4 Question 5 Question 6 Question 7 Question /105/EC Do you have any Other observations Transposed? exceedences for Yes/ existing MAC-EQS? Do you use SM for MAC- EQS compliance? Yes/ If no Do you intend to use SM in the future? (Yes / / Unsure) If yes What percentile /confidence level do you intend to introduce? Austria Yes Yes - 90%ile 1in 12 allowed to exceed Belgium (Flemish region) Belgium (Wallonia) Describe monitoring practice for MAC- EQS Monthly sampling one year period Yes Unsure - Monitoring practices comply with the Annex V-1.3 from the WFD. Yes (every measuremen t must comply) Bulgaria Yes Unsure at present but will use them in the future Czech Republic - 6times per year and a periodicity of 6years Unable to say at present Yes Unsure 95%ile 95% conf In accordance with WFD requirements Denmark Yes Unsure - Germany Yes - 12 per year monthly full Describe experience in checking compliance with SM Transpositio n in Jan 11 so no real experience Statistical methods are for the time being not used We don t use SM to check MAC- EQS experience using SM Each value is compared with the MAC-EQS t normally used See return for Italy Yes Unsure - monthly application of statistical methods Latvia Yes Unsure - 12 per year monthly experience experience See Section 1.1 of document attached to return In period most problematic are IPU, diuron and benzo(a)pyrene Unable to comment at present Unable to comment at present Very few exceedences TBT and Hg Frequent failures for IPU, Hg, TBT and locally Cd Some other failures see return for Can your laboratories meet minimum QA/QC requirements? Yes/ TBT pesticides and PCP in GW Data not available Yes - Exceedences for TBT It would be good to apply statistical methods and to standardize practices for all countries for comparability. It would be good to apply statistical methods and to adjust and standardize these practices for all countries in order to have comparability

19 Country Question 1 Question 2 Question 3 Question 3.1 Question 4 Question 5 Question 6 Question 7 Question /105/EC Do you have any Other observations Transposed? exceedences for Yes/ existing MAC-EQS? Do you use SM for MAC- EQS compliance? Yes/ If no Do you intend to use SM in the future? (Yes / / Unsure) If yes What percentile /confidence level do you intend to introduce? Describe monitoring practice for MAC- EQS Lithuania Yes Unsure - Portugal Yes Unsure - Monitoring every two months during two year period. For PPP a program set according to the crops spring time applications. Slovak Republic Describe experience in checking compliance with SM experience Exceedences for Cd and Hg in Baltic - MAC-EQS exceedences Yes Yes Yes 90%ile 12 per year monthly problem Exceedences for Cd and Hg (LoQs non compliant) Sweden Yes Unsure TBT, NP and Cadmium based only upon an initial review United Kingdom Yes Yes Yes 95%ile 95% conf Summary 13 countries responded including 2 regional returns from Belgium Do you use SM for MAC- EQS compliance? Yes/ If no Do you intend to use SM in the future? Yes//Unsure If yes What percentile /confidence level do you intend to introduce? See return for Can your laboratories meet minimum QA/QC requirements? Yes/ Laboratory capacity an issue- contracted out TBT Question 1 Question 2 Question 3 Question 3.1 Question 4 Question 5 Question 6 Question 7 Question /105/EC Transposed? Yes/ Other observations Yes: 13 : 1 Yes: 3 : 11 Yes: 2 : 2 Unsure: 9 answer: 1 95%ile: 2 90%ile: 2 Unsure: 1 answer: 9 Describe monitoring practice for MAC- EQS Describe experience in checking compliance with SM Do you have any exceedences for existing MAC-EQS Can your laboratories meet minimum QA/QC requirements Yes/ Yes: 1 : 13 19