Sewer Monitoring Projects: Data Collection, Data Handling and Data Quality in Arnhem H.J.Liefting 1 * and J.G. Langeveld 1,2 1 Royal Hasoning, Barbarossastraat 35, P.O. Box 151, 6500 AD, Nijmegen, the Netherlands 2 Delft University of Technology, Faculty of Civil Engineering and Geosciences, P.O. Box 5048, 2600 GA, Delft, the Netherlands *Corresponding author, e-mail e.liefting@royalhasoning.com ABSTRACT The municipality of Arnhem has set up a monitoring program to investigate water quality in storm sewers as well as the efficiency of several end-of-pipe solutions for stormwater treatment. Four pilot installations have been set up: a sandfilter, a lamella-separator, a soil filter and a helophyte filter. Flow rates and water levels are real-time measured. The obtained data are tested on reliability, accuracy and completeness by automated tests. Logical and statistical tests are used. During the project the necessity of data validation was proved. The detected problems appeared to be divers. Heavy noise in flow data was distinguished with a chec on autocorrelations. The total measured volume was largely distorted by the noise. Multiple errors in data logger software, causing faulty flow data, were discovered. Suspicious data were sometimes caused by unexpected disturbance of the conditions. Data validation must regularly tae place from the very beginning of a monitoring project to be alert on any of such disturbances. KEYWORDS Case Study, Data Collection, Data Handling, Data Quality, Data Validation, Stormwater Treatment INTRODUCTION In the last decade, water quantity sensors and to a certain extent water quality sensors have become better and cheaper. Combined with a rapid development of the information and communication technology this has led to a widespread application of continuous monitoring in sewer systems. The municipality of Arnhem has set up a monitoring program to investigate water quality in storm sewers as well as the efficiency of several end-of-pipe solutions for stormwater treatment. Flow rates and water levels are real-time measured. An automatic sampling system starts when runoff flow is measured. After the runoff period the samples are analyzed on water quality parameters. Unfortunately, a sewer is a harsh environment for sensitive measuring equipment. This may cause that the obtained data have a poor quality in terms of availability, reliability and accuracy. Therefore, data have to be validated. The current practice of data validation in the field of wastewater engineering is largely manual and visually based, often using spreadsheets. The application of continuous monitoring has caused the amount of data to grow greatly. Also the Arnhem monitoring project produces a lot of data. For about two years, Liefting et al. 1
every minute, at four locations, flow and water level data are sampled. After two years of measuring each sensor produced over one million data points. There are no standard methods for data validation (Bertrand-Krajewsi and Muste, 2008). It may cost great efforts to do analyses manually, if at all possible (Van Bijnen and Korving, 2008). For example, maing a table and plot in MS Excel is impossible, because the maximum number of rows in this program is exceeded by the amount of data. Therefore, data management in this project has been largely automated. The used calculation program for data validation is Matlab. The results of the validation show that validation was necessary indeed. METHODS Monitoring setup Four pilot installations have been set up: a sandfilter, a lamella-separator, a soil filter and (later added to the project) a helophyte filter. At these locations, the measured flow is used to control the automatic sampling system. An automatic sampling system starts when flow above a threshold is measured. Location sandfilter. The flow sensor at location sandfilter is a combined ultrasonic flow sensor (OCM PRO), with ultrasonic velocity measurement. This sensor uses a correlation technique to recognize reflecting particles and to determine their traveled distance in time. The flow sensor is installed in a 700 mm diameter sewer. The measuring range of the sensor is adjusted to 0 200 m 3 /h. Location lamella filter. The flow sensor at location lamella filter is an ultrasonic flow measuring system (Endress + Hauser PROMAG 50 W). This sensor uses a Doppler shift to determine the velocity in the fully submerged pipe. The flow sensor is installed in a 250 mm contraction at the end of a 700 mm diameter sewer. Location soil passage. The flow sensor at location soil passage is the same type ultrasonic flow sensor as at location lamella filter. The flow sensor is installed in a 400 mm contraction in a 800 mm pipe. Besides the flow in the sewer, also the water level in the soil passage is measured by pressure gauges. The used sensors are pressure gauges, type ATM/N. Location helophyte field. At the helophyte field the flow is derived from water level measurements at an orifice construction. In this case, the setup of the pilot installation was as shown in figure 1. All used sensors are pressure gauges, type ATM/N. Langeveld et. al., (submitted) provides more information about the setup of the monitoring program. 2. Automatic sampling influent 6. Manual sampling effluent after runoff period 3. Helophyte filter with level measurements (H) H H 1. Flow is derived from level measurements (H) before and after orifice H 4. Effluent drain pipe 5. Manhole with level measurement (H) 7. Surface water Figure 1. System layout of pilot installation helophyte filter H H 2 Sewer Monitoring Projects: Data Collection, Data Handling and Data Quality in Arnhem
Data collection and management The used sensors are connected to a Campbell Scientific CR200 logger at the pilot locations (CR2100 at location helophyte filter). These loggers are equipped with a GSM / GPRS transmitter for data transport to a receiver at the Royal Hasoning office in Nijmegen. The module has an alerting service via SMS. The data transmission is managed from the office with the Campbell software LoggerNet. The transmitted data are added to a data file per location on the connected computer. Depending on the weather circumstances and any SMS alerts, the data are plotted, analyzed, and stored in a bacup several times per wee to several times per day. After the analysis of the obtained measurement data, it is decided how any taen samples will be analyzed. Principles of data quality and data validation Bertrand-Krajewsi and Muste (2008) state that measurements are wrong until there are sufficient and objective reasons to admit that the data are representative and reliable. Data validation is obtaining such reasons. In general, data quality can be judged on 6 aspects (Liefting 2006): 1. Reliability. The reliability of a measurement is influenced by systematic differences between the actual state and the state as measured (Clemens, 2001). Due to various causes systematic errors can occur. 2. Accuracy. Accuracy is the variation range directly related to the measuring method and the measuring equipment applied (Clemens, 2001). Note that it is useless to improve accuracy when there are systematic errors. 3. Completeness. Completeness of the data is the amount of acquired data, compared to the amount that was aimed at in the measuring plan. 4. Actuality. Actuality of the data is the usability in the present of data obtained in the past. Changed conditions may have the effect that data are not actual any more. 5. Verifiability. Verifiability of data is the possibility of comparing the data with other information, lie other measured data or metadata. 6. Practical usability. Practical usability of the data may be influenced by the data format, the data being digitalized or not, the used date and time notation and similar characteristics. The obtained data in the Arnhem monitoring project are tested on reliability, accuracy and completeness by automated tests in Matlab. The tests are logical and statistical tests. The method of data transmission and filing maes the raw data immediately accessible, so they do not need (manual) pre-processing before validation. Actuality and verifiability are secured by eeping logboos and archiving all relevant information digitally. Logical tests for data validation The following logical tests are used in the automated data validation: Exceeding of boundary values. The chec for exceeding of boundary values is used to discover values outside the range of the measuring instrument and physically improbable values. Per location the boundaries are set by the analyst. Absence of data. The completeness of the dataset is checed calculating the intervals between the registrations. If the time registration is non-equidistant, data lac. Events with too small timescale. During dry weather the flow sensors in a storm sewer normally produce zero-values. Under wet weather conditions, the flow will become significant until the last runoff is transported and the system returns to dry weather conditions again. The duration of such a runoff pea is somewhere in between Liefting et al. 3
a quarter of an hour and multiple hours. Deviations from dry weather values smaller than 15 minutes are found with a logical test and mared. Statistical data validation. Autocorrelation chec. If for every time lag at least one pair of measured data separated by that lag exists, then the estimate of each covariance is (Hamilton, 1994; Liefting, 2006): T T 1 γ = ( yt µ ) It ( yt+ µ ) It+, T c t= 1 where is the sample mean; T is the total number of observations; is a coefficient defined as: c c = T t= 1 I I t t+ (Jones 1971); I is an indicator sequence defined as: I t =1 if y t is observed; I t =0 if y t is missing. The division / 0 is called the -th sample autocorrelation. A moving window is used to localize high and low correlations in the time series. The used method neglects missing data, while the observation times are still taen into account (Jones, 1971). Provided that the measuring interval is chosen small enough, the successive data (=1) in a time series should be significantly (positively) correlated. If no significant autocorrelation exists between successive data, this is either an indication that the real process has such a small timescale that the measuring interval is too large to record the process, or the data contain noise. In both cases, the quality of the data is low. Depending on the presence of periodical events (lie a daily pattern) data also are correlated for higher order. For storm water sewers no daily pattern should be expected in flow and water level data. Cross correlation chec. A cross correlation test is used to chec for a relation between data from different time series. An obvious example is the correlation between rain intensity and flow rates in the nearby sewer system. The cross correlation function for two time series (x and y) of equal length at a certain time lag is (Liefting, 2006): Γ T 1 = T c where c x, y, T ( xt µ x ) I x ( yt µ y ) t= 1+ T x, y, = I x, ti y, t+. t= 1 γ 0, x γ 0, y I y, The value of time lag at which the correlation is maximal, is determined by the hydraulic traveling time between the two measuring locations. 4 Sewer Monitoring Projects: Data Collection, Data Handling and Data Quality in Arnhem
RESULTS AND DISCUSSION During the monitoring project in Arnhem the necessity of data validation was proved. The detected problems appeared to be divers. Location sandfilter Almost directly from the start of the monitoring project, the flow sensor showed a signal as shown in figure 2. Flow data were not correlated with rain data. The time-scale of the measured flow peas is much smaller than the time-scale of storm weather runoff. The flow exceeds 100 m 3 /h, which is more than half the measuring range. Furthermore it is remarable that the wild peas stop when a recognizable runoff flow is passing the sensor. If the shown pea flows were to be a real process, we might see here a pattern of frequent voluminous discharges into the sewer system or fluctuations caused by waves entering the sewer from the canal. More liely however, is that the shown pattern is non-physical noise. Figure 2. Flow peas at location sand filter after the start of the monitoring period with precipitation in the same period. Figure 3. The flow data in figure 2 tested on autocorrelations. The size of the moving window is 30 observations. If the first autocorrelation 1 / 0 < 0.7, the data are mared with an x. Zero-values are neglected. The data have been tested with an autocorrelation test with =1 (figure 3). It appears that the shown pattern has no significant positive autocorrelation; the measuring value after 1 minute is not correlated to the measuring value before. Also longer-term autocorrelation does not Liefting et al. 5
exist. If the shown pattern is physical, it has a characteristic timescale smaller than 1 minute. That is practically impossible, because of the inertia of the water volume in the fully submerged 700 mm sewer. Thus, the shown pattern must be noise. The outcome of the validation is confirmed by a test on the location. The sewer was closed with clinchers, so that actual flow became impossible. The sensor continued to produce noise during the test. The total volume measured by the flow sensor is largely distorted by the noise. The total volume measured by the flow sensor between 7 September 2006 and 27 November 2006 during measured flows higher than 50 m 3 /h, is 6147 m 3. However, when the noise is removed, the total volume is only 963 m 3 (table 1). Table 1. Flow data location sandfilter between 7 September 2006 and 27 November 2006 divided in good data and noise, with test statistic for noise 1 / 0 < 0.7. absolute percentage good noise good noise number of measurements 95,929 20,963 82% 18% total measured volume (m 3 ) flow data Q 50 m 3 /h 963 5,184 16% 84% The the sandfilter installation and the sampling program start when the flow exceeds 50 m 3 /h. Since the noise peas regularly exceeded 50 m 3 /h, the installation often started needless. On 27 November 2006 a numerical dissipation is installed in the measuring equipment which damps the noise. The side effect is that the dissipated noise is now artificially autocorrelated. Thus it cannot be detected with the autocorrelation test any more. Location lamella filter With sensors and loggers, always software is included. The supplier of the equipment often adjusts this software. At location lamella filter, multiple errors were discovered in the software of the data logger, causing faulty flow data. Software error nr. 1. Soon after the start of the project on 3 July 2006, a fault was discovered, because the measured flows exceeded the confidence interval. Velocities over 6 m/s were reached in the measurements. The cause was a software error (table 2). On 7 September 2006 the software was corrected. Table 2. Small part of the software code in the logger of location lamella filter. Six flow measurements per minute should be averaged. But the measurements were totalized, causing the flow registration to be six times too high. old code corrected code DataTable (TS_Fast,1,-1) DataInterval (0,reg_tabel_1,reg_unit) Totalize (1,debiet_m3,False) EndTable DataTable (TS_Fast,1,-1) DataInterval (0,reg_tabel_1,reg_unit) Average (1,debiet_m3,0) EndTable Software error nr. 2. The flow sensor has a 4-20 ma analogue output. This output is linearly converted into a digital value. The conversion function is described by an offset value and a direction coefficient. Further all values smaller than a cutoff value are set to zero, leaving no negative values in the data. About 6 months after the start of the monitoring program a second fault in the flow sensor software was discovered by pure coincidence. It appeared that the offset value of the conversion function was incorrect. This fault caused all produced data to be 10 m 3 /h too low. Between and 8 September 2006 and 14 February 2007 a total volume of 6 Sewer Monitoring Projects: Data Collection, Data Handling and Data Quality in Arnhem
3,130 m 3 was measured by the faulty calibrated flow sensor. After a correction for the faulty offset value, the volume had become 4,986 m 3. Thus all measured values contained a structural error of averaged 37%. On 8 September 2006, after the first software correction, the flow sensor had been locally calibrated. This was done by pumping a fixed flow of 230 m 3 /h into an upstream manhole. This method appeared to be insufficient to detect a 10 m 3 /h bias in the measurements. Noise. Figure 4 shows flow data of location lamella filter during dry weather. This pattern has no significant autocorrelation for the first time step. The density spectrum of the data is grouped around 6 values, all multiples of 1/6 times the cutoff value of 7.5 m 3 /h. Since the data are means of 6 measurements per minute, the shown pattern is probably caused by 1 to 6 small exceedings of the cutoff value per minute. The conclusion is that these values are noise. Figure 4. Flow data during storm weather and dry weather at location lamella filter. Figure 5. The flow data in figure 4 tested on autocorrelations. The size of the moving window is 30 observations. If the first autocorrelation / 0 < 0.7, the data are mared with an x. Zero-values are neglected. Liefting et al. 7
Location helophyte filter Bertrand-Krajewsi and Muste (2008) distinguish two causes of anomalies in data: 1. the sensor response is unreliable or inaccurate under normal conditions, 2. the measurand (i.e. the phenomenon to be measured) is disturbed due to unexpected events. Data validation may lead to the outcome that not the data are faulty, but that the measurand is unexpectedly disturbed. At location helophyte filter, multiple examples of the last occurred. Level drift. Based on the system layout in figure 1 there was no trend expected for the water level in the manhole (5) at the effluent side of the filter. The surface water level should constantly be below the invert level of the pipe. However, the water level data in the first part of the monitoring period showed a trend (figure 6), even during dry weather. In first instance it was thought that a level drift of the sensor caused this trend. A field survey one wee later showed the real cause: the PVC pipe from the manhole to the surface water was closed with a lid under the surface water level. Thus the outlet of the filter was loced and the filter could not function hydraulically at all. If the suspicious data had not been discovered early in the project, this problem might not have been detected for a long time (figure 7 demonstrates the visibility of a small outlet under water level). Figure 6. Trend of effluent water level after the start of the monitoring project. Figure 7. The effluent pipe was closed with a lid under the surface water level. The removed lid is mared. 8 Sewer Monitoring Projects: Data Collection, Data Handling and Data Quality in Arnhem
Reverse flow. In the weeend following on the discovery of the closed effluent pipe, due to a pump failing, the nearby combined sewer was filled and ultimately overflowed into the pond near the helophyte filter. After the pond was filled with foul water it also entered the helophyte filter and the storm sewer. Due to the absence of staff during the weeend, the failure was discovered only on Monday (figure 8). After checing the measurement data, the possible cause was guessed well and Royal Hasoning informed the municipality about the incident. Figure 8. Removal of PVC lid on Friday 4 January. Filling of the pond as result of a sewer overflow in the following weeend. Discovery of overflow on Monday 7 January. On 8 January, the storm water sewer was emptied to remove the foul water. Flow without rain. In the same month, the measurements at the location showed a significant runoff pea (figure 9). Automatic sampling started and an SMS alert was sent. The strange thing was no single drop of rain was falling in the area. A direct telephonic inquiry learned that the flow was caused by the connection of a pond to the storm water sewer. Figure 9. Stormwater event without storm weather on 25 January CONCLUSIONS From the results of the different locations, the following conclusions can be drawn: Data validation is a crucial part of any monitoring project. Liefting et al. 9
Checing for autocorrelations appeared to be a good method to detect if certain measured values are noise. An accurate local calibration of a newly installed flow sensor is, though practically difficult, an absolute must. All used sensor software and adjustments of the used equipment have to be filed as metadata with the measured data, so that any signal conversion or other digital operations can be checed afterwards. Suppliers and installers of measuring instruments should be professionally distrusted (in our case, the supplier was a well-nown firm specialized in sensors for soil and water measuring). If measuring data appear to be suspicious, the cause may be unexpected disturbance of the measurand as well as sensor failing. Data validation must regularly tae place from the very beginning of a monitoring project, not afterwards, to be alert on any disturbances of the measurand. ACKNOWLEDGEMENT This monitoring program is initiated by the municipality of Arnhem. Water board Rivierenland is partner of the project. The project is supported by the EU Interreg program Urban Water. François Clemens (Delft University of Technology) contributed to the development of the validation tools during the corresponding author s MSc graduation. REFERENCES Bertrand-Krajewsi, J.-L. and Muste, M. (2008). Data validation: principles and implementation. In: Fletcher, T.D. and Deleti (eds.), Data Requirements for Integrated Urban Water Management, Urban Water Series, UNESCO. Butler, D. and Davies, J.W. (2000). Urban Drainage. Spon Press, London. Bijnen, M. van and Korving, H. (submitted). Application and results of automatic validation of sewer monitoring data. 11 th Int. Conf. on Urban Drainage. Clemens, F.H.L.R. (2001). Hydrodynamic models in urban drainage: application and calibration. PhD thesis, Delft University of Technology. Hamilton, J.D. (1994). Time Series Analysis. Princeton University Press, Princeton, New Jersey. Jones, R.H. (1971). Spectrum Estimation with Missing Observations. In: Annals of the Intitute of Statistical Mathematics, vol. 23, Kluwer, Toyo, pp. 387-398. Langeveld, J.G., Liefting, H.J. and Velthorst, H., (submitted). Storm water sewers: pollution levels and measured removal rates of storm water treatment techniques. 11 th Int. Conf. on Urban Drainage. Liefting, H.J. (2006), Validation of Water Quality Data from In-Sewer Measurements. MSc thesis, Faculty of Civil Engineering and Geosciences, Delft University of Technology. 10 Sewer Monitoring Projects: Data Collection, Data Handling and Data Quality in Arnhem