Digitalization and Data Quality Control of Historical Data at ZAMG

Transcription

1 Digitalization and Data Quality Control of Historical Data at ZAMG Paul A.(1), Adler S.(1), Albenberger J.(1), Galavics H.(1), Lechner W.(1), Lipa W.(1), Manhard A.(1), Mandl A.(1), Teuschler D.(1) (1) ZAMG - Zentralanstalt für Meteorologie und Geodynamik

2 BACKGROUND Weather effects nature and people

3 BACKGROUND

4 BACKGROUND People are more informed about weather and climate People are more interested in weather and climate -> What is in range and what is left outstanding? -> Has this historically been a trend and can we determine trends based on this data? -> Is the climate, in general, getting more extreme? Investigation in climate change studies and analysis Conditio-sine-qua-non: Long term data in high temporal resolution Our task: Collection, digitalization, quality control of relevant data sets

5 PAST Complete data inventory of our ZAMG-Climate Data Archive includes: Austrian Climate Sheets (->Project Austrian Climate Rescue) Third Party Climate Data Sheets (primary countries that belonged to Austria in times of Austrian-Hungarian Monarchy) Historical Austrian Yearbooks ( ) Austrian Climate Data Rescue Initiative ( ) Outcome: -) Complete Data inventory of ZAMG-Climate Sheets -) Digitalization of over Climate Sheets as pdf (80% in DB) -) Data Quality Control of over 6000station years Continuous Improvement of quality control procedures ordinary tests to more sophisticated checks (DCT Software)

6 PAST Complete data inventory of our ZAMG-Climate Data Archive includes: Austrian Climate Sheets (->Project Austrian Climate Rescue) Third Party Climate Data Sheets (primary countries that belonged to Austria in times of Austrian-Hungarian Monarchy) Historical Austrian Yearbooks ( ) Climate Sheet Dobratsch 1878 August Regenmesser wurde von den mutwilligen Bleiberger= Knappen in der Nacht zerbrochen und verunreinigt important Metainformation: The glas usedforrain measurements was soiled and destroyed by a wilful bleiberger knave at night

7 PAST Complete data inventory of our ZAMG-Climate Data Archive includes: Austrian Climate Sheets (->Project Austrian Climate Rescue) Third Party Climate Data Sheets (primary countries that belonged to Austria in times of Austrian-Hungarian Monarchy) Historical Austrian Yearbooks ( ) We found stations in: Bosnia and Herzegovina Bosnia and Herzegovina Bulgaria Croatia Czech Republic Germany Greece Hungary Italy Romania Russia Slovenia (partly digitalized) Slovakia (partly digitalized) Switzerland Ukraine

8 PAST Overview of lost and found: information is given on data one country (y-axis) has in their archives from stations that are now located in another country (x-axis). Information is available for those cells in which a bold X can be seen.

9 PAST Complete data inventory of our ZAMG-Climate Data Archive includes: Austrian Climate Sheets (->Project Austrian Climate Rescue) Third Party Climate Data Sheets (primary countries that belonged to Austria in times of Austrian-Hungarian Monarchy) Historical Austrian Yearbooks ( ) Data inventory of historical yearbooks (station-wise, parameter-wise) Conception phase: Educational training and first test scans at geological federal institute using Bookeye (specialized image access scanner)

10 PRESENT + FUTURE Complete data inventory of our ZAMG-Climate Data Archive includes: Austrian Climate Sheets (->Project Austrian Climate Rescue) Third Party Climate Data Sheets (primary countries that belonged to Austria in times of Austrian-Hungarian Monarchy) Historical Austrian Yearbooks ( ) Digitalization and Data Quality Control Digitalization of historical yearbooks ( ) with book scanner (book eye/ zeutschel) into pdf Digitalization of pdf files with OCR Software correction of not plausible values due to transformation Data quality control with DCT (Multi-stage-quality control) correction of not plausible values due to meteorological aspects

11 PRESENT + FUTURE Historical Austrian Yearbooks ( ) Digitalization and Data Quality Control Data Quality Control of historical data Data Quality Control of (sub-)daily data with Multi-stage-quality control (DCT) performed step by step - new digitalized data - already checked data (with old methods) COMPLETENESS CHECKS CLIMATOLOGICAL CHECKS A B TEMPORAL CONSISTENCY CHECKS INNER CONSISTENCY CHECKS SPATIAL CONSISTENCY CHECKS STATISTICAL CHECKS C D E F

12 PRESENT COMPLETENESS CHECKS A test if all values of the stations are available complete datasets to calculate sums cloud amount-cloud density CLIMATOLOGICAL CHECKS threshold tests climatological basis checks B neg. tminin summer [06-09] under 1000m Plausibility checks: extreme values TEMPORAL CONSISTENCY CHECKS C test on constant values wind and freezing test on missing values no sun duration over a predefined period (considering astronomical components, effective sun duration) measurements of wind speed (cup-anemometer) and wind direction (weather vane)

13 PRESENT INNER CONSISTENCY CHECKS logical checks -test of different parameters of one time series nsnow>=0 and no prec nsnow>=0 and no snow depth nsnow>0 with a decreasing snow depth D #ERROR: F39 fresh snow existing, but no snow depth ---please correct #WARNING: W81 calculated fresh snow<>measured fresh snow ---please check

14 PRESENT SPATIAL CONSISTENCY CHECKS E Comparison of (sub-)daily values of one station (ZAMG observational network) with a variable number of reference stations (ZAMG and hydrographical observational network) predefined amount of reference sites weighting factor (height, distance) user-defined up to 5 sites weighting factors for temperature, precipitation, snow, pressure

15 PRESENT STATISTICAL CHECKS F quantile-tests standard deviation/ variance tests IN GENERAL: HANDLING tests produce ERROR/ WARNING-list ERRORS must be corrected, if confirmed an entry in the meta DB has to be done WARNINGS have to be checked Every value is flagged and stored in our DB IN GENERAL: TOOLS original records thermo-hygrograph stripes climate sheets radar, satellite,

16 FUTURE Implementation of more sophisticated checks Implementation of model data Continuous validation and enhancement of quality control criteria and routines Standardization of data quality control procedures (historical and recent data) FURTHER INFORMATION:

17 OUR TEAM THANK YOU FOR YOUR ATTENTION!!!! HVALA NA POZORNOSTI!!!!

18 Any questions? datum druck07 druck14 druck19 druckmit tmax tmin t7 t14 t19 t tf7 tf14 tf