CEN/TC Some open issues Ari Karppinen Athens 6/2017

Transcription

1 CEN/TC Some open issues Ari Karppinen Athens 6/2017

2 How to use MQO in real life? Station representativeness Demonstration of eqivalence Minimum number of stations needed Focus area vs. validation area

3 Example:Modelling system - FMI Weather prediction models ECMWF HARMONIE (2.5) HIRLAM (7.5) LAPS;PALM Dispersion models - long-range, regional SILAM LRT, meso-scale, radioactivity, pollen HILATAR LRT, meso-scale Aerosol process models: SALSA(UH,UKU,FMI) SILAM-APMs Dispersion and effects models urban, local CAR-FMI, roadside UDM-FMI, urban OSPM (NERI), street canyon BUOYANT, fires ESCAPE, chemical accidents PALM, LESCAPE (DNS-code development) EXPAND (FMI, YTV) population exposure MPP-FMI, Meteorological pre-processing model

4 Ideal situation Validation area = whole country Measurements used for assesment of the modelling system = all stations available (~150) Focus area = any area inside the country Number of stations in focus area irrelevant (?!)

5 Typical situtation Different model for different regions Different/completly separate models for e.g. urban vs. rurual areas Challenge : number of stations available for evaluation (taking into account the type of model) very limited => definition of minum number of stations needed for model evaluation Suggestions: ~5, not based on any statistical jsutification, but more on experince/the real/existing status with measurements

6 Some additional comments Certain types of models are still automatically left out from this type of evaluation (even without discussing the problems with long term time series /statistics): e.g. CFD/LES modelling of urban areas UNTIL the measurement networks are able to support these very high resolution models.. Or.. eg. Wind-tunnel validation would be accepted as a feasible oprtion for Validating these

7 Data assimilation Models which take full use of the available measurement data will be of fastly growing importance everywhere <-> compare with opreative meteorological models : any reasonable model uses heavily data-assimilation Models utilizing data assimilation or data fusion need to have clear guidance for proper evaluation: multitude of different assessment methods (leave n-out/bootstrapping etc..) Good: don t need to invent anything really new Bad: not trivial to force a single evaluationmethod, which would be the optimal choice for all different DA/DF-modelling system Additional concern: lot of modelling sytems use hidden data-assimilation..e.g. bias correction -> should we jsut accept thtat this is feature of the model, or should we evaluate teh mdoel without bias-correaction?!

8 Operational modelling system For selected cities in Finland FMI-ENFUSER produces high resolution (15x15 m) AQ information - Includes AQ forecasts for the next hours - Model data accessible through a Web Portal Calibration update (1-4 times per year) Information extraction (every 10 minutes) FMI-ENFUSER (runs every 1-6h) Model output (figures, animations, gridded data, etc) Web Portal NO2, PM2.5, PM10, O3, AQI Amazon Web Service, HerokuApp, Dropbox 8

9 Online /real-time evaluaton

10 Some concerns about the delta-formulation: beta-parameter In order to assess and classify model quality in a scientific way, a criterion based only on β or β90 does not seem sufficient. There are many situations where an assessment using only β or β90 will lead to disputable or simply wrong classifications, compared to an assessment based on standard regression techniques. Requiring values for β or β 90 less than 2 in the MQO will result in more possibly wrong assessments.

11 Some concerns about the delta-formulation : normality of residuals) In practically all cases the tail of the distribution was long and sometimes also skewed. o o There is a larger possibility of outliers. The usual assumption that 95% of residues is between ±2 times the standard deviation may not hold. Determining the meaning of this finding for the MQO requires more tests and research

12 Some concerns about the delta-formulation ( temporal and spatial shift) NO2, Germany, ½year switch +12 h, switch PM10, Germany, ½year switch +12 h, switch O3, Germany, ½year switch +12 h, switch

13 Some concerns about the delta-formulation ( temporal and spatial shift) PM10, Germany, correct location vs. wrong location

14 Preliminary conclusions ( temporal and spatial shift) The shape of the diurnal/seasonal variation is crucial: different pollutants have completely different sensitivity to temporal shift : e.g O3 very sensitive whereas NO2/PM10 not for this specific tested shift Additional systematic analysis is needed e. g.: more drastic temporal shift e.g.randomly distribute the time dimension of the modeled concentration field more drastic spatial shift e.g spatial averaging of the modeled concentration field optimal stringency of MQI und MPI with respect to spatial and temporal shift should be clarified (long-term vs short-term, different pollutants)

15 Summary CEN/TC 264/WG 43 has still several interesting open issues to discuss, study & finally close Reasonable amount of studies & work & discussions still needed before we can propose the final suggestions for model evaluation standards Some of the open issued are highly relevant/tightly connected to FAIRMODE/DELTA tool development