STS Use case SDMX implementation and validation services

Transcription

1 STS Use case SDMX implementation and validation services Item 5.3 of agenda Standards Working Group meeting 2017 Simo Pasi Eurostat, Unit G3

2 Contents I. Who are we? II. STS data transmissions III.SDMX-ML implementation IV. Validation of STS SDMX-ML files V. Conclusions 2

3 I Who are we? Eurostat (Directorate General of the European Commission) Mariana Kotzeva Acting Director General Directorate A:... Directorate G: Global business statistics Maria Helena Figueira Director STS team produces: 4 monthly news release Industrial producer prices Retail trade volume Industrial production Production in construction 4 other PEEIs Many more STS indicators, e.g. on labour STS Team Hakan Linden Team Leader... Unit G3: Business cycle, tourism and registers August Götzfried Head of Unit

4 II a. Data transmissions (04/ /2017) 4 domains, 39 different STS datasets 7040 data transmissions (incl. revisions) 5490 "first" data transmissions 4

5 II b. Eurostat transmissions per domain year to 09/2015 5

6 II c. STS data transmissions throughput time Data files are validated and fed on reception day and published in Eurobase at 11am the next weekday (if not under embargo); Data are transformed into lower frequencies, missing calendar and seasonal adjustment is performed, European aggregates and the rates are calculated. 6

7 II d. STS data transmissions - issues Files "re-feeding" rate: +25%; Reasons for re-feedings: Acceptance high revisions Manual embargo handling Addition of new time-series Rebasing Manual correction by Eurostat of coding errors in the files (few cases) 7

8 III a. SDMX-ML implementation - formats GESMES/TS Also called SDMX-EDI Used since the beginning of the STS Regulation Until end-2018 Based on UN EDIFACT Conversion from csv with Genedi SDMX-ML Can be used from January 2017 Based on XML Conversion from csv with SDMX Converter or created directly with SDMX Reference Infrastructure tools 8

9 III b. SDMX-ML implementation - DSD 01 FREQ 02 REF_AREA 03 ADJUSTMENT SEASONAL_ADJUST 04 INDICATOR 05 ACTIVITY FREQ REF_AREA ADJUSTMENT STS_INDICATOR STS_ACTIVITY STS_INSTITUTION BASE_YEAR TIME_PERIOD TIME_FORMAT OBS_VALUE OBS_STATUS OBS_CONF OBS_PRE_BREAK STS_INSTITUTION Not needed 06 BASE_YEAR BASE_PER 07 TIME_PERIOD 08 OBS_VALUE 09 OBS_STATUS 10 CONF_STATUS 11 COMMENT_OBS 12 EMBARGO_TIME 13 COMMENT_DSET 14 UNIT_MULT 15 UNIT UNIT_MEASURE 16 DECIMALS 17 TRANSFORMATION 18 TIME_FORMAT ESTAT+STSALL COMMENT_TS 20 PRE_BREAK_VALUE 9

10 III c. SDMX-ML implementation - guidelines GESMES/TS and SDMX-ML Available on CIRCABC 10

11 IV a. Validation of SDMX-ML files From CNA to Eurostat EDAMIS flow and file format STRUVAL Feedback in EDAMIS CONVAL 11

12 IV b. From CNA to Eurostat CNA Eurostat No STS xml STS ges EDAMIS Input hall XML? STRUVAL OK? fame2 prod Feedback on STRUVAL results No 12

13 IV c. EDAMIS flow and file format When sending a file over EDAMIS, need to be selected: o File o EDAMIS flow (=dataset). Any file type can be transmitted over EDAMIS, but only GESMES/TS, SDMX-ML and NOTE files are of interest for STS. 13

14 IV d. Feedback in EDAMIS (1/2) A validation report is made available over the EDAMIS feedback mechanism to the sender of the SDMX-ML file. 14

15 IV e. Feedback in EDAMIS (2/2) 15

16 IV f. STRUVAL The structure of SDMX-ML files (file extension.xml) is validated before files are forwarded to the STS team: against the data structure definition (DSD) ESTAT+STSALL+2.1 against the dataflow specific constraints, e.g. CR_STSIND_TURN_M (see "Cube Regions" in the Euros SDMX Registry for acceptable values) All other files, including GESMES/TS (.ges) and NOTES (.docx), by-pass STRUVAL 16

17 IV g. Issues in STS SDMX-ML files File type EDAMIS flow Data Structure Definition Header Time format Unit multiplier Decimals Absolute values Activity 17

18 IV h. CONVAL coming File name vs. contents Indices (prices) vs. absolute values (building permits) Consistency between fields: e.g. if UNIT_MEASURE='IX' then UNIT_MULT='0'and and BASE_PER in {2005,, 2016} Average of base year; averages of different presentations To be tested after National Accounts 18

19 V Conclusions The structure of STS SDMX-ML files is automatically validated with feedback over EDAMIS - and the files pass the same validation process as the GESMES/TS files Quality of SDMX-ML files is guaranteed 'ad hoc' solutions may be needed to get "bad" files through SDMX-ML is richer than GESMES/TS not all new fields are though checked and used A solution for recording embargo date and time implemented SDMX-ML implementation, with CONVAL, is expected to reduce the "re-feeding" of files (caused by coding errors and some inconsistencies in files) Full effect as from 2019 when GESMES/TS is abandoned 19