SDMX global conference of SDMX framework 2.0. Laura Vignola Italian National Institute of Statistics

Transcription

1 SDMX framework 2.0 Laura Vignola Italian National Institute of Statistics

2 SDMX framework 2.0 SDMX framework 2.0 SDMX Metadata framework (Mauro Bianchi Capacity building C) SDMX Data framework (Dario Camol Capacity building B)

3 SDMX Metadata Framework Istat production of questonnaires FMI (document) EUROSTAT STS (document) EUROSTAT ESMS (xml)

4 SDMX Metadata framework Store the reference metadata into a database Create a single access point to insert metadata for all the questionnaires Reuse the common information among various questionnaires

5 SDMX Metadata framework Manager web application (creates the environment using an xml file) Web application to insert data Desktop application to produce output Web service application Rss provider

6 <SDDS> <CONFIG> <CONNECTION> <STATUS>1</STATUS><VALUE> connection string;</value> </CONNECTION> <CONNECTION>...</CONNECTION> </CONFIG> <TEMPLATES> <TEMPLATE> <VALUE>1</VALUE> <AGENCY_NL>DQAF</AGENCY_NL><AGENCY_EN>DQAF</AGENCY_EN> <SDMX_STRUCTURE> <PK_SDMX> </PK_SDMX> <DESCRIPTION>Source data definitions, scope,..</description> <FK_SDMX>01.04</FK_SDMX> <FK_TEMPLATE>1</FK_TEMPLATE> <INTERFACE>1</INTERFACE> <COMMENTNL>...</COMMENTNL> <COMMENTEN>.</COMMENTEN> </SDMX_STRUCTURE> <SDMX_STRUCTURE>...</SDMX_STRUCTURE> </TEMPLATE> <TEMPLATE> </TEMPLATE> </TEMPLATES> <DECODE> <SDMX_DECODE> <PK_SDMX>03</PK_SDMX> <LEVEL> </LEVEL> </SDMX_DECODE> <SDMX_DECODE> </SDMX_DECODE> </DECODE> <DOMAIN> <DATA_FLOW> <PK_DATAFLOW>SSTSIND_ORDR_MS</PK_DATAFLOW> <DATAFLOW_NL>New order received </DATAFLOW_NL> <DATAFLOW_EN>New order received...</dataflow_en> <RSS_DESC /> </DATA_FLOW> <DATA_FLOW> </DATA_FLOW> </DOMAIN> </SDDS>

7 SDMX Metadata framework Output DQAF.doc SDDS.xml Manager STS.doc ESMS.xml FMI ESMS rss EUROSTAT STS Loader Web Service ESMS SDMX query

8 SDMX DATA framework Partecipation in several Eurostat projects(sodi, Demography, Census Hub) Istat scenario Time (Long terms solution, Short terms solution)

9 SDMX DATA framework Create an environment structured with SDMX elements Follow data from loading to dissemination

10 SDMX DATA framework Manager desktop application Loader desktop Application Web service Rss provider

11 SDMX DATA framework Manager Creates the environment Creates the tables for loading data

12 SDMX DATA framework Loader Loads datafile (csv, fixed length record, gesmes TS) Creates SDMX datafile and rss (pull method without WS)

13 SDMX DATA framework Web Service Query parser SQL generator SDMX Generator

14 Meeting of the OECD global Expert conference Group Statistical Data and Metadata Exchange Query parser Basic types of SDMX query nodes Time And Or Dimension Attribute Dataflow Data Provider Complex nodes Simple nodes

15 Meeting of the OECD global Expert conference Group Statistical Data and Metadata Exchange Query parser <Time> <StartTime> Date 1</StartTime> <EndTime> Date 2</EndTime> </Time> <And> <StartTime> Date 1</StartTime> <EndTime> Date 2</EndTime> </And> <And> <And> Cond 1, Cond 2 </And> </And> <And> </And> Cond 1 Cond 2

16 Meeting of the OECD global Expert conference Group Statistical Data and Metadata Exchange Query parser <Or> <Or> Cond 4, Cond 5 </Or> </Or> <Or> </Or> Cond 1 Cond 2 <And> Cond 1 <Or> Cond 2, Cond 3</Or> </And> <Or> <And> Cond 1, Cond 2</And> <And> Cond 1, Cond 3 </And> </Or>

17 <Query> <query:datawhere> <query:and> <query:dataflow>stsind_prod_m</query:dataflow> <query:dimension name="frequency">m</query:dimension> <query:dimension name="indicator">prod</query:dimension> <query:or> <query:and> <query:dimension name="adjustment">s</query:dimension> <query:or> <query:dimension name="sts_activity">n100da</query:dimension> <query:dimension name="sts_activity">n100c0</query:dimension> <query:dimension name="sts_activity">n100d0</query:dimension> </query:or> <query:time> <query:starttime> </query:starttime> <query:endtime> </query:endtime> </query:time> </query:and> <query:and> <query:dimension name="sts_activity">n100c0</query:dimension> <query:time> <query:starttime> </query:starttime> <query:endtime> </query:endtime> </query:time> </query:and> </query:or> </query:and> </query:datawhere> </Query>

18 Meeting of the OECD global Expert conference Group Statistical Data and Metadata Exchange Query parser AND START_TIME END_TIME DATAFLOW AJUSTMENT FREQUENCY INDICATOR STS_ACTIVITY SSTSIND_PROD_M S M PROD N100DA SSTSIND_PROD_M S M PROD N100D SSTSIND_PROD_M S M PROD N100C SSTSIND_PROD_M M PROD N100C0 OR Problems with long query

19 Query parser <And> Cond 1 <Or> Cond 2, Cond 3</Or> <Or> Cond 4, Cond 5</Or> </And> <And> Cond 1 <Or> Cond 2-3</Or> <Or> Cond 4-5</Or> </And> AGE CAS SEX GEO TOT TOT NST - NAP TOT IT - ITC - ITC1 - ITC11 - ITC11_ ITC11_ ITC12 - ITC12_ ITC12_ ITC3 - ITC31 - ITC31_ ITC31_ ITC32 - ITC32_ ITC32_009002

20 Meeting of the OECD global Expert conference Group Statistical Data and Metadata Exchange SQL generator START_TIME END_TIME DATAFLOW AJUSTMENT FREQUENCY INDICATOR STS_ACTIVITY SSTSIND_PROD_M S M PROD N100DA SSTSIND_PROD_M S M PROD N100D SSTSIND_PROD_M S M PROD N100C SSTSIND_PROD_M M PROD N100C0 SELECT METADATI_STS.*,SSTSIND_PROD.* FROM METADATI_STS,SSTSIND_PROD WHERE ADJUSTMENT= S AND FREQUENCY= M AND INDICATOR= PROD AND STS_ACTIVITY=N100DA AND TIME_PERIOD BETWEEN AND UNION SELECT METADATI_STS.*,SSTSIND_PROD.* FROM METADATI_STS,SSTSIND_PROD WHERE ADJUSTMENT= S AND FREQUENCY= M AND INDICATOR= PROD AND STS_ACTIVITY=N100D0 AND TIME_PERIOD BETWEEN AND

21 Meeting of the OECD global Expert conference Group Statistical Data and Metadata Exchange The architecture ESA DEM Manager rss EUROSTAT STS Query parser SDMX query csv Loader GES rss SQL generator FLR SDMX compact SDMX generator SDMX cross sectional SDMX compact SDMX cross sectional

22 Meeting of the OECD global Expert conference Group Statistical Data and Metadata Exchange Sql Mapping generator START_TIME END_TIME DATAFLOW AJUSTMENT FREQUENCY INDICATOR STS_ACTIVITY SSTSIND_PROD_M S M PROD N100DA STS_MAPPING_TABLE NOME_TABLE CONISTAT NOME_FIELD CATEGORIA TIPO VS ATECO U_MIS DATAFLOW ADJUSTMENT FREQUENCY INDICATOR STS_ACTIVITY OBS_STATUS OBS_CONF

23 Meeting of the OECD global Expert conference Group Statistical Data and Metadata Exchange Sql Mapping generator ESA Sdmx db DEM STS Manager rss EUROSTAT csv GES Loader Query parser SDMX query FLR SDMX cross sectional SDMX compact CONISTAT SQL generator SQL generator SQL Mapping generator SDMX generator SDMX cross sectional SDMX compact SDMX generator

24 Meeting of the OECD global Expert conference Group Statistical Data and Metadata Exchange Links for download