How to implement an SDMX infrastructure for dissemination and reporting Session 7: Capacity building technical stream Bengt-Åke Lindblad Eurostat Francesco Rizzo Istat 1
Agenda Overview of Eurostat's SDMX Reference Infrastructure Tutorial on how to set up an SDMX architecture through Eurostat SDMX Reference Infrastructure and related "plugins" Installation of tools, mapping of statistical concepts, transcoding between code lists. etc) Use of tools acting as client for access, browsing and presentation of data and metadata 2
Overview of the SDMX Reference Infrastructure 3
What is the SDMX-RI Universal framework for modern data provision and exchange Set of pick-and-choose reusable building blocks allowing a statistical office to expose data to the external world based on access rights Designed to provide data and structural metadata based on mappings to each organization's dissemination data warehouse Uses SDMX standards incl. one for Web Services 4
SDMX-RI building blocks Web Service - a service that accepts both SDMX Query messages for data and structural metadata. Web Client - a web application that allows users to create basic SDMX queries to be used for retrieving data from a Mapping Store database and dissemination databases using a SDMX-RI Web Service instance. Mapping Assistant - provides mapping information between SDMX artefact and the local data storage scheme. Test Client - meant to test the SDMX-RI building blocks and the Web Service endpoint. 5
Why use SDMX-RI Developed to simplify the exchange of data Provides standard software and components, allowing individual statistical organisations to interact and exchange their data using the same software and methodology Modular approach, use part or the entire infrastructure, extend it by adding new modules or modify it in any other way to suit their own purposes Developed in both Java and.net 6
Where is the SDMX-RI used As of Sept 2013 deployed in 23 EU countries Tested across EU Member States (2011 population census) Running in Mexico Expression of interest: The Caribbean, Latin America, OECD and Russia Autumn 2013: Expected to run in EU28 Member States Expected to run in EFTA countries In DSWS (Eurostat dissemination web service) 7
SDMX-RI development strategy 2013-2014 Systems must continually be adapted to the changing environment; otherwise their utility will progressively decline. Architectural changes (two step approach) Intermediate solution to solve performance & out of memory errors "Ultimate" solution to provide a common API and implement SDMX 2.1 Implement new user requests and correct defects Widen the scope and usage of SDMX-RI among data providers ESS.VIP-programme such as the ICT project Reuse for other statistical data collections DSWS (Eurostat dissemination web service) 8
Web Service Provider WS extension to SDMX 2.1 new interfaces SOAP Request 2.0 SOAP Request 2.1 REST Request 2.1 NSI_Service_2.0 NSI_Service_2.1 NsiRestService Controller (1)Structure Retriever (6)Common SDMX API (3)Data Retriever (streaming) (7)SDMX (7)SDMX 2.0 (7)SDMX API 2.1 Implementation Implementation Implementation 9
SDMX-RI development timeline JAVA SDMX-RI ultimate API Integrated v 0.9.25 SDMX v2.0 compliant JAVA SDMX-RI ultimate API v 0.9.25 + compression SDMX v2.1 compliant JAVA SDMX-RI ultimate + enhancements SDMX v2.1 compliant JAVA SDMX-RI intermediate +compression SDMX v2.0 compliant Jan.2013 Feb Mars Apr. May Jun. July Aug. Sept. Oct. Nov. Dec. 2013 Jan.2014.NET SDMX-RI ultimate API Integrated v 0.9.15 SDMX v2.0 compliant.net SDMX-RI intermediate +enhanced performance and compression SDMX v2.0 compliant.net SDMX-RI ultimate API Integrated v 0.9.25 SDMX v2.0 compliant.net SDMX-RI ultimate API v 0.9.25 +compression SDMX v2.1 compliant Mapping Store SDMX v2.1 compliant.net/java SDMX-RI ultimate + enhancements SDMX v2.1 compliant 10
How to create an SDMX infrastructure for dissemination and reporting Step by step live demonstration 11
Steps to be executed during the live demo 1. Present the scenario as part of the demonstration (some datasets already pre-mapped for the demonstration purposes) 2. Use client tools to interact with the Web Service (SDMX-RI) to browse, present and extract data and structural metadata. This action is performed using the following tools: SDMX Structural Metadata Browser (developed by Istat) Web Client application (part of the SDMX-RI) SDMX MS Excel Plug-in (developed by Istat) 3. Use the Mapping Assistant (SDMX-RI) and Test Client (SDMX-RI) to prepare a new dataset (Production in industry) to be exposed in SDMX through the Web Service (SDMX-RI) 4. Use client tools, like in point 2, to browse, present and extract the new dataset 12
The scenario Mapping Store DB SDMX Structural Metadata Browser Mapping Assistant Test Client Web Service Web Client Excel Plug-in Dissemination database (.STAT DB schema) 13
Mapping Assistant workflow STEP 1 MSDB Connection DDB Connection STEP 2 Load SDMX Structures (CategoryScheme, DataFlow, DSD ) STEP 3 Map local database schema (Dataset) STEP 4 Map local Concepts to SDMX ones (Mapping Set) STEP 5 Map local Codes to SDMX ones (Transcoding) Query Editor Custom Query DB Tables 14
Live demonstration (The slides that follow will only be used in case of technical problem) 15
Step 1 MSDB connection 16
Step 1 DDB connection 17
Step 2 Load SDMX Structures 18
Step 3: Map local database schema (Dataset) - Query Editor 19
Step 3: Map local database schema (Dataset) Custom Query 20
Step 4: Mapping local Concepts to SDMX ones Local concepts SDMX concepts - FREQ ATECO_2007 ACTIVITY2007 ITTER107 AREA_EE CORREZ ADJUSTMENT TIPO_DATO7 IND_TYPE - BASE_PER - MEASURE VALUE OBS_VALUE TIME TIME_PERIOD 21
Step 5: Mapping of local codes to SDMX ones (transcoding) Local CL_ADJUSTMENT Code Description R Row D Seasonally, not correct C Calendar Y Seasonally and correct SDMX CL_ADJUSTMENT Code Description N Row S Seasonally, not correct W Calendar Y Seasonally and correct Local SDMX Description R N Row D S Seasonally, not correct C W Calendar Y Y Seasonally and correct 22
Questions & Answers 23