SDMX implementation in Istat: progress report Meeting of the Expert Group on Statistical Data and Metadata Exchange 27-30 October 2014, Korea Francesco Rizzo Department for Integration, Quality, Research and Production Networks Development (DIQR)
Aim of presentation Stat2015 Istat modernization multiannual program Enterprise Architecture in Istat SDMX strategy in Istat Istat SDMX Architecture Funding Collaboration with other organizations Streamlining the internal workflow Capacity building actions SDMX in the National Statistical System SDMX in production state of art
Stat2015 Istat modernization multiannual program Industrialization and standardization of the statistical processes in order to overcome the stovepipes statistical production model Harmonization of statistical information Foster the efficiency and reuse of processes, and the reuse of collected data Raising the quality of statistics and reduce the statistical production costs
Stat2015: Enterprise Architecture in Istat Defines what the industry does and how it is done: Business Function Business Process Business Service Business Architecture Conceptual layer GSBPM Describes the information, its flows and uses across the industry, and how that information is managed Information Architecture Logical Layer SDMX GSIM DDI Other CSPA Describes the set of practices used to select, define or design software components and their relationships Application Architecture Application Layer SOA, WS, Tools Describes the infrastructure technology underlying (supporting) the other perspectives Technical Architecture Physical Layer Web Service and Databases on load balancing/business continuity
Stat2015: SDMX Istat Strategy
Implementing the SDMX strategy (2013-2015) 1. From a tactical to a strategic approach 2. Design a suitable architecture that could satisfy the whole SDMX strategy; 3. Looking for the necessary funds to support the implementation; 4. Collaborating with other organizations; 5. Streamlining the internal workflow; 6. Capacity building actions;
From a tactical to a strategic approach SODI Demography EGR Census Hub OECD SIS - CC ESSnets collaborative sharing of knowledge and software SDMX for standardization and industrialization (Stat2015) getting experience from ESS pilot exercises SDMX Istat Strategy beyond the data exchange
The overall architecture SUM RSS feed Eurostat National Accounts Reference Metadata Structural Metadata I.Stat Data warehouse Structural Metadata Web GUI Tablet Smartphone Excel Plug-in Eurostat ICT survey Eurostat Census Hub OECD STES SDMX-RI SEP Pull Bank of Italy... Subject matter domain production units RDF channel Google DSPL channel Push JSON channel Private enterprises Researchers Other Italian Institutions
Financing the implementation: Eurostat ISTAT grant Total estimated cost: 464.983,87 Duration: 24 months (Extended to August 2015) 24 deliverables split among 4 work packages: WP1 Coordination WP2 Enhancement of the SDMX-RI, and its integration into the Istat information system WP3 Enhancement of the Istat Metadata Management System WP4 Contribution to the development of SDMX standard and implementation of capacity building actions Involved: 2 Directorates, 3 Services and 7 Units (19 internal people involved for a total of 778 man/days) > 1000 man/days subcontracting
Collaboration with other organizations OECD SIS-CC Statistical Network ISTAT INEGI (MEX) SDMX-RI User group SDMX TWG/SWG
Streamlining the internal workflow: Centralized Vs Decentralized Eurostat DSDs Subject Matter Domain actions Reporting Mapping (MA) Production DBs (macro) Reporting (Push) OECD (STES) IMF (SDDS+) DSDs Reporting Mapping (MA) Metadata and Mapping Repositories Dissemination Data modelling (DSW) Mapping (MA) Centralized actions TCI SDMX WS I.STAT (macro) Reporting (Pull) Dissemination (Pull)
Capacity building actions (2014) Internal training Introduction to SDMX 0.5 day 2 sessions SDMX Data modeling 2 days 2 sessions How to use the SDMX-RI 3 days 3 sessions Programming SDMX 5 days 1 session Other actions Workshop on SDMX Source.NET 3 days 10--12 March 2014 How to use the SDMX-RI for reporting to International Organizations (OECD SDMX Expert Group) 2 days 29-30 October 2014 ABS-ISTAT-SNZ 15 days 24 Nov 5 Dec 2014
SDMX in the National Statistical System SISTAN Hub Objectives and phases Disseminate from a single web application data stored in a distributed environment (data are not stored centrally but grabbed from the peripherals nodes upon request of the end users) End users could access and compare harmonized datasets (DSDs will be defined centrally by Istat), or navigate data that refers to a specific peripheral node (e.g. a Region or another SISTAN Institution) Proof of concept already started: Lombardi and Sicily Region involved Reused/adapted the SDMX-RI Web Client to simulate the Hub (it acts as a Dispatcher Istat Regional Offices involved for supporting Training already started Prototype in 2014-2015 Central Hub Application under design (the development should start in Autumn)
SDMX in the National Statistical System SISTAN Hub ** DB DB DB SDMX-RI I.Stat WEB GUI Istat Web ** DL ** HUB SDMX-RI SDMX-RI SDMX-RI SDMX-RI File CSV e/o SDMX DB DL * Existing Data warehouse.stat SISTAN Institutions
SDMX in production state of art Metadata repositories Data flows: 318 I.Stat + 1071 Agriculture Census + 481 Industry Census) Code lists: 412 I.Stat + 102 Agriculture Census+ 69 Industry Census Concept schemes: 56 I.Stat + 23 Agriculture census + 5 Industry Census) Data dissemination (machine to machine) All cubes of I.Stat + censuses databases available from the SEP Data reporting Census Hub ICT VIP pilot project National Account pilot project Job Vacancy Survey (in testing) Labour Cost Indicator (in testing) STS VELA (in testing) OECD STES IMF SDDS plus (prototype)
Thanks