SDMX Tools Task Force Progress report. Item 3.1 of the agenda IT Working Group meeting 2017

Transcription

1 SDMX Tools Task Force Progress report Item 3.1 of the agenda IT Working Group meeting 2017

2 Introduction & Objectives SDMX Tools Task Force Scope & priority areas Status of the work and planning Participants are asked to: Acknowledge the work of the Task Force Provide feedback 2

3 SDMX tools Task Force Under DIME/ITDG governance Reports to Working group IT Establishment and composition operational since October 2015 (first meeting) 9 National Statistical Offices (Italy, Slovenia, Luxembourg, Netherlands, Austria, Serbia, Finland, Estonia and Macedonia) 1 National Bank (National Bank of Belgium) joined in International organisation (OECD) 3

4 Purpose Narrow the gap between SDMX experts inside ESS Sharing of experience, knowledge and best practices towards a more efficient statistical production and data sharing Lower the barrier for adoption and implementation of SDMX standard Governs and support the development, sharing and adoption of the SDMX software Represents central forum and coordination of SDMX related activities, tools and planning 4

5 Scope and priorities SDMX strategy Tools DSW, SDMX Registry SDMX Converter; SDMX-RI SDMX-RI, Census/Data Hub, ESS-MH STRUVAL, EDIT DLBB DSWS, SDMX-RI, Census Hub Evolution Collaborative development Improved quality Communication SDMX Tools TF Priority areas Capacity building OSS, supported by Eurostat For data collectors and data providers Cover different use case Based on SDMX Standard: Registry, WS, Reference Architecture, IM Re-usable and shared building blocks Java mainly (SDMX-RI also in.net) GUI, API and/or WS Based on Common API (CAPI) Extensible 5

6 Priority Area 1: Evolution Evolution of SDMX tools and SDMX standard by new use cases and requirements; SDMX based systems and processes; adoption; explore synergies with existing systems; pilot of ideas Work Project 1: Generic pull based transmission Project 2: SDMX-RI as a generic SDMX reporting platform Project 3: Work with other task forces, groups Project 4: Synergies with other tools Results Architecture established and pilot running SDMX-RI used in more domains Timely cover the planned work, prepare the tools in advance, ensure coordination and information sharing Common SDMX based systems; Reduce effort 6

7 Priority Area 2: Collaborative development Contribute to the development of SDMX software by developing new functionalities, extending features, integration of tools Work Project 1: Shared development with OECD Project 2: Shared development with ISTAT Project 3: Framework for collaborative development Project 4: SDMX OSS Results SDMX-JSON, reusable and extensible components Registry APIs in.net Establish a platform and workflow for distributed software development Code available in nuget.org, github.com 7

8 Priority Area 3: Improved quality Improve the quality of the SDMX software by contribution to the testing activities Work Project 1: Pre-releases available for community usage and test Results Multiple issues discovered and corrected; Support to the public release cycle; Increased test coverage; Resolution of issues in timely manner 8

9 Priority Area 4: Communication Improve the communication quality or establish communication channels, if not existing, by establishing means for structuring and sharing the information in clear and timely manner Work For SDMX developers For SDMX tools For projects and SDMX work Results Developers space; Information pages and Wiki; SDMX Info Space; Release calendar; Collaboration and sharing of information between different task forces and groups; Reporting 9

10 Priority Area 5: Capacity building Build, share and ease SDMX knowledge gathering by establishing more & high quality trainings; access to materials; extend the scope and relevance of the trainings Work Information sharing Easy access Developers training Results SDMX info space and training calendar; Information pages and Wiki; Free and easy access to training materials; Webinars; Visible training programme; Cooperation programmes 10

11 Work plan Established per year based on priorities and needs Priorities area confirmed for 2017 Ongoing & planned activities Conclude the ongoing pilots in order to derive best practices & propose improvements Develop, test and release new SDMX Converter and SDMX-RI (Mapping Assistant) Assess the work (re-shape if needed) 11

12 ISTAT s experience in the SDMX tools Task Force

13 Summary 1. Istat s SDMX implementations: milestones and key products 2. Istat participation to the task force activities: scope, actions and results 3. ISTAT s recent projects and activities positively influenced by the participation to the Task force 13

14 ISTAT s SDMX Implementations SDMX UMS Istat NSS Metadat Reporting Metadata a SISTAN HUB SEP Web Browser Metadata GUI SEP: Single Exit Point Dissemination End users Other formats UMS: Unified Metadata System RDF DSPL CSV JSON 16

15 Architecture packaging: SDMX Istat Framework A set of pick-and-choose building blocks allowing a statistical organisation to handle (macro)data and metadata (subject-matter domain independent) It extends the SDMX-RI in order to streamline the dissemination and reporting business processes: push and pull; distributed data warehouse SDMXbased; web presentation of data and metadata It is built using the SDMX Common API (SdmxSource.NET) 15 Eurostat

16 SDMX Istat Framework tools SDMX DDB Databases Web Browser Caching (Windows application) SSO WS SDMX Builder & Loader DDBs Configuration & Cache DB SDMX Web Browser MSDB / Registry WS SDMX Mapping Repository Mapping Assistant Data Flow Registration Structure Metadata Repository Metadata Web GUI 16 Eurostat

17 SDMX Builder loader 17 Eurostat

18 18 Eurostat

19 19 Eurostat

20 SDMX Data Browser 20 Eurostat

21 SDMX Data Browser 21 Eurostat

22 SDMX Data Browser 22 Eurostat

23 ISTAT s participation to the SDMX tools Task force Reasons: Share experiences and implementations related to SDMX tools with the other participating organisations Participate to collaborative development initiatives (planning, design, and software implementations) in coordination with other members of the task force (i.e. Eurostat and OECD) Address and share in a more structured way new functional requirements and/or bug reports related to the SDMX tools (i.e. the SDMX Reference Infrastructure tools or the SdmxSource libraries) 23 Eurostat

24 ISTAT s partiticipation to the SDMX tools Task force Collaborative developments: ISTAT EUROSTAT OECD 24 Eurostat

25 ISTAT s projects positively influenced by the task force activity: Hub of the Public Statistics Web statistical dissemination system composed by a web GUI (hub) allowing end users to browse, view and download data of the national public organisations, bodies (i.e. ministries, NSS organisations, national central bank) and agencies participating to the project. Data are stored in the organisations (nodes) and made available to the hub via the SDMX web service The architectural design was inspired by the Eurostat s Census Hub project, and the system is based on the tools of SDMX-RI and of the SDMX Istat Framework Is compliant with the open data initiatives at Government level, and facilitates semantic interoperability among public organisations 30 Eurostat

26 Hub of the Public Statistics: architecture HUB 31 Eurostat

27 ISTAT s projects positively influenced by the task force activity: international cooperation At present ISTAT is involved in the following projects as far as concerns SDMX activities and workpackages: Tunisia, Serbia, Macedonia, Laos Support to the design and implementation of an SDMX architecture for dissemination and reporting, based on the SDMX-RI and the SDMX Istat Framework. Capacity building actions for strengthening the knowledge of the SDMX standard and for allowing the colleagues of these countries to be autonomous in configuring, using and managing and customising the SDMX tools for dissemination and reporting purposes. 32 Eurostat

28 Thank you for your attention Questions? Do you have particular interest in some of the priority areas defined by the Task Force? Would you like to participate in pilot initiatives? 28