SDMX Technical Working Group Activity Report

Transcription

1 SDMX Technical Working Group Activity Report Sami Airo 7th Meeting of the Expert Group on SDMX October 2014, Seoul, Korea SDMX Technical Working Group 1

2 Background & composition Provides the forum for the active participation of the SDMX community in the management and development of the SDMX technical standards Started in May 2011, with the SDMX Global Conference in Washington Currently 24 members Central banks, statistical offices & international organisations 16 countries and 7 international organisations 4 regions => Representative of the SDMX technical community 2

3 Tasks Improves the SDMX Technical Standards Owns the SDMX technical standards issues list Reviews the issues list and sets priorities Prepares proposals for revised versions of the standard Oversees a set of SDMX reference tools Cooperates with the SDMX Statistical Working group 3

4 SDMX Technical Standards Humans Computers 1. Framework Document 2. Information Model Specification (UML) 3. Exchange Formats 3.1 SDMX-ML Specification XML Schema 3.2 SDMX-EDI Specification SDMX-JSON Specification JSON Schema 4. Services (=APIs) 4.1 Registry Services Specification 4.2 Web Services Guidelines WADL + WSDL 5. Technical Notes Validation & Transformation Language Specification BNF Grammar 4

5 Active Work Packages WP2 - Interoperability of IT tools WP4 - Improved technical documentation WP5 - Interaction of SDMX with other technical standards (i.e. DDI) WP6 - Expressions and calculations (VTL) WP7 - SDMX-JSON WP8 - SDMX Technical Issues List 5

6 WP2: Interoperability of IT Tools The main aim is to foster the interoperability of the SDMX software tools. This allows adopters of the SDMX standards to combine various tools developed by different organisations and build the system they need. SDMX Software Components 6

7 WP2: Interoperability of IT Tools Report to be published on Set of normative interfaces to be fulfilled by SDMX-compliant tools ( SDMX Common APIs ) would help in the development of custom implementations. Current de-facto SDMX common API is implemented in the SdmxSource framework (Java and.net versions). Creating and maintaining APIs as technical standards would not be practical as programming platforms are substantially different, and the software development work is substantial. Open-source implementations of technical standards will continue to be important for the SDMX initiative and their further development needs stable funding. Areas for further work Investigate automated tools for testing web service compatibility with SDMX 2.1 Web Services specifications ( compatibility tool kit ). Maintain mapping of the component list to existing software and state of art implementations (in the report appendix) when new software applications and new versions of existing applications are released. 7

8 WP4: Improved technical documentation Often pointed out that the SDMX technical documentation is complex Need to suggest possible improvements and enhancements Focused on SDMX 2.1 Close co-operation with the SWG Some deliverables Updated user guide Implementation model Technical vocabulary 8

9 WP5: Interaction of SDMX with other technical standards Contribute to the discussion of the possible interaction of SDMX and other standards (such as DDI, GSIM, etc.) Assess current capabilities (SDMX vs. other standards) Suggest how SDMX could evolve Discuss the possible integration with other standards Final report in preparation, some preliminary conclusions SDMX standard is broadly able to represent both unit and dimensional data in a homogeneous way, by using the DSD. Representing unit and dimensional data through the same basic artefacts may give significant benefits. Some areas for improvements: representation of multiple measures, relationships and integrity constraints, time validity of data and metadata 9

10 WP6: Validation and Transformation Language (VTL) Allows definition and exchange of data validation and transformation rules Based on the existing SDMX Information Model framework and the GSIM Data Model Usable by statisticians Supports dimensional and unit-record data Two main parts: general description and library of operations Draft version published on the public review period until 8 November 10

11 VTL - Example D1 := get( POP/ US, keep(ref_date, BIRTHS, DEATHS) ) D2 := get( POP/ EU, keep(ref_date, BIRTHS, DEATHS) ) DR := D1 + D2 DS := put( DR, POP/ ALL ) This simple VTL program retrieves two data sets, sums the data sets and stores the result. 11

12 VTL - Data Retrieval Example D1 := get( POP/ US, keep(ref_date, BIRTHS, DEATHS) ) D2 := get( POP/ EU, keep(ref_date, BIRTHS, DEATHS) ) Result Data Sets 12

13 VTL - Simple Transformation Example DR := D1 + D2 Source Data Sets Result Data Set 13

14 VTL - Data Storage Example DS := put( DR, POP/ ALL ) Input and Result Data Sets 14

15 VTL - Transformation Examples DR := D1 + D2 DR := log( D1 - D2 ) DR := ln( abs(d1) + D2 ) - abs(d1) DR := union( D1, D2 ) DR := diff( D1, D2) DR := D1 <> D2 DR := D1 between 1000 and 2000 DR := isnull(d1) DR := check( D1 > D2, imbalance( abs(d1 - D2), erlevel(8) ) DR := if D1 > 4 then D1 else D2 + 3 DR := D1[ filter REF_DATE = 2013 ] DR := D1[keep(BIRTHS)][aggregate sum(births)] 15

16 WP7: SDMX-JSON Exchange Format New exchange format for data dissemination on the Web Conforms to the JSON standard specification, and supports the SDMX 2.1 Information Model Supports all features of the SDMX RESTful API for data queries Allows web developers to write software that requests data responses in JSON from SDMX RESTful API and processes data and metadata in the JSON response 16

17 WP8: Issue List Issue list is for bug fixes and minor improvements Examples StructureMap defines wrong multiplicity for ComponentMap Ability to reference a Code List Map not contained in the same Structure Set Current status Bug reports: 1 open, 5 resolved Improvements: 7 open, 2 resolved New features: 2 resolved Minor updates to 2.0 and 2.1 Technical Standards 17

18 WP8: Updated Processes and New Tools for the Standards Maintenance Updated Technical Standards revision process Types of changes: bug fix, improvement, new feature Semantic version numbering: xx.yy.zz Process steps: working draft, candidate, proposed, final Testing new tools (Github) for the maintenance of standards documents and related source code files Github provides efficient source code management and collaborative features such as bug tracking Should also improve transparency and openness of the standards maintenance process Example: 18

19 That s all, thank you! sami.airo@bof.fi twg@sdmx.org 2010 Alec Perkins 19