SDMX IT Tools SDMX Converter

Transcription

1 SDMX IT Tools SDMX Converter Nadezhda VLAHOVA Jean-Francois LEBLANC OLIVIER LECLERE May 2018 Eurostat, Unit B3 IT solutions for statistical production

2 Table of Contents 1. Objectives 2. What is the Converter 3. Interfaces 4. SMDX Converter vs SDMX-RI 5. Installing the Converter 6. Hands-on exercise 7. Further information 2

3 1. Objectives Illustrate how to use the Converter Explain all the supported formats Clarify its interfaces Point out when it is recommended State its limitations 3

4 2. What is the SDMX Converter? The Converter is a Java application that converts statistical data files between different formats. SDMX-ML XML GESMES/TS GES Version 2.0 Generic Compact Utility Cross-Sectional Message Group* Version 2.1 Generic Structure-Specific Generic Time Series Structure-Specific Time Series Time-Series Flat Files CSV and FLR Ad-hoc Excel templates Plain Text MS Office SDMX-CSV Plain Text Conversion to/from some formats may require to add or to subtract information. Limitations may apply. 4

5 2.1 How does the Converter work? 1. Takes Input data file (in source format) and input format Output path and format Data Structure Definition (DSD) o o Either stored in a file (offline operation), or Given as a reference to be fetched from an online SDMX Registry Additional information (format specific, optional) o E.g., mappings, parameters, delimiters, message headers. Specification of additional operations (optional) o E.g., transcoding, SDMX validation. 2. Performs the processing 3. Generates Converted data file (in destination format)

6 2.2 Validation The Converter can also validate SDMX files: Validate your input files before/after conversion* Validation only SDMX files only can be validated: SDMX-ML SDMX-CSV 6

7 3. Interfaces (modes of use) User interface Command line Locally installed Web App Web service API cd io «interface» Reader + readdata(inputstream, InputParams) : DataMessage + setinputstream(inputstream) : void + setinputparams(inputparams) : void + setwriter(writer) : void + isready() : boolean + readdata(inputstream, InputParams, Writer) : void + readdata() : void «interface» Writer + writedata(datamessage, OutputStream, InputParams) : void + setoutputstream(outputstream) : void + setinputparams(inputparams) : void + isready() : boolean + writeheader(header) : void + writeemptydataset(dataset) : void + writegroup(groupkey) : void + writeseries(timeserieskey) : void + closewriter() : void «realize» «call» «realize» CompactDataWriter GenericDataReader + readdata(inputstream, InputParams) : DataMessage + setinputstream(inputstream) : void + setinputparams(inputparams) : void + setwriter(writer) : void + isready() : boolean + readdata(inputstream, InputParams, Writer) : void + readdata() : void + setoutputstream(outputstream) : void + writedata(datamessage, OutputStream, InputParams) : void + setinputparams(inputparams) : void + isready() : boolean + writeheader(header) : void + writeemptydataset(dataset) : void + writegroup(groupkey) : void + writeseries(timeserieskey) : void + closewriter() : void 7

8 3.1 GUI (Converter 6.14) 1. Operation selection: Convertion Sdmx file validation Convertion with validation 2. Selection of the input/output files and their format 3. Possibility to load a template saved during a previous convertion, it contains: Input Output Formats Etc 8

9 3.1 GUI (Converter 6.14) 1. Structure selection: DSD OR Dataflow 2. It is possible to use a remote structure stored on the registry 3. Selection of the structure 9

10 3.1 GUI (Converter 6.14) 1- Validation result 2- Conversion status Actions 10

11 3.1 GUI (Converter 6.14) 2- For a CSV conversion to SDMX format you need a SDMX header: - load a SDMX header previously saved - manual encoding 1- The converter adapts the GUI depending of the format selected as input/output 4- CSV parameters: Csv header ( different from SDMX header) Column mapping with DSD component Use of quotes or not 5- Transcoding 11

12 3.1.2 Header Erase Fields : To empty all fields. Load Defaults : To load default values from a configuration file (set during installation). 12

13 3.1.2 Header - Don t mix headers! Don t get confused between SDMX headers and CSV headers SDMX Header is mandatory (xml file) CSV header is optional (CSV file) 14

14 3.1.3 Change mapping 14

15 3.1.4 Transcoding 15

16 3.2 Command Line Interface (CLI) Windows command line Unix/Linux Shell For automated conversion Scripts / external systems No human intervention Useful for batch processing Can run locally or call the converter web service 16

17 3.2 CLI Converter Windows OS: converter-cli.bat Unix OS: converter-cli.sh using cmd.exe using the terminal Exemple of a conversion converter-cli.bat -from COMPACT_SDMX -to GENERIC_SDMX -inputfile data_compact.xml -outputfile data_generic.xml -dsd_file NA_MAIN_1_6.xml Exemple of a validation converter-cli.bat -validation true -inputfile compact_namain_idc.xml -dsd_file NA_MAIN_1.6_V2.1.xml 17

18 3.2 CLI Options (they correspond to GUI buttons and controls) -inputfile** -outputfile* -from* -to* -reg -dsd_file** -dsd_id -dsd_agency -dsd_version -df -df_id -df_version -df_agency -header_file -date_format -level -mapping_file -ordered_input -trans-file -delimiter -header_row -unescapeinputcsv -escapeoutputcsv -header_writing -nuri -nprefix prefix -ges_technique -validation -mapping_param_file -parameter_file -excel_template -exceltype -generated_file_comment -quote_cha -maxerrornumber -wsdl **mandatory *mandatory for conversion only For further information check the User Manual page 6. Eurostat 18

19 3.3 Common API cd io «interface» Reader + readdata(inputstream, InputParams) : DataMessage + setinputstream(inputstream) : void + setinputparams(inputparams) : void + setwriter(writer) : void + isready() : boolean + readdata(inputstream, InputParams, Writer) : void + readdata() : void «interface» Writer + writedata(datamessage, OutputStream, InputParams) : void + setoutputstream(outputstream) : void + setinputparams(inputparams) : void + isready() : boolean + writeheader(header) : void + writeemptydataset(dataset) : void + writegroup(groupkey) : void + writeseries(timeserieskey) : void + closewriter() : void «realize» «call» «realize» CompactDataWriter GenericDataReader + readdata(inputstream, InputParams) : DataMessage + setinputstream(inputstream) : void + setinputparams(inputparams) : void + setwriter(writer) : void + isready() : boolean + readdata(inputstream, InputParams, Writer) : void + readdata() : void + setoutputstream(outputstream) : void + writedata(datamessage, OutputStream, InputParams) : void + setinputparams(inputparams) : void + isready() : boolean + writeheader(header) : void + writeemptydataset(dataset) : void + writegroup(groupkey) : void + writeseries(timeserieskey) : void + closewriter() : void 19

20 3.4 Web Service A web service also exists for the Converter Based on Java Can be installed on Tomcat or WebLogic server 2 endpoints o Converter o Struval (validation) 20

21 3.4.1 Usage the webservice WSDL definition (url depends on your configuration) : Example of conversion: 21

22 3.5 Web application Modern user interface Same operations as in the desktop application Installed on a server and access via web browser Can be installed on Tomcat or Weblogic server No local installation No local update Based on Java 22

23 3.5.1 User interface 1. Operation selection: Convertion Sdmx file validation Convertion with validation 2. Selection of the input/output files and their format 3. Possibility to load a template saved during a previous convertion, it contains: Input Output Formats Every parameters 23

24 4. SMDX Converter vs SDMX-RI SDMX Converter SDMX-RI Standalone application / on a server Local files Needs to be installed on a server Connected to a dissemination DB Generates SDMX files from input files If you are a Data Provider Generates SDMX files from customized SDMX queries Get the DSD from the SDMX Registry Data stored as files Data in a DDB DB SDMX Converter SDMX-RI Push via edamis Expose the data to be pulled (WS, HUB) 24

25 5. Installing the SDMX Converter You can download the latest version of the SDMX converter on CIRCABC Available packages SDMX Converter Wizard application SDMX Converter Web Service SDMX Converter CLI SDMX Converter WebApp SDMX Converter API 25

26 5.1 Common issue: Java_home 26

27 6. Hands-on exercises Conversion using the GUI Conversion using the webapp Conversion using the webservice Conversion using the CLI (command line) Validation 27

28 7. Further information SDMX official website Eurostat SDMX Info Space Tools Tutorials Trainings And much more!

29 7.1 Need help? Contact us! If you have any problem converting, please send: Input file DSD Screenshot with the error, if possible Log file, if possible Don t you find the log? Try looking at C:\Users\<your_user>\AppData\Roaming\ SDMX Converter x.x.x_repository\converter_data\logs

30 SDMX Converter 30