ESSnet on SDMX. Technical issues. Workshop on ESSnets 31/5 1/6 2010, Stockholm

ESSnet on SDMX Technical issues Workshop on ESSnets 31/5 1/6 2010, Stockholm

ESSnet on SDMX Technical issues Technical coordination Work packages Dissemination (wiki, web site)

ESSnet on SDMX Technical issues Objectives The main contents of the terms of reference: Implementation of SDMX at international and national level The further development of the SDMX standards used for data and metadata The further development of the SDMX Content- Oriented Guidelines The further development of the SDMX IT architecture and IT tools The increase of SDMX capacity and knowledge

ESSnet on SDMX Technical issues Objectives Broadening and strengthening a NSI standard SDMX framework (WP8); Making standard tools available to a more easy implementation of the standard SDMX in the processing and exchange of data and metadata files between national and international organizations (WP3, WP4, WP7); Improving the semantic harmonization of concepts and terminology supporting this standard (WP2); Disseminating related tools and documentation widely (WP9);

ESSnet on SDMX Technical issues Objectives Results have to be broadly reusable within the ESS Software tools developed on an Open Source basis and registered under European Union Public licence (EUPL); Ensures the following rights: Obtain the source code of the software from a free access repository; Use the software in any circumstances and for all usage Reproduce (copy, duplicate) the software Modify the original software The development of software should take care about: A modular design The use of a version control system Clear documentation Detailed information at http://ec.europa.eu/idabc/eupl

ESSnet on SDMX Technical issues Success factors Communication: all the members of the project shall communicate anything that might affect or contribute to the project; Good inter co-ordination between Work Packages, namely between WP s responsibles, in order to minimize the overlapping of work and to assure interoperability as always as needed; Strong involvement of all the participants and stakeholders; Allocation of the right people for each task ; Stick to the approved working plan, reporting as soon as possible any problem that could compromise it;

ESSnet on SDMX Technical issues Main achivements Involvement and increase of knowledge on NSI s about SDMX; Better inter co-ordination between Work Packages; Improvement of ROI ( Return On Information); A common framework to support integrated and interoperable solutions; Dramatic reduction of developments costs on each NSI, due to common tools and solutions;

ESSnet on SDMX Technical issues Technical coordination WP1 Management of the Action (PT) Work: Meetings: Monitoring the work in the different WP s, the coordination between WP s and the quality of the deliverables; To Identify problems and make proposals to overcome them; To Produce progress reports; Organizing coordination meetings; Reports: Kick-off meeting 16/10/2009 - Eurostat and partner countries: Start the project, make a project review, establish the project governance, define roles and responsibilities, clarify the expectations of all parties; 2 nd coordination meeting 26/03/2010 - progress evaluation Quarterly progress reports (January, April)

Technical coordination Difficulties: Sound knowledge of the state-of-the-art is needed In SDMX field there are too many software tools not yet broadly known and commonly accepted There are other standards for the same purpose Oportunities: Increase knowledge on SDMX An oportunity to implement SDMX in the Organization The possibility of influencing the way forward on the short and medium run

ESSnet on SDMX Technical issues Proposals A group of experts to assist the coordinator on technical evaluation of WP s would be desirable; A repository of specialized tech skills existing on each NSI, could resolve eventual gaps on a particular task, i.e. sharing knowledge not just information; Improvement on the current practise of reusing existing pieces of software already developed by someone in any project.

ESSnet on SDMX Technical issues Work Packages WP2 MCV Ontology (PT) Responsible: Statistics Portugal Goal: To create an ontology of statistical metadata based on Metadata Common Vocabulary (MCV) concepts Deliverables: 1 st version of the ontology - July 2010 2 nd version of the ontology November 2010 Final version of the ontology March 2011 Report describing the methodology used March 2011

ESSnet on SDMX Technical issues Work Packages WP2 MCV Ontology (PT)

ESSnet on SDMX Technical issues Work Packages WP2 MCV Ontology (PT) Terminological point of view Improve the quality of the definitions: More complete and accurate More clear More coherent among them

ESSnet on SDMX Technical issues Work Packages WP2 MCV Ontology (PT) Technological point of view Semantic Web Technologies(*) Web services software tools designed to support interoperable machine-to-machine interaction over a network Ontology Set of concepts, the relations among them and software for computing them Knowledge representation languages RDF, OWL (Web Ontology Language) Turn an ontology from a model of concepts into something that is computational Make inferences with the concepts in an ontology (*) The ideas and definitions in this slide were extracted from Open issues on the semantic web, Dan Gillman, Work session on statistical metadata (METIS), March 2010

ESSnet on SDMX Technical issues Work Packages WP3 SDMX Data Explorer (PT, IT, ES) Responsible : Statistics Portugal Participants: ISTAT, INE-SP Goal: To deliver a tool (using a generic SDMX class library) to visualize SDMX data files under different perspectives (html, CSV, Charts, SDMX format conversion) in a more user friendly way. Progress: Functional and technical specifications (03.D-1 finished); Analysis and design documentation (03.D-2 finished); User Interface prototype description(03.d-3 finished); The development of SDMX-ML Parser has been started; Source code available for partners in repository

ESSnet on SDMX Technical issues Work Packages WP3 SDMX Data Explorer (PT, IT, ES) Deliverables: Application package intended to explore statistical data using an SDMX data file and related DSD; Documentation associated with the tool: Functional and technical specifications Analysis and Design documentation User interface prototype User Manual Implementation report Application package: Web application, class library and web services

ESSnet on SDMX Technical issues Work Packages WP3 SDMX Data Explorer (PT, IT, ES)

ESSnet on SDMX Technical issues Work Packages WP4 Generalized SDMX Framework (IT, PT, SK) Responsible: ISTAT Participants: Statistics Portugal, INFOSTAT (Slovak Republic) Goal: To complete the Generalized SDMX Framework already implemented by ISTAT, with more functionalities, in order to make it available as a complete product;

ESSnet on SDMX Technical issues Work Packages WP7 PC-Axis with SDMX-ML (SE, NO, IT) Responsible: SCB (Sweden) Participants: ISTAT, Statistics Norway Goal: To create a PCAxis < > SDMX converter, based on PCAxis files and on an SQL database structured according to the Nordic Metadata Model

ESSnet on SDMX Technical issues Work Packages WP8 SDMX in Survey Design and Data Collection (SK, PT) Responsible: INFOSTAT (Slovak Republic) Participants: Statistics Portugal Goal: To implement SDMX in the survey design and data collection Pilot implementation on a selected statistical domain (tourism and census); Progress: Design of system architecture and system functionality (May 2010) 1st deliverable Deliverables: Pilot implementation results System documentation Report on the methodological results achieved

ESSnet on SDMX Technical issues Work Packages Implementation of SDMX in survey design and data collection processes at NSI METADATA MANAGER METADATA MDSD FORMS SURVEYS DATA STRUCTURE DEFINITIONS DOCUMENTS GENERATOR e FORMS GENERATOR RESPONDENTS LOADER SDMX DB EXTERNAL METADATA eforms REPOSITORY RESPONDENTS REGISTRY RESPONSES MANAGER REGISTER DATA LOADER SOURCE DATA DATA EDITING IMPUTATION SOURCE DB System architecture

WP8 SDMX in survey design and data collection Proposal for e-form (e-questionnaire) standard Preamble section Identification section (zero modul) Information section Data section (modules) Declaration section Interpretative note section

WP8 SDMX in survey design and data collection Structure of preamble section LOGO Official name of the statistical office Statistical survey name Form ID Surveyed period Subbmitt to: Address of form submitting Form registration No. Until: Date of submitting Declaration of data protection Contact:: Contact address

WP8 SDMX in survey design and data collection Matrix Data Strucure Heading of rows Heading of columns Numbering of columns Numbering of rows Matrix Variable

REGISTRATION_NUMBER REGISTRATION_DATE 1.. N IDENTIFICATION_KEY Identification Items ROW_IDE KEY_ORDER WP8 SDMX in survey design and data collection KEY_NAME KEY_SIZE INSTRUCTFILL Report structure for describing e-form 1.. N INDICATION_DATA Indicator data ROW_IND The structure of report on concepts in Concept KEY_ORDER Scheme for filling the tables describing statistical e-form - Heading level and Modules level (descriptive metadata for database tables) KEY_NAME Relationships Concepts Sub-concepts The new semantic role of the KEY_SIZE concept, which is used from another INSTRUCTFILL concept 1...1 N ORGANISATION_RESPOSIBLE KCOMPLEMENTARY The Descriptive organization information responsible the for survey identifier and indicator CONTACT_ORGANISATION COMPLEMENTARY Abbreviation COMPL_ORDER 12.1 Responsible organization LOGO 1..1 FORM_IDENTIFICATION CONFIDENTIALITY Identification Confidentiality Form CONTACT_NAME CONF_POLICY 12.3 Statement founded SURVEY_TITLE CONF_DATA_TR 1..1 INST_MANDATE SURVEY_CODE Institutional mandate IKF_CODE Addressing SURVEY_YEAR INST_MAN_LA_OA SURVEY_PERIOD INST_MAN_SHAR 1..1 DELIVER_CONTACT REGISTRATION_PLACE REGISTRATION_NUMBER DELIVER CONTACT_NAME REGISTRATION_DATE Data for service form Deliver - description 12.3 Deliver - the person's name 1.. N IDENTIFICATION_KEY CONTACT_ORGANISATION Identification 12.1 Deliver - Organization Items (Regional 1:1 METHODOLOGICAL_ EXPLANATION ROW_IDE CONTACT_MAIL KEY_ORDER KEY_NAME KEY_SIZE INSTRUCTFILL EXPLANATION_HEAD Report) 12.5 Deliver - postal address Methodological notes on title 1.. N INDICATION_DATA EXPLANATION Indicator data 1.. N SURVEY_MODULES ROW_IND KEY_ORDER MODULE_ORDER KEY_NAME MODULE_SIGN KEY_SIZE MODULE_NAME INSTRUCTFILL 1.. N KCOMPLEMENTARY Descriptive information on the identifier and indicator COMPLEMENTARY

WP8 SDMX in survey design and data collection Concept Scheme

WP8 SDMX in survey design and data collection DSD for data collected

ESSnet on SDMX Technical issues Work Packages WP9 Dissemination (PT, all co-partners) Responsible: Statistics Portugal Participants: All partner countries Goal: Dissemination of the results of this Action within and outside the European Statistical System Progress: ESSnet on SDMX web site (available April 2010) http://sdmxessnet.ine.pt Deliverables: Training Plan on the tools produced Presentations about the project (ITDG, STNE, MWG) Final workshop (March 2011, Lisbon)

ESSnet on SDMX Technical issues Dissemination WIKI and WEB SITE WIKI Populated by the participants on the ESSnet Authentication needed Documents can be uploaded on draft status Collaboration WEB SITE Populated by the co-ordination Public access Documents available with on final status No collaboration

ESSnet Main page Structure of the wiki content at 19/03/2010 ESSnet on SDMX Coordinator Project details Project description Main objectives Main milestones Work Packages Participants

Structure of the wiki content at 19/03/2010 WP Main page WP name of the WP (what is expected): Participants Main objectives Description of the work Deliverables (In progress):

Web site http://sdmxessnet.ine.pt

Web site