7th Meeting of the Expert Group on SDMX. SDMX Implementation. in Statistics Korea. Youngok PARK

Transcription

1 7th Meeting of the Expert Group on SDMX SDMX Implementation in Statistics Korea Youngok PARK

2 [Contents] 1. KOSTAT s SDMX Progress 2. Korean Data Provision System (KODAPS) 3. KOSIS Data Sharing Service (Open-API) 4. Comments on SDMX Experience

3 1. KOSTAT s SDMX Progress Development of G20 Dashboard (use ECB, Flex-CB) - Data visualization SDMX under review Re-development of existing system not an option Select priority areas for implementation 2011 Development of KODAPS - Data provision SDMX download service available when looking up statistical tables in KOSIS - Provide data query SDMX Sophistication of KODAPS Participation in OECD pilot short term economic statistics - Use of KODAPS SDMX 2.0 SDMX 2.1 English version of KODAPS launched Participation in OECD pilot short term economic statistics continued)) KOSIS Sharing Service (Open-API) developed - Provide data query SDMX 2.1 3

4 2. KODAPS (Korean Data Provision System) A SYSTEM FOR SUPPORTING DATA PROVISION TO OECD, UN AND OTHER INTERNATIONAL ORGANIZATIONS 4

5 2.1 KODAPS Korean Data Provision System (KODAPS) Supports business related to providing statistical information i.e. managing history of information requested, data provision, data transmission to agencies including international organizations Data request management: manage data request by international organization Data management: Linking to KOSIS, direct input, formula, file Data provision management: register data by request, generate data to be supplied, provide data Create file for each data to be supplied in Excel, CSV, SDMX Support send out (push mode) and web service (Pull mode) Manage user, log and different codes 5

6 2.2 KODAPS Business Process Data User A Data Request Data Provider C Data provision Request for Data Use (register new user) Data download Support direct extraction of SDMX data Direct download of SDMX data via URL access (Web-Service REST) c B Data request and data management Data management Link to KOSIS Database Direct input Arithmetic expression File User Screen Manage user & log Manage information Manage request Manage data Manage data provision Approval for Use (user management) Information management Manage agency users Mange SDMX DSD Manage code Manage data for provision System Administrator a 6

7 2.3 KODAPS SDMX Implementation Steps STEP 1 STEP 2 STEP 3 STEP 4 STEP 5 Register DSD requested by international organizations Register Data Request - Select SDMX as format : Version, Agency, DSD, Data format Define SDMX Attribute - Assign attribute by selecting code for each data and dimension/attribute Create SDMX data file KODAPS Web Service 7

8 [Step 1 Register DSD] Add/Revise/Delete menu for all SDMX DSD Version Agency List, DSD list by organization, DSD version list Detailed information on DSD: Dimension, Attribute, Code List, and relevant information each kept as a tab 8

9 Check for DSD Registration Download DSD in excel to check the contents 9

10 [Step 2 Register Data Requests] Select SDMX(2.0) as configuration type, then select Agency ID, DSD ID, DSD Version and Data Format and click on save. Data format has two choices: compact and generic (v2.0) Compact: attribute compressed for each data Generic: attribute value for each data shown 10

11 [Step 3 Define SDMX Attributes ] 1 2 Register attribute values for dimension and attribute of each data. 1 - Use code list already registered in DSD Freq registered in SDMX does not cover irregular freq, bimonthly or cycles, thus Apply SDMX frequency must be checked. 2 11

12 [Step 4 Create SDMX data file ] Under request management system, registered information is selected as default value. Click on Creation to create an SDMX data file and send via . Or user agency can self-extract data from provider agency using the web. 12

13 [Step 5 KODAPS Web Service ] User accesses KODAPS web service to directly extract data(pull mode). 13

14 3. KOSIS Data Sharing Service (Open-API) Standardized apis (application programming interfaces) which are open to the public so that public and private developers can build their own services on statistical database in the KOSIS SDMX 2.1 VERSION Provides numerical data, list of statistics and statistical metadata 14 14

15 3.1 Why Offer KOSIS Open-API? Without Open-API, users had to access KOSIS every time they need to download new data. Open-API allows KOSIS users to automatically retrieve and utilize KOSIS data (i.e. mobile app). Use SDMX with development of API Service KOSIS? Paving the way for the Korea s Government 3.0 initiative, KOSIS is the national statistical database, operated by Statistics Korea. As a gateway for Korea s official statistics, KOSIS offers a convenient onestop service to full range of major domestic, international and North Korean statistics. Currently, official statistics produced by over 120 statistical agencies covering more than 500 subject matters. 15

16 3.2 Open-API Coverage Information to be provided over Open-API List of Statistics: List of statistics provided over KOSIS across subject areas, themes, etc. Metadata: Meta information on survey Statistics: Tables provided over KOSIS API Method Restful Service Output format : json, sdmx, xml *SDMX(V2.1) : DSD, GenericTimeseries, StructureSpecificTimeseries 16

17 [KOSIS Sharing Service ] URL (available in Korean only) 17

18 3.3 Procedure for Data Access User login required Auto approval on list of statistics, general statistical data and statistical metadata Mass volume data are approved based after review 18

19 3.4 List of statistics Provision of information on list of available tables Data format: SDMX(Category), JSON SDMX JSON 19

20 3.5 Statistical Data Provides information about the numerical values in statistical table and structural metadata (i.e. code, source, unit) Time series (single series, multiple time points) Cross-sectional (multiple series, single time point) Screen for selecting statistical table (query builder) Data provided in: JSON, SDMX (DSD GenericTimeseries StructureSpecificTimeseries) Screen for selecting table 20

21 3.6 Statistical Metadata Provides detailed information about surveys pertaining to statistical table Title of survey, type of statistics, current or discontinued data, legal authority, purpose of survey Periodicity of enumeration/dissemination, survey structure, scope of dissemination, contact point, etc. Reference Metadata service Data provided in: JSON, XML Screen for Selecting Survey JSON XML 21

22 3.7 Strategies for DSD Design Keep existing statistical database structure in the KOSIS * DSD definition not possible on all data: data volume, structure - 80,000+ tables, 250 million series 1:1 mapping relation to KOSIS statistical database structure Classifier, item, time period: dimension Unit: attribute 22

23 [DSD Design for Statistical Data] Type Description Type ID Codelist ID Category Note SDMX Version SDMX(2.1) ITEM CL_ITEM Item Agency ID DSD ID DSD Name Sender KOSIS OrgCode_TableID Ex) 101_DT_1DA7001 TableName(Period) Ex) Economically Active Population by Sex (Monthly, quarterly, yearly, ~ ) Survey administering team and contact person Dimension C_Classifier Code1... C_Classifier Code8 CL_Classifi er Code1... CL_Classifi er Code8 UNIT CL_UNIT Unit Classifier1... Classifier8 FREQ CL_FREQ Frequency Use Statistical DB Classifier Code *Number of classifiers vary depending on table layout Source KOSIS(Data producer, survey name) (Ex) KOSIS (KOSTAT, Economically Active Population Survey) TimeDimension MeasureList TIME_PERIOD OBS_VALUE Time Period Numerical data

24 3.8 Development strategy Offer time series only Offer cross sectional data as well Insufficient use of SDMX standard (DSD) Apply standard rules Apply only data format rules Apply data standards as well 24

25 4. Comments on SDMX Experience 25 25

26 Comments on SDMX Experience KOSIS aims to shift Korea s decentralized statistical production system to centralized data service system Data producers design their own output tables and send them to KOSIS, and hence, it is difficult to enforce use of standard data code. Massive volume of data in detail are included so harmonized DSD design was not feasible. We need to explore business process that can satisfy the SDMX initiatives Follow standards, provide ample amount of metadata By expanding DSD globally, we can anticipate innovation in statistical production and service 26

27 감사합니다 THANK YOU