Modernization of Statistical Information systems Global initiatives Eric Hermouet, Statistics Division, ESCAP 1 Presentation Why modernization? What is modernization? What is modernization? How: Global initiatives and collaboration mechanisms Main outputs of those mechanisms 1
Modernization Modernization of statistical information systems Industrialization of statistics Modernization of statistical production and services The data deluge The internet had 1800 exabytes of data in 2011 Exabyte=10 18 Or 1 million terabytes 2
The data deluge 50 000 exabytes by 2020 Even if only 0.1% of these data is useful, that leave millions of terabytes for possible statistical purposes New data providers Google Real-time price indices Public data explorer First point of reference for the data generation Facebook Telecom companies 4 billion mobile phones in Asia and the Pacific How can statistical offices access and use those data sources? 3
Changing user expectations Expectation of data available at a faster rate Real-time Ability to customize datasets Linking coherently datasets across domains With a high degree of detail Data presentation addressing different target groups Governments General public IT progress IT developments offers new solutions In processing huge amounts of data In processing huge amounts of data In accessing new sources of data In a more efficient way In terms of speed In terms of financial costs 4
So what is modernization? Common generic processes Common tools Common methodologies Recognizing that all statistics are produced in a similar way No special domains Increased flexibility to adapt y To access new data sources To generate new statistical products 5
A shared production environment From Custom design Built solutions Design for a collection mode designed-in quality Survey cycles with clear start and end points To Use of information bank and modular design Assembled solutions Source independent design Working with existing data and varying level of quality Continuous approach of ongoing collection, processing, and release A shared production environment From Direct data collection supplemented with data from administrative source Individually crafted data structure Management of data A large field workforce To Tapping into existing data, using direct data collection to link sources and bridge gaps Use of agreed standard approaches Management of data, metadata, and paradata Smaller field workforce with specialized interviewing skills 6
A shared production environment From Spending effort and resources on collection and processing Understanding user needs and designing collection instruments To More emphasis on specifying needs, design, and analysis Understanding data characteristics and negotiating solutions that bridges gaps between existing data and user requirements What is modernization and why is it needed? Questions so far? 14 7
Global initiatives & collaboration mechanisms 15 Global initiatives & collaboration mechanisms High-Level Group for the Modernization of Statistical Production and Services (HLG) Formerly HLG-BAS Objectives To promote common standards, models, tools and methods to support the modernization of official statistics; To drive new developments in the production, organization and products of official statistics, ensuring effective coordination and information sharing within official statistics, and with relevant external bodies; 8
HLG Gosse van der Veen (Netherlands) - Chairman Brian Pink (Australia) Eduardo Sojo Garza-Aldape (Mexico) Enrico Giovannini (Italy) Mr Park Hyungsoo (Republic of Korea) Irena Križman (Slovenia) Katherine Wallman (United States) Walter Radermacher (Eurostat) Martine Durand (OECD) Lidia Bratanova (UNECE) HLG reference documents Strategic vision http://www1.unece.org/stat/platform/display/hlgbas/s trategic+vision Strategy to implement the vision of the HLG http://www1.unece.org/stat/platform/display/hlgbas/ HLG+Strategy 9
Technical groups - MSIS Management of Statistical Information Systems (MSIS) Objectives: To provide, through regular meetings and other means such as the MSIS Wiki, a forum for exchange of experiences and good practices among information systems managers from national and international statistical organizations. To contribute to the coordination of activities of different national and international organizations in the area of statistical information systems. To facilitate and encourage implementation of international standards and recommendations in the field of statistical computing among national and international statistical organizations. Technical groups - MSIS Annual meetings jointly organized by UNECE, Eurostat, OECD. And ESCAP for this 2013 MSIS OECD. And ESCAP for this 2013 MSIS MSIS meetings consider issues related to information technology governance and management, system architecture, accessibility and usability. First meeting in 2000 Secretariat: UNECE O llun d l Open to all UN countries and international organizations Reports to the Conference of European Statisticians 10
Technical Group - METIS Steering Group on Statistical Metadata METIS Objectives: Promote the implementation ti of metadata t systems by developing advocacy targeting the senior management level and subjectmatter staff of NSOs; Oversee the maintenance of the Common Metadata Framework (CMF) directing it towards a practical guide serving national statistical offices; Facilitate collection, discussion and dissemination of best practices in the field of statistical metadata Established in 1990 Work sessions every 2-3 years Workshops in-between Organized with Eurostat / OECD Open to all UN member countries and international organizations Technical Groups - SAB Sharing Advisory Board Objectives To promote harmonization of business and information systems architectures; To support collaboration for the development of statistical software To provide guidelines and tools to assess new statistical software tools and components To assist in the improvement of the technical statistical infrastructure of countries both within and outside the UNECE region as required. 11
Technical groups - others Statistical Data editing Statistical Data collection Not directly overseen by HLG Statistical Network SDMX Expert Group Statistical Open Standards Group Working ggroup on quality in statistics. Main outputs Emerging statistical industry standards GSBPM GSBPM GSIM SDMX DDI 12
GSBPM Generic Statistical Business Process Model To define and describe statistical processes in a coherent way To standardize process terminology To compare and benchmark processes within and between organisations To identify synergies between processes To inform decisions on systems architectures and organisation of resources GSBPM Applicability All activities undertaken by producers of official statistics which result in data outputs National and international statistical organizations Independent of data source, can be used for: Surveys / censuses Administrative sources / register-based statistics Mixed sources 13
GSBPM Not a linear model Sub-processes do not have to be followed in a strict order Sub-processes do not have to be followed in a strict order It is a matrix, through which there are many possible paths, including iterative loops within and between phases Some iterations of a regular process may skip certain subprocesses 14
GSBPM Examples of use Harmonizing statistical computing systems Harmonizing statistical computing systems Facilitating sharing of statistical software Framework for process quality management Structure for storage of documents Measuring operational costs 15
GSIM GSIM contains objects which specify information about the real world 'information objects Examples include data and metadata (such as classifications) as well as the rules and parameters needed for production processes to run (for example, data editing rules). GSIM identifies around 150 information objects, which are grouped into four top-level groups GSIM Generic Statistical Information Model To describe data and metadata objects and flows within the statistical business process Implementation of GSIM, in combination with GSBPM, allow Creating an environment to prepare for reuse and sharing of methods, components and processes; Implementing rule based process control, thus minimizing human intervention in the production process; Economies of scale through development of common tools by the community of statistical organizations. 16
SDMX and DDi Standard Data and Metadata exchange UNSD/DFID project in Cambodia, Laos, Thailand, Viet Nam Data Documentation Initiatives World Bank Microdata Management Toolkit SDMX and DDI 17
What s next? Launch of Plug and Play project February 2013 To create a common statistical production architecture To create a standardized architecture for statistical production solutions, including processes, information and systems, coherent with the Generic Statistical Business Process Model (GSBPM) and the Generic Statistical Information Model (GSIM), To enable and advance the sharing of production processes or components, thus reducing costs. To provide the basis for a central inventory or repository with life cycle management of sharable production processes and components. 18