Adopting Big Data Technologies in the Support of Official Statistical Production: Opportunities, Experiences and Lessons Learned

Size: px

Start display at page:

Download "Adopting Big Data Technologies in the Support of Official Statistical Production: Opportunities, Experiences and Lessons Learned"

Cecilia Randall
5 years ago
Views:

1 Adopting Big Data Technologies in the Support of Official Statistical Production: Opportunities, Experiences and Lessons Learned Antonino Virgillito Istat Istituto Nazionale di Statistica 31/07/2017 1

2 Introduction The use of Big Data sources in the context of production of official statistics has been at the core of several initiatives at both national and international level in recent years Among all the questions that were raised by the use of Big Data for statistics, a specific one is the use of the novel IT tools that are available for handling Big Data In this presentation we give an overview of Big Data technology and presents the different ways in which it can be used to support statistical production, analysing the experiences made in Istat and at international level

and loose structure What are the characteristics of these

3 Overview of Big Data Tools Big Data Tool Specifically designed to cope with aspects such as large size of data and loose structure What are the characteristics of these tools? How can they be used in the context of statistical production?

4 Categories of Big Data Tools Distributed Computing Platforms Clusters of interconnected machines working as a whole in order to store and process data Hadoop Spark

5 Categories of Big Data Tools Massively Parallel Processing DB Provide real-time querying with fast response times over large data sets Dremel Drill Impala

6 Categories of Big Data Tools NoSQL Databases Not based on the traditional tabular data model but capable of handling non-structured data HBase MongoDB Elasticsearch

7 Uses of Big Data Tools RDBMS Offload Big Data staging Experiments Heavy processing

8 Case Study: UNECE Sandbox Web-based collaborative environment, hosted in Ireland by ICHEC (Irish Centre for High-End Computing) UNECE Big Data Project ( ) High-Level Group for the Modernisation of Official Statistics (HLG-MOS) Objective: to better understand how to use the power of Big Data to support the production of official statistics Both tools and data sources available Currently used as Training platform for the ESTP courses on Big Data Shared test environment for the ESS Big Data project promoted by Eurostat

9 Case Study: Istat Big Data Platform In-house Big Data platform 8-node Hadoop Cluster + Spark + MPP DB Designed for use in both production and experimental projects Scanner data Population estimates with mobile phone data Completely hosted on-premise and managed by internal staff Motivated by privacy constraint over datasets Costly solution evaluate cloud when possible

10 Lessons Learned: The Problem with Size Volume is the dimension having highest practical impact in real Big Data projects Datasets in the order of Tb of size, constantly growing No absolute threshold over which data becomes big strictly related to the kind of processing that they should be subject to Real datasets are smaller than what many companies are used to treat daily, but not common for an NSI

Lessons Learned: IT Tools as Enablers of Advanced Analytics Big data tools enable operations and analysis that are not possible with standard tools Or, they can provide better

11 Lessons Learned: IT Tools as Enablers of Advanced Analytics Big data tools enable operations and analysis that are not possible with standard tools Or, they can provide better performance with respect to traditional tools when dealing with large data sets Example of execution times of real operations Istat Scanner Data project Big data tools vs RDBMS

12 Lessons Learned: Skills and Roles The relationship between IT and statisticians within a statistical organization is a particularly critical issue when working with Big Data How to exploit the potential of technology without compromising the autonomy of the research? Complete separation of concerns as commonly experienced in NSIs, may lead to inefficiencies. Capacity building is crucial New mentality needed! Mix of competences, collaborative approach

13 Conclusions The rising of Big Data created new problems for statistical organizations, that lie at the intersection of statistical analysis and IT Technology is now mature enough to provide us with a huge potential for easily and help organizations to give the answers to new and more complex questions A paradigm shift is needed in the approach to those aspects of the statistical business that need to exploit new technologies More cohesive and mixed approach between IT and statisticians No matter how big the datasets we will manage, this is only a part of the general problem of how statistical organizations would initiate the transition from traditional methods to the modern challenges of data science.

Experiences in the Use of Big Data for Official Statistics

Think Big - Data innovation in Latin America Santiago, Chile 6 th March 2017 Experiences in the Use of Big Data for Official Statistics Antonino Virgillito Istat Introduction The use of Big Data sources