Workshop Almaty 28 30 November 2017 Jan Byfuglien
Introductory assumptions Non traditional data sources when used for official statistics will have to be collected and handled in line with international guidelines and best practices for official statistics However, there might be new challenges in relation to existing legislation, organisational setup and methodological approach
The basis for my input: UN Fundamental Principles of Offical Statistics (UNFPOS) European Statistics Code of Practice (ESCOP) Generic law on official statistics (GLOS), including explanatory notes (ECE/CES/2016/8)
What are some main legal issues? 1. What is official statistics? The borderline to administrative data and other statistics 2. Professional independence, impartiality and objectivity 3. Exclusive use for statistical purposes, confidentiality 4. Mechanisms and procedures for planning and coordination 5. Principles, methods and tools for editing and quality control
1: What is official statistics and the system of official statistics? Is national legislation clear about the borderline between official statistics, and administrative data or statistics for sectoral purposes? Statistical legislation sometimes seeks to cover all types of public data collection at all regional levels This is mirrored by the many concepts used in present legislation: Statistics State statistics National statistics Horizontal statistics Basic statistics Central statistics
Why clarify the definition and scope of official statistics? To qualify as official statistics some specific criteria should be fulfilled (FPOS, GLOS, ESCOP): Professionally independent, impartial and objective Accurate and reliable, coherent and comparable Exclusively used for statistical purpose, protected by confidentiality Clear and transparent dissemination, equal access Administrative data/other statistics on the other hand: Based on specific laws and produced for a specific administrative purposes often monitoring individuals Not necessarily coherent and comparable Often other rules for dissemination and access
Why and how to develop use of administrative and other non traditional sources? Administrative sources and systems are of basic importance for developing an efficient and sustainable statistical system also for census like statistics Costs: statistical surveys and censuses are expensive to conduct and process (human and financial resources) Response burden: using administrative sources reduces the burden on respondents and allow more frequent data collection Coverage: in some cases administrative sources offer a better coverage of the target population and allow the production of reliable information for small-area and small population group Timeliness: administrative data might be faster to process than survey data
The need to differentiate between administrative and statistical registers: Administrative registers based on specific laws and for specific purposes Statistical registers based on law on statistics and only for statistical purposes However: Building statistical business registers and population registers without a parallel development of the administrative counterparts is a difficult if not impossible task Necessary to build parallel administrative solutions to feed the statistical system with clear boundaries and interfaces
An example from Norway The basic administrative registers True copies The statistical base registers Legal entities CCRLE CRE Brønnøysundregistrene Cadaster CAD SN-CAD Mapping authority Persons CPR SPR Tax authorities Statistics Norway 9
What is the common identifier? Three base registers in Norway Businesses (legal entities) Id. No. + Address ++ People Address Id. No. + Address + Property id, Building id + Address Properties Addresses Buildings 10
2: How to ensure professional independence, impartiality and objectivity? Why? Political influence and fake data will destroy trust in statistics How? Protection from political and other interference clearly stated in law and proven in practice The status of the Chief Statistician and transparent procedures for appointment/dismissal The challenge producing official statistics within Ministries or administrative authorities: Who decides on what and how data should be collected? Who approves what should be disseminated and when? Is there a clear borderline in relation to administrative usage?
3: Confidentiality: keep individual information secret, exclusive use for statistical purposes Clear rules for handling individual microdata, and small aggregates in tables? One way principle implemented? Identifiable microdata only to the official statistical system, not from. Few countries are able to implement exchange of microdata between the NSO and other producers of official statistics. Easier to integrate data within a strong NSO being able to combine data from different sources?
4: Mechanismes and procedures for planning and coordination? Well established routines for long term and short term planning of surveys for official statistics? Well functioning statistical councils (users) and/or council of producers of official statistics? A system of advisory working groups in different subject areas? Cooperation agreements between the NSO and other partners, based on the law on official statistics?
5: Principles, tools and methods for quality management? The differences between statistical and administrative data collection: Statistical data collection: Using statistical theory and methodology Allowing estimation and handling of non response Administrative and other non traditional data; Concepts not adapted to general statistical concepts? Raises quality issues not covered in general statistical methodology?
How to improve the quality of statistics based on administrative sources? Responsible authority: Provide detailed documentation of source in cooperation with the NSO Implement well defined control procedures in cooperation with the NSO NSO: Implement additional control procedures and provide feedback to the data provider Match and link different data sources; surveys, administrative data ++ Edit and impute data based on documented procedures Clear division of tasks and responsibilities: cooperation agreements between NSO and administrative authority
Some legal and organisational issues: Will non traditional sources require new ways of working within the National statistical system? New partners? The role of the National Stat. Office? Different challenges related to administrative data geospatial information big data? Will non traditional sources require revision of national legal frameworks? Laws prohibiting access for statistical purposes? Compensation for access to data from these sources? New issues related to confidentiality?