ESSnet KOMUSO Workshop on "Quality of Multisource Statistics"

Transcription

1 ESSnet KOMUSO Work Package 3 (SGA-3) ESSnet KOMUSO Q u a l i t y i n M u l t i s o u r c e S t a t i s t i c s h t t p s : / / e c. e u r o p a. e u / e u r o s t a t / c r o s / c o n t e n t / e s s n e t - q u a l i t y - m u l t i s o u r c e - s t a t i s t i c s - k o m u s o _ e n M a t e r i a l f o r ESSnet KOMUSO Workshop on "Quality of Multisource Statistics" Copenhagen December 6th to 7th Program 2. Presentation and workshop questions on the handbook Quality Guidelines for Multisource Statistics (QGMSS) 3. Presentation and workshop questions on Revision of Quality Guidelines for Frames in Social Statistics (QGFSS ) 4. Presentation and workshop questions on Quality Measures and Calculation Methods (QMCMs) [1]

2 ESSnet KOMUSO Work Package 3 (SGA-3) Program The workshop consists of sessions related to the work of the ESSnet KOMUSO on the development of quality guidelines for multisource statistics including statistics based on data from administrative registers. Participants are expected to contribute actively by discussing the questions put forward by the KOMUSO team in a number of roundtable discussions. A cover note for each roundtable including questions for the participants' preparation is included in this document. Additional material including full versions of the KOMUSO networks deliverables can be found at: The workshop is organized so that all participants will take part in discussions of all the topics. Participants will be divided in three groups. Each group will deal with the three issues QGMSS, QGFSS, QMCMs in turn. By organizing it this way we ensure not only that everybody will be able to discuss all the issue but also that it can be done in a dialogue with the responsible work package leaders from the KOMUSO network. Thursday December 6 th 12:00-13:00 Arrival, registration and lunch 13:00-13:15 Welcome by Statistics Denmark 13:15-14:45 Introduction to each of the three rounds of discussion by KOMUSO work package leaders from Statistics Italy, Austria and The Netherlands 14:45-15:00 Break 15:00-16:00 1. Roundtable 16:00-16:30 Report from each table on the first round of discussions 16:30-16:45 Break 16:45-17:45: 2. Roundtable 17:45-18:00 End of day one Friday December 7 th 9:00-9:30 Report from each table on the second round of discussions 9:30-10:30 3. Roundtable 10:30-11:00 Report from each table on the third round of discussions 11:00-11:15 Break 11:15-12:15 Plenary on dissemination and the potential content of the course in September :15-12:30 Wrap up and goodbye 12:30-13:30 Lunch [2]

3 ESSnet KOMUSO Work Package 3 (SGA-3) 2. Presentation and worksho p questions Quality G uidelines for Multisource Statistics (QGMSS) preparatory for the Worksho p Version October 2018 Prepared by Giovanna Brancato and Gabriele Ascari (Istat, Italy) ESSnet co-ordinator: Niels Ploug (DST, Denmark), npl@dst.dk, telephone Presentation of the handbook Quality Guidelines for Multisource Statistics (QGMSS) Workshop reference version: QGMSS version 0.81 (November 2018) 1 The aim of the handbook is to provide practical and applicable quality guidelines supporting the design, implementation and quality assessment of multisource statistics, shared at the European Statistical System. 1 Please note that the initial workshop invitation linked a previous version 0.8 (September 2018), which was in the meantime further revised. Links on the workshop homepage are up to date. [3]

4 ESSnet KOMUSO Work Package 3 (SGA-3) Quality guidelines are agile and relative short manuals providing indications on what should be carried out in the statistical process in order to guarantee general quality principles. Usually they are not documents describing the methodologies to be applied, which instead are treated in more extensive methodological manuals. These quality guidelines are oriented to statistical processes practitioners of the National Statistical Institutes (NSIs) of European Statistics System (ESS) member states, who can use them as a reference to be guided in the production of multisource statistics at the highest quality standards. They also support the increase of quality awareness in multisource statistics. Finally, they can be used as a reference to assess the quality of the statistics produced in the multisource framework. In the end, they can be considered as a tool to reinforce the quality of the European statistics developed using multiple sources. Many NSIs have developed quality guidelines. However, a manual considering the multisource perspective with quality guidelines commonly agreed at European level was still in the interest of both the member states and Eurostat. Multisource statistics are statistics produced using several complementary data sources for direct estimation, i.e. direct tabulation or substitution and supplementation for direct collection (Statistical Network, MIAD, A1, 2014). The source data can range from survey data (sample or census), administrative to any other kind of data obtained from public or private data owners. For the purpose of this manual, the combinations of sources are restricted to one survey and one or more administrative datasets. The indirect usage of administrative data, e.g. for validation, as well as big data and administrative data with big data features are excluded. The handbook is organised into two parts. Part 1 describes the quality framework underlying the guidelines and provides some hints on quality management in multisource context. The quality framework links together different quality facets: output quality dimensions (assumed to be the European Statistical System quality dimensions), process quality activities (mapped using the UNECE Generic Statistical Business Process Model, GSBPM), statistical s arising from the processes using both administrative and survey sources (e.g., coverage, measurement, ). In Part 2, for each Eurostat output quality dimension general quality principles and guidelines are provided. The principles reflect general quality requirements. Principles and guidelines are developed around three main objectives of process quality activities: i) the prevention; ii) the monitoring/correction/adjustment of possible s during the statistical production process; iii) the assessment/estimation of the impact of the s on the final estimates. Some activities, e.g. the assessment of the relevance via users satisfaction surveys, are independent from the multisource nature of the statistical production process. Other activities aimed to prevent and correct the s are carried out separately on the administrative and survey components. For the sake of completeness, these are reported together with the activities more tailored to the multisource context or performed on the joint sources. References to standard quality indicators and output quality measures and calculation methods are included in the texts of the guidelines. Output quality measures and calculation methods are summarised at the end of each chapter. According to the goals of ESSnet KOMUSO SGA2 2, the handbook is complete with respect to part I and the chapters on Relevance, Accuracy, Timeliness and punctuality, Accessibility and clarity of part II. The chapters on Reliability and Coherence and Comparability are foreseen during SGA3. 2 SGA: Specific Grant Agreement. The KOMUSO project has been organised into three SGAs. [4]

5 ESSnet KOMUSO Work Package 3 (SGA-3) Organisation of the discussion during the workshop Participants to the workshop are asked to read the handbook beforehand and be prepared to contribute to the handbook enhancement by providing feedback on the following issues: deletion or reformulation of the existing guidelines or inclusion of new ones; short descriptions of practical experiences that can be framed in existing or new prosed guidelines. General questions: 1. Would you consider useful to have the handbook clearly split into two parts, so that readers interested more to the theoretical issues can read the first part and readers more interested to the practical activities can read only the second part? 2. Considering the output quality measures and calculation methods, do you find useful to have summaries at the end of each chapter or would you consider sufficient just to have a list and have the fully description in the annex? Specific questions: 3. Would you delete/add/reformulate any guideline to Chapter 2.1. Relevance: Delete: Add & motivate: Reformulate & motivate: Practical experience (guideline n....): 4. Would you delete/add/reformulate any guideline to Chapter 2.2. Accuracy: Delete: Add & motivate: Reformulate & motivate: Practical experience (guideline n....): [5]

6 ESSnet KOMUSO Work Package 3 (SGA-3) 5. Would you delete/add/reformulate any guideline to Chapter 2.4. Timeliness and punctuality: Delete: Add & motivate: Reformulate & motivate: Practical experience (guideline n....): 6. Would you delete/add/reformulate any guideline to Chapter 2.6. Accessibility and clarity: Delete: Add & motivate: Reformulate & motivate: Practical experience (guideline n....): Questions for the future activities 7. Do you have any suggestion on guidelines or practical experiences for the quality dimension Reliability? Guidelines: Practical experience: 8. Do you have any suggestion on guidelines or practical experiences for the quality dimension Coherence and Comparability? Guidelines: Practical experience: 9. Do you find the guidelines usable in practice? If not (fully), how can they be made more usable? Suggestions: 10. Let's now consider the ESTP course that is going to be delivered in the third quarter of We intend to provide an overview of the theoretical quality framework and of the more significant guidelines for preventing, [6]

7 ESSnet KOMUSO Work Package 3 (SGA-3) monitoring and assessing the s, with reference only to those concerning specifically the multisource production environment. Some time will be devoted to a short group work on these subjects. Do you agree with this structure? Given the limited time, would you prefer to have an overview of all the quality dimensions or a deeper focus on some of them? Would you consider useful to have the complete overview also on the guidelines that are applicable to each single component (i.e. only to the survey or only to the administrative data component? [7]

8 ESSnet KOMUSO Work Package 3 (SGA-3) 3. Presentation and worksho p questions Revision of Quality Gui delines for Frames in Social Statistics (QGFSS ) preparatory for the Workshop Version October 2018 Prepared by Thomas Burg and Magdalena Six (Statistics Austria, Austria) ESSnet co-ordinator: Niels Ploug (DST, Denmark), npl@dst.dk, telephone Revision of Quality Guidelines for Frames in Social Statistics (QGFSS) Workshop reference version: QGFSS version 1.0 (May 2018) During SGA II of ESSnet KOMUSO a version 1.0 of the Quality Guidelines for Frames in Social Statistics (QGFSS) has been delivered. On the way to the version 1.0 a draft version 0.91 was subject on written consultations in the Working Group Quality and the Steering Group ADMIN. The document was presented and discussed in the Working Group Quality and was as well subject of a presentation in a special session of the Q2018 in Cracow. The document consists of five chapters starting with an introduction outlining the purpose and the objectives of the document as well highlighting the special objectives related to the European Code of Practice. The second chapter sets the fundament by developing a definition for frames in social statistics and investigating the questions which processes are related to frames. It is a main objective of the chapter to define a terminology, taking into account the different situations in the NSIs, and to develop a common language as a basis for the subsequent chapters. The different terminology may simply originate in different customs. But also the possibilities to use registers and the legal frameworks among the countries in the ESS vary considerably and may influence the [8]

9 ESSnet KOMUSO Work Package 3 (SGA-3) usage of specific terms. (Does a central population register exist? Do common identifiers exist? How do the legislative restrictions look like to avoid privacy issues?). Chapter three to five can be seen as the central part of the document re containing the guidelines. Table 1: Chapters 3 to 5 - topics for guidelines and subtopics 3. Construction and Maintenance 4. Use of frames 5. Quality 3.1 Sources for constructing 3.2 Organization, planning coordination 4.1 Sampling Sampling frames Contact variables 4.2 Support for processing Weighting and calibration Editing and imputation 5.1 Methods to assess quality 5.2 Quality and Metadata 3.3 Methods for construction 4.3 Direct use for statistics 3.4 Possible outputs As table 1 shows, the main chapters Construction and Maintenance, Sampling, and Quality are further divided in various sup topics and each sub-chapter follows the same structure: The first part gives a general overview and a general description of the topic Subsequently challenges are described: What kind of s can occur due to problems with the topic of the sub-chapter? Which quality dimensions are affected due to these problems? In the third part of each sub-chapter the concrete quality guidelines in so called grey boxes are presented. One general remark is that the aim is to provide guidelines for frames in social statistics. This means it is not the intention to advice on processes itself. If we look for instance at the chapter dealing with the use of frames in sampling we do not like to give guidelines on sampling (stratification or how to comply with precision requirements). This created sometimes a kind of rendering problem because it turned out difficult to separate the one for the other. What now is available is sometimes a compromise solution in this regard. Another important aspect is the question of minimum requirements. For some guidelines currently elements are listed showing what is seen as the basic prerequisite for compliance to a guideline. However it seems evident that not every guideline is suitable to be enhanced by minimum requirements. Although wit is not intended to go into further detail here it seems worth to be mentioned that subchapter 4.3 Direct use for Statistics addresses the possibility to use frame data as direct input to compile statistical outputs as part of statistical products. Since frames are nowadays composed on the basis of a broad variety of data sources using frames for direct estimation, i.e. direct tabulation or substitution and supplementation for direct collection can be seen as a new idea for frame usage compared for instance to the classical one as simply used only for sampling purposes. The idea in the document is to provide guidelines how the aim of optimizing the six criteria (relevance, accuracy, timeliness and punctuality, accessibility and clarity, comparability and coherence) can be achieved when frame data are used as direct input source for statistics. [9]

10 ESSnet KOMUSO Work Package 3 (SGA-3) For a document like the QGFSS it is important to receive as much acceptance as possible. Therefore a need for the involvement of those who will be working with the guidelines seems to be a necessary prerequisite in order to test the practical usability of the guidelines in an appropriate way. Given that the participants of the workshop are encouraged to provide input regarding the questions provided by the next chapter. The results of this workshop together with input already received will form the basis for a revised version of the QGFSS. Organisation of the discussion during the workshop Participants to the workshop are asked to read the document beforehand and be prepared to contribute to the preparation of a new version by providing feedback on the following questions. The aim of this exercise is to come to 1. a common understanding for definitions used in the document A list of common definitions, incl. frame and sampling frame (a schema would be preferable) which is as well consistent to other relevant documents and glossaries 2. a minimal set and optional set of variables in a frame of Social statistics having first of all in mind one single frame for all social surveys as an optimal solution 3. quality indicators to be included into the chapters where appropriate. Now we have a set of quality indicators defined in Annex I of the document. The idea is to include quality indicators in relevant sections of the QGFSS where appropriate. 4. proposals for new/deleted/rephrased quality guidelines. This concerns not only guidelines but as well the matter of minimum requirements related to guidelines. 5. usable in practice. Do you find the guidelines usable in practice? If not (fully), how can they be made more usable? Based on the objectives listed above participants are encouraged to prepare input by answering the following five questions on the two subsequent pages. It should be mentioned that each of the boxes can be replicated if there is more than one proposal. [10]

11 ESSnet KOMUSO Work Package 3 (SGA-3) 1. The document should be enhanced by a list of common definitions. Which definitions should be integrated or reformulated? Term to define: Proposed definition: 2. What would you see a minimal/optimal set of variables included in which should be contained in a frame for social statistics (provided the fact that we use a single frame for all social surveys)? Variable to be included in the frame: Definition: Seen as a minimal requirement Seen as an optional requirement Argumentation: [11]

12 ESSnet KOMUSO Work Package 3 (SGA-3) 3. Quality Indicators: What quality indicators should be integrated directly into the chapters containing the guidelines? Quality indicator to be integrated into the document: Definition: To be integrated into chapter: 4. Would you prefer to delete/add/reformulate any guideline in the document? Chapter: Guideline: To be added To be deleted Reformulate: 5. Do you have any further ideas how the practical usability of the guidelines could be improved? Suggestions: [12]

13 ESSnet KOMUSO Work Package 3 (SGA-3) 4. Presentation and worksho p questions Q uality Measures and Calculation Methods (QMCMs) Version Prepared by Ton de Waal (Statistics Netherlands) ESSnet co-ordinator: Niels Ploug (DST, Denmark), npl@dst.dk, telephone Workshop reference versions QMCMs: Hands-on examples: [13]

14 1. Introduction In this report we give a brief overview of the results of Work Package 3 ( Quality measures and indicators ) of Specific Grant Agreement (SGA) 3 of KOMUSO (ESSnet on quality of multisource statistics). The results are a set of output quality measures, supported by applications and computation details, that will integrate and complement the ESS Quality Guidelines for Multisource Statistics, where they are referred to and provided as an annex. Those ESS Quality Guidelines for Multisource Statistics will be produced by Work Package 1 ( Guidelines on the quality of multisource statistics ) of the same ESSnet. The ultimate aim of the ESSnet is to produce usable quality guidelines for National Statistical Institutes (NSIs) that are specific enough to be used in statistical production at those NSIs. The guidelines will cover the entire production chain (input, process, output). They aim to take the diversity of situations in which NSIs work and the restrictions on data availability into account. The quality of the final output will depend both on the existing data sources and on the use and processing of the data. It is therefore clear that general decision rules and single thresholds do not suffice. Instead the guidelines list a variety of potential indicators/measures, indicate for each of them their applicability and in what situation it is preferred or not and provide an ample set of examples of specific cases and decision-making processes. For this reason, the first SGA of the ESSnet identified several basic data configurations for the use of administrative data sources in combination with other sources, for which it proposed, revised and tested some measures for the accuracy of the output (see KOMUSO, 2017). In the second SGA Work Package 3 of the ESSnet continued this work on indicators/measures, by developing further quality indicators/measures related to process and output needed for the use in practice of the guidelines. In particular, we documented the examined quality indicators/measures and accompanying calculation methods. These documents are referred to as Quality Measures and Calculation Methods (QMCMs). The QMCMs, and a number of related hands-on examples, are the results of Work Package 3 that form the Annex to the Quality Guidelines for Multisource Statistics. The remainder of this report is organized as follows. Section 2 lists the QMCMs and related hands-on examples that have been developed in Work Package 3. Section 3 concludes the report by briefly describing these QMCMs. 2. Overview of QMCMs and examples Below we present quality indicators/measures for which we have produced QMCMs. The QMCMs are listed in three tables: for accuracy (Table 1), for timeliness (Table 2) and for coherence (Table 3). In the first column of these tables the sources the QMCMs aim to quantify are given. Each QMCM is given a code consisting of the letters QMCM, a letter referring to the quality dimension ( A stands for accuracy, T for timeliness and C for coherence ), and a number. In the column QMCM we give that number. In the same column we also indicate whether we have written a hands-on example for that QMCM in SGA 2. The QMCMs and the related examples themselves are available as separate documents. The column Data config. refers to the basic data configurations that have been identified in the first SGA of the ESSnet (see KOMUSO, 2017). The QMCMs and related hands-on examples are made available on the CROS portal: 14

15 Table 1. Measures and indicators on Accuracy Error type(s) 3 Data Sources Quality config. 4 measure / indicator More information QMCM Sampling ; Coverage ; ; Non-response ; Processing Frame and Selectivity s; and processing s Sampling ; Sampling ; All Administrative data only 2 Combination of several administrative registers and survey datasets 5 Combination of aggregated data 5 Combination of aggregated data (administrative and/or survey data) Expert opinion based on questionnaire Bias, variance and validity Variance Mean squared The Quality Assessment Tool described in the paper is based on a set of questions to be answered by the statistical agency on the one hand and by the administrative agency on the other hand. In particular, the statistical agency has to document their quality expectations to the data source whereas the data supplier has to document the actual characteristics of their data or processes. This is a generic framework for disentangling sources. The development of specific estimation methods for different sources remains. In principle, the approach can be used for any statistics. The approach has been used on employment status production process. The approach can be used to estimate the variance of reconciled totals and the reconciliation has done by means of a macro-integration technique. applied to: Reconciled data on International Transport and Trade Small test datasets applied to assess the quality of estimates for municipal unemployment based the Labour Force Survey. QMCM_A_1 QMCM_A_2 QMCM_A_3, QMCM_A_4 Examples are provided ( Example QMCM_A_3 and Example QMCM_A_4 ) QMCM_A_5 Example is provided ( Example QMCM_A_5 ) Sampling 4 Combination of microdata and Variance The approach can be used to estimate the variance of cells QMCM_A_6 3 The types that we distinguish in this document are based on the ESS Handbook for Quality Reports (2014). For information on (the definition of) these types we refer to that Handbook. 4 The data configurations are based on the basic data configurations identified in SGA 1 (see KOMUSO, 2017). 1 means complementary microdata sources, 2 overlapping microdata sources, 3 overlapping microdata sources with under-coverage, 4 microdata and macrodata, 5 only macrodata, and 6 longitudinal data. 15

16 Error type(s) 3 Coverage (frame ) (Validity ) Data config. 4 Sources Quality measure / indicator aggregated data 2 Several administrative or survey sources with overlapping units and variables 2 Combination of several administrative registers, using microintegration Coverage 3 Two or more (administrative) datasets ; Processing 2 Combination of several data sources with overlapping units and variables Bias, variance Bias Confidence interval Qualitative indicator of quality More information in tables obtained by so-called repeated weighting. applied to the Dutch Population and Housing Census, which is based on a mix of administrative and survey data. applied to the Quarterly Survey on Earnings. The approach measures accuracy of the estimates based on the predicted values. applied to register-based employment data and Labour Force Survey data. The approach estimates the confidence interval for the population size and its domain size. applied to an automated system of decentralized population registers (with information on people that are legally allowed to reside in the Netherlands and are registered as such) and a Central Police recognition system where suspects of offences are registered. The approach combines quantitative information with expert knowledge to compute quality indicators for the whole data editing process. Two kinds of situations are distinguished: (1) the output value comes from a data source and there are misclassifications in all data sources, or (2) the output value was imputed. applied to: a register-based census register-based labour market statistics QMCM Example is provided ( Example QMCM_A_6 ) QMCM_A_7 QMCM_A_8 QMCM_A_9 QMCM_A_10 Example is provided ( Example QMCM_A_10 ) 16

17 Error type(s) 3 ; Processing (linkage ) Data config. 4 Sources Quality measure / indicator 2 Two data source Bias, variance (administrative data and/or survey data) 1 Combination of administrative data and survey data (business data) 2 Combination of several data sources with overlapping units and variables (categorical data) 2 Business register (with delayed information) and survey data 2 Two or more datasets with overlapping units and the same target subject to measurement Bias, variance Bias, variance Bias, variance Confidence intervals More information The approach can be used to measure the impact of linkage s (and methods to correct for these s) on the quality of estimated frequency tables. applied to Census data linked to a settlement database. The approaches examine the effect of incorrect NACE classifications in the Business Register on the quality of the output. applied to Quarterly VAT data and survey data. Two kinds of approaches have been studied. In one approach it is assumed that all data sources may contain s. In the other approach it is assumed that one data source is free and the other data sources contain auxiliary data. The approaches have been applied to: Employment status derived from Labour Force survey (LFS) data and administrative data Employment status from LFS with administrative data as auxiliary variables The approach measures the impact of the frame s on bias and variance of the estimator of a total in the case enterprises may join, split or change their kind of activity during the year. applied to enterprise data on turnover. The approach can be applied to measure the quality of reconciled microdata when both data sources can contain classification s. applied to estimate the quality of home-ownership status observed in several datasets. QMCM QMCM_A_11 Example is provided ( Example QMCM_A_11 ) QMCM_A_12 Example is provided ( Example QMCM_A_12 ) QMCM_A_13 QMCM_A_14 QMCM_A_15 Example is provided ( Example QMCM_A_15 ) 17

18 Error type(s) 3 (Validity ) Data Sources Quality config. 4 measure / indicator 2, 6 Combination of several longitudinal data sources with overlapping units and variables (categorical data) 2 Several administrative datasets 2 Combination of administrative data with survey data, with overlapping units and variables (numerical data) 6 Combination of longitudinal administrative data and survey data Misclassification rate Aggregated predicted person place probabilities for housing units Validity of observed variable as indicator for target variable, Bias due to measurement Bias, variance More information The approach measures the misclassification rate for observed variable with respect to target variable. applied to: administrative data and survey data on homeownership register data and survey data on jobs and benefits The approach can be used to assess the effect of classification s on the output. To this end, an assessment of the so-called ROC curve (plot of the true positive rate against the false positive rate) is used. applied to Census enumeration. The approach estimates the effect of measurement s in administrative and survey variables by structural equation models. applied to VAT data and survey data for turnover. The approach derives analytical expressions for the accuracy of growth rates as affected by classification s. applied to quarterly turnover growth rates based on business register and survey data (Short Term Statistics). QMCM QMCM_A_16 Example is provided ( Example QMCM_A_16 ) QMCM_A_17 QMCM_A_18 Example is provided ( Example QMCM_A_18 ) QMCM_A_19 Example is provided ( Example QMCM_A_19 ) Table 2. Measures and indicators on Timeliness Error type(s) s Data config. Sources 2 Combination of several administrative Quality measure / indicator Bias More information Effect of progressiveness (delayed input data). applied to Employment Status data. QMCM QMCM_T_1 18

19 Table 3. Measures and indicators on Coherence Error type(s) Sampling ; Coverage ; ; Non-response ; Processing Coverage ; ; Non-response ; Processing,; Model assumption ; (Specification ) Model assumption Data config. Sources 4, 5 Combination of microdata and aggregated data All data configu rations All data configu rations Any type of data sources, where an external source is available All kinds of data sources Quality measure / indicator Scalar uncertainty measure Indicators on cross-domain coherence Mean Absolute Revision (MAR); Relative Mean Absolute Revision (RMAR); Mean Revision (MR) More information The approach measures the uncertainty in reconciled estimated accounting equations. applied to quarterly and annual supply and use tables. In this approach released data are compared to estimates from other sources. Application to real data in which a revision indicator, used to measure Reliability, is also used to measure the coherence between several related datasets. QMCM QMCM_C_1 Example is provided ( Example QMCM_C_1 ) QMCM_C_2 Example is provided in ST_C_4 QMCM_C_3 An example is already provided in the QMCM itself 19

20 3. Descriptions of QMCMs In this section we give brief descriptions of the QMCMs mentioned in Section 2. The descriptions are largely based on the description of these QMCMs in Quality Guidelines for Multisource Statistics produced by Work Package 1 of the ESSnet. 3.1 Accuracy QMCM_A_1: Questionnaire with open questions The decision to acquire an administrative source must be taken by the statistical organization on the basis of the available information regarding the source. Interaction and feedback between the organisation and the data producer should aim to make the quality of the source and the expectations of the organisation meet. However, this process may be time-consuming and prone to s. The tool described QMCM_A_1 tries to solve this issue and make the communication between the two entities easier. It is a questionnaire that both the statistical administration and the data producer should answer, preferably at the beginning of the data acquisition step. The answers from both parties should clarify what is expected from the data, what uses are considered for the data, what can be done to improve future releases and so on. Besides accuracy, other quality dimensions such as coherence and reliability are investigated. Finally, it has to be noted that no quality measures are computed in the questionnaire so it is not a quantitative tool. QMCM_A_2: Modelling of total in multisource statistical data QMCM_A_2 describes a two-phase framework to measure the total for the situation where data from multiple sources are integrated to create a statistical micro dataset. This total comprises the s occurring during the construction of each input source (first phase) and the s occurring during the integration process (second phase). QMCM_A_2 suggests obtaining at least qualitative descriptions for the envisaged s and their nature; quantitative indicators depend on the nature of the s and the availability of a target dataset and the statistical knowledge and metadata needed for the computation. QMCM_A_3 and QMCM_A_4: Variance-covariance matrix for a reconciled vector An issue that may arise when using multiple sources directly for producing estimates is that the estimates may differ and be in need of reconciliation. This problem, from a macro-data point of view, has been studied in QMCM_A_3. The proposed methodology, applied by the authors to international trade and transport statistics, considers that, besides the initial set of estimates of the variables of interest, there may also be additional information on them derived by other sources and linear equalities between them that should hold by definition. The additional information and its covariance matrix can be used to update the initial estimates and variance matrix. This leads to reconciled estimates and an updated variance matrix. Obviously, the smaller the updated variances, the more accurate the reconciled estimates can be considered, as the updated variance estimates can be considered a quality measure for the accuracy after the reconciliation process. If the variables of interest are also subject to inequality restrictions, the methodology described should be integrated with the so-called border method or the so-called approximate moments method to obtain the vector of reconciled variables estimates. In QMCM_A_3, data are assumed following a normal distribution and initial estimates should be available for all variables of interest, albeit not necessarily from a single source. 20

21 Another application dealing with the same problem (QMCM_A_4), but within a quadratic programming framework, has also been developed; here the assumptions behind the model are similar and the calculations to be computed depend on the type of restrictions that have to be obeyed. QMCM_A_5: Mean squared of small area estimates Small area estimation is a technique aimed at improving the accuracy of estimates for samples involving small sections of the target population. This methodology is based on the use of auxiliary data and on some assumptions regarding the input estimates and the distribution of the s. If such assumptions do not hold true, the final small area estimates may be biased. QMCM_A_5 illustrates the application of small area estimation methodology on municipalities. QMCM_A_6: Variance of cell values in estimated frequency tables Sampling variance is a major component of the total mean squared. As such, in a statistical process based on multiple sources, there is the need to compute and compare measures of the variance obtained from the various sources. QMCM_A_6 approaches this issue when the sources are multiple surveys used to estimate frequency tables. In this situation, variance of each cell value in the frequency tables and their covariances can be considered a measure of the accuracy of the estimates. Moreover, since the estimates in the table are derived from different sources, there is the need to assure their consistency. The repeated weighting (RW) estimator used in the application ensures the consistency among tables estimated from different surveys and gives an estimation of the variances. The procedure basically consists in repeated applications of the calibration estimator, where results are calibrated on previously estimated figures. In particular, the variance-covariance matrix of the RW estimator, whose diagonals provide estimated variances for the frequency tables, is the product of so-called super-residuals, linear combination of ordinary residuals. Indeed, these computations can get quite complex since the calibration is based on already estimated figures. QMCM_A_7: Effect of the frame under-coverage / over-coverage on the estimator of total and its accuracy measures It is often the case that an administrative source is updated constantly for non-statistical purposes and thus ends up more complete than a traditional frame such a sampling list or a fixed register. In situations like this the frame that is used for sampling will probably be affected by under-coverage or over-coverage, for it does not take into account the units that have entered or exited the population. So, it may be a legitimate choice to adopt the complete administrative archive as an auxiliary source. QMCM_A_7 studies the effect of coverage issues on the business register frame and the relation with the social insurance inspection database, characterised by perfect coverage. The database contains an auxiliary variable, correlated with the variable of interest that can be used for the computation of the estimator when the changes in the population size are considered. QMCM_A_8: Quality assessment of register-based outcome variable in the presence of a sample survey for the same variable If a variable can be estimated from an administrative-based register, bias of the estimate not only gives an indication of the accuracy of the estimator, but may also indicate a possible lack in validity. In the context of QMCM_A_8 it is impossible to distinguish validity from measurement. In order to assess the validity, along with the overall bias of the register-based statistic, QMCM_A_8 proposes a comparison with an estimate of the same variable from a survey, if such source is available. The proposed quality measure is the estimated bias of the register-based subpopulation estimator. This bias is estimated by 21

22 applying a small area model on the survey data in combination with the register data. The proposed quality measure is a weighted average of the directly observed difference between the register-based and surveybased subpopulation estimates, and the average bias of the register-based estimator across all subpopulations. Assumptions of the proposed quality measure are no variance for the register-based estimator and no bias for the survey-base estimator. QMCM_A_9: The confidence interval for population/domain size estimator In the presence of under-coverage in a frame, not only the estimates of the population variables are affected by greater inaccuracy, but the population size itself is subject to uncertainty. Therefore, it is useful to obtain a confidence interval for this latter quantity, if this quantity is estimated. The method proposed in QMCM_A_9, which is based on capture-recapture techniques, assumes the presence of two separate lists: one from the population census, the other from a post-enumeration survey. The method on four main assumptions: (i) independence of inclusion (the probability of inclusion in one list is independent of the probability of inclusion in the other list), (ii) inclusion probabilities are homogeneous for at least one list, (iii) the population is closed, and (iv) elements in the two lists can be perfectly linked. QMCM_A_9 proposes the use of the confidence interval estimate for the population/domain size as a quality measure for the estimated population/domain size. The narrower the confidence interval is at a given confidence level, the more accurate is the population size estimate. The analysis can also be generalised to the case of three data sources for which the computation of parametric bootstrap confidence intervals for the population size is suggested. QMCM_A_10: Combined quality assessment indicator Using multiple data sources to generate statistics needs several process steps. The framework described in QMCM_A_10 introduces an indicator between 0 and 1 assessing the quality in every stage of the data processing (raw administrative data; the combined dataset, i.e. the integration of registers; and the final dataset, i.e. after imputation of missing data) for each attribute. Due to the modular design, every step of the framework could be applied individually. The approach for the assessment of administrative data relies on four quality-related hyper-dimensions (Documentation, Pre-Processing, External Sources and Imputations). Documentation describes quality-related processes as well as the documentation of the data (metadata) at the administrative authorities. The degree of confidence and reliability of the data source keeper was monitored by using a questionnaire. Pre-Processing refers to the proportion of data records that cannot be used. In the External Source dimension the administrative data source is compared with another source, for example the Labour Force Survey, by matching individual records and computing the share of consistent observations per variable and administrative data source. The entire information from the registers is combined with the central database which covers all attributes of interest. At this level, a quality indicator for each attribute across all data sources is computed. If a variable is only derived from one administrative data source, then the quality of this attribute on raw data level is the same as in the central database. If several administrative data sources are combined in order to derive a variable or to establish the most plausible value, then the quality indicator is calculated. This is done by using the Dempster-Shafer theory in order to combine quality indicators from different data sources. In addition, a comparison with an external source is carried out. In the last step of the data processing, missing values in the central database are imputed. For the assessment of the data quality in the final dataset, the quality indicator for Imputation is computed. 22

23 QMCM_A_11: Variance of a bias-corrected estimator which aims to correct for bias due to linkage s When two datasets are linked through a non-unique identifier, s may occur and the resulting estimates may be biased. QMCM_A_11 deals with this issue by adopting a probabilistic linkage procedure and computing an estimator aimed to correct for the linkage bias. The variance of this estimator is a measure of the accuracy of the estimates. The probabilities of linkage s are grouped in a matrix, which enters in the computation of the bias-corrected estimator and its variance. The variance itself is made up of three components, two of which can be estimated through a bootstrap procedure and the third analytically. One of the main advantages of this method is that it can be applied to more than one probabilistic linkage model; on the other hand assumptions such as the homogeneous distribution of the linkage s probabilities may be violated in practice. QMCM_A_12: Mean squared of level estimates affected by classification In QMCM_A_12, stratum estimates are obtained by adding up the data within each stratum. However, the variable on which the division of the strata depends is affected by classification s. This leads to s in the stratum totals. Classification s are described by a transition matrix, containing classification s probabilities estimated through an independent and -free collection of data. Once such probabilities have been estimated, the following step concerns the assessment of the bias and variances through a bootstrap procedure (when dealing with level estimates of stratum totals, analytic formulae can be used instead). The main obstacle in this method may be obtaining a sample of data for the estimation of the transition matrix that are clean of classification s. QMCM_A_13: Relative bias and relative mean square QMCM_A_13 proposes, in a model-based approach, a measure of accuracy with respect to measurement, in particular the in classifying individuals with respect to employment status. Contrary to classical approaches, this method entails an unsupervised approach to the use of administrative data along with a traditional survey sample. This is done by considering the target variables as latent variables, of which researchers can only obtain imperfect measures. The application has been used on Italian market labour data, specifically from the Labour Force Survey and related administrative data. Data are used to draw g estimates for a target variable, where g is the number of available sources; such estimates are part of the measurement model, which may be affected by measurement s and model misspecification. Data are modelled following Hidden Markov Models and estimates are obtained through likelihood methods. Simulations are also carried out by the authors to assess the robustness of the methodology with respect of departures from the model assumptions. Furthermore, distributions of the model parameters can be used to assess the quality of each source. QMCM_A_14: Effect of stratum changes, joining and splitting of the enterprises on the estimator of a total QMCM_A_14 concerns the changes, and the consequent measurement, that may occur in a sample after the units have been selected. Changes may be acquired with delays in a register, resulting in temporarily wrong information. For example, in a business population, in a stratified sample design some businesses may be assigned to a wrong stratum due to changes in their number of employees. Specifically, QMCM_A_14 focuses on three types of measurement s deriving from delayed information: s due to sampling units joining; s due to sampling units splitting; and s caused 23

24 by changes in a classification variable. The three s are treated and measured separately, however they all share the distinction between the selected sample and the observed sample, the latter being the sample after the changes have occurred. The total of the variable of interest in the observed sample is the quantity to be estimated. QMCM_A_14 gives analytical formulas to quantify the effect of these changes on the estimate of a population total, and hence measure the quality of the estimated population total. In particular, these formulas estimate bias and variance of the estimated population total. QMCM_A_15: Variance of estimates based on reconciled microdata In QMCM_A_15, a latent class model is used to produce estimates based on a registers of addresses and buildings and a survey component. Estimates are computed after the reconciliation of microdata; an observed categorical variable is considered to be an expression of the true, but hidden, target variable. This application also introduces restrictions on the latent classes in order to have results that make sense logically (for example, a rent benefit receiver cannot be a home owner): the restricted model is referred to as MILC. While the use of latent class methods can represent a benefit for the accuracy of the estimates, the drawbacks are the complications involving the calculations and the possibility of biased estimates when the covariates contain a classification. QMCM_A_16: Misclassification rates of observed categorical variables in longitudinal data In a situation where two or more linked longitudinal datasets contain the same categorical variable which may be subject to misclassification, it is plausible to represent the true values (i.e. categories) of the variable at different time points by introducing a vector of latent variables. The approach adopted in QMCM_A_16 estimates, for each unit in the sources and for each time point considered, the probability that the unit belongs to the true category. The development of the latent variable through time is described by a Markov model, under the assumption that the classification s are independent. Since this assumption may not be true in practice, various adaptations of the model are proposed, for example introducing a dependency of a classification on certain time points, which is reasonable for the case that the data supplier repeats the same under the same circumstances. In any case, if the assumptions are correct, the model can be used to obtain an estimation of the misclassification rates in all the variables simultaneously. QMCM_A_17: Aggregate predicted person-place probabilities for housing units The indicator described QMCM_A_17 is a measure of the quality of an address variable in an administrative source containing addresses information, specifically data on housing units. Thus, the application focuses on contact address s and linkage s and hence potential coverage having consequences on the accuracy of the estimates. In the method individual probabilities for each person in the housing units are computed and then aggregated. The individual probabilities convey the likelihood for a person-place combination of being correct and can be calculated through various methods, including model-based ones. Then, such probabilities are aggregated (through a minimum function or a mean function), resulting in an overall indicator that ranges from 0 to 1: the closer to 1, the higher the quality of the address variable. 24