ESSNET USE OF ADMINISTRATIVE AND ACCOUNTS DATA. WP6 Quality Indicators when using Administrative Data in Statistical Outputs

Transcription

1 ESSNET USE OF ADMINISTRATIVE AND ACCOUNTS DATA IN BUSINESS STATISTICS WP6 Quality Indicators when using Administrative Data in Statistical Outputs List of Quality Indicators: Draft for Stage 2 User Testing November, 2010

2 Quality Indicators when using Administrative Data in Statistical Outputs Introduction One of the aims of the Admin Data European Statistical System Network (ESSnet AdminData) project is the development of quality indicators for business statistics involving administrative data with a particular focus on developing quantitative quality indicators. Some work has already been done in the area of quality of business statistics involving administrative data and some indicators have been produced, namely under the preparation of the Quality Report Framework for Business Statistics under Regulation (CE) no. 295/2008. However, the work conducted thus far refers to qualitative indicators or is based more on a descriptive analysis of administrative data (see Eurostat, 2003). The quality indicators that have been produced have been more to do with the quality of the administrative sources (Daas & Fonville, 2007) or have been to develop a quality framework for the evaluation of administrative data (Daas, Arends-Toth, Schouten & Kuijvenhoven, 2008). These do not address the quality of the production of the statistical output however. In fact, almost no work has been done on quantitative indicators of business statistics involving administrative data, which is the main focus of this project. The ESSnet aims to develop quality indicators of statistical outputs that involve administrative data. These indicators are for the use of members of the European Statistical System; producers of statistics. Therefore, the list contains indicators on input and process because these are critical to the work of the NSIs and it is the input and process in particular that are different when using administrative data. Moreover, the list of indicators developed is specifically in relation to business statistics involving administrative data. Indicators (e.g. on accessibility) that do not differ for administrative vs. survey based statistics are not included in this work because they fall outside the remit of WP6. To address some issues of terminology, a few definitions are provided below to clarify how these terms are used in WP6 and throughout the ESSnet AdminData. What is administrative data? Administrative data is data derived from an administrative source, before any processing or validation by the NSI. What is an administrative source? A data holding containing information collected and maintained for the purpose of implementing one or more administrative regulations. The list of quality indicators A list of quantitative quality indicators has been developed on the basis of research which took stock of work being conducted in this field across Europe. This list was then user tested within five European National Statistical Institutions (NSIs) and the next step is to engage in wider user testing across a broader range of NSIs. The current list of indicators has been grouped into two main areas: o Background Information these are indicators in the loosest sense. They provide general information on the use of administrative data in the statistical output in question but do not, directly, relate to the quality of the statistical output. This information is often crucial in understanding better those indicators that measure quality more directly. This list has been sub-divided into information relating to the input and information relating to the process. ESSnet Admin data: Quality Indicators 2

3 o Quality Indicators these provide information directly addressing the quality of the statistical output in terms of the input and process involved. Within this grouping, the list is split into input and process indicators. A short description of each indicator is included in the attached list along with a formula on how to calculate the indicator (if applicable). Links between this and other work on Quality The work being carried out under this project should not be seen as independent of other work already in place. When analysing the list of indicators, one can conclude that some other information is useful in regard to the quality of administrative data. However, some of that very useful information can not be (or has not been) translated into quantitative indicators. The main aim of the current project is not to discuss all the issues related to quality when using administrative data. The aim, at this stage, is to discuss basic quantitative quality indicators. In addition, these indicators are for the benefit of the members of the European Statistical System (ESS); the producers of statistics. Consequently, the end result of the ESSnet AdminData work in this area should be integrated with the work already in place on the production of Eurostat Quality Reports. For this reason, quality indicators that can be applied in the same way when using administrative data or survey data are not included in this list. Many of these latter indicators are those specifically related to the statistical output or the publication. For example, indicators in relation to accessibility of the statistics are out-of-scope for this project because accessibility is not impacted by whether survey or administrative data are used. In contrast, this project focuses on the quality of the input and process in the aim of producing the statistical output. This is because input and process indicators are critical to the work of the NSIs and it is the input and process in particular that differ when using administrative data. Doing this for business statistics involving administrative data sets this list of indicators apart from other work in this field. Future work In addition to the list of basic indicators, the ESSnet also aims to investigate the creation of more complex, composite quality indicators that aggregate some of the more basic indicators, allowing easier interpretation of the data and facilitating a global picture of the quality assessment. This work will also integrate the list and the newly created indicators more broadly within the quality framework and other work in this area. References Eurostat, (2003). Item 6: Quality assessment of administrative data for statistical purposes. Luxembourg, Working group on assessment of quality in statistics, Eurostat. Daas, P.J.H., Arends-Toth, J., Schouten, B. & Kuijvenhoven, L. (2008). Quality framework for the evaluation of administrative data. Paper presented at the Q2008 European Conference on Quality in Official Statistics. Rome, Italy. Daas, P.J.H. & Fonville, T.C. (2007). Quality control of Dutch administrative registers : An inventory of quality aspects. Paper presented at the Seminar on Registers in Statistics methodology and quality. Helsinki, Finland. ESSnet Admin data: Quality Indicators 3

4 Input information: Background Information 1 2 This indicator provides information on the number of admin sources used in each statistical output. The number of sources should include all those used in the statistical output whether the Admin data is used as raw data, in imputation or to produce estimations. Admin source refers to a data holding containing information collected and maintained for the purpose of implementing one or more administrative INPUT (1) regulations. Statistical output refers to a statistic produced by the NSI whether based on a specific variable (e.g. no. of employees) or a set of related variables (e.g. total turnover; domestic market turnover; external market turnover). In the broadest sense, statistical output would also apply to the whole STS or SBS output. Number of admin sources used Periodicity (frequency of arrival of the admin data) INPUT (17) This indicator provides information about how often the admin data is received by the NSI. This indicator should be provided for each admin source. Number of admin sources used in statistical output Note. Where relevant, a list of the admin sources may also be helpful for users along with a list of the variables included in each source. Periodicity Note. If data is provided via continuous feed from the admin source, this should be stated in answer to this indicator. ESSnet Admin data: Quality Indicators 4

5 3 4 This indicator provides information on the extent that admin data are used in the statistical output as a proxy or are used in calculations rather than as raw data. This indicator should be calculated on the basis of the statistical output the number of required variables derived indirectly from admin data (because not available INPUT (3) directly from admin or survey data) should be considered. Required variables refer to those necessary to calculate the statistical output. % of required variables derived indirectly from admin data % of required variables obtained exclusively from admin data INPUT (5) This indicator provides information on the proportion of required variables only obtained from admin data, whether directly or indirectly, and where survey data is not collected. This includes where admin data is used as raw data, as proxy data, in calculations, etc. This indicator should be calculated on the basis of the statistical output the number of variables obtained exclusively from admin data (not by survey) should be considered. Required variables refer to those necessary to calculate the statistical output. No. of required variables derived indirectly from admin data No. of required variables Note. If a combination of survey and admin data is used, this indicator would need to be weighted (by number of units). If double collection is necessary (e.g. to check quality of admin data), some explanation should be provided. This indicator could also be weighted in terms of whether or not they are key variables to the statistical output. No.of required variables obtained exclusively from admin data No.of required variables Note. If a combination of survey and admin data is used, this indicator would need to be weighted (by number of units). If there is a combination of survey and admin data, or double collection (admin and survey) this should be explained. This indicator could also be weighted in terms of whether or not they are key variables to the statistical output. ESSnet Admin data: Quality Indicators 5

6 Process Information: 5 6 This indicator relates to the combination of one or No. of relevant common units in the more admin sources. No. of relevant unique units This indicator provides information on the proportion of common units across more than one admin source. Only units relevant to the statistical output should be considered. This indicator should be calculated pairwise, for each pair PROCESS (11) of admin sources and then averaged. Unit refers to enterprise, enterprise group, kind-ofactivity unit, etc. Common units refer to those units that are included in the Admin and survey data. % of common units in more than one admin source % of common units when combining admin and survey data PROCESS (13) This indicator relates to the combination of admin and survey data. This indicator provides information on the proportion of common units across admin and survey data. Linking errors should be detected and resolved before this indicator is calculated. This indicator should be calculated for each admin source and then aggregated based on the number of common units (weighted by turnover) in each source. Common units refer to those units that are included in the Admin and survey data. admin sources 100 % Note. The unique units in the denominator means that units should only be counted once, even if they appear in multiple sources. If only one admin source is available, this indicator is not relevant. This indicator could also be weighted (e.g. by turnover or number of employees) in terms of the % contribution of these units to the statistical output. No. of common units in admin and survey data No. of units in survey Note. If there are few common units due to the design of the statistical output (e.g. a combination of survey and admin data), this should be explained. This indicator could also be weighted (e.g. by turnover or number of employees) in terms of the % contribution of these units to the statistical output. ESSnet Admin data: Quality Indicators 6

7 7 8 This indicator relates to the combination of admin and survey data. This indicator provides information on the double No. of common items obtained by admin collection of data, both admin source and surveys. Thus, No. of items in survey it provides an idea of redundancy as the same data is being obtained more than once. PROCESS (12) This indicator should be calculated for each admin source and then aggregated. Item is a value for a variable in a specific unit. % of items obtained from admin source and also collected by survey % reduction of sample size when moving from survey to admin data PROCESS (33) This indicator relates to the combination of admin and survey data This indicator provides information on the reduction in sample size because of an increased use of admin data. Only changes to the sample size due to using admin data should be included in this calculation. The indicator should be calculated for each survey and then aggregated (if applicable). and survey data Note. Double collection is sometimes conducted for specific reasons, e.g. to measure quality. If this is the case, this should be explained. Sample size after increase in use of admin data - sample size before Sample size before increase in use of admin data ESSnet Admin data: Quality Indicators 7

8 Input indicators: Quality Indicators 9 10 Time from the end of reference period to receiving Admin data This indicator provides information on the proportion of Time from end of reference period to publication date the time from the end of the reference period to the Note. If a continuous feed of data is received, the last publication date that is taken up waiting to receive the dataset used to calculate the statistical output should be admin data. This is calculated as a proportion of the used in this indicator. overall time between reference period and publication If more than one source is used, an average should be date to provide comparability across statistical outputs. calculated, weighted by the sources contributions to the final INPUT (15) This indicator should be calculated for each admin source estimate. and then aggregated. If the admin data is received before the end of the reference period, this indicator would be 0. Delay to accessing / receiving data from Admin Source Item non-response (% of units with missing values for key variables) INPUT (7) Although there are technically no responses when using admin data, non-response (missing values at item or unit level) is an issue in the same way as with survey data. This indicator provides information on the extent of missing values for the key variables. This indicator should be calculated for each of the key variables and for each admin source and then aggregated based on the contributions of the variables to the overall output. Unit refers to statistical units enterprise, legal unit, local unit, etc. Key variables are those that are the most important and have the largest impact on the statistical output (e.g., turnover, number of employees, wages and salaries, etc.) No. of relevant units in the admin data with missing value for X variable 100 % No. of units relevant for X variable This indicator could also be weighted (e.g. by turnover or employment) in terms of the % contribution to the output. ESSnet Admin data: Quality Indicators 8

9 11 12 This indicator provides information on the proportion of units in the admin data which are incorrectly coded. For simplicity and clarity, activity coding as recorded on the Business Register (BR) is considered to be correct. No. of relevant units in admin data The level of coding used should be at a level consistent with the level used in the statistical output (e.g. if the statistical output is produced at the 3-digit level, then the accuracy of the coding should be measured at this level). INPUT (21) This indicator should be calculated for each admin source and then aggregated based on the number of relevant units (weighted by turnover) in each source. Relevant units refer to those businesses that are within the scope of the statistical output (e.g. units from the services sector should be excluded from manufacturing statistics). Misclassification rate Undercoverage / Unit non-response INPUT (22) This indicator provides information on the undercoverage of the admin data. That is, units that should be included in the admin data but are not (for whatever reason). This indicator should be calculated for each admin source and then aggregated based on the number of relevant units (weighted by turnover) in each source. Relevant units refer to those enterprises that are within the scope of the statistical output (e.g. units from the services sector should be excluded from manufacturing statistics). Reference population refers to the set of units about which information is wanted and estimates are required. This might be the entire Business Register (BR) or some part of the BR, e.g. manufacturing sector. No. of relevant units in admin data with different NACE code to BR Note. If the activity code from the admin data is not used by the NSI (e.g. if coding from BR is used), this indicator is not relevant. If a survey is conducted to check the rate of misclassification, the rate from this survey should be provided and a note added to the indicator. This indicator could also be weighted (e.g. using turnover or employment) in terms of the % contribution to the output. No. of relevant units in reference population but NOT in admin data No. of relevant units in reference population Note. This could be calculated for each relevant publication of the statistical output, e.g. first and final publication. This indicator could also be weighted (e.g. using turnover or employment) in terms of the % contribution to the output. ESSnet Admin data: Quality Indicators 9

10 13 14 Overcoverage This indicator provides information on the overcoverage of the admin data. That is, units that are included in the admin data but should not be (e.g. are out-of-scope). This indicator should be calculated for each admin source and then aggregated based on the number of relevant No. of relevant units in admin data but NOT units (weighted by turnover) in each source. No. of relevant units in reference Relevant units refer to those enterprises that are within the scope of the statistical output (e.g. units INPUT (23) from the services sector should be excluded from manufacturing statistics). Reference population refers to the set of units about which information is wanted and estimates are required. This might be the entire Business Register (BR) or some part of the BR, e.g. manufacturing sector. % of units in the admin source for which reference period differs from the required reference period INPUT (19) This indicator provides information on the proportion of units that provide data for a different reporting period than that required in the statistical output. If the periods are not those required, then some imputation is necessary, which may impact quality. This indicator should be calculated for each admin source and then aggregated based on the number of relevant units (weighted by turnover) in each source. Required period is the reporting period used within the statistical output. in reference population population 100 % This indicator could also be weighted (e.g. using turnover or number of employees) in terms of the % contribution to the output. No. of relevant units in Admin data with different reporting period from required period No. of relevant units in Admin data 100 % This indicator could also be weighted (e.g. using turnover or number of employees) in terms of the % contribution to the output. ESSnet Admin data: Quality Indicators 10

11 15 16 This indicator provides information on the consistency of any common variables across sources (either admin or survey). Only variables directly required for the statistical output should be considered basic information (e.g. business name and address) should be excluded. No. of common items (within to Values within a tolerance should be considered Total no. of items consistent the width of this tolerance (1%, 5%, 10%, etc.) would depend on the variables and methods used in calculating the statistical output. This indicator should be calculated for each of the key INPUT (10) variables and aggregated based on the contributions of the variables to the overall output. Item is a value for a variable in a specific unit. Common items refer to those values for a variable in a specific unit that are available in more than one source (i.e. are common across sources). % of consistent items for common variables in more than one source Size of revisions from the different versions of the admin data RMAR Relative Mean Absolute Revisions INPUT / PROCESS (30) This indicator assesses the size of revisions from different versions of the admin data, providing information on the reliability of the data received. With this indicator it is possible to understand the impact of the different versions of admin data on the results for a certain reference period. When data is revised on other information (e.g. survey data) this should not be included in this indicator. The indicator should be calculated for each admin source and then aggregated. lerance) for X variable for X variable Note. If only one source is available or there are no common variables, this indicator is not relevant. This indicator could also be weighted (e.g. using turnover or employment) in terms of the % contribution to the output. X X Lt Pt T t = 1 X X Lt Pt T X t = 1 Pt = Latest data for X variable = First data for X variable Note. This indicator should only be calculated for estimates based on the same units (not including any additional units added in a later draft). If only one version of the admin data is received, this indicator is not relevant. ESSnet Admin data: Quality Indicators 11

12 Process indicators: This indicator provides information on the extent to which data fail some elements of the checks (automatic or manual) and are flagged by the NSI as suspect. This does not mean that the data are necessarily adjusted No. of relevant units in admin data checked (see Indicator 18), simply that they fail one or more No. of relevant units in admin check(s). This checking can either be based on a model, checking against other data sources (admin or survey), internet PROCESS (20) research or through direct contact with the businesses. This indicator should be calculated for each of the key variables and aggregated based on the number of relevant units (weighted by turnover) in each source. Key variables are those that are the most important and have the largest impact on the statistical output. % of units in admin data which fail checks % of units for which data have been adjusted PROCESS (24) This indicator provides information about the proportion of units for which the data have been adjusted (a subset of the units included in Indicator 17). These units are those that are considered to be erroneous and are therefore adjusted in some way (missing data should not be included in this indicator see Indicator 10). Any changes to the admin data before arrival with the NSI should not be considered in this indicator. This indicator should be calculated for each of the key variables and aggregated based on the number of relevant units (weighted by turnover) in each source. Key variables are those that are the most important and have the largest impact on the statistical output. or validat ed Data Note. If the validation is done automatically and the system does not flag or record this in some way, this should be noted. This indicator should also be weighted, e.g. by turnover or employment. No. of relevant units in the Admin data with adjusted data No. of relevant units in Admin Data This indicator could also be weighted (e.g. using turnover or number of employees) in terms of the % contribution to the output. ESSnet Admin data: Quality Indicators 12

13 This indicator provides information on the impact of the values imputed by the NSI. These values are imputed because data are missing (Indicator 10) or data items are No. of imputed items in the relevant unreliable (see Indicator 18). This indicator should be calculated by variable for each No. of relevant items in admin admin source and then aggregated based on the PROCESS (26) contributions of the variables to the overall output. Relevant items refer to values for units on relevant variables that should be included in calculating the statistical output. % of imputed values (items) in the admin data % of relevant units in admin data which have to be adjusted to create statistical units PROCESS (36) Cost of converting admin data to statistical data PROCESS (34) This indicator provides information on the proportion of units that have to be adjusted in order to create statistical units. For example, the proportion of data at enterprise group level which therefore need to be split to provide reporting unit data. Reference population refers to the set of units about which information is wanted and estimates are required. This might be the entire Business Register (BR) or some part of the BR, e.g. manufacturing sector. This indicator provides information on the estimated cost (in person hours) of converting admin data to statistical data. The indicator should be calculated for each admin source and then aggregated based on the contribution of the admin source to the statistical output. S U S + U os Relevant units in the reference population that are adjusted to the statistical concepts by the use of statistical methods Relevant units in the reference population that correspond to the statistical concepts U admin data data This indicator should be weighted (e.g. using turnover or number of employees) in terms of the % contribution of the imputed values to the statistical output. U S U os This indicator should be weighted (e.g. using turnover or number of employees) in terms of the % contribution of these units to the statistical output. (Estimated) Cost of conversion in person hours Note. This should only be calculated for parts of the admin data relevant to the statistical output. ESSnet Admin data: Quality Indicators 13

14 22 This indicator provides information on the efficiency gain in using admin data rather than simply using survey data. For example, collecting admin data is usually cheaper than collecting data through a survey but this benefit Production cost of might be offset by higher processing costs. PROCESS (35) Production cost should include all costs the NSI is Note. Estimated costs are acceptable. able to attribute to the production of the statistical output. Efficiency gain in using admin data Production cost of adminbased statistic - production cost of surveybase d statistic surveybase d statistic ESSnet Admin data: Quality Indicators 14