Assessing the quality of integrated data (ESSnet on quality of multisource statistics)

Transcription

1 Assessing the quality of integrated data (ESSnet on quality of multisource statistics) Sorina Vâju, European Commission

2 Summary 1. Problem statement quality in a multisource environment 2. ESS.VIP ADMIN 3. ESSnet on quality of multisource statistics 2

3 How to measure quality in a new environment? Cost and burden surveys admin data New demands Newly available sources Big data Multisource statistics Are we still measuring quality in a proper way? 3

4 Quality facets Input Process Output Quality of raw data Whether and how a given data source can be used on a regular basis to produce statistics Whether final data is real Magnitude of errors introduced in processing stage Analyse of statistical process User easy to understand information on the quality of the final data 4

5 Output quality assessment by input and process Process step Linkage and determination of the target population Risk Missed link, wrong link: under/over coverage Impacted quality dimension Accuracy, comparability Error measurement Bias, confidence range of the target population Concept/ definition Aggregation of different concept/definitions Relevance, accuracy, comparability Bias, Variance error, qualitative assessment Imputation/ estimation Classification Estimation error Wrong classification Accuracy Relevance, accuracy, comparability below a certain level of aggregation Bias, variance error Bias, variance error 5

6 Is it feasible to assess output quality through input/process quality? Multiple sources Surveys Admin data Big data Multiple uses Direct use Sampling frame Auxiliary information Calibration Multisource output Complex processes International level ESS aggregation International aggregation 6

7 Is input and process assessment enough? Useful for Less useful for Deciding on which sources to use Is the output good enough? Designing the process Which final data is better? Improving the process Will the user understand? 7

8 Alternative: assessment based on the output itself Exclusively ach based on output Time series/cross sectional data Based on reference source Comparison with other statistics/ sources Bootstrapping based on admin data Primary/ complementary data Breaks in series Revisions Outliers Quality surveys Support sampling Auxiliary information 8

9 Why the ESSnet on quality of multisource statistics? Problem Complex environment Input + process assessment is cumbersome and not sufficient Suggestions Direct assessment on the basis of the output Use of reference source Bootstrapping Needs Develop quality indicators based on output Develop step-by-step algorithms for implementation Cost-benefit analysis Update templates for quality reports 9

10 ESS.VIP ADMIN areas of work Access to data Use of Commission administrative data Quality measurement ESSnet on quality of multisource statistics Frames for social statistics Methodology for integrating sources 10

11 ESSnet on quality of multisource statistics Input quality Output quality Quality of frames for social statistics 11

12 Work area 1: quality of input Tasks: Critical review and testing of existing methodology Consolidated version of input checklist Gap analysis Delivery: June

13 Work area 2: quality of frames for social statistics Tasks: Review of literature and comparative analysis of frame types Gap analysis Development and test of some quality measures Proposal for further work Delivery: November 2016, April

14 Work area 3: quality evaluation of statistical output based on multiple sources Tasks: Critical review of existing quality measures and approaches Tests of the suitability of existing measures and approaches in several domains Action plan for developing a theoretical framework for measuring output quality Delivery: November 2016, June

15 Additional work Communication and dissemination: CROS portal will be used for dissemination Workshop on quality of multisource statistics, April, Budapest Further work up to 2019 Framework and indicators for assessing the quality of frames for social statistics Framework and indicators for assessing the quality of output Recommendations for the ESS Handbook for Quality Reports 15

16 Contact ESS. VIP ADMIN Sorina Vâju (): ESSnet on quality of multisource statistics Niels Ploug (Statistics Denmark): Sorina Vâju (): 16