Mr Anders Norberg, Statistics Sweden (SCB)

Transcription

1 Selective Data Editing The Third Baltic-Nordic Conference on Survey Statistics BaNoCoss June 2011 in High Cost area, Sweden Mr Anders Norberg, Statistics Sweden (SCB)

2 If we only want information from businesses that we know they have, and we ask for that information so they understand, and we motivate them to deliver as good quality in data as possible, and we help them to avoid accidental errors in answering questionnaires, then editing would be a minor process! 2

3 Editing Editing is an activity of detecting, resolving and understanding errors in data and produced statistics

4 Where errors are introduced Errors in raw data delivered by respondents to the statistical agency are typically nonresponse and measurement errors Errors in data transmissions The statistics production process is a mixture of many activities with risks of introducing errors

5 Editing activities A. Respondent editing B. Manual editing before data registration C. Data registration editing D. Production editing / micro editing 1 Traditional editing 2 Selective editing E. Coherence analysis F. Output editing / macro editing G. Evaluation H. Delivery control

6 Types of errors Obvious errors / Fatal errors Item non-response Non-valid values Data structure- or model errors, total sum of components Contradictions Suspected data values Deviation errors (Outliers) Suspiciously high/low values, data outside of predetermined limits Definition errors (Inliers) Many respondent miss-understand a question in the same way Many respondents fetch data from info-systems with other definitions

7 Suspected data values Deviation errors Manual follow-up takes time and is expensive Few deviation errors have impact on output statistics (low hit-rate, many changes in data have very little impact) Editing must have impact on the output! Remember response burden!

8 Suspected data values Definition errors (Inliers) Difficult to find Ways to find them: Combined editing for several surveys Deep interviews in focus groups Use statistics from FAQ and from re-contacts with respondents High proportions of item non-response Graphical editing Good examples

9 The new role of editing Quality Control of the measurement process Find errors (use efficient controls) Consider every identified error as a problem for the respondent to deliver correct data by our collection instrument Identify sources of error (process data) Analyse process data communicate with cognitive specialists Contribute to quality declaration Adjust (change/correct) significant errors Granquist (1997). The New View on Editing. International Statistical Review

10 The Process Perspective Audit and improve data collection measurement instrument collection process and the editing process itself Un-edited data must be saved in order to produced important process indicators, as hit-rate and impact on output!

11 Process indicators Sources of errors (problem for the respondents) Prop. of flagged units and variables Prop. of manually and automatically reviewed units and variables Prop. of amended values and impact of the changes, per variable Hit-rate for edits

12 Traditional data editing

13 Traditional data editing An EDIT is a checking rule / edit rule, a logical condition or a restriction to the value of a data item or a data group which must be met if the data is to be considered correct. An EDIT has: Test-variable Edit group Acceptance region if Occupation = Doctor and not (2900 < Salary_Month < 7100) then Errcode_A01 = Flag ;

14 Suspicion / Traditional edits Finding acceptance limits: Data from previous survey rounds Hourly wage distributed by SNI code at one-digit level.

15 Selective data editing A procedure which targets only some of the micro data variables or records for review by prioritizing the manual work. Potential impact Flagged 0 1 Suspicion

16 Selective data editing Criteria for prioritizing variables and records for review: Limited bias Limited variance imagining that 100% would yield best quality Hedlin, D. (2008). Local and global score functions in selective editing. Invited paper, UNECE Work Session on Statistical Data Editing, Wien, Austria, April.

17 Selective data editing Construct a score function for prioritizing variables and records: Potential impact on statistics for records flagged by traditional edits Expected impact on statistics for variable values flagged to be suspected by edits Norberg, A. et al. (2010): A General Methodology for Selective Data Editing. Statistics Sweden

18 Selective data editing The purpose of selective data editing is to reduce cost for the statistical agency as well as for the respondents, without significant decrease of the quality of the output statistics.

19 Selective data editing Latouche, M. and Berthelot, J.-M. (1992): Use of a score function to prioritize and limit re-contacts in business surveys. Journal of Official Statistics, Vol. 8, pp Lawrence, D. and McDavitt, C. (1994): Significance Editing in the Australian Survey of Average Weekly Earnings. Journal of Official Statistics, Vol. 10, pp Granquist, L. (1995): Improving the Traditional Editing Process. In Business Survey Methods, eds. Cox et.al., Wiley Granquist, L. (1997): The New View on Editing. International Statistical Review Granquist, L. and Kovar, J. (1997): Editing of survey data: How much is enough? In Survey measurement and process quality (p ) eds. Lyberg et al., Wiley Hedlin, D. (2008): Local and global score functions in selective editing. Invited paper, UNECE Work Session on Statistical Data Editing, Wien, Austria, April. Norberg, A. et al. (2010): A General Methodology for Selective Data Editing. Statistics Sweden Ilves, K. (2010): Probability Approach to Editing. Workshop on Survey Sampling Theory and Methodology, Vilnius, Lithuania, August 23-27, 2010

20 Selective data editing Statistics Sweden has developed a generic IT-tool for selective editing, SELEKT 1.1 It is based on a documented methodology. SELEKT 1.1 is flexible but require your understanding of the methodology. Norberg, A. et al. (2010): A General Methodology for Selective Data Editing. Statistics Sweden Norberg, A. et al. (2011): User s Guide to SELEKT 1.1, A Generic Toolbox for Selective Data Editing. Statistics Sweden

21 The survey environment -Coding Sum of wages by Industry -Decision making Respondent (u) has one or several sampled units -Editing Industry -Information -Imputation A Sampled unit (k) -Estimation B Observed Background variable Measurement var. (j) C unit (l) Industry Gender Occup. 1 2=Wage D 1 E y jkl 2 B M 2 F - Z 3 Input Throughput Output Use 4 Sum of wages by Occupation and Gender Gender Occupation Men Women Sum Sum

22 The survey environment -Coding Sum of wages by Industry -Decision making Respondent (u) has one or several sampled units -Editing Industry -Information -Imputation A Sampled unit (k) -Estimation B Observed Background variable Measurement var. (j) C unit (l) Industry Gender Occup. 1 2=Wage D 1 E y jkl 2 B M 2 F - Z 3 Input Throughput Output Use 4 Sum of wages by Occupation and Gender Gender Occupation Men Women Sum 1 Suspicion Sum

23 Predicted (expected) values Edit groups Data / predictor Time series Previous value Forecast Cross section Mean/standard error Median/quartile Monthly pay Blue collar workers Weekly pay Profession=3111 Payment by the hour All data Profession=3112 Monthly pay White collars Weekly pay Profession =1 Profession= 2 Profession= 3 Profession=9 Profession=3113 Payment by the hour Profession=3480 Women Men 21

24 Suspicion R= L U L L z~ KAPPA z~ z~ z /( ~ z z~ ) if z z~ KAPPA z~ z~ L U 0 if z~ KAPPA z~ z~ z z~ KAPPA z~ z~ U U L U U z z~ KAPPA z~ z~ /( ~ z z~ ) if z~ z~ KAPPA z~ z~ Suspicion=R/(TAU+R) KAPPA = 0. The ratio R is the distance between t and the centre ~ t divided by the ( ) ( ) dispersion range r = ~ U ~ L t R = a/r: t, a * * * r KAPPA = 1. The ratio R is the distance from the nearest range limit divided by the range. Hence R = a/r. For data between the lower and upper limits of the dispersion range the suspicion is zero. a * r Susp * *

25 Impact Actual impact = w ( y une y edi ) for an observation is the impact on estimated domain-total of variable Y if y une is kept instead of making a review to find y edi Potential impact = w (y une y pred ) is a proxy for actual impact to be used in practice. y pred is a prediction (expected value) for y edi Expected impact (per domain, variable, observation) is the product of suspicion and potential impact

26 Score function (1) Local score nr 5, by domain d, variable j, observed unit k,l is the expected impact related to an appropriate measure of size for the domain/variable, say standard error of estimate. VIOLIN j = weight for variable j CLARINET c(d) = weight for classification (domains) c(d) OBOE j = adjustment for size of estimated total or its standard error for variable j Score5 d, = Suspicion Potential impact d, CELLO d(c),j CELLO d(c ),j = ( { ( )}) OBOE j maximum VIOLIN j ALFA CLARINET j Tˆ d,j,t 0,SE c(d) Tˆ d,j,t 0 27

27 Score function (2) Global scores are aggregated local scores by domain, variable, second stage units (opt.) to a score for the primary unit and finally to respondent unit (opt.) Methods: sum, sum of squares, maximum etc. by (Minkovsky s distance) Score2 ( { }) = max 0,Score3-3 k k,l Threshold3 l 3 Hedlin, D. (2008): Local and global score functions in selective editing. Invited paper, UNECE Work Session on Statistical Data Editing, Wien, Austria, April.

28 Evaluation Relative pseudo-bias is a measure of error in output due to incomplete data review () RPB q = Tˆ q - Tˆ SE Tˆ 100 ( ) 100

29 Evaluation Psedobias for PPI-survey relative to the overall price index. PSUs ordered in descending order of score 0,2 0,18 0,16 0,14 0,12 0,1 0,08 0,06 0,04 0, Antal ändringar

30 Cut-off or probability sampling? Say that 821 of the total sample (n=4 000) have a score >0. There are two options for manual review: Cut-off sampling: Score2 >Threshold2, assuming the remaining bias is small Two-phase sampling: ps-sampling and design-based estimation of measurement errors to subtract from initial estimates Ilves, K. (2010): Probability Approach to Editing. Workshop on Survey Sampling Theory and Methodology, Vilnius, Lithuania, August 23-27, 2010

31 SELEKT 1.1 Raw+edited past (cold) survey data Survey specific cold adapter (SAS code) Data preparation SAS data set Input (hot) survey data Edits SNOWDON -X analysis of edits CLAN estimation software Table of Parameters Table of Estimates PRE-SELEKT Parameter specifications, Analysis of cold data AUTOSELEKT Score calculation & record flagging Survey specific hot adapter (SAS code) Data preparation SAS data set Records to FOLLOW-UP Records to IMPUTATION Accepted records Process data and reports

32 Editing remaining methodology issues Confidence (respondents and clients) Do we make a differrence between new and old respondents Editing in earlier processes Web-questionnaires Scanned paper questionnaires Fatal errors Classifying variables Survey variables Data and methods for computing predicted values etc. Homogenous edit groups How to decide threshold values Aggregating scores Sampling below threshold Inference Data for evaluation