The Business Survey Framework 2: Quality issues

Transcription

1 Designing and Conducting Business Surveys for Official Statistics, Tbilisi, November 2016 The Business Survey Framework 2: Quality issues 1

2 Topics Quality Dimensions Unit and measurement related quality issues Cost efficiency calculations Response burden issues 2

3 Framework and Framework elements Total Survey Error Approach Survey error Survey cost 3

4 TSE approach: Quality = 1 (Bias 2 + Variance) 2 1 [(BSpecificat ion BFrame BNonresponse BMeasurement BProcessin g ) (VSampling VMeasurement VProcessin g )] WHO? Extended Survey Survey Cycle Cycle HOW? WHAT? Coverage errors Sampling errors Nonresponse errors Population Sample frame Sample Specification Development & testing Survey communication & management Construct Measurement instrument Validity Measurement errors Adjustment errors Respondents Data delivery & documentation Response Processing errors Adjusted Coding, cleaning & data integration Edited Data delivery 4

5 Sampling Business Surveys Sampling Not eligible Unit nonresponse Item nonresponse Varying sample variances Inactive included Active excluded One counted as several (multiplicity) Several counted as one (clustering) Inaccurate classification Misclassification Mandatory? 5

6 The Relationship Between Business Units and Survey Units Quality of Business Register Confusion about sample unit (Stratification issues) Unit or item nonresponse? Competence? Reference? 6

7 Representation Measurement = Mismatch Accessibility Delineation of... Likely outcomes Concept Unit Reference period Easy accessible Exact datum Difficult accessible Approximation Generable Solid estimate Estimable Rough estimate Inconceivable Blunder Nonexistent Item nonresponse 7

8 Sources of Response Burden E-technical E-Usability Other Layout No of questions Wait for information Terms Calculations Need of help Mismatch Response categories

9 Survey Elements and Business Responses Survey Communication Questionnaire Communication More steps increase the error risk More complicated structure increases the error risk Information distributed Mismatch between Between people organizational Between sources structure and Over time question flow increases error risk 9

10 Survey Coordinator Take the role Manage the role 10

11 Prominent Errors Sources in Business Surveys WHO? WHAT? Coverage/ Identification Population Sample frame Sample Management decisions Construct Measurement instrument Mismatches Transformations Integration Respondents Response Information system complexity Adjusted Edited Data delivery 11

12 Cost Efficiency Quality Cost 2 1 (BSpecificat ion BFrame BNonresponse BMeasurement BProcessin g ) (VSampling VMeasurement VProcessin g ) ( Consumables TimeInside NSI ) ( TimeInterviewers TimeBusin ess management TimeRe spondents ) Monetary Cost = Internal cost + «Actual Response Burden» Non Monetary Costs: Perceived Response Burden Technology Ethics Time 12

13 Response burden effects on cost efficiency Corrected key variable Corrections made on key structural variable by perceived response burden and one/multi business enterprise. Percent Number of reminders Average number of reminders by perceived response burden and one/multi business enterprise 100 % 90 % 80 % 70 % One business enterprise Multi business enterprise Total 2 1,8 1,6 1,4 One business enterprise Multi business enterprise Total 60 % 1,2 50 % 1 40 % 0,8 30 % 0,6 20 % 0,4 10 % 0,2 0 % Very easy Easy Neither easy, nor difficult Difficult Very difficult 0 Very easy Easy Neither easy, nor difficult Difficult Very difficult Cost Efficiency Quality Cost Edits indicate small quality reduction Reminding cost slightly increase Response time increases 13

14 What perceptions matter most? 14

15 User Evaluation Criteria Macro Economists Mediators Decision makers Relevance Accuracy Timeliness Punctuality Accessibility Clarity Comparability Coherence (Eurostat Quality Dimensions, 2011) You and I 15

16 User Evaluation Criteria Macro Economists Mediators Relevance to The National Account Timeliness Coherence Comparability News criteria relevance Punctuality Timeliness Trustworthy Decision makers Trustworthy Political or economic relevance You and I Trustworthy Political or economic relevance 16

17 Quality issues in Business Surveys: Key Takeaways Relevance to National Account Representation Coverage/ identification Measurement Mismatch Cost Efficiency Monetary «Actual» & and non monetary Perceived business costs Response Burden Taking the role as survey coordinator 17