Controlling the Quality of Surveys in the Gulf

Transcription

1 RTI International The following slides are the property of their authors and are provided on this website as a public service. Please do not copy or redistribute these slides without the written permission of all of the listed authors. Controlling the Quality of Surveys in the Gulf March 1, 2011 Dr. Paul Biemer (ppb@rti.org ) - RTI International and University of North Carolina Ms. Lisa Thalji - RTI International Ms. Ashley Richards - RTI International 1

2 Controlling the Quality of Surveys in the Gulf Paul Biemer RTI International and University of North Carolina Lisa Thalji Ashley Richards RTI International

3 Outline What is survey quality? Why is it important? How can survey quality be controlled? What do we know about Gulf survey quality? Where to start to improve survey quality? 3

4 User and Producer Have Very Different Perspectives on Survey Quality Producers place high priority on Accuracy total survey error is minimized Credibility credible methodologies; trustworthy data 4

5 User and Producer Have Very Different Perspectives on Survey Quality Producers place high priority on Accuracy total survey error is minimized Credibility credible methodologies; trustworthy data but many data users place higher priority on 5

6 User and Producer Have Very Different Perspectives on Survey Quality Producers place high priority on Accuracy total survey error is minimized Credibility credible methodologies; trustworthy data but many data users place higher priority on Timeliness data deliveries adhere to schedules 6

7 User and Producer Have Very Different Perspectives on Survey Quality Producers place high priority on Accuracy total survey error is minimized Credibility credible methodologies; trustworthy data but many data users place higher priority on Timeliness data deliveries adhere to schedules Relevance data satisfy user needs 7

8 User and Producer Have Very Different Perspectives on Survey Quality Producers place high priority on Accuracy total survey error is minimized Credibility credible methodologies; trustworthy data but many data users place higher priority on Timeliness data deliveries adhere to schedules Relevance data satisfy user needs Accessibility access to data is user friendly 8

9 User and Producer Have Very Different Perspectives on Survey Quality Producers place high priority on Accuracy total survey error is minimized Credibility credible methodologies; trustworthy data but many data users place higher priority on Timeliness data deliveries adhere to schedules Relevance data satisfy user needs Accessibility access to data is user friendly Interpretability documentation is clear; meta-data are well-managed 9

10 User and Producer Have Very Different Perspectives on Survey Quality Producers place high priority on Accuracy total survey error is minimized Credibility credible methodologies; trustworthy data but many data users place higher priority on Timeliness data deliveries adhere to schedules Relevance data satisfy user needs Accessibility access to data is user friendly Interpretability documentation is clear; meta-data are well-managed Comparability valid demographic, spatial and temporal comparisons 10

11 Total Survey Quality is achieved by optimally balancing dimensions This is done by identifying measurable and achievable objectives for each user-defined dimension of quality determine costs/resources required to achieve these objectives maximize survey accuracy with remaining budget 11

12 This is done by Total Survey Quality is achieved by optimally balancing these dimensions identifying measurable and achievable objectives for each user-defined dimension of quality determine costs and resources required to achieve these objectives maximize survey accuracy with remaining budget Survey Budget = Cost of Accuracy Accuracy + Cost of User-Defined Quality Other 12

13 Accuracy of an estimate is achieved by minimizing total survey error Estimate Parameter ˆ Total survey error is the difference between an estimate and the target population parameter 13

14 What contributes to TSE? Sampling sample size, scheme, choice of estimator 14

15 What contributes to TSE? Sampling sample size, scheme, choice of estimator Specification question measures wrong concept 15

16 What contributes to TSE? Sampling sample size, scheme, choice of estimator Specification question measures wrong concept Nonresponse missing units, missing values, imputed values, weight adjustments, etc. 16

17 What contributes to TSE? Sampling sample size, scheme, choice of estimator Specification question measures wrong concept Nonresponse missing units, missing values, imputed values, weight adjustments, etc. Frame frame omissions (undercoverage), duplications, ineligible units. 17

18 What contributes to TSE? Sampling sample size, scheme, choice of estimator Specification question measures wrong concept Nonresponse missing units, missing values, imputed values, weight adjustments, etc. Frame frame omissions (undercoverage), duplications, ineligible units. Measurement due to respondents, interviewers, faulty questions or data collection methods 18

19 What contributes to TSE? Sampling sample size, scheme, choice of estimator Specification question measures wrong concept Nonresponse missing units, missing values, imputed values, weight adjustments, etc. Frame frame omissions (undercoverage), duplications, ineligible units. Measurement due to respondents, interviewers, faulty questions or data collection methods Data Processing due to editing, coding, disclosure limitation methods, weight creation 19

20 Total survey error is measured by the mean squared error Total Survey Error Sampling Error Sampling scheme Sample size Estimator choice Mean Squared Error (MSE) MSE = Bias 2 + Variance Systematic Bias Nonsampling Error Specification Nonresponse Frame Measurement Data processing Variable Variance 20

21 Sampling error is usually a small part of the error Fraction of MSE that is sampling error for three levels of bias 0.9 5% Bias 0.8 3% Bias 0.7 1% Bias Sample size 21

22 Controlling data quality requires the use of proven design principles methods that reduce frame error, nonresponse, and measurement error provisions for monitoring costs and error as the survey process is being implemented 22

23 Controlling data quality requires the use of proven design principles methods that reduce frame error, nonresponse, and measurement error provisions for monitoring costs and error as the survey process is being implemented real-time cost and error reduction strategies continuous quality improvement statistical process control techniques adaptive design and implementation strategies 23

24 Controlling data quality 24 requires the use of proven design principles methods that reduce frame error, nonresponse, and measurement error provisions for monitoring costs and error as the survey process is being implemented real-time cost and error reduction strategies continuous quality improvement statistical process control techniques adaptive design and implementation strategies reliable data on costs and error collect and maintain detailed costs data by process develop, monitor, and document metrics from key survey processes conduct quality evaluation studies

25 Survey quality and Gulf surveys: a literature review Some surveys in our review Arab Barometer Global Youth Tobacco Survey Global Adult Tobacco Survey Ipsos MENA 2010 Omnibus Survey (Qatar U) Gallup World Poll Zogby International Survey Burson-Marsteller Arab Youth Survey World Values Survey UAE Media Survey of Internet Users Jordan Media Survey of Internet Users Egypt Internet Users and e- commerce Survey Household Socio-Economic Survey for Iraq Oman 2003 General Census of Population and Housing Balkhy, et al (2010) survey of Riyadh and Jeddah Bener, et al (2004) survey of PHCs in Al-Ain, UAE 25

26 Survey quality and Gulf surveys Summary of the literature review 26

27 Survey quality and Gulf surveys Summary of the literature review Sophisticated sample designs often used 27

28 Survey quality and Gulf surveys Summary of the literature review Sophisticated sample designs often used Standard errors and response rates are usually computed and reported 28

29 Survey quality and Gulf surveys Summary of the literature review Sophisticated sample designs often used Standard errors and response rates are usually computed and reported However, there appears to be little/no documentation on nonsampling error 29

30 Survey quality and Gulf surveys Summary of the literature review Sophisticated sample designs often used Standard errors and response rates are usually computed and reported However, there appears to be little/no documentation on nonsampling error some evidence of risky practices; e.g., convenience sampling substitution of nonresponding households questionnaire items seldom validated little attention given to controlling interviewer error lack of data collection and data processing quality control 30

31 Survey quality and Gulf surveys 31 Summary of the literature review Sophisticated sample designs often used Standard errors and response rates are usually computed and reported However, there appears to be little/no documentation on nonsampling error some evidence of risky practices; e.g., convenience sampling substitution of nonresponding households questionnaire items seldom validated little attention given to controlling interviewer error lack of data collection and data processing quality control considerable variation in quality across surveys

32 Some Recommendations for Improving Gulf Survey Quality 32

33 Adopt the TSE Paradigm for Survey Design Compile information on TSE (e.g., quality profiles) Identify major contributors to TSE Allocate resources to control these errors Use results from the literature and other similar surveys to guide the design Develop an effective process for modifying the design during implementation to achieve optimality Embed experiments and conduct studies to obtain data on TSE for future surveys 33

34 Plan to modify the design during implementation The initial survey design must modified or adapted during implementation to control costs and maximize quality. Three strategies for reducing costs and errors in real-time: Continuous quality improvement Applying Six Sigma tools and principles Adaptive total design and implementation Initial quality Final quality Quality Level 34

35 Make Error Evaluation a Priority Why should survey error be evaluated? Addresses both user and producer dimensions of total survey quality. Is needed to compare the quality of alternative methods. Provides valuable information on data quality for gauging uncertainty in estimates, interpreting the analysis results, and building confidence and credibility in the data. Is essential for optimizing resource allocations to reduce the errors (i.e., minimizing the TSE). 35

36 Summary of Recommendations Use best practices in survey design Control quality during data collection using CQI Conduct quality evaluations Document the results of quality evaluations in reports and in the literature. 36