Better Data for Better Development Policies

Transcription

1 Better Data for Better Development Policies The Role of Household Survey Data in the Global Development Agenda Alberto Zezza, World Bank 17 Aprile 2018, La Sapienza, Roma Why are we here today? Show some of the (cool) things we are doing in survey work to improve the way we measure development outcomes Entice you to use our data for your work, master, PhD thesis or your next research project Lure you to possibly work in the field of data production Make you wary of the data you are working with, whether collected by others, or by yourself 1

2 It is cool: you can be in the movies! The Crowd & The Cloud A PBS Documentary on Citizen Data Science LSMS Segment on Episode 4: Citizens4Earth Every data point has a human story Household Surveys and the Sustainable Development Goals 2

3 The Demand for Data 1. Performance-based management The Demand for Data 1. Performance-based management has created pressures on developing countries to improve the quantity and quality of their macro and micro-data: Is the public sector delivering good services? Are country policies/poverty reduction strategies reducing poverty? Is aid supporting poverty reduction? Are project and programs having the impact they were designed to have? 3

4 The Demand for Data 1. Performance-based management 2. Poverty Reduction Strategies, national development plans Measure welfare/poverty Identify problems--magnitude, causes Alternative policies Cost/benefit Monitor Evaluate The Demand for Data 1. Performance-based management 2. Poverty Reduction Strategies, national development plans 3. SDGs 4

5 History Adopted in September 2015 by the UN member countries Following on the Millennium Development Goals (MDGs) 2030 Agenda for Sustainable Development To end poverty, protect the planet, and ensure prosperity for all Overview 17 goals 169 targets 232 indicators Themes People Planet Prosperity Peace Partnerships 5

6 Indicators: Three Tiers Tier I: conceptually clear, methods established, data regularly produced Tier II: conceptually clear, methods established, no regular data Tier III: no established methods / methods being developed 82 Tier 1, 61 Tier II and 84 Tier III indicators as of April 2017 (+5 with multiple tiers) Some updates following IAEG-SDG meeting in Bahrain in Nov

7 SDGs and Data Needs Where will all the data come from to monitor the 232 indicators? Administrative data Civil registration and vital statistics (CRVS) Geospatial data Big data? Censuses Household surveys (about one third of the indicators) SDG Indicators by Goal and Tier 77 indicators in total identified as coming from household surveys Goal 3 with highest number followed by goals 16, 8, 5, 7, 1 and 2 About 80% are either Tier I or Tier II, 13 of the indicators are Tier III By Goal: Tier I Tier II Tier III Mixed Total Goal 1: Poverty Goal 2. Hunger Goal 3. Health Goal 4. Education Goal 5. Gender equality Goal 6. Water and sanitation Goal 7. Energy Goal 8. Decent work Goal 9. Infrastructure Goal 10. Inequality Goal 11. Cities Goal 16. Justice Goal 17. Partnership Total Mitra and Walsh,

8 SDG Indicators by Custodian Agency and Tier Custodian agency totals are broadly in line with the goals Tier III indicators are spread widely Tier II are more heavily concentrated 11 are under joint custodianship 1 has national governments as custodians Tier I Tier II Tier III Mixed Total WHO UNICEF ILO World Bank UNESCO/UIS UNODC FAO UN-Habitat DESA Population Division ITU OHCHR UNDP UNAIDS UNFPA UNIDO UNWTO National Governments Joint Custodians Total Mitra and Walsh, 2017 SDG Indicators: Current Coverage through HH Surveys About half of the indicators are fully covered currently Another 27 partially covered and 10 not covered at all Tier I Tier II Tier III Mixed Tier Total Fully covered and household survey preferred Fully covered but alternative source preferred Partially covered and household survey preferred Partially covered but alternative source preferred Not covered Total Mitra and Walsh, 2017 Suggests development work extends beyond the 13 tier III indicators (19+10=29 or which only 12 are Tier III) 8

9 Measuring the SDGs Distinguish statistical reality from aspiration Reconciling national priorities with international reporting Need clearer roadmap and prioritization Let s focus on targets 1.1 and 1.2 SDG 1: End poverty in all its forms everywhere Targets 1.1 By 2030, eradicate extreme poverty for all people everywhere, currently measured as people living on less than $1.25 a day 1.2 By 2030, reduce at least by half the proportion of men, women and children of all ages living in poverty in all its dimensions according to national definitions Indicators Proportion of population below the international poverty line, by sex, age, employment status and geographical location (urban/rural) Proportion of population living below the national poverty line, by sex and age Proportion of men, women and children of all ages living in poverty in all its dimensions according to national definitions 9

10 First LSMS Survey WDR Poverty MDGs 24/4/2018 Poverty data availability is better than it used to be Serajuddin et al., 2016 Deprivation in Household Survey Data Data points or more 2, interval <=5 years 2, interval >=6 years Only 1 No data Countries 92 low/middle income countries do not have a multi-topic survey every 3 years, as per the President s commitment No data: mainly in EAP and LAC small countries Only 1 point: mainly in AFR 77 with extreme deprivation (> 5-year interval) Irregular (ad hoc) survey implementation But also, beyond data deprivation, issues with: Uncertainty of funding: many more (IDA) countries at risk Data quality (reliability, comparability) and accessibility E.g., only 27 of 48 SSA countries have at least two comparable surveys between Note: number of data deprived countries estimated based on surveys conducted during

11 Measuring Poverty Multidimensional concept Non-monetary measures Consumption vs. income Consumption vs. Income 11

12 CONSUMPTION 24/4/2018 Measuring Consumption Lisa C. Smith, Olivier Dupriez and Nathalie Troubat. Assessment of the Reliability and Relevance of the Food Data Collected in National Household Consumption and Expenditure Surveys. International Household Survey Network Working Paper No. 008, February Different Users and Definitions of Food Consumption Economists Food purchases Food security analysts Food available for consumption Nutritionists Food ingested 12

13 Poor Harmonization Across All Criteria 13 % Poor Harmonization of Recall Periods 13

14 Do Methods Matter? Beegle et al. (2012). Methods of household consumption measurement through surveys: Experimental results from Tanzania. Journal of Development Economics Volume 98, Issue 1, May 2012, Pages Lessons Learned Measuring SDGs will require concerted effort and prioritization Household surveys are the sole source to measure a large number of SDGs Measuring a large number of the indicators presents many challenges Methods matter, thus harmonization is key! 14

15 Suggested Reading Sustainable Development Goals 2030 Agenda for Sustainable Development SDGs Indicators List Tier Classification for Global SDG Indicators Inter-Secretariat Working Group on Household Surveys International Household Survey Network: Measuring Food Consumption Methods of Household Consumption Measurement through Surveys: Experimental Results From Tanzania Going beyond indicators: Understanding development policy and outcomes 15

16 Surveys and Policy Analysis Gov t Programs Conditional Cash Transfers Day care centers Public Health Campaign Households Individuals Firms Social Goals Increase enrollment Increase female LFP Lower infant mortality 33 Surveys: Going Beyond Rates Understanding secondary school enrollments, year olds, Albania 2002 Percent 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Average In almost all countries we have a single statistic: mean enrollment at the national level. In this case it is 61%. This is interesting for monitoring purposes, but it doesn t say much about poverty or other factors.... A regional disaggregation would be useful 16

17 Understanding secondary school enrollments, year olds, Albania 2002 Percent 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Urban Rural Average In some countries we have regional breakdowns, with marked contrasts The contrast between urban and rural rates emphasizes the disadvantages faced by rural communities. Other breakdowns would be useful Understanding secondary school enrollments, year olds, Albania 2002 Percent 100% 90% 80% 70% 60% 50% 40% Male Male Female Female Urban Rural Average possibly, official statistics can add the gender dimension the figures show that, in urban areas, there is no gender differential but a large gap in rural areas. 30% 20% 10% But we still don t know much about who sends their children to school 0% 17

18 Understanding secondary school enrollments, year olds, Albania 2002 Percent 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Q1 Q2 Q3 Q4 Q5 Consumption quintile Male, rural Female, urban Male, urban Female, rural Average With a survey we can show enrollment rates broken down by consumption level- -and thus understand an additional dimension Only by measuring we can cross the River of Myths Hans Rosling ( ) 18

19 Gathering information through surveys (household or other) There is a range of options They can be ordered along two main dimensions: degree of representativeness subjective/objective dimension 39 Degree of Representativeness Case study Purposive selection Quota sampling Small prob. sample Large prob. sample Census 19

20 Subjective/Objective Dimension Direct measurement Questionnaire (quantitative) Questionnaire (Qualitative) Case study Purposive selection Structured interview Quota sampling Small prob. sample Large prob. sample Census Open meetings Conversations Subjective assessments Participatory Poverty Assessments Case study Wonderful World of Surveys Purposive selection Participant observation Sentinel Site Surveillance Quota sampling Beneficiary Assessment Direct measurement Questionnaire (quantitative) Questionnaire (Qualitative) Structured interview Open meetings Census Household Budget Survey LSMS/ IS CWIQ/PS Small prob. sample Community Surveys Large prob. sample Census Windscreen Survey Conversations Subjective assessments 20

21 Wonderful World of Surveys: Statistical Surveys Direct measurement Questionnaire (quantitative) Questionnaire (Qualitative) Census Household Budget Survey LSMS/ IS CWIQ/PS Case study Purposive selection Structured interview Quota sampling Small prob. sample Large prob. sample Census Open meetings Conversations Subjective assessments Summary Expanding demand for timely, relevant data Need to determine the range of data needs to begin to define a system of information Surveys are one, important, source of information among many No one survey can meet all data needs: System of Household Surveys 44 21

22 Common concerns with survey instruements Accuracy Precision Cost, cost-effectiveness Burden on respondents Periodicity Timeliness What LSMS does Established in the 1980s Country-owned, nationally representative surveys Multi-topic, household-level and community data Monitor, but more importantly understand, analyze Typically every 3-5 years Technical assistance to NSOs, capacity building & knowledge transfer Support on survey design/implementation to World Bank teams Research on best practices in survey methods & latest technology Documents & publicly releases open-access data 22

23 LSMS: Integrated, multi-topic questionnaires Household Expenditures Food & Nonfood Education Health Labour Nonfarm Enterprises Durable Assets Anthropometry Food Security Shocks Agriculture Plot Details Inputs Use & Access; Labor & Non-Labor Alike Crops Cultivation & Production Implements & Machinery Extension services Livestock, Fisheries Forestry? Community Demographics Services Facilities Infrastructure Governance Organizations & Groups Prices LSMS Surveys are innovative Geo-referenced: Integration with other spatially explicit data sources Use of sensors for direct measurement : GPS of land area Soil testing Water quality testing Rain gauges Accelerometers for physical activity Phone apps for commuting in urban areas Computer-assisted: Using Survey Solutions CAPI platform Use of mobile phones to integrate face-to-face data collection 23

24 and open access 100+ LSMS Surveys available on the World Bank s Microdata Catalog Enhance data usability via automated analytical tools (ADePT) Pushing the technology frontier in household surveys TECHNOLOGY FOR 24

25 Measuring Land Area: Methodological Options Farmer self-reported estimate Compass and rope (aka traversing) GPS Remote Sensing (?) PROS - Inexpensive - Less missingness PROS -Traditional gold standard for accuracy - Eliminates subjectivity PROS - Significantly quicker than traversing with advantages of objective measurement PROS - Potential to eliminate plot visits CONS - Subjective - Complicated by traditional units -Potential ulterior motives CONS - Time/labor intensive - Requires travel to plot CONS - Questions of accuracy on small plots - Requires travel to plot CONS - Resolution limitations - Feasibility of boundary identification 3 Methodological experiments: Ethiopia (n=1798), Tanzania (n=1945), Nigeria (n=494) Total N=4237 Comparison of Methods (National Surveys) Farmer self-reported estimates Potentially sensitive to: - Respondent characteristics - Perceived use of the data (taxation, program eligibility) - Traditional/local units of measurement - Rounding Source: Carletto, Savastano, Zezza (2013). Fact or Artifact: the Impact of Measurement Errors on the Farm size - Productivity Relationship, Journal of Development Economics. May result in Large errors, systematic biases 25

26 GPS vs. Compass & Rope vs. Subjective Ethiopia GPS Tanzania SR 4 CR CR 1 Correlation between GPS and CR measurements: (about 0.5 between SR and CR) GPS SR Nigeria CR GPS SR GPS much, much faster (cheaper) than CR Average Measurement Duration Plot Size Level (CR) & Minutes GPS CR Ethiopia Tanzania Total Ethiopia: GPS = 13.7 minutes CR = 56.8 minutes Tanzania: GPS = 7.4 minutes CR = 29.3 minutes 26

27 Integrating data via geo-referencing 2010 Rainfall as % of Normal Rainfall (mm) Integrating data via geo-referencing 2010 Max EVI Deviation from Mean NDVI 500 sparse moderate dense >

28 YIELD MEASUREMENT ON THE GROUND MAPS Ground Truth for Maize Yields: Crop Cutting 28

29 /4/2018 Your Many Averages of Maize Yields (KG/Ha) in Your Average Country in Africa Source 1 Source 2 Source 3 Source 4 Source 5 0 Country 1 Country 2 Country Uganda National Maize Yield (Ton/Ha) FAOSTAT Annual Time Series

30 MAPS Ground Truth for Maize Yields: Crop Cutting New, Cost-Effective, High-Resolution Imagery Options Sensor Wavelengths Spatial Resolution Revisit frequency Launch year Terra Bella Optical 1m ~Weekly 2013 Planet Optical 5m ~Daily 2014 Sentinel-2 Optical 10m 5 day 2017 Source: Hand, Science News,

31 Average Farm Size USA: 434 Ha Uganda: 2.5 Ha m resolution products suitable for national or regional analysis 31

32 20-30 m resolution products suitable for subnational analysis Plot-level analysis in Africa requires spatial resolution of 1 2 m 32

33 Yield estimation requires additional spectral bands (NIR, TIR), and multiple cloud-free observations during the growing season May June

34 June MAPS Remote Sensing Validation Compute Green Chlorophyll Vegetation Index (GCVI) values across the landscape for each date of imagery GCVI = Reflectance in Near Infrared Wavelength Reflectance in Green Wavelength 1 Build a plot-level yield prediction model based on the crop cutting yield & GCVI relationship in the purestand domain Obtain yield predictions for the entire sample; compare to sub-plot & full-plot crop cutting based yield measures 34

35 Sample of MAPS Round I Plots over GCVI Output COMPUTER ASSISTED PERSONAL INTERVIEWING (CAPI) Researchers design questionnaires using visual tools and upload them to the central server Questionnaires with no errors are uploaded to the HQ central distributes server the sample lists across teams of Internet enumerators Internet Supervisors monitor the submissions Interviewers Supervisors assign synchronize households to their devices individual and interviewers upload completed questionnaires Enumerators WiFi repeat interviews if errors are detected WiFi Interviewers visit households and collect data 35

36 Summary Demand for data: Household surveys a key components of SDG monitoring Understanding outcomes, designing and evaluation policy requires more than indicators Survey methods matter Careful implementation Harmonization Introduce technology Opportunities for data integration (Satellite, big data...) Computer Assisted Personal Interviewing (quality, timeliness, cost) Better Data for Better Development Policies The Role of Household Survey Data in the Global Development Agenda Alberto Zezza, World Bank azezza@worldbank.org 17 Aprile 2018, La Sapienza, Roma 36