New Methods in Household Surveys

Transcription

1 New Methods in Household Surveys TALIP KILIC Living Standards Measurement Study Team Poverty & Inequality Group Development Research Group The World Bank PREM Learning Days DEC Course The Living Standards Measurement Study: Innovation in Survey Data for Better Policy Making

2 Outline Computer-Assisted Personal Interviewing Geo-referencing Methodological Survey Experiments on Agriculture

3 COMPUTER-ASSISTED PERSONAL INTERVIEWING (CAPI)

4 Computer-Assisted Personal Interviewing (CAPI) PAPI: Paper-based personal interviewing, coupled with computer-assisted field-based data entry (CAFE) pioneered by the LSMS CAPI: Integration of interviewing & data entry through the use of a handheld device, preloaded with an electronic questionnaire Household (HH) surveys implemented on CAPI platform since the late 80 s in high- & middle-income countries, inc. the Netherlands, the UK, the US, Norway & Turkey Increasing number of applications in low-income setting in recent years Mobile Phones, PDAs vs. Netbooks & Tablet PCs

5 LSMS Experience LSMS operations marked by a gradual transition to CAPI CAPI survey experiment (~200 households) (Albania) Application developed in CSProX CAPI survey (~500 households) (Ngara District, Tanzania) Application developed in CWEST Kagera Health and Development Survey (KHDS) (Tanzania) Application developed in CWEST Uganda National Panel Survey (UNPS) ( ) Supported by LSMS-ISA, implemented by Bureau of Statistics Partial transition to CAPI in 2010/11 (in CWEST); CAPI transition completed in 2011/12 (on-going; in CWEST & Surveybe); Next round 2013/14 Ethiopia Rural Socioeconomic Survey (ERSS) ( ) Supported by LSMS-ISA, implemented by Central Statistical Agency CAPI application developed in Surveybe for the Ag Questionnaire, implemented in a subset of EAs in 2011/12; Next round in 2013/14

6 Hardware Options General Features: 7-10 stylus-friendly screens Rapid navigation across questionnaire Several questions displayed at one time Samsung Q1b Ultra (KHDS; UNPS) $ Camera, microphone, virtual keyboard & hand-writing recognition software 5-7 hours of (initial) battery life Extended battery pack, external battery pack & daily charge of batteries recommended Generators in low-electrification settings Multiple ports: Internet dongles, GPS units, external keyboards Asus Eee PC T101MT (UNPS; ERSS) $

7 Software Options Traditional DE software designed for transfer from paper questionnaire to computer Benefits of relying on CAPI better realized working with software packages designed for interactive interviewing CAPI software packages make up a small market, with varying degrees of cost effectiveness & types of strengths Key players: Blaise, CASES, CSProX, MMIC & Surveybe LSMS-commissioned comparative assessment of software programs for the development of CAPI applications (available on

8 Why Contemplate Transition to CAPI? Enhanced tools for in-field & remote management of mobile teams Headquarters & Team Leaders: Assigning work, tracking progress, immediate & comprehensive feedback Expected gains in timeliness of data availability Data entry, checking & exportation in one application Expected gains in data quality Accommodation of non-linear/integrated questionnaires Automated routing reduces the incidence of missing data Data checking, reporting & revision facilities during the interview Range & consistency checks, flags for missing fields Improvements in quantification of nonstandard units Instructions on questions, note taking facilities

9 Non-linear Navigation

10 Automated Routing

11 Consistency Checks

12 Consistency Checks (Cont d)

13 Use of Media for Better Quantification

14 Use of Media for Better Quantification

15 Managing Expectations Data quality control principles in CAPI set-up no different than surveys based on PAPI with CAFE CAPI tools useful as much as enumerators & field supervisors take advantage of available facilities & act on inconsistencies Relative impact of CAPI on data quality: Open question Limited evidence on improved data quality with respect to a wellsupervised survey based on PAPI with CAFE Fafchamps, M., McKenzie, D., Quinn, S., and Woodruff, C. (2010). Using PDA consistency checks to increase the precision of profits and sales measurement in panels. CSAE Working Paper Series No Caeyers, B., Chalmers, N., and De Weerdt, J. (2012). Improving consumption measurement and other survey data through CAPI: Evidence from a randomized experiment. Journal of Development Economics, 98, pp

16 Cost Implications CAPI generates (minimal) savings in printing costs & data entry Savings increase with the complexity & frequency of survey Significant up-front costs in hardware procurement More cost-effective if machines are used in other survey operations Transition into CAPI also driven by field work structure Size of the enumerator corps may be prohibitively large Gradual transition to CAPI as part of the LSMS operations primarily underlined by demand for increased data quality & availability

17 Uganda National Panel Survey (UNPS) CAPI Experience Teams quick to adapt, instrumental in training & knowledge sharing Required change in institutional thinking on surveys: Greater upfront work (& costs) with respect to PAPI with CAFE Prep of Wave I (PAPI) data uploaded onto Wave II (CAPI) application Hundreds of intra/inter-module consistency checks, in addition to range & default checks for missing values Programming of rules on generation of household & individual identifiers for new additions to the sample Training of UBoS Headquarters staff on case management suite

18 UNPS CAPI Experience (Cont d) In-country procurement problems Lags assoc. with operating within Government systems/unreliable suppliers US procurement by the LSMS-ISA project: Not straightforward either Anti-virus software critical to maintaining the hardware integrity Application glitches even after piloting three times: Need for more intensive testing in comparison to PAPI with CAFÉ CAPI application platform based on multiple software packages: CWEST & CSPro (in 2010/11); CWEST & Surveybe (in 2011/12) Dependence on the CWEST application developer for adjustments Continued reliance on multiple software packages necessitated by lack of case management features on Surveybe Timely communication of bugs that might compromise the integrity of incoming data critical: No paper questionnaires to re-enter

19 UNPS CAPI Experience (Cont d) Continuing improvements to the CAPI application on a rolling basis throughout the field work Even with internet dongles, slow internet speeds & lack of service in certain areas Affects timely headquarters review of data sent from the field Receipt of application updates by the survey teams not always timely Regular backup of interview files in the field & at the HQ crucial Lags associated with Surveybe data export Still need a paper questionnaire for dissemination purposes: CAPI application dictionary is not more than a linear questionnaire report

20 Comparative Assessment of Software Programs for the Development of CAPI Applications Initially twofold objective: Inform internal decision making on the choice of surveys for upcoming surveys planned under the LSMS-ISA project, in Uganda, Ethiopia, and Nigeria Fill the gap in public knowledge on the relative performance of available software packages for the development of CAPI applications for multi-topic household surveys Peer-reviewed report, managed by the LSMS team, compiled by the IRIS Center at the University of Maryland, reviewed by software developers prior to release Available on

21 Comparative Assessment (Cont d) Software packages screened as suitable for the development of CAPI applications for multi-topic household surveys & evaluated by the report include: Software Blaise CASES Developer CSProX Serpro, S. A. Entyware MMIC Open Data Kit Pendragon Forms Surveybe Westat & Statistics Netherlands Computer-Assisted Survey Methods Program at the University of California, Berkeley Techneos RAND Labor and Population The University of Washington s Department of Computer Science and Engineering Pendragon Software Corporation Economic Development Initiatives

22 Comparative Assessment (Cont d) Structure of the report Brief overview of each software package Comparative assessment of each software package in 12 areas: Evaluation Areas Programming Questionnaire Development Questionnaire Implementation Interface for Field Users Questionnaire Navigation Case Management Data Transfer Data Exporting Support & Documentation Hardware & Software Needs Pricing & Upgrades Extensibility Detailed evaluation of each software package, accompanied with full functionality check lists for each evaluation area

23 Comparative Assessment (Cont d) No single software package is an unequivocal frontrunner in all evaluation areas Ideal approach to questionnaire design: Marrying menu-driven development environment for novice users with a command line for more experienced users, replicating functionality in the menu environment, accommodating customization needs Missing across all evaluated programs Non-trivial task in this set-up: Allowing for simultaneous questionnaire development by several survey designers & being able to integrate each piece into an application Positive relationship between quality/scope of documentation & proprietary nature of the software (MMIC, ODK vs. Blaise) Top contenders: Surveybe: Ease to use (menu-based development environment) but lacks case management suite & only allows for sequential workflow for qx development MMIC & Blaise: Powerful & expansive in scope but steep learning curve (command line driven development environment) & high need for technical assistance Differences in quality of documentation & user community, in favor of Blaise Differential cost structures Open source (TA needs?) vs. per software installation (corporate licensing) vs. data points

24 Where Next? Sustainability of adoption relies on availability of a user-friendly, yet highly customizable, public solution around which in-country capacity could be built LSMS and Development Economics Computational Tools (DECCT) Unit of the World Bank supporting the development of a publicly available, closed-source CAPI software platform Informed by LSMS field experience & comparative CAPI software assessment Core interface components: Builder (for Survey Designers), Manager (for Survey Managers & Team Leaders), Client (for Interviewers) Approach to Builder: Coupling a menu-driven development environment for a core set of functionality (common across LSMS-type household surveys) with a command-line for programming more complex features & supporting customization Key decisions: Target hardware/software platforms Hardware is not independent of OS! Mouse/keyboard vs. Stylus vs. Finger Touch: Implications for questionnaire design OS platform independence: Implications for software development, robustness of interface components

25 GEO-REFERENCING

26 Geo-referencing Recording longitude and latitude of households & other POI (plots, markets, schools, health centers) GPS-based data collection not new Technology is fairly cheap, wider appreciation for usefulness of spatial data (GoogleEarth, GoogleMaps, remote sensing data) Innovation in uses of GPS data: Survey Management Evaluation of Survey Responses Data Integration Dataset Characterization Research Questions

27 Survey Management Standard mapping-grade GPS units (Garmin etrex, Trimble Juno) should produce fairly accurate readings Visit verification (timing & location) Navigation to & positive identification of sample household in successive panel survey rounds

28 Data Evaluation Internal consistency checks & validation Distance from household to agricultural plots HHID Plot ID EST_KM GPS_KM 1234 M M M

29 Data Evaluation (Cont d) Distribution of household responses on the occurrence of drought / irregular rains shows large local variation

30 Data Integration Having GPS locations enables integration with other spatial datasets, making available large range of additional variables. Distance HH to Plot HH to Market HH to Major Road Environmental Climatology Landscape typology Soil Elevation Terrain Time Series Rainfall Vegetation Indices

31 Data Characterization Representativeness of the Tanzania National Panel Survey (TZNPS) 2008/09 sample across agro-ecological zones Agro-Ecological Zone Coastal, Islands & Alluvial Plains Arid & Semi-Arid Lands # of Households Northern, Southern & Western Highlands 688 Plateau 512 TOTAL 3266

32 METHODOLOGICAL SURVEY EXPERIMENTS ON AGRICULTURE

33 Methodological Survey Experiments on Agriculture Identification process Alignment with Global Strategy Field experience Iterative Peer-review Methodological survey experiments in the pipeline on the measurement of 1. Agricultural land areas 2. Soil fertility 3. Water resources 4. Agricultural labor input 5. Continuous/extended harvest crop production

34 Methodological Survey Experiments on Agriculture (Cont d) As part of a DFID-funded 3-year ( ) program led by the LSMS team, conducted in collaboration with the Statistics Department of the Food & Agriculture Organization of the United Nations (Component 1 & 5) World Agroforestry Centre (Component 2) World Bank Environment Department (Component 3)

35 Component 1: Agricultural Land Areas Why is it important? Fundamental component of agricultural statistics (forecasting production and yield measurement) Priority #1 of Global Strategy Recent research (Carletto et al. 2011) documents bias in farmerreported land areas with respect to GPS-Based counterparts Small (large) farms shown to over-(under-)report: Implications for the inverse farm size-productivity relationship Available measurement methods Farmer Reporting P 2 /A method GPS (already utilized in LSMS-ISA surveys) Traversing (Compass & Rope)

36 Component 2: Soil Fertility Why is it important? Physical soil characteristics remain key unobserved variables for analysis of agricultural productivity Available measurement methods Farmer Evaluation Spectral soil analysis (SSA) Conventional soil analysis (CSA)

37 Component 3: Water Resources Why is it important? Water essential input into production & agriculture in sub-saharan Africa is predominantly rainfed Large discrepancies across data sources on water availability Publicly available data sources defined at low resolutions Higher resolution data not publicly available Available measurement methods Farmer Reporting Remote Sensing Weather Stations Communal Rain Gauges

38 Component 4: Agricultural Labor Input Why is it important? Essential for accurate labor productivity measurement Existing data very poorly measured Available methods Recall Computer-Assisted Telephone Interviews Labor input diaries

39 Component 5: Continuous/ Extended Harvest Crops Why is it important? Continuous/extended harvest crops are major staples in many African countries Inaccuracy of recall: May extend across seasons, harvest on an ongoing, at times need, basis Available methods Recall Crop card (with local monitors) On-going work with the Uganda Bureau of Statistics CATI (data transmission/supervision)

40 New Methods in Household Surveys TALIP KILIC Living Standards Measurement Study Team Poverty & Inequality Group Development Research Group The World Bank PREM Learning Days DEC Course The Living Standards Measurement Study: Innovation in Survey Data for Better Policy Making