DIME Impact Evaluation Workshop. Innovations for Agriculture June 2014, Kigali, Rwanda

Transcription

1 DIME Impact Evaluation Workshop Innovations for Agriculture June 2014, Kigali, Rwanda

2 Impact Evaluation Workshop Innovations for Agriculture Learn from Surveys Data Collection Experiments Florence Kondylis Siyao Jessica Zhu (DIME) June 2014 Kigali, Rwanda

3 Overall Motivation What are the tradeoffs in measuring agricultural outcomes? Lessons from 3 impact evaluation data collections Measuring the diffusion of a technology knowledge and adoption Measuring farming input and labor usage Plot by plot / input by input vs. Aggregate measures

4 Measuring technology diffusion Project: Mozambique Market Led Smallholder Development in the Zambezi Valley Impact Evaluation (Kondylis, Mueller, Zhu, 2014) 4,000 HHs sampled in remote central Mozambique Compare three measures of knowledge Know by memory Know by name Knowledge score Compare measures of adoption self-reported objective

5 Knowledge Know by memory vs. knowledge score Farmers understate actual knowledge Know by name vs. knowledge score Farmers understate actual knowledge These differences varies widely by techniques 50-90% of correct reporting Cost Benefit of knowledge score Additional 50 cents on each survey percent increase in correct answers Up to one third of the magnitude of your effect LOST to measurement!!! Worth administering a knowledge score!

6 Adoption Self-reported vs. Observed adoption Farmers overstate their adoption About 80% of correct reporting This difference varies by technique Cost benefit additional $6 on each survey 7-27 percent increase in correct answers, depending on the technique May be cheaper to sub-sample

7 Results: False Reporting

8 Results: Cost-Effective Analysis Cost Effectiveness of the Inclusion of the SLM Knowledge Exam and Measuring SLM Objective Adoption (2012 USD)

9 Correlates of misreports Knowledge Adoption # of plots + + If many plots then get objective Interview duration + 0 Respondent fatigue Trained - + Social desirability bias

10 Measuring Input Use 2 IEs [Jones, Kondylis, Mobarak, Stein, Zhu, ongoing] Rwanda Land Husbandry, Water Harvesting and Hillside Irrigation (LWH) Project Sample size: 2802 households. each household has few plots, average: 1 plot/hh. Bangladesh Integrated Agricultural Productivity (IAPP) Project Sample size: 4820 households. each household has a lot of plots, average: 7 plots/hh.

11 Randomized survey design Questions at household level vs. plot level: Did the household apply composting in Season A 2013? Did the household apply composting on Plot 1,2,3 in Season A 2013? Questions that aggregate/not prompted vs. breakdown/prompted key words: Which inputs did you apply to this plot? Did you apply composting to this plot? (repeat for each input)

12 Quiz Right or Wrong? Asking questions plot by plot always increases the length of a survey WRONG!

13 Interview Time Interview time: no statistically measurable difference. in minutes By Aggregate By Plot All plots sample HH with 1 plot sample HH with more than 1 plot sample Male sample Female sample

14 Quiz Right or Wrong? Asking questions plot by plot always changes the mean of the outcome WRONG!

15 it depends!

16 it depends!

17 it depends!

18 Quiz Right or Wrong? The way inputs are recorded will likely affect the sample size required to detect an impact RIGHT!

19 Example: Rwanda (labor), Bangladesh (all inputs) Finding: Different ways of collecting input data significantly affects the variance of outcomes Implication: To detect the same effect size (all else being equal) the required sample size may be very different when collecting input use by plot / by input Example: Labor use, H: power=0.8, alpha=0.05, MDES 10% Aggregate measure: N>=377 By Plot: N>=1150

20 When designing your survey When choosing N using secondary data, check the survey design! You may end up really under/over powered! You may not detect the effect size you bargained for, even if the outcome is realized OR You may overspend on your data collection

21 Overall Conclusion There are clear cost/precision tradeoffs in collecting agricultural data More intensive ways of collecting data may be too costly (e.g. adoption in the field) inexpensive enough to be worthwhile (e.g. knowledge scores, plot by plot / input by input records) There is some interaction between the way you measure and the sample you need Be mindful! Re-run your power calculations at baseline to make decisions on your follow-up sample

22 In case you are still here with me

23 Rwanda DS: Inputs Mean Farming Inputs Aggregate By Plot Difference Mean Mean of Mean Dummy Variables Applied compost ** Applied manure Applied fertilizer Applied lime Applied pesticides Applied any farming inputs Continuous Variables (winsorized at high 1%) Total amount of compost purchased (kg) Total amount of manure purchased (kg) Total amount of fertilizer purchased (kg) * Total amount of lime purchased (kg) Total amount of pesticides purchased (kg) Total amount farming inputs purchased (kg) Total amount spent on compost (RWF) ** Total amount spent on manure (RWF) *** Total amount spent on fertilizer (RWF) Total amount spent on lime (RWF) Total amount spent on pesticides (RWF) Total amount spent on farming inputs (RWF) ***

24 Rwanda DS: Labor Mean Labor Inputs Aggregate By Plot Difference Mean Mean of Mean Dummy Variables HH hired labor to assist proudcing compst HH hired labor to assist preparing fields HH hired labor to assist planting HH hired labor to assist applying inputs HH hired labor to assist weeding HH hired labor to assist irrigating HH hired labor to assist harvesting HH hired labor to assist processing crops HH hired labor to assist farming Continuous Variables (winsorized at high 1%) Total number of labor days HH spent on proudcing compst *** Total number of labor days HH spent on preparing fields *** Total number of labor days HH spent on planting *** Total number of labor days HH spent on applying inputs *** Total number of labor days HH spent on weeding *** Total number of labor days HH spent on irrigating *** Total number of labor days HH spent on harvesting *** Total number of labor days HH spent on processing crops *** Total number of labor days HH spent on farming ***

25 Rwanda DS: Inputs Variance Farming Inputs Aggregate Plot Level Levene s test (winsorized at high 1%) Std. Deviation Std. Deviation P-value Total amount of compost purchased (kg) Total amount of manure purchased (kg) Total amount of fertilizer purchased (kg) Total amount of lime purchased (kg) Total amount of pesticides purchased (kg) Total amount farming inputs purchased (kg) Total amount spent on compost (RWF) Total amount spent on manure (RWF) Total amount spent on fertilizer (RWF) Total amount spent on pesticides (RWF) Total amount spent on farming inputs (RWF)

26 Rwanda DS: Labor Variance Labor Inputs Aggregate Plot Level Levene s test (winsorized at high 1%) Std. Deviation Std. Deviation P-value Total number of labor days HH spent on proudcing compst Total number of labor days HH spent on preparing fields Total number of labor days HH spent on planting Total number of labor days HH spent on applying inputs Total number of labor days HH spent on weeding Total number of labor days HH spent on irrigating Total number of labor days HH spent on harvesting Total number of labor days HH spent on processing crops Total number of labor days HH spent on any other task Total number of labor days HH spent on farming

27 Rwanda MDES: Inputs Farming Inputs (winsorized at high 1%) Aggregate Plot Level (power=0.8, alpha=0.05) N N Total amount of compost purchased (kg) Total amount of manure purchased (kg) Total amount of fertilizer purchased (kg) Total amount of lime purchased (kg) Total amount of pesticides purchased (kg) Total amount farming inputs purchased (kg) Total amount spent on compost (RWF) Total amount spent on manure (RWF) Total amount spent on fertilizer (RWF) Total amount spent on pesticides (RWF) Total amount spent on farming inputs (RWF)

28 Rwanda MDES: Labor Labor Inputs (winsorized at high 1%) Aggregate Plot Level (power=0.8, alpha=0.05) N N Total number of labor days HH spent on producing compost Total number of labor days HH spent on preparing fields Total number of labor days HH spent on planting Total number of labor days HH spent on applying inputs Total number of labor days HH spent on weeding Total number of labor days HH spent on irrigating Total number of labor days HH spent on harvesting Total number of labor days HH spent on processing crops Total number of labor days HH spent on any other task Total number of labor days HH spent on farming

29 Rwanda Distribution: Inputs Farming Inputs Combined K-S P-Value Applied compost Applied manure Applied fertilizer Applied lime Applied pesticides Applied any farming inputs Total amount of compost purchased (kg) Total amount of manure purchased (kg) Total amount of fertilizer purchased (kg) Total amount of lime purchased (kg) Total amount of pesticides purchased (kg) Total amount of farming inputs (RWF) Total amount spent on compost (RWF) Total amount spent on manure (RWF) Total amount spent on fertilizer (RWF) Total amount spent on lime purchased (RWF) Total amount spent on pesticides purchased (RWF) Total amount spent on farming inputs purchased (RWF)

30 Rwanda Distribution: Labor Labor Inputs Combined K-S P-Value HH hired labor to assist producing compost HH hired labor to assist preparing fields HH hired labor to assist planting HH hired labor to assist applying inputs HH hired labor to assist weeding HH hired labor to assist irrigating HH hired labor to assist harvesting HH hired labor to assist processing crops HH hired labor to assist any other task HH hired labor to assist farming Total number of labor days HH spent on producing compost Total number of labor days HH spent on preparing fields Total number of labor days HH spent on planting Total number of labor days HH spent on applying inputs Total number of labor days HH spent on weeding Total number of labor days HH spent on irrigating Total number of labor days HH spent on harvesting Total number of labor days HH spent on processing crops Total number of labor days HH spent on any other task Total number of labor days HH spent on farming

31 DIME Impact Evaluation Workshop Innovations for Agriculture June 2014, Kigali, Rwanda