Impact evaluations of agricultural innovations

Transcription

1 Impact evaluations of agricultural innovations Kasetsart University, Bangkok 5 th of June, 2015 Dr. Jo Puri Deputy Executive Director and head of evaluation, 3ie

2 Agricultural productivity remains low

3

4 The big questions still remain What works to increase productivity in agriculture? How makes farmers adopt new technologies? How can information What works? be delivered so farmer decisions are improved? For whom? Efficacy of ISFM When? for increased agricultural productivity and incomes. Why? Contractual arrangements for increased market power, food security, marketed surplus and returns. Cost effective mechanisms to improve use of technologies and inputs.

5 WHAT IS AN IMPACT EVALUATION?

6 PROGRESA Well targeted program Incidence of illness down by 23% School enrollment up by 0.7 years Anthropometric (weight) measures improved

7 So what was different with Progesa? Why was it different? Why did policy makers take note? It was an IMPACT EVALUATION!. What is an IMPACT EVALUATION??

8 PROGRESA Random sample of of 506 eligible communities in 7 states - evaluation sample Random assignment of benefits by community: 320 treatment communities (14,446 households) First transfers distributed April control communities (9,630 households) First transfers November 1999

9 What is common to all these evaluations? Did the program cause the change? Would it have happened anyway?? If the program caused the effect, how much was the effect? Are there other ways, that are cheaper to get the same impact? Attributable impact Counterfactual counterfactual.. Counterfactuals!

10 What do we need to measure impact? Agricultural technology adoption programme Before After Project (treatment) 92 comparison The majority of evaluations have just this information which means we can say absolutely nothing about impact

11 Before versus after single difference comparison Before versus after = = 52 Before After Project (treatment) comparison the technology programme was successful because a large number of farmers attended the training programmes (and perhaps income increased) This before versus after approach is outcome monitoring. Outcome monitoring has its place, but it is not impact evaluation

12 Post-treatment comparison comparison Single difference = = 8 Before After Project (treatment) 92 comparison 84 But we don t know if they were similar before though there are ways of doing this (statistical matching = quasi-experimental approaches)

13 Double difference = (92-40)-(84-26) = = -6 Before After Project (treatment) comparison Conclusion: Longitudinal (panel) data, with a comparison group, allow for the strongest impact evaluation design (though still need matching). SO WE NEED BASELINE DATA FROM PROJECT AND COMPARISON AREAS

14 What do we need to measure impact? Before After Project Comparison

15 So in fact Before After Project Comparison

16 What is an impact evaluation? The holy grail statement: A xxxx program caused a 7% increase (amount of change and direction of change) in the income (some measure) of farmers. So we use implicit or explicit We can t see both the treatment counterfactuals and the absence in the same group. Has a theory of change. Uses mixed methods. CONTRIBUTION of a group can also be measured rigorously using these methods.

17 Step I Eligible units Step II Evaluation sample Step III Random Assignment Control Treatment External Validity Internal Validity Ineligible Eligible

18 EXAMINING THE TRADE-OFFS BETWEEN AGRICULTURE EXPANSION AND DEFORESTATION A QUASI EXPERIMENTAL APPROACH

19 Large tropical forest loss Protected areas are an important instrument Road building represents trade-offs.

20 Some important questions Do agricultural innovations programmes work? Do ag technology ALWAYS lead to increased income? Are agr programmes mean trade offs for the environment? How much? How MUCH did income increase by? WHO is best targeted through these programmes?

21 STUDY IN THAILAND

22 Most protected areas and forest reserves in Thailand are in the north.

23 Variable 1: Protected Areas

24 Variable 2: Road building

25 EXAMINING EFFECTIVENESS OF PROTECTED AREAS AND ASSESSING TRADE OFFS

26 Selection bias NORTH THAILAND Elevat.shp Elevat.shp Elevat.shp feet feet feet feet Areas that get protected are those that HAVE forests. Areas with high elevation and slopes and bad soils are where Usually they have LOW agricultural productivity. protected aeas are located. So forest programmes usually bring in land with low productivity. So do protected areas and forestry programmes really help? This is selection bias.

27 The Econometric model The econometric model that we estimate is thus given by Zi : Plot attributes (Slope, Elevation, Impedance weighted travel time, Soil Dummy, Population density) Y1i*: Net profit from clearing Y2i*: Net utility from protecting a plot otherwise 0 0; * 2 if * 2 otherwise 0 0; * 1 if * 1 = > = + + = = > = + + = i Y i Y i e i W B i Z i Y i Y i Y i e i Y B i Z i Y α γ

28

29 Thailand protected areas NORTH THAILAND Elevat.shp Elevat.shp Elevat.shp feet feet feet feet Roads (1982) and Forests Of North Thailand (1986)

30 Method: Instrumental variables Probability of land getting cleared = determined by soil fertility, slope, elevation, distance to the market, administrative factors, population pressure etc. Probability of land being protected = determined by some of the same factors AND closeness to a watershed area. Watershed (+) Protection x (-) Clearing

31 Cleared Land (Y1 = 1) T- Stats Slope (degrees) Elevation (ms.) Population density1990 (people/km 2 ) Log(cost) (1982)** Soil and Province Dummies Not Shown Protected Area dummy (1986) Protected Area (Y2 = 1) Constant Equation Slope (Degrees) Elevation (ms.) Population density1990 (people/km 2 ) Log(cost) (1982) Soil and Province Dummies Not Shown Watershed dummy Constant Log Likelihood No. of observations

32 Cleared Land (Y1 = 1) T- Stats Slope (degrees) Elevation (ms.) Population density1990 (people/km 2 ) Log(cost) (1982)** Soil and Province Dummies Not Shown Protected Area dummy (1986) Protected Area (Y2 = 1) Constant Equation Slope (Degrees) Elevation (ms.) Population density1990 (people/km 2 ) Log(cost) (1982) Soil and Province Dummies Not Shown Watershed dummy Constant Log Likelihood No. of observations

33 Thailand Protected Areas: Results Naïve model: Protection has a large effect on preventing deforestation. After you account for selection bias, in the static model, there is no effect. Protected lands would not have been cleared even if they had not been protected.

34 Accuracy of Predictions Actual Predicted Cleared Forested % of Predictions Correct Cleared % Forested % % Correctly predicted 57% 91% 34

35 Predicted Threatened areas Nam Nao Khao Sanam Phriang Long way still... Thung Salaeng Suang Chae Son Mae Yom Wiang Kosai Areas predicted to clear after cost is reduced by 100 units 35

36 SO HOW MANY IMPACT EVALUATIONS ARE THERE?

37 IMPACT EVALUATIONS REMAIN WOEFULLY INADEQUATE

38 2. WHAT DO WE WANT FROM THE WINDOW?

39 FARMER FIELD SCHOOLS

40 The adoption S-curve

41 Farmer field schools: from agricultural extension to adult education Farmer field schools (FFS) have been implemented since the 1980s reaching 12 million farmers in over 90 countries The question of effectiveness of FFS has long been a subject of debate Large evaluation literature: 3ie systematic review draws on over 500 documents. 28,000 study titles and abstracts screened 460 relevant FFS studies screened at full text Review of global project portfolio: 337 projects Review of targeting: 92 projects ( including 34 projects not covered in other reviews) Review of effectiveness: 134 studies (71 projects) Review of barriers and enablers: 27 studies (20 projects ) Review of benefit-cost estimates: 2 studies (4 projects )

42 No evidence of diffusion to FFS neighbours

43 Implications for policy, programmes and research FFS projects have changed knowledge and practices and raised yields in pilot projects FFS has not been effective when taken to scale. FFS requires skilled facilitation, which is difficult to sustain beyond pilot programmes FFS requires hands-on experience to encourage adoption, so diffusion has rarely occurred in practice Insufficient evidence on empowerment outcomes

44 Aims for the Agricultural Window High quality studies produced Policy makers and implementing agencies notice Better policy gets made. - Higher agricultural productivity, - Greater food security, - Increased welfare of (smallholder and women) farmers.

45 3. WHAT ARE HIGH QUALITY STUDIES?

46 What are 3ie high quality studies? - Plan ahead; Use other sources of data. - Discussed plan with implementers - Strong theory of change (formative work) - Use process evaluations & monitoring data - MAIN outcomes and indicators in a pre-analysis plan - Robust identification designs - Ensure sample size and power (for TOT) - High fidelity qualitative data, baseline and endline data and analysis - Well written timely final report Because Murphy s law works! Engagement with program staff Engagement with policy makers Engagement with 3ie

47 Thank you. 47