DIME Impact Evaluation Workshop

Transcription

1 DIME Impact Evaluation Workshop DIME Impact Evaluation Workshop Innovations for Agriculture Innovations for Agriculture June 2014, Kigali, Rwanda June 2014, Kigali, Rwanda

2 Impact Evaluation Workshop Innovations for Agriculture Pastoral Community Development Project-III Ethiopia June 2014 Kigali, Rwanda

3 What is the intervention? Objective of PCDP III: To improve access to community demand-driven social and economic services for pastoralists and agro-pastoralists of Ethiopia. The project has two primary components: o Community Driven Service Provision (CDSP) o Community investment fund (prioritizing community needs), institutionalizing the CDD, community-level self-monitoring and learning o Rural (Economic) Livelihoods Program o Promotion of SACCOs, identification and development of livelihood opportunities, promotion of adaptive research and innovation Complemented by o Development Learning and Knowledge Management o Project Management and M&E

4 Evaluation Questions Evaluation Question 1: Is a community demand-driven (CDD)* approach effective in improving** pastoral communities access to key services? Evaluation Question 2: What elements of the CDD process are the most effective in improving outcomes? Should we scale it up? *CDD approach is where decision-making is transferred to beneficiary communities ** concept of improved access includes whether services are those that address beneficiary communities priority needs *** concept of higher returns includes improved HH welfare as well as economic rates of return

5 Evaluation Question 1 Is a community demand-driven* approach effective in improving** pastoral communities access to key services? 1. Do pastoral communities have increased access to key services as a result of the project? 2. Do available services better address the communities priority needs as compared with externally-provided public services? 3. Do households invest in economic activities with higher returns*** as a result of enhanced access to financial, advisory and research services 4. Do pastoral households in targeted communities register improved health, nutrition, education and economic outcomes? 5. Do these effects differ by different sub-groups? (agro-pastoralists, mobile community groups, men/women)

6 Evaluation Question 2 What elements of the CDD process are the most effective in improving outcomes? Should we scale it up? 1. What aspects of the CDD approach work well? 2. Is there greater convergence between kebele (communityled) and woreda (district-led) public spending with integration of the CDD approach into regular government systems? 3. Are pastoral-specific financial products provided through SACCOs better than traditional products? In terms of uptake and changing economic livelihoods? 4. Is innovation promoted through the association of farmers with researchers more effective in improving pastoralists livelihoods?

7 Evaluation Design (for Evaluation Question 1) Randomized phase-in with difference-in-differences o Within each woreda, kebeles will be randomly assigned to start either in the first year (treatment) or second year (control) o Data will be collected during the baseline and midterm from these treatment and control kebeles o This will measure shorter-term impacts Differences-in-differences with matching: o Within each woreda, we will compare the treatment kebeles (chosen by the woreda) with similar control kebeles o Data will be collected before, during and after the project from treatment and control kebeles o This will measure longer-term impacts For evaluation question 2, a separate evaluation design will be developed

8 Sample and data There are 90 new project woredas and 9 kebeles per woreda, for a total population of ~900 treatment kebeles Within each region, we will randomly choose evaluation woredas from among these woredas o We will choose a sample of treatment kebeles from this population o We will also choose a sample of control kebeles The evaluation will collect baseline, midterm and final evaluation data from among: o Treated kebeles in the first and second year (shortterm impacts) o Treatment and control kebeles (longer-term impacts)

9 Timeline Design mission and concept note review (July-August) Baseline survey should begin 4-6 months from now (September-November) Endorsement by client of preliminary design (end of June)

10 Murakoze! Thank you! Merci! Obrigada! Asante! Gracias!

11 DIME Impact Evaluation Workshop DIME Impact Evaluation Workshop Innovations for Agriculture Innovations for Agriculture June 2014, Kigali, Rwanda June 2014, Kigali, Rwanda

12 Impact Evaluation Workshop Innovations for Agriculture Smallholder Agricultural Productivity Enhancement & Commercialization (SAPEC) Project Liberia June 2014 Kigali, Rwanda

13 Intervention Summary Objective: Rural Poverty; Food Security Income of smallholder farmers, rural entrepreneurs Target: Women/Youth Focus/Physically Challenged Components: Intensification of Rice, Cassava, & Vegetable Production Introduction of Agribusiness National Capacity Building

14 Impact Evaluation Question 1: program dissemination stage Problem: possible low participation rates Question: What is the impact of Soft Entry on participation rates? What is the impact of working with farmer leaders ahead of village dissemination? What is the impact of working with rural women leaders ahead of village dissemination? Method: Random assignment of different soft entry approaches in different villages

15 Sample and Data Treatment Group Villages Treatment 1 Conventional dissemination Treatment 2 Soft entry using farmer leaders Treatment 3 Soft entry using rural women leaders Pseudo Dissemination in Control Areas Data: administrative

16 Evaluation Question 2: Extension stage Problem: low capacity for extension services Question: What alternative extension services are effective on improving learning, technology adoption and crop yield and value? Conventional Extension Farmer Field School Lead Farmer E-Extension Method: Random assignment across villages

17 Evaluation Design RCT with assignment at the Village Level Outcomes of Interest: Participation rates Knowledge score Technology adoption rates Crop Yield (kg/ha 2015 & 2016) Crop value

18 Sample and Data Treatment Group Treatment 1 Conventional Extension Treatment 2 Farmer Field School Treatment 3 Lead Farmer Treatment 4 E-extension Villages Pure Control 50 Data: Agricultural Household Surveys

19 Timeline (Tentative) Activity Period Actors Identify Eligible Participants July October 2014 MOA via NGO Partners Baseline Survey October-November 2014 MOA, DIME, Survey Firm Extension Begins February October 2015 MOA, NGO Partners Planting Season April 2015 Farmer! Harvest October-November 2015 Farmer! Follow up Survey 1 October-November 2015 MOA, DIME, Survey Firm Follow up Survey 2 October-November 2016 MOA, DIME, Survey Firm

20 Open Questions? Limited Rural Finance for Inputs Rural Roads not rehabilitated yet Markets not repaired yet

21 DIME Impact Evaluation Workshop DIME Impact Evaluation Workshop Innovations for Agriculture Innovations for Agriculture June 2014, Kigali, Rwanda June 2014, Kigali, Rwanda

22 Impact Evaluation Workshop Innovations for Agriculture Agriculture Sector Development Project II (ASDP-II) Tanzania June 2014 Kigali, Rwanda

23 Impact Evaluation Workshop Innovations for Agriculture The Main Objectives ASDP II Increasing Income and crop yields (rice and maize farmers) ASDP II has 3 Components which include Development of Sustainable Smallholder Irrigated Rice Value Chain Support to Collective Warehouses Based Marketing (COWABAMA) Institutional Strengthening and Project Management June 2014 Kigali, Rwanda

24 COWABAMA - For the IE The Main Objective of COWABAMA: creating a market pool for cereals through FBOs (good price hence income of the farmers increase). The COWABAMA 2 Sub-components Improving Access to Markets which will be realize through improve feeder roads, construct/rehabilitate of warehouses, markets information access by farmers, etc) Enhancing Access to Productivity Enhancing Drivers such as improved technologies.

25 Challenges & Proposed Sub-Interventions Challenge: Farmers may not store due to immediate need of cash just after harvest Solution: Granting financial loans at the harvest time. The Loan will be conditional to storing the crops in the warehouses (i.e. commitment loan) To test its impact we will randomly assign some FBOs to access the loans.

26 Evaluation Question What is the impact of COWABAMA and the proposed commitment Loans on:- on crop yields Amount of crops stored in the warehouses Overall farmers incomes

27 Evaluation Design The COWABAMA will be rolled out in two phases: 145 warehouses(123 vs. 22) and then 100 warehouses(40 vs. 60 ). Baseline study for the entire 245 warehouses. For the Commitment Loans (Randomly assign some FBOs/warehouses to the treatment group) Now in order to estimate the Impact, 2 methods will be used:- Diff-in-diff for COWABAMA Randomized Control Trial methods for the Commitment Loans

28 Timeline Baseline study in Year 1 (November 2014), Phase I (145 warehouses : 123 vs. 22) 3 Years At the end of year 3 will conduct intermediate Survey and Evaluation. Phase II (100 warehouses: 40 vs. 60) begins at year 4 for a period of 2 years. End-line Survey Year 6 (2019) followed by IE. Upon return, the team will prepare IE for the wholly project (ASDP II).

29

30 DIME Impact Evaluation Workshop DIME Impact Evaluation Workshop Innovations for Agriculture Innovations for Agriculture June 2014, Kigali, Rwanda June 2014, Kigali, Rwanda

31 Impact Evaluation Workshop Innovations for Agriculture The Impact of evouchers and Extension on Input Uptake Uganda Beatrice Byarugaba, Robert Khaukha, Apollo Kamugisha, Edward Kirumira, Joseph Oryokot June 2014 Kigali, Rwanda

32 Intervention Summary Objective: Enhance correct utilization of improved inputs. Components evoucher: Varying discount vouchers off the price of improved inputs Provision of extension Government Private Community led

33 Evaluation Questions 1. Which type of extension delivery service is the most effective? 2. What is the impact of evoucher provision on uptake and use of improved inputs (fertilizer and seeds)? 3. What is the additional effect of combining extension delivery services and evoucher provision? a. Which extension delivery services facilitates greater uptake and use of inputs through evouchers?

34 Evaluation Design: Sub-county level randomization Randomly chosen villages: Within-village assignment to evoucher schedule E1: Government (75 subcounties) E2: Private sector (75 sub-counties) Randomly chosen villages: Within-village assignment to evoucher schedule Randomly chosen villages: Within-village assignment to evoucher schedule E3: Community based (75 sub-counties) E0: Control (75 subcounties) Randomly chosen villages: Within-village assignment to evoucher schedule

35 Evaluation Design: Household-level randomization Schedule A Schedule B Schedule C Season 1 75% subsidy Cash transfer No subsidy Season 2 50% subsidy 50% subsidy 75% subsidy Season 3 25% subsidy 25% subsidy 50% subsidy Season 4 No subsidy No subsidy 25% subsidy Within-village, household-level random assignment to Schedule A, B, or C Use public lottery to assign households to a subsidy scheme

36 Sample and data 300 sub-counties Data Randomly chosen villages with sample subcounties Farmer surveys: baseline, follow-up surveys Input use, knowledge Administrative data Extension performance monitoring forms

37 Timeline Planning: Jul Dec 2014 Baseline: Dec Jan 2015 Intervention Season 1: Feb 2015 Follow-up 1: June 2015 Intervention Season 2: Sept 2015 Follow-up 2: Jan 2016 Intervention Season 3: Feb 2016 Follow-up 3: Jun 2016 Intervention Season 4: Sep 2016 Endline: Jan 2017

38 DIME Impact Evaluation Workshop DIME Impact Evaluation Workshop Innovations for Agriculture Innovations for Agriculture June 2014, Kigali, Rwanda June 2014, Kigali, Rwanda