Sustained and Emerging Impacts Evaluations IFRC, Geneva 5 March 2018 Jindra Cekan, PhD

Transcription

1 Sustained and Emerging Impacts Evaluations IFRC, Geneva 5 March 2018 Jindra Cekan, PhD

2 Sustained and Emerging Impacts.. Valuing Voices 1. Close your eyes and think about a successful project. 1. Do you know if the results you achieved continued after the project ended? 3. Now think about a person you encountered in that project in a village, town, or partner. 4. Would they know what continued?

3 Concept of Sustainability in Definitions of Final and Post Project Evaluation Final Evaluation: The systematic and objective assessment of [a] completed project or programme, its design, implementation and results. to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact and [projected] sustainability. Post Project (Sustained Impacts, Ex-Post) Evaluation: The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long-term benefits. The resilience to risk of the net benefit flows over time. (Both OECD-DAC)

4 Change our perspective, look beyond evaluating only the finite aquarium of development

5 to the sea of sustained impacts! Development is a Process, not a Result Ian Davies

6 Expenditures: $5 trillion spent on foreign aid since 1945 $137 billion was spent in 2014 alone on development projects After much M&E, we re not sure if what we re doing has actually been sustainable Less than 1% of all projects have been evaluated for sustained impacts

7 From:

8 Who is doing the most sustained impacts (post project) evaluation? Japan s JICA, OECD, some European Commission Who the least? Multilaterals, most Bilaterals, INGOs, NGOs, national governments see

9 Why do Post Project Sustained Impacts Evaluations? I do post-project sustained impacts evaluation Wow. That must be depressing YET Agencies do so to: Learn from actual, not projected results for the next design and implementation Inform replication or scale Provide justification for future funding Promote their success in industry Evaluations can surface surprising (emerging) outcomes. There is more to learn: Lessons to improve the rigor of accountability to participants Data for actual sustained Return on Investment Advocacy strategy

10 Post Project Evaluations & Sustained and Emerging Impacts Evaluation (SEIE) Data and Theories of Sustainability

11 Negative Water/ Sanitation Outcomes: Dismal rates of feces disposal (below baseline), handwashing (below midterm). Nearly doubled open defecation (from midterm) Discouraging rates of Latrine use (decreases by 2/3 and ¾). USAID/ CRS/ CARE/ CARITAS, and local NGOs Madagascar, RANO-HP project

12 Negative Conflict Resolution Outcome: Unsustained resolution of land conflicts amongst herders, farmers, and hunters/ gatherers as it dropped potentially to zero after project s architecture for conflict resolution ended. No local LDP conflict resolution plans updated or financed World Bank Nigeria, Fadama II Development Project,, IDA-38380

13 ADRA, CARE, FH, Save the Children Bolivia USAID Exit Strategies, 4-country study, Tufts Friedman School of Nutrition and FHI/ Positive & Negative Child Growth Monitoring Outcome: Improved growth monitoring by 3% (CARE) Decreased growth monitoring from end line 2-3 years postproject only of 4-16% (ADRA, FH, SC)

14 Positive Food Consumption Impact: Participating households reported an increase in the number of months of food self sufficiency post project by 1/3. USAID/ CRS Niger, Nutritional and Food Security Program (PROSAN Rayuma)

15 Theory of Sustainability example by Tufts/ FHI 360 (2016)

16 (1) Source: Rogers and Coates Tufts/ FHI360 study for USAID Food for Peace, 2016 (2) Source: Cekan Valuing Voices study for CRS/ Niger, 2016 Positive Sustained Outcomes and Impact Why? CARE Bolivia (1) LINKAGES for RESOURCES: CARE encouraged and expected community health workers (CHWs) to continue working after project exit, by establishing links between CHWs and government public health system to provide supervision, training, materials. CRS Niger (2) 1. MOTIVATION: CRS M&E was scaled back to a handful of indicators and all staff and partners were trained in tracking and learning from data 2. RESOURCES: Local government allocated resources in district budget for CRS health and agric activities to continue and found new private and NGO funding to continue 3. CAPACITY: CRS Planned for exit halfway through, at 2.5 years of 5 year project by building capacity of local government to implement project after take over 4. LINKAGES: CRS linked farmers to traders directly for agric produce sale so food security benefits continued to grow

17 Valuing Voices Theory of Sustainability

18 Emerging Outcomes from Nepal and Niger: 1. LINKAGES & MOTIVATION: Half of the members of the all-women Village Banks reported helping one another deal with domestic disputes, including domestic violence. (Pact/Nepal) 2. MOTIVATION BECAME LOCAL VERSION: Participants exposed to messages about the benefits of clinic-based birthing reported this being due to the introduction of locally created incentives. (CRS/Niger) 3. CAPACITY AND RESOURCES: While a food security project improved assets and consumption for some, it was better water access that led to surprising decreases in gender violence (LWR/ Niger) Sources: PACT/ Nepal 2013, Cekan Valuing Voices study for CRS/ Niger, 2016, Cekan for LWR 2008

19

20 IFRC/ ERCS Ethiopia Design, M&E: Asked participants at project end what they consider selfsustainable assets overall agreement across three communities but against donor and GOE priorities of oxen, dairy cows, bees Implement: Communities knew what indicators to monitor for design-for-impacts Source: Cekan Valuing Voices IFRC study 2014

21 Why the dearth and what to do next? Valuing Voices Vision for Sustainable, Development 1 MAKE participant input mandatory when funding, designing, monitoring and evaluating programs 2 INVEST in sustained and emerging impact evaluation through allocating 5% of the project budgets for such evaluation, learning and dissemination 3 4 BUILD regional research centers to distill global lessons while promoting and institutionalizing local ownership DESIGN for sustained impacts and long-term country-ownership through evaluating what continued, what failed, and why and adapting funding and design for sustained impacts

22 QUESTIONS AND ANSWERS Join us! to advocate, evaluate and network