Managing Risk through Economic Development Phase I: Impact Evaluation

Transcription

1 Managing Risk through Economic Development Phase I: Impact Evaluation MERCY CORPS SPRING 2016

2 Causal Design PBC partners with international development clients to provide rigorous independent program evaluation, expand cultures of evidence within organizations, and join them in efforts to relieve human suffering and end poverty. 720 Village Center Dr. Colorado Springs, CO 80919, USA Tel: +1 (720)

3 TABLE OF CONTENTS 1.0 EXECUTIVE SUMMARY BACKGROUND Overview M-RED Program Area and Hazard Profile M-RED Structure and Activities Beneficiary Selection IMPACT EVALUATION DESIGN & METHODOLOGY Evaluation Design Theory of Change Research Questions Key indicators Estimating Household Impact Cost-Benefit Analysis Sampling and Data Quality RESULTS Nepal Terai: Sugarcane Intervention Purpose and Activities Results Impact and Discussion of Results Nepal Hilly Region: Fodder Intervention Purpose and Activities Results Impact and Discussion of Results Timor-Leste Purpose and Activities Results Impact and Discussion of Results CONCLUSIONS Program Recommendations Limitations Future Research 54 APPENDIX A: PRE-PROGRAM COMMUNITY MATCHING 52 APPENDIX B: RESULTS OF PROPENSITY-SCORE MATCHING 53

4 LIST OF FIGURES AND TABLES Table 1: Disaster Risks and Nexus Activities by Context 6 Table 2: Program Impacts 8 Figure 3: M-RED Nepal Process Chart 12 Figure 4: M-RED Theory of Change 14 Figure 5: Levels of Sustainability 15 Table 3: Disaster Risk Reduction Output Indicators 17 Table 4: Disaster Risk Reduction Outcome Indicators 18 Table 5: Economic Development Output Indicators in Nepal Terai 18 Table 6: Economic Development Output Indicators in Nepal Hilly Region 19 Table 7: Economic Development Outcome Indicators 20 Table 8: Resilience Capacity and Outcome Indicators 21 Figure 7: Sugarcane Nexus - Contribution of Plantings to Harvest 25 Table 9: Quantitative Sample Size and Attrition 26 Table 10: Qualitative Sampling 26 Figure 10: M-RED Terai - Theory of Change 29 Table 11: Terai DRR Output Results 30 Table 12: Terai DRR Outcome Results 31 Table 13: Terai Economic Development Output Results 32 Table 14: Terai Economic Development Outcome Results 33 Table 15: Terai Other Resilience Capacity Results 35 Figure 12: M-RED Nepal Hilly Region - Theory of Change 38 Table 16: Hilly DRR Output Results 39 Table 17: Hilly DRR Outcome Results 40 Table 18: Hilly Economic Development Output Results 41 Table 19: Hilly Economic Development Outcome Results 42 Table 20: Hilly Resilience Capacity Results 43 Figure 15: M-RED Timor-Leste Theory of Change 46 Table 21: Timor-Leste DRR Output Results 47 Table 22: Timor-Leste DRR Outcome Results 48 Table 23: Timor-Leste Economic Development Outcome Results 49 Table 24: Timor-Leste Resilience Capacity Results 50 Figure 16: Predicted Sustainability 52

5 ACRONYMS AND TERMS Agro-vets Local level input and technical service providers Bigha Standard unit of land measurement in Nepal; one bigha is equivalent to 2/3 hectare CBA Cost Benefit Analysis DADO District Agriculture Development Office DLSO District Livestock Support Office DMC Disaster Management Committee DRR Disaster risk reduction ED Economic development EWS Early warning system FFS Field Farmer School Gabion Structural mitigation walls, boxes or fences IGA Income-generating activity Katha Standard unit of measurement; one katha is 1/20 bigha MFI Micro-finance institution M-RED Managing Risk through Economic Development MSD Market Systems Development Nexus intervention Uniquely designed interventions to incentivize disaster risk reduction through economic opportunity NGO Non-Governmental Organization River-cutting Erosion of riverbanks Quintal Unit of weight measurement in Nepal; equivalent to 100 Kgs

6 1.0 EXECUTIVE SUMMARY Natural disasters present a serious risk for communities in many parts of the world. Acute shocks like floods and landslides take a sharp toll in lives lost and homes destroyed. Over a longer time period, they present a lasting threat to communities existence. Livelihoods become untenable as farmland is washed away, roads are blocked, and productive equipment and herds of livestock are destroyed. Families who lose their income sources may find themselves trapped in a cycle of poverty, unable to reinvest in their livelihood and unable to protect themselves from future shocks. Even people who are not directly affected by disasters are still forced into making inefficient choices, such as leaving fertile farmland untended out of fear that seeds and crops will be destroyed. Traditional measures to prevent this cycle of loss focus on structural mitigation, generally using retaining walls to bolster riverbanks and protect hillsides. This approach, while effective, is also expensive. The initial construction requires labor and materials, and afterwards continual maintenance is needed for them to remain effective. Communities cannot always marshal the resources needed to maintain these works, so they depend on long-term assistance from government agencies or non-governmental organizations. Mercy Corps Managing Risks through Economic Development (M-RED) program was conducted from It sought to provide sustainable disaster mitigation to vulnerable communities in Western Nepal and Timor-Leste. Crucially, M-RED used an innovative approach built around nexus activities that tied economic incentives to mitigation activities. The nexus activities used high-value crops, planted in strategic locations, to strengthen hillsides and reinforce riverbanks. In doing so, they replaced or reinforced traditional mitigation structures. Since the crops could be sold on the market, community members were also able to earn income by maintaining the plantations. If this approach succeeded, then communities could be expected to continue the nexus activities even without external support. Table 1 below details the disaster risks and nexus activities in each context. Table 1: Disaster Risks and Nexus Activities by Context Region Disaster risks identified pre-program Nexus Activity Income Source Nepal Terai (Floodplains) Flooding and river-cutting (rivers washing away nearby land) Sugarcane plantations along vulnerable riversides Sale of sugarcane Nepal Hilly Region Landslides, flooding, and river-cutting Fodder plantations on fragile hillsides, and introduction of high-productivity dairy cows that consume fodder Sale of milk from dairy cows Timor-Leste Acute: Flooding, drought, fires, heavy wind, earthquakes Chronic: Climate change and loss of soil fertility Bananas and lumber trees planted on erosion-prone slopes. Legumes planted on low-fertility and erosion-prone areas Sale and/or consumption of vegetables, fruits, and legumes, future sale of lumber Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 6

7 METHODOLOGY This study examines M-RED s impact on communities disaster affectedness and economic development, and its prospects for future sustainability. The program worked in three separate geographic contexts: the lowland Terai (floodplains) of Nepal s Far West region, the Hilly region of Western Nepal, and Timor-Leste. We treat each of the three as a separate case-study, since the program targeted different types of disasters and employed different nexus activities in each region. We use propensity-score matching to estimate M-RED s impact on beneficiary households. This approach allows us to compare beneficiary households who participated in the nexus activities against households from other communities with similar demographic and disaster-risk profiles before the project began. Any difference between the two groups can then be attributed to the program. We study M-RED s effect on households disaster losses, economic status, and other resilience capacities. The household-level analysis is reinforced through qualitative discussions with program beneficiaries and key stakeholders. RESULTS In the Terai, M-RED bolstered measures of disaster readiness, such as households use of structural mitigation and bio-engineering to protect fields. It also increased the amount of sugarcane that beneficiaries planted near riversides, and led to greater cohesion among their communities and stronger connections to government and market actors. However, this did not translate into reduced disaster impacts or economic development. There are two reasons for this disconnect: first, during M-RED s implementation the disaster profile of the Terai shifted from flooding and river-cutting to drought, rendering M-RED s flood prevention measures less effective. Second, the sugarcane nexus activities were managed at the community level, and the majority of the income earned was reinvested into sugarcane or used for group projects. Within the program period, individual households didn t receive any significant income from the activities. In Nepal s Hilly region, M-RED also increased disaster preparedness in beneficiary communities, and led to greater milk production. These effects led to a 80% reduction in the value of land and property lost to disasters, relative to the comparison group. However, there was no effect on measures of economic growth or stability during the project s implementation. The results were similar in Timor-Leste, where M-RED built disaster readiness and led to a 66% decrease in the value of disaster losses, relative to comparison communities. Program beneficiaries also had higher levels of food security and nutrition than others, and were better able to maintain their income sources in the face of disasters. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 7

8 Table 2: Program Impacts Region Disaster Risk Reduction Economic Development Other Resilience Capacities Nepal Terai - Sugarcane No statistically significant result No statistically significant result Increased social cohesion, strengthened connections to government and market actors Nepal Hilly Region Fodder and Dairy Reduced value of disaster losses by 80% No statistically significant result No statistically significant result Timor-Leste Bananas, Legumes, Agro-forestry Reduced value of disaster losses by 66% Increased income stability Increased food security and nutrition, and increased connections to government actors CONCLUSIONS Targeting specific disasters matters In the Nepal Hilly and Timor-Leste areas, M-RED effectively reduced the disaster impact on beneficiary households. In both these areas, the disaster profile over the implementation period largely matched expectations. However, in the Terai the program had a more limited effect on disaster losses, since the intervention specifically targeted flooding and river-cutting rather than the drought that communities actually experienced. Economic and market development requires a longer timeframe During the two years from the beginning of the nexus activities to the program s end, none of the three contexts showed clear economic growth. Many of the nexus activities revolved around new income-generating activities or new techniques: sugarcane planted on reclaimed riverbanks, new breeds of dairy cows, and agroforestry. This is unsurprising, since two years is considered a short timeframe for market development programs, where participants must learn new skills and market systems need time to adjust. A lack of economic growth does not make the nexus activities unsustainable, since communities still benefited from reduced disaster losses while earning income. However, a program seeking to introduce new livelihood activities must be prepared to support communities for more than two years before they are economically profitable on their own. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 8

9 The level at which nexus activities are conducted matters Although the sugarcane nexus did not show an economic advantage over the other interventions, it nonetheless had unique benefits: improving social cohesion among beneficiary communities. This reflects the community-level management of the sugarcane work, which required groups to handle financial and logistical matters together. Most communities also used some of the income from sugarcane sales to create credit schemes to benefit local farmers, or funds for the healthcare of pregnant women and infants. In contrast, the more individually-oriented nexus activities in the Hilly region and Timor-Leste did not have a clear effect on social cohesion. RECOMMENDATIONS Prepare for multiple disaster risks M-RED conducted substantial research into common disaster types and possible nexus interventions before programming began. This research led to the choices of nexus activities, which were intended to mitigate the most common disasters in each region. However, during the implementation phase the disaster profile shifted in both Nepal contexts, particularly in the Terai which saw a drastic shift from flooding to drought. This type of unexpected change cannot be foreseen, but it raises the risk that communities will discount the benefits of the program during its implementation. When it comes time for them to maintain the sugarcane plantations and other mitigation works without M-RED s support, their decision-making may be biased by the recent drought. While M-RED cannot program toward every eventuality, there should be funding available to adapt to unexpected circumstances. Otherwise, if a community did not benefit from disaster protection or from increased income, it is unlikely that they will be motivated to sustain program activities after funding ends. Assess livelihood landscape in targeted communities Nexus interventions were chosen primarily for the disasters they could mitigate. However, injecting a new income source into the complex web of a market, even a local one, may have unexpected effects. First of all is the opportunity cost: households may be sacrificing time or resources they would have spent on other income opportunities to participate in the nexus activity. This lowers the relative benefit of the nexus activity compared to other livelihoods. There may be specific interventions for which there would be little opportunity cost. Communities may also be able to bring new participants into the labor force rather than drawing on those who would otherwise be working elsewhere. There is some anecdotal evidence that this has already happened in Nepal, as several focus groups reported women entering the labor market for the first time as a part of the nexus activities. M-RED could explore these examples further to see if they can be replicated in other contexts, thereby increasing the economic gains of the program. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 9

10 2.0 BACKGROUND 2.1 Overview The Managing Risks through Economic Development (M-RED) program aimed to reduce the human and economic toll of disasters in a sustainable manner. From 2013 to 2016 the program worked with communities in Nepal and Timor-Leste that are routinely affected by shocks, such as floods and landslides, and long-term stresses like erosion and soil degradation. The continuous disasters can cause personal harm and combine to make common livelihoods like farming untenable. Previous disaster risk reduction (DRR) programs tried to combat this decline through structural mitigation, often through retaining walls to bolster riverbanks and protect hillsides. This approach, while effective, was also expensive, and many communities have been unable to maintain the mitigation works after prior program s completion. M-RED sought to move beyond this approach by connecting income sources directly to disaster mitigation, thereby making DRR efforts economically sustainable. The program identified certain nexus activities which use valuable crops to replace or reinforce mitigation structures. This approach allowed communities to invest in protective mitigation as a source of future income, rather than one that requires continual maintenance and resources without providing an economic return. This double-benefit approach combining income and DRR is clearly preferable in theory, and this evaluation tests its effectiveness in practice. Additionally, M-RED worked to develop communities own capacities to protect themselves and respond to local disasters. By building greater organization and cohesiveness, communities become less reliant on outside assistance for expensive structural mitigation. They are better able to pool their human and financial resources together for both structural maintenance and disaster response. The program also connected communities with government and private sector actors, widening the net of resources they can draw on. Figure 1: M-RED Nepal Program Areas This evaluation tests whether M-RED s approach has been effective in practice. Additionally, the study compares the results between different environmental and economic contexts to identify particular practices which worked well or encountered challenges. 2.2 M-RED Program Area and Hazard Profile Within Nepal, M-RED worked in 34 hazard-prone and poverty-affected communities located in four districts of the Far West region, totaling a population of 18,000. An additional 6 communities were added after the two-year mark of the program, and are excluded from the evaluation because we do not expect significant benefit after only one year of implementation. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 10

11 These four districts cover two geographic areas: Kailali and Kanchanpur sit in the lowland Terai, an area of plains known for its productive agriculture despite being frequently affected by floods and river cutting. Baitadi and Dadeldhura are part of Nepal s Hilly region and struggle with frequent landslides and drought, in addition to being affected by flooding and erosion around rivers. deforestation that has led to erosion, loss of soil nutrients, and even damaging winds. In Timor-Leste, the program targeted 29 disaster- and poverty-affected communities located in three districts of the western part of the country: Dili, Ermera and Ainaro. Their total population is over 22,000, and they suffer from many of the same disasters as the communities Nepal. Flooding, landslides, and river-cutting are common, but these are further exaggerated by Figure 2: M-RED Timor-Leste Program Areas 2.3 M-RED Structure and Activities M-RED touched on two complex systems in its target areas: the local disaster environment and the local market system. To operate effectively in these systems, the program expanded its activities as needed to overcome challenges and constraints that arose. For example, when community members were hesitant to invest in expensive new breeds of cows, M-RED connected them with government-backed insurance providers to reduce the risk. Similarly, when sales of sugarcane gave lower-than-expected prices, the program forged connections between communities and larger sugar mills. As a result, activities were conducted at the level of regional markets, community leaders, and individual households. Community-Level Capacity Building In partnership with local disaster management government officials, M-RED worked at the community level with Disaster Management Committees (DMCs) to build local capacity to prepare for, mitigate, and respond to these disasters. The average DMC consists of roughly 7-15 members. Exact activities varied between communities but generally included training of first responders, creation of disaster response plans and early warning systems, and maintenance of physical mitigation structures. Some of these activities included the broader community as well, particularly trainings on basic preparation for and response to disasters. Household-Level Nexus Interventions In addition to this community-level programming, M-RED also tested the effect of providing an economic incentive (through income-generating activities, or IGA s ) for certain DRR measures. This combination was brought about through nexus interventions: a set of agriculture products and planting techniques chosen for their ability to simultaneously mitigate hazard risks and generate income. Through a rigorous assessment process involving stakeholders from communities, government, the private sector, and local partner organizations, the program selected the following set of nexus interventions : Sugarcane as a cash crop, planted on silted and erosion-prone riverbanks in Nepal s Terai region of the Far West. Fodder planted on erosion- and landslide-prone slopes in Nepal s Far West Hilly region, to be used as an input in the dairy sector. Bananas for consumption and sale, planted on erosion-prone slopes and riverbanks in Timor-Leste. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 11

12 Legumes for consumption and sale, planted on erosion-prone and marginal (poor fertility) lands in Timor- Leste. Agro Forestry including fruit trees for short-term consumption and sale and hardwood trees for long-term sale, planted with land-management techniques on erosion-prone slopes in Timor-Leste. Market Development To ensure that there was a viable market opportunity for these nexus products, M-RED partnered closely with local and regional private sector market actors from the beginning of the program. These market actors range in value chain roles from input supply to technical advice, collection, storage, sales points, transportation, processing, value addition, and retail outlets. The program also worked with financial service providers including local cooperatives, micro-finance institutions (MFI s), micro-insurance, and banks on a limited basis to expand access to appropriate savings, loans, and insurance services for targeted nexus sub-sector growth. The financial context at the time of implementation varied dramatically between the two countries. In Timor-Leste there was limited access to micro-credit and no existing micro-insurance system. In response, the program focused on village savings and lending. In contrast, Nepal had a more developed financial services market with MFIs, cooperatives, and nascent micro-insurance offerings that M-RED was able to leverage. Figure 3: M-RED Nepal Process Chart below shows the varieties of interaction between these actors in Nepal: Figure 3: M-RED Nepal Process Chart 2.4 Beneficiary Selection Starting in 2013, the program identified 2,000 households (1,000 per country) to work with on the nexus intervention strategy. M-RED worked with local authorities and the DMC to identify the most hazard-prone and poverty-affected households. These most vulnerable families were then organized into farmer groups through which Mercy Corps could deliver the nexus interventions. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 12

13 3.0 IMPACT EVALUATION DESIGN & METHODOLOGY 3.1 Evaluation Design M-RED was a far-reaching program, and within each of the three geographic regions it conducted activities aimed at government officials, the private sector, and communities. However, the ultimate goal was to build the welfare of disaster-affected communities through protection from natural shocks and stable economic development. This evaluation, then, focuses on the households participating in nexus activities - the downstream beneficiaries of every aspect of the program. To estimate M-RED s effect on households well-being, we compare beneficiary households against households from similar communities in the same regions. These comparison communities were selected for their similarity to the treatment communities in terms of economic activity and disaster experience prior to the program 1. The matching process is described in Appendix A. We also use propensity-score matching 2 to compare specifically between similar households. The basic estimation of M-RED s effect is the change among the treatment group from the beginning (baseline) of the program to the end (endline), minus the change for the comparison group from baseline to endline. This lets us control for broader trends in the local climate and regional markets which might have affected both groups equally. For some of the outcome measures we use, no baseline data are available and we can only compare the levels at endline between the two groups. While this gives a less precise estimation of the program s impact, by comparing between similar communities and households we can still account for unobserved characteristics that might bias our results. The exact outcome measures are defined in Section 3.4 below. M-RED s goal of sustainable DRR with economic incentives was consistent across all three of its target areas, but the specific activities conducted varied between them. Likewise, the environmental and economic contexts were different in Nepal s Terai region, the Hilly region, and the western portion of Timor-Leste. For these reasons, we estimate M-RED s effect separately for each of the three regions effectively treating them as three separate subprograms. However, we can also make broader conclusions about the program by comparing between these subprograms to gain insight into how and why particular program components were more successful or faced challenges. 3.2 Theory of Change Figure 4 below depicts the two paths through which M-RED s activities were expected to affect community members well-being outcomes. The first chain was through DRR and other resilience capacities - such as social capital and the use of financial tools - which allowed households to protect themselves and their assets from disasters. When shocks and disasters did occur, these capacities also helped to maintain households well-being by protecting their property and livelihood from the disaster s impact. The second path to enhanced well-being was through economic development, by bolstering income sources and market linkages for communities. This fed directly into economic measures of well-being, including wealth and income. There were two components to these economic measures: first, M-RED was intended to increase households income and overall economic capability what we describe as economic growth. Second, the combination of the DRR and ED pathways was also meant to protect income sources and welfare in the face of disasters which we describe as economic stability. Sustainability then springs from the combination of these two streams, reduced disaster impact and improved economic well-being M-RED s novel contribution to the field of DRR. We cannot measure future sustainability, but we anticipate that if the communities perceive the benefits of the program activities to outweigh the costs, they will continue these activities even without outside support. 1 Conducted by the original research partner, the Asian Disaster Preparedness Center (ADPC) 2 Rosenbaum, Paul R. and Donald B. Rubin The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika 70: Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 13

14 Figure 4: M-RED Theory of Change Program Activities Outputs Outcomes Sustainability DRR Outputs: Mitigation Capacity Building Early Warning Systems Disaster Response DRR Outcomes: Nexus activities Disaster management training Building linkages Other Capacities: Social Capital Use of Financial Inputs Migration Loss of land and property Coping strategies Communities perceive benefits of activities to outweigh costs ED Outcomes: Economic stability ED Outputs: Economic growth Planting on Marginal Land Production The conflux of economic incentive and disaster mitigation predicts how sustainable the program activities will be. Figure 5 shows these possible outcomes: if M-RED was able to provide a profitable and stable income source while mitigating the effects of a disaster, we expect it to be highly sustainable. It may only have provided one of those components, either acting as a solid economic investment or a straightforward DRR program. In this case, the prospects for sustainability depend on the balance between the two, as well as the community s perception of any other benefits or costs beyond our consideration. The program becomes more unsustainable as both the economic and disaster mitigation effects become negative, although the community may still feel that other benefits make it worthwhile to continue. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 14

15 Figure 5: Levels of Sustainability Increased Economic Development Medium Sustainability: Profitable No disaster impact Low Sustainability: No economic impact or economic loss No disaster impact High Sustainability: Profitable Reduces disaster impact Medium Sustainability: No economic impact or economic loss Reduces disaster impact Reduced Disaster Impact 3.3 Research Questions This study is intended primarily as an evaluation of M-RED s impact on the beneficiary population. However, one of the key motivations behind M-RED was to test the nexus interventions as a theoretical model for future DRR programs. If the combined mitigation and economic benefits of the nexus crops outweighed their costs, it would open the possibility of truly self-sustaining disaster mitigation maintained by communities themselves, with no need for continued funding from government or NGO sources. Because of this possibility, we conduct additional analysis to better understand the functioning of the nexus activities: how they worked, as well as their ultimate results. We therefore seek to use rigorous quantitative analysis, supported by qualitative analysis and the contextual understanding of the program teams, to better understand the overall effect of various aspects of the M-RED intervention addressed below: QUESTION 1: DOES THE INTEGRATED DRR AND ECONOMIC DEVELOPMENT MODEL IMPLEMENTED IN M-RED ACHIEVE MEANINGFUL IMPACT ON BOTH RISK REDUCTION AND ECONOMIC DEVELOPMENT? TO WHAT EXTENT DO THE INTERVENTIONS IN M-RED ACHIEVE RISK REDUCTION IMPACT? TO WHAT EXTENT DO THESE SAME INTERVENTIONS IN M-RED ACHIEVE ECONOMIC DEVELOPMENT IMPACT? This question addresses M-RED s ultimate impact on beneficiary households, divided into DRR and economic benefit. If M-RED s activities mitigated disasters while fostering stronger economic opportunities for the nexus households, we would expect to see beneficiaries with fewer losses to disasters and greater economic well-being. At a minimum, we would expect that the nexus interventions allowed households to maintain their income streams in the face of a disaster, so that over time they remain on a growth trajectory. We analyze these results separately for each of the three geographic contexts: The Nepal Terai, Nepal Mid-Hilly region, and Timor-Leste, although we Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 15

16 use the same analytical framework for all three. This disaggregation reflects the differences in risk profiles, market systems, and the implementation activities between the three groups. QUESTION 2: IN WHAT WAYS DOES THE INTEGRATED DRR/ED MODEL CONTRIBUTE TO ENHANCING OTHER CAPACITIES OF HOUSEHOLDS IN HAZARD-PRONE, LOW-INCOME COMMUNITIES TO NATURAL DISASTERS AND OTHER MAJOR RECURRENT SHOCKS? Household and community resilience is related to, but distinct from, M-RED s primary goals of DRR and ED. We use Mercy Corps definition of resilience as the capacity that ensures adverse stressors and shocks do not have long-lasting adverse development consequences, focusing on specific categories of capacities Mercy Corps has previously identified as critical to managing shocks. This question goes beyond M-RED s specific DRR and economic objectives to test for broader effects on social capital and financial inclusion. We then test the usefulness of these capacities by examining households disaster loss and economic status when they were affected by a shock. We expect that resilient households were able to draw on their capacities to recover in spite of the shock, compared to less-resilient families. QUESTION 3: DOES THE INTEGRATED DRR AND ECONOMIC DEVELOPMENT MODEL HOLD POTENTIAL FOR SUSTAINABILITY BY OVERCOMING BARRIERS FOR COMMUNITIES TO INDEPENDENTLY CONTINUE PROGRAM ACTIVITIES, PARTICULARLY IN PROVIDING ECONOMIC BENEFITS GREATER THAN THE COST OF CONDUCTING ACTIVITIES? ARE THERE EXAMPLES OF REINVESTMENT, INDEPENDENT REPLICATION AND/OR SCALE-UP OF THE MODEL WITHIN THE TARGET COMMUNITIES OR IN OTHER COMMUNITIES? Evaluating M-RED s success in regards to sustainability presents challenges, as we try to proactively measure the duration of communities DRR activities. The evaluation approaches this question by calculating the costs of continuing the nexus activities against the benefit accrued by the households and communities which engage in them. If the combination of the nexus risk-reduction and economic development was successful, we expect the value of land protected and income earned to be greater than the costs spent planting and harvesting the crops. A positive benefits-to-cost ratio suggests that households and communities will be sufficiently incentivized to continue conducting the nexus activities independently after the completion of the project. For the purposes of this study, we limit our cost-benefit analysis (CBA) to the sugarcane intervention since more reliable data are available on the quantity of harvest and sales. The sub-question addresses whether individual households used their own resources to scale-up their plantations in treatment communities, and whether non-treatment communities have been motivated to replicate the sugarcane nexus intervention. Since these non-treatment groups would bear the full costs of adopting the nexus approach, they would provide a strong yardstick for measuring the perceived benefits. We use discussions with non-treatment replicators to better understand their motivations. QUESTION 4: WHAT ARE THE KEY FACTORS OF THE INTEGRATED MODEL THAT CONTRIBUTED TO ITS SUCCESSES AND FAILURES? WHAT ARE THE BEST METHODS TO INTEGRATE DRR AND ED IDENTIFIED IN THE PROGRAM? WHAT TOOLS AND PROCESSES COULD BE ADOPTED BY OTHER PROGRAMS TO ACHIEVE SIMILAR INTEGRATED RESULTS? The final research question moves from asking what M-RED did to why its program elements were effective or ineffective. This analysis primarily draws on qualitative reporting and implementing context drawn from the program teams to identify best practices for use in future iterations of the nexus concept. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 16

17 3.4 Key indicators Output and outcome indicators are chosen to reflect M-RED s anticipated impact on households within the treatment communities. They are divided by topic disaster risk reduction, economic, and other resilience capacities though due to the interplay of disaster risk and economic status, these categories are related. DISASTER RISK REDUCTION We first look at a set of output indicators reflecting how M-RED s components directly affected households. These outputs do not indicate an overall change in the households disaster risk, but rather the channels by which M-RED may have affected it. The connection between outputs and outcomes can be elusive for disaster mitigation programs; it depends not only on the outputs efficacy, but also on the disaster context of the particular region and time period under evaluation. For example, a well-planned flood prevention system will not be expected to benefit communities undergoing a drought. Generally, the final outcome measurements depend on both the program s success and outside factors. The set of output indicators used is given in Table 3 below: Table 3: Disaster Risk Reduction Output Indicators Topic Mitigation Techniques DMC Activity Disaster Response Indicator Use structural mitigation to protect land Use bio-engineering to protect land Use improved agricultural techniques to protect land Community has active DMC Frequency of DMC meetings in past year Community has disaster response plan Community has early warning system Household knows how to respond in case of disaster Household has disaster response plan (formalized) Availability of Data (Baseline/Endline/Both) Endline Endline Endline Endline Endline Endline Endline Endline Endline The set of indicators used to measure M-RED s impact on risk reduction directly address the ways that households within M-RED s targeted regions were affected by disasters. These include losing productive land or agricultural inputs like seeds to a variety of disasters, losing physical property and assets, and the total value of a household s disaster losses. The indicators used to asses M-RED s impact on disaster risk are given in Table 4 below. Since not all indicators were collected at both baseline and endline, the timing of each indicator is also shown: Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 17

18 Topic Table 4: Disaster Risk Reduction Outcome Indicators Indicator Loss of land or agricultural inputs to disaster over previous year Availability of Data (Baseline/Endline/Both) Both Loss of Property to Disasters Use of Adverse Coping Strategies Loss of property or assets to disaster over previous year Total value of losses to disaster Value of losses to last disaster Weighted coping strategies index (CSI 3 ) addressing strategies used to maintain food supply at the cost of long-term well-being. Compared between treatment and comparison households with similar levels of disaster impact Endline Endline Both Endline ECONOMIC IMPACT We examine several indicators to understand how M-RED affected households economic behavior. These outputs represent how M-RED may have influenced economic status, but not a change in status itself. For example, if beneficiaries adopted new income sources, there is no guarantee of an overall increase in income since they may have abandoned their older sources. The specific ED pathways differ between the three program areas, so Table 5 contains ED outputs for Nepal s Terai only. These indicators focus on households growth of sugarcane in general, and sugarcane planted on marginalized land specifically. Table 5: Economic Development Output Indicators in Nepal Terai Topic Sugarcane Indicator Households grows sugarcane (any location) Household grows sugarcane on marginal land Area of sugarcane grown Area of sugarcane grown on marginal land Availability of Data (Baseline/Endline/Both) Both Both Both Both Table 6 lists the ED outputs used for Nepal s Hilly region. There, M-RED focused on fodder for cattle and the development of the dairy industry, and accordingly the indicators focus on both amount of fodder planted in marginal areas and the total value of dairy and milk production. 3 Caldwell, R and Daniel Maxwell The Coping Strategies Index Field Methods Manual Second Edition. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 18

19 Table 6: Economic Development Output Indicators in Nepal Hilly Region Topic Fodder and Dairy Indicator Household grows fodder on marginal land Number of fodder grass slips planted on hillsides Value of dairy production Value of milk production Availability of Data (Baseline/Endline/Both) Both Endline Endline Both In Timor-Leste, M-RED drew on a variety of nexus interventions including legumes, agro-forestry, and banana plantations. We do not have sufficient data on the many pathways through which these could affect economic wellbeing, so in our analysis of M-RED s effect on ED in Timor-Leste we focus on outcomes only (described below). We next look at the program s ED outcomes, in the form of household economic status. There are two components to these outcomes: growth and stability. For the purposes of this study, we examine growth as the overall change in outcomes among the treatment group compared to the change in the comparison group. A positive change for beneficiaries, greater than the change among comparison households, would indicate that M- RED improved the overall economic status of targeted households over its three years of implementation. We also measure M-RED s effect on economic stability by comparing the outcomes for beneficiary households against others who experienced a similar level of disaster impact over the course of the program. If the program helped households to maintain their economic status in spite of being affected by disasters, then when we control for disaster impact we would expect higher levels of the outcome indicators among the treatment. The selected outcome indicators include both high- and low-sensitivity measures of economic status and food security. To measure economic growth, two sensitive measures are used. The first is the Household Dietary Diversity Index (HDDI), which emphasizes food access and the household s ability to afford a quality and varied diet a common proxy for general well-being. The second sensitive measure is reported monthly expenditure, which is a superior measurement of economic well-being to income itself 4. We use the log of the expenditure to account for outliers. The least sensitive measure is the likelihood of living under the poverty line, as predicted by the Progress out of Poverty Index (PPI). The PPI consists of 10 basic questions on household composition, living conditions, and asset ownership, which are then able to predict the likelihood of poverty using the Grameen Foundation s algorithm. A major advantage of the PPI is its ease of use: the questions are extremely straightforward and can be answered reliably by any household member. However, as it relies on asset ownership and living conditions, it is less sensitive than other measures. For example, even households whose income has increased over the previous year may not have purchased a new radio or refurbished their home. A secondary indicator of economic growth is also used: household income. This is the component of economic status most directly affected by the nexus interventions, but we consider it secondary because it is also the least reliable question to collect from a household. Many may be hesitant to report their full income, either out of concern that they will no longer benefit from the program, or simply from a common hesitancy to fully disclose one s wealth. Even those who are willing to share may not know exactly how much they earned. Income is reported over the previous month, which is easier to recall, and over an average month to account for seasonal variation. 4 World Bank Website, Defining Welfare Measures (accessed May 2016) Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 19

20 To measure economic stability, we use several of the most sensitive indicators while controlling for the impacts of disaster on a household. These include the HDDI and reported income over the previous month, which we would expect to change quickly for disaster-affected families. Two other measures are also added to better define economic stability. The first is the Coping Strategies Index (CSI), which measures strategies used by a household to maintain food security. These strategies are considered adverse, meaning the household makes a long-term well-being sacrifice by using them. For example, they may reduce portion size to children or women, or remove children from school to use as labor. The second added measure is simply the household s report of being cut-off from an IGA by a disaster. The indicators used to asses M-RED s impact on economic development are given in Table 7 below: Table 7: Economic Development Outcome Indicators Topic Primary Indicators of Economic Growth Secondary Indicator of Economic Growth Primary Indicators of Economic Stability Secondary Indicators of Economic Stability Indicator Average expenditure (log value) HDDI 5 Likelihood of living under the national poverty line (calculated through PPI 6 ) Income over previous month (log value) Income in an average month (log value) HDDI, comparing between households with similar shock experience Household cut off from income-generating activity by disaster, compared between treatment and comparison households with similar levels of disaster impact Weighted coping strategies index (CSI 7 ) addressing strategies used to maintain food supply at the cost of long-term well-being. Compared between treatment and comparison households with similar levels of disaster impact Income from previous month, comparing between households with similar shock experience in presence of shock (log) Availability of Data (Baseline/Endline/Both) Both Endline Both Endline Both Endline Endline Endline Endline OTHER RESILIENCE CAPACITIES M-RED also targeted other resilience capacities beyond its DRR and ED components. These are measured using indicators for stronger market linkages, greater use of financial instruments, and enhanced social capital within communities and between communities and the government. These capacities are functionally similar to the outputs in the DRR and ED categories they show the direct effects of the program, but not the final impact on households well-being. We generally expect that the development of positive capacities allows households, when 5 Bilinsky, P and Anne Swindale Household Dietary Diversity Score (HDDS) for Measurement of Household Food Access: Indicator Guide Version 2 6 Created by the Grameen Foundation. Website: 7 Caldwell, R and Daniel Maxwell The Coping Strategies Index Field Methods Manual Second Edition. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 20

21 impacted by disasters, to respond in a way that maintains the stability of their income sources and food access without negative effects over the long-term. Table 8: Resilience Capacity and Outcome Indicators Topic Indicator Availability of Data (Baseline/Endline) Market Access Able to buy desired agricultural inputs Endline Financial Inclusion Social Capital Migration Loans Formal loans Micro-insurance (Agriculture/livestock) Savings Formal savings Bonding: Trust Bonding: Can rely on others for assistance Linking: Connection to local and national government Number of HH members migrating for work Duration of migration (months) Both Both Both Both Endline Endline Endline Endline Endline Endline 3.5 Estimating Household Impact This study focuses on M-RED s effects on beneficiary households, as the fundamental purpose of the program was to improve the well-being of marginalized and vulnerable individuals. Improved market linkages and enhanced local government capacity were ultimately means to that end, rather than objectives in their own right. For this reason, the primary estimation of program impact is conducted at the household level, using the estimation strategy described below. However, there are numerous reasons why activities conducted at the community- or market-level may not lead to improved household outcomes, and throughout the analysis further context is provided based on qualitative and community-level data collection. Pre-Implementation Matching M-RED was targeted at specific communities which met certain criteria: a high degree of risk and history of damage from natural disasters, and limited economic opportunities. Randomized assignment, although a best practice in reducing bias, was not feasible for a DRR program that would only benefit communities with specific characteristics. To limit bias as much as possible, communities selected for treatment were matched with comparison communities that exhibited similar levels of disaster risk and economic development. This comparison group was excluded from the program because of limited resources not because they didn t meet M-RED s selection criteria. The criteria used for selection are provided in Appendix A. This process did not exactly replicate randomization, since there may have been undocumented differences between the two groups that prompted M-RED staff to choose some communities rather than others. However, it did ensure that the types of communities included in our treatment and comparison groups are generally similar. We further built upon this technique using propensity-score matching, discussed below. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 21

22 Controlling for General Time Trends During the period of the program, contextual changes may have affected both groups. For example, as the government constructs roads, communities throughout the region may have had better access to markets and been able to sell their produce for higher prices, or pay less for fertilizer and seeds. Examining the treatment group alone would overestimate the program benefits. Alternatively, severe weather may have impacted the region, making the treatment group worse off compared to their starting point. To account for these larger trends, a difference-in-difference approach is used when baseline data are available, estimating the change for the treatment group held against the change for the comparison group. Potential Endogeneity M-RED treatment communities were not selected randomly, and a naïve comparison of outcomes for treatment and comparison households may be biased by endogeneity, particularly variables which are related both to selection into M-RED and the various outcomes. Like all DRR programs, M-RED specifically identified hazardprone communities which would benefit from risk mitigation. If the treatment communities were substantially more vulnerable to disasters, or less developed economically, than the comparison groups, this could be due to unobservable variables that cannot be controlled for when estimating the results. Alternatively, the program may have focused on communities with cohesive, well-functioning leadership in the belief that activities would be more effective and sustainable than in lower-functioning groups. Households who joined the program may also have been either more vulnerable, or more entrepreneurial and motivated, than their counterparts. For these reasons, a simple comparison of the two groups at endline would provide a biased estimate of the program s impact. Propensity-Score Matching To account for this bias, a propensity-score matching approach is used to create a valid comparison group for the treatment households 8. The likelihood of receiving treatment (propensity-score) is calculated for each household, based upon factors which influence both the probability of involvement in the program and the relevant outcomes. This strategy estimates the causal effect of M-RED by comparing the outcomes of households who benefited from program activities against those of households with similar propensity-scores who did not benefit from the program. These two groups should then be comparable. Propensity-score matching also has the advantage of accounting for a large number of confounding variables without creating a dimensionality problem. The propensity for treatment is calculated separately for each of the three contexts: Nepal Terai, Nepal Hilly, and Timor-Leste. The exact prediction model varies depending upon the context, and on the data available for each sample, but included disaster risk and livelihood status prior to the program, as well as time-invariant household demographic variables. The results of the matching for each context are presented in Appendix B. Controlling for Disasters We face a dilemma in controlling for the effects of disasters on other outcomes. It is clear that a disaster s strength will affect all of our chosen outcomes for example, a household which was struck by a heavy flood is more likely to lose their property, and to have reduced income and expenditure. We would normally account for this by controlling for disaster-affectedness, giving a result in the form of M-RED reduced the probability of losing property for households that were equally affected by disasters. However, we also think M-RED will make households less likely to be affected by disasters at all. A community sheltered by a gabion wall or sugarcane plantation is less likely to have floods or river-cutting. If we re correct and disaster-affectedness is itself a result of the program, then by controlling for it we would underestimate M-RED s effect. Our solution is to run two different analyses, one with and one without disaster controls. The former gives the average result across all households, which we use as the estimate of growth in well-being outcomes. The latter looks at M-RED s effects specifically for households that were both equally affected by disasters, which we interpret as a household s ability to maintain stable well-being outcomes despite the disaster. Functional Forms 8 Rosenbaum, Paul R. and Donald B. Rubin "The Central Role of the Propensity Score in Observational Studies for Causal Effects". Biometrika 70 (1): Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 22

23 We use the functional forms below to estimate M-RED s effect on treatment communities relative to the comparison communities. In all of these forms, we use ordinary least squares regression for continuous variables such as income and expenditure, and logit models for binary outcomes like losing land to a disaster. We also cluster standard errors by community, because we believe that individuals within a community will be similarly affected by disasters and market forces. F1: Endline-Only Cross-Section: ϒ i = β 0 + β 1 Τreatment + µ it This estimate is inversely weighted by the propensity score to estimate the Average Treatment Effect: W ATE =!!! for the treatment group, and W ATE =! (!!!! ) for the comparison group. The cross-sectional endline-only estimate is considered the best possible estimate for outcomes measured at the endline only, since it benefits from the propensity-score s bias reduction but does not account for baseline values and time trends. This estimate has the same form for both the repeated cross-sectional sample and the panel sample. F2: Cross-Sectional Difference-in-Difference: ϒ it = β 0 + β 1 Τreatment it + β 2 Endline it + β 3 Endline Treatment it + ε it This estimate also uses propensity score weighting for the Average Treatment Effect, as in F1. The difference-in-difference estimate is considered the best possible estimate for outcomes measured at both baseline and endline. It also benefits from the propensity-score s bias reduction, while accounting for general time trends. This estimate is only used for the repeated cross-sectional analysis. F3: Matched Panel Controlling for Baseline: ϒ it = β 0 + β 1 Τreatment i + β 2 ϒ it-1 + ε it This third form controls for each household s baseline value of the outcome variable, which is considered to provide the most precise estimate and increase statistical power. This form is preferred because it allows us to identify a smaller treatment effect than difference-in-difference so even if the treatment causes a small change we are better able to confirm it. This estimate also uses propensity score weighting for the Average Treatment Effect, as in F1 and F Cost-Benefit Analysis Emphasis on Sugarcane Nexus Activity We conduct a cost-benefit analysis (CBA) for the sugarcane nexus only. Among the three approaches tested in Nepal and Timor-Leste, the sugarcane intervention was believed to have the strongest evidence of dual benefit for DRR and ED. It is also easier to measure the beneficial mitigation effects of sugarcane planting in vulnerable lands are discrete and clearly visible. Communities can distinctly identify areas washed away through river-cutting, or silty areas recovered from the river, and track their production and sales each year. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 23

24 Figure 6: Aerial View of River-Cutting Destroying Arable Land in the Nepal Terai We believe that beneficiary communities in the Terai region have a clearer calculation to make as they consider whether or not to continue their nexus activities once outside aid ceases and they assume the full costs. In contrast, the other nexus activities had more roundabout connections to DRR and ED. The fodder, planted on hillsides in the Hilly region of Nepal, directly reduced the risk of landslides, but did not directly lead to income rather, it provided feed for dairy cows. Vegetable and fruit plantings in Timor-Leste, on the other hand, were primarily targeted at slower-paced soil degradation and erosion, meaning the immediate DRR benefits might not be internalized by community members. To evaluate the economic viability of the sugarcane nexus intervention, a separate case study was conducted, applying a CBA that treats the sugarcane plantations as an investment. The full results are available in the report Managing Risks through Economic Development (M-RED): A cost benefit analysis of the sugarcane nexus intervention introduced in the Terai region of Far West Nepal (Mercy Corps, May 2016). The CBA is structured according to the following research questions: A. Over a three-year period, did income from the sugarcane nexus exceed the marginal costs? B. Did communities where the nexus was profitable demonstrate different characteristics than those where it was not? C. Over a three-year period, did the total benefits of the sugarcane nexus activity exceed the total costs? How does the return compare to the fixed investment of structural mitigation? To address these questions, the following techniques are used to assess profit and net benefit at the community level: QUESTION A: OVER A THREE-YEAR PERIOD, DID INCOME FROM THE SUGARCANE NEXUS EXCEED THE MARGINAL COSTS? Looking at net profit through a purely financial lens, the income from sale of sugarcane is compared against the costs of planting, harvesting, and other agricultural inputs. Sugarcane functions on a three-year cycle, with greater inputs required in the first year and relatively low marginal costs afterwards. The three-year profit is calculated for each community with the following formula: 3 Year Profit = Year 1 Income Year 1 Costs + (Year 2 Income Year 2 Costs) + (Year 3 Income Year 3 Costs) New plantations were created in 2014 and 2015, with the 2014 plantation being harvested twice, in 2015 and 2016, while the 2015 plantation was only harvested in The profitability of the two plantations are expected to differ due to changes in communities technical capacity as well as environmental factors, and so are estimated separately. Since even the 2014 plantation has only gone through two harvests, income for the third harvest of the three-year cycle is estimated based on the remaining area of active plantation and the productivity of the previous years. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 24

25 Figure 7: Sugarcane Nexus - Contribution of Plantings to Harvest QUESTION B: DID COMMUNITIES WHERE THE NEXUS WAS PROFITABLE DEMONSTRATE DIFFERENT CHARACTERISTICS THAN THOSE WHERE IT WAS NOT? Since communities were not consistently profitable, the case study looked for a clear pattern as to which communities had experience a profit, or were expected to profit, within the three-year cycle. This ad-hoc analysis examines characteristics of the 21 communities which had sugarcane plantations, comparing characteristics between those which were and were not profitable. These characteristics include geographic factors like proximity to service providers, mills, and roads, and community characteristics like social cohesion and previous disaster experience. This same basic analysis is conducted for productivity and reduction in productive plantation area as well. QUESTION C: OVER A THREE-YEAR PERIOD, DID THE TOTAL BENEFITS OF THE SUGARCANE NEXUS ACTIVITY EXCEED THE TOTAL COSTS? HOW DOES THE RETURN COMPARE TO THE FIXED INVESTMENT OF STRUCTURAL MITIGATION? The nexus interventions were designed to mitigate disaster risk and protect community land, in addition to their economic benefits. The case study uses another estimate to account for the added benefit of protected land, valuing the land at its approximate rental rate: Total Value of Land = Rental Price Units of Land Duration Protected The value of land is added to the value of income earned, to assess the net benefit of the plantation against its costs. This analysis is conducted for each community. Lastly, to further understand the viability of sugarcane nexus plantations as a private venture, the net benefit is compared against the cost of structural mitigation required to begin the plantation in each community. Although the mitigation works have been provided by government and non-profit actors, this estimate of the time needed to pay off a theoretical upfront investment is still useful in assessing whether communities may adopt the nexus approach independently in the future. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 25

26 3.7 Sampling and Data Quality Quantitative Sampling Household data were collected at the beginning of the nexus activities, in December 2013, and again at the program s end in December 2015 and January Simple random sampling was used to collect an unbiased representation. In treatment communities, respondents were drawn from a sampling frame of all households participating in the nexus activities. This was not possible in comparison communities, where no nexus activities took place. Instead a similar list was created by community leaders that included all households who were economically vulnerable and at risk to disasters. We tried to follow the same respondents from baseline to endline, and in the two Nepal contexts we largely succeeded. In Timor-Leste we found that many of those surveyed at baseline did not actually go on to participate in the nexus activities. This meant that we had little data on the actual treatment group from baseline, so instead of following the respondents we drew a new sample at the endline and limited our analysis to endline only. Table 9: Quantitative Sample Size and Attrition9 below shows the number of communities and households surveyed in each of the three contexts. Table 9: Quantitative Sample Size and Attrition Context Baseline Households Endline Households Attrition Rate Nepal Terai % Nepal Hilly % Timor-Leste Discarded 497 N/A Qualitative Data Collection We also drew on qualitative discussions with community groups and key stakeholders to better understand the broader context and how M-RED affected communities. Our goal for this piece was to speak to well-informed people who could shed light on our questions, so we used deliberate rather than representative sampling. Table 10: Qualitative Sampling10 below shows the groups spoken to in each context. Table 10: Qualitative Sampling Context DMC Farmers Group Government Actors Market Actors Non-Treatment Replication Groups Nepal Terai Nepal Hilly Timor-Leste Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 26

27 4.0 RESULTS The following sections present the findings from each of the three implementation regions. Within each of the regions, the program s effects are divided into DRR, ED, and other resilience capacities in accordance with M- RED s Theory of Change. For each indicator the results of the most precise and least biased models are reported. For indicators available at both baseline and endline, this is represented by panel analysis weighted by propensity score, controlling for baseline values. For indicators only collected at endline, cross-sectional analysis of panel data is weighted by propensity-score 9. Three types of coefficients are presented: Level coefficients, used for continuous variables such as area of land cropped, or number of household members migrating. These coefficients have a direct interpretation, meaning a treatment coefficient of 1.7 household members migrating over the past year indicates that the treatment s causal effect was 1.7 additional members migrating. Odds-ratio coefficients are used for binary variables, where the outcome is either 0 or 1. An odds ratio can be interpreted as the treatment group was X times as likely to do something. For example, a coefficient for the use of formal savings of 3 would indicate that the treatment made beneficiaries 3 times as likely to use formal savings. A coefficient less than 1 indicates a negative effect, for example 0.5 would indicate that treatment made beneficiaries only half as likely to use formal savings. Log coefficients indicate a log outcome variable. Directionality and statistical significance are straightforward, but the exact interpretation requires further calculation. Specifically, when the outcome variable is in log form, the difference in the actual level of the outcome variable between treatment and comparison groups is equal to: % y = 100 (e!"#$%&#'% 1) Statistical significance is indicated by stars, with * meaning significant at the 5% level, ** at the 1% level, and *** at the 0.1% level. We consider findings significant at 5% or below to be statistically significant, in line with standard convention. However, we also report findings which approach the 5% level and have an economically large effect, in line with a growing thought that using the 5% level as a binary threshold is arbitrary and may lead to a distorted interpretation of results Other models used include unweighted cross-sectional models and unweighted panel analysis controlling for baseline levels. However, these are considered more biased than the models that use propensity-score matching. For this reason they are used only as a robustness check. 10 Wasserstein, Ronald L., and Nicole A. Lazer The ASA s Statement on p-values: context, process, and purpose. American Statistical Association. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 27

28 4.1 Nepal Terai: Sugarcane Intervention Purpose and Activities Nepal s lowland Terai region is a fertile agricultural center, but each year s monsoons also cause heavy floods and shift the course of rivers until they eat into productive fields. M-RED sought a sustainable solution in the form of sugarcane plantations along the riverbanks and even in reclaimed riverbeds. This nexus intervention would shelter communities and their fields from the river, while at the same time providing additional income from the sugarcane Figure 8: Bamboo fencing reclaims riverbanks sugarcane is planted harvest. This dual benefit is critical to the project s sustainability, as communities have an additional economic incentive to protect themselves from disasters. In the Terai the sugarcane plantation was managed by communal farmers groups, with income going back to the group rather than individual households. The majority of this income was reinvested to expand the sugarcane plantations, and several groups created a system of small loans for community members to pay healthcare costs. In this study, we test whether community members see both protection from disasters and improved economic status as a result of the program. If they realize these dual benefits, we believe the project is uniquely sustainable because communities are essentially paid to continue protecting themselves from disasters. Because the sugarcane harvest was managed at the community level, we also look at the group level to see how the benefits of the nexus activity compare against its costs. Outside of the nexus, M-RED built general disaster management capacity by training community members, forming Disaster Management Committees (DMCs), and constructing mitigation works. The program also connected communities to local input suppliers and trainers called agro-vets, and built links to sugar and molasses mills as purchasers of the sugarcane. Figure 10 reiterates the Theory of Change specifically for the Terai context and presents a brief summary of our findings. Areas where M-RED achieved the desired results are highlighted in green, and areas where we found no effect are colored grey. Overall, M-RED bolstered DRR capabilities, led to more plantations of sugarcane on marginal land, and built other resilience capacities among the treatment communities. However, these outputs didn t translate into reduced disaster impacts, increased economic stability, or economic growth. There are two reasons for this disconnect: first, during M- RED s implementation the disaster profile of the Terai shifted from flooding and river-cutting to drought, Figure 9: Weather Trends in Nepal rendering M-RED s flood Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 28

29 prevention measures less useful. Figure 9 shows the program team s report of weather trends prior to and during the project, which matches reports from community discussions. Second, the sugarcane nexus activities were managed at the community level, and the majority of the income earned was reinvested into sugarcane or used for group projects. Within the program period, individual households did not receive any significant income from the activities. These findings are expanded on after the Theory of Change. Figure 10: M-RED Terai - Theory of Change Program Activities Outputs Outcomes Sustainability DRR Outputs: Mitigation Capacity Building Early Warning Systems Disaster Response DRR Outcomes: Sugarcane Disaster management training Links to mills and agro-vets Other Capacities: Social Capital Use of Financial Inputs Migration Loss of land and property Coping strategies ED Outcomes: Economic stability Communities perceive benefits of activities to outweigh costs ED Outputs: Economic growth Planting on Marginal Land Production Results Disaster Risk Reduction We first look at how the program impacted community members preparations for disasters. M-RED had a strong effect on protection techniques households were almost 10 times as likely to protect their fields with structural mitigation, and also much more likely to use bio-engineering and advanced planting techniques. The program also increased the prevalence and activity of DMC s and spread the use of early warning systems in treatment areas. As a result, households both knew their communities disaster response plans and had their own household plans to deal with a disaster. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 29

30 Table 11: Terai DRR Output Results Topic Indicator Result Mitigation Techniques Use structural mitigation to protect land Use bioengineering to protect land 9.487*** (3.55) 5.191*** (3.41) Statistical Significance M-RED beneficiaries vs. comparison households 9.5 times as likely as comparison households to protect their fields with structural mitigation (gabion walls, fences) at endline 5.2 times as likely as comparison households to protect their fields with bio-engineering (sugarcane plantations) at endline Use improved agricultural techniques to protect land 3.263** (2.60) 3.3 times as likely as comparison households to protect their fields by using planting techniques like intercropping at endline DMC Activity DMC active 523.0*** (7.54) Frequency of DMC meetings in past year 13.82*** (10.31) Vastly more likely than comparison households to say their community had an active DMC at endline Reported their DMC s meeting 14 more times per year than comparison households at endline Community has disaster response plan 125.6*** (5.32) Much more likely than comparison households to say their community had a disaster response plan at endline Disaster Response Community has EWS HH knows how to respond in disaster 393.1*** (5.47) 136.4*** (10.53) Vastly more likely than comparison households to say their community had an early warning system at endline 136 times as likely as comparison households to say their family knew how to respond in a disaster at endline HH has formal response plan 107.1*** (8.76) 107 times as likely as comparison households to say their family had a formal plan to respond to a disaster at endline Although the program clearly bolstered communities protections from natural disasters, our analysis doesn t show a clear link to DRR outcomes. At the endline, treatment households were no less likely to have lost land or property to a disaster than the comparison group, and the value of their losses was not statistically different. This is not surprising, since the Terai experienced a sharp drought during the program period. Its activities were targeted at floods and river-cutting, and the protections it created were not meant to mitigate the impact of a drought. For this reason, the treatment group was also no less likely to have been cut off from an IGA by a disaster than the comparison group. However, treatment households did report a slightly higher use of adverse coping strategies to maintain their food supply (such as cutting portions or feeding males first) relative to comparison households with a similar level of disaster impact. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 30

31 Table 12: Terai DRR Outcome Results Topic Indicator Result Lost land or agricultural inputs to disaster over previous year (-0.10) Statistical Significance Not M-RED beneficiaries vs. comparison households Loss of Property to Disasters Lost property or assets to disaster over previous year Total value of losses to disaster (log) (-0.52) (-0.16) Not Not Value of losses to last disaster (log) (1.15) Not Economic Development We next examine M-RED s effect on economic development, the second branch of the program. We expect the subsidies for sugarcane and the connections to agro-vets and mills to make participants more productive farmers and to increase the size of their sugarcane plantations. Through increased agricultural production, and by protecting their other fields and assets from disasters, M-RED would then lead to an overall increase in economic well-being. The analysis shows that, in the Terai, treatment households did slightly increase their area of sugarcane cultivation relative to the comparison group, from baseline to endline. The program also increased growth of sugarcane specifically on marginalized riverside land by its beneficiaries. Farmers also reported turning to agro-vets for technical advice and agricultural inputs, reflected in the use of improved farming techniques. The analysis does not prove that a greater portion of the treatment group actually planted sugarcane, but the household reports we use may also underestimate M-RED s effect since many plantations were run at the community level. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 31

32 Table 13: Terai Economic Development Output Results Topic Indicator Result Statistical Significance Interpretation: M-RED beneficiaries Household begins to grow sugarcane (any location) (1.60) Not Household begins to grow sugarcane on marginal land (1.72) Nearing Significance Were 3.5 times as likely to grow sugarcane on marginal riverside land, relative to comparison households at endline Sugarcane Change in area of sugarcane grown (1.69) Nearing Significance Grew 18 more katha (0.9 bigha) of sugarcane at endline, relative to comparison households with similar-sized sugarcane plots at baseline Change in area of sugarcane on marginal land 73.48* (2.28) Grew 73 more katha of sugarcane (3.65 bigha) relative to comparison households at endline We would expect the increase in planting of sugarcane caused by M-RED to translate into increased income and overall economic stability and growth. However, within the two-year timeframe since the beginning of the plantations, we do not yet see conclusive results. There was no clear growth in expenditure or dietary diversity both strong indicators of economic status. Similarly, there was no change in the likelihood of living under the national poverty line as calculated through the PPI. This is not surprising, given the low sensitivity of the PPI s asset indices in the short-term. Likewise, treatment households did not have higher dietary diversity or income compared to other households equally affected by disasters meaning they were not more economically stable. As discussed, these results may not present the full extent of the program s effect since income from the communal plantations was managed at the group level but we have measured individual level effects. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 32

33 Table 14: Terai Economic Development Outcome Results Topic Indicator Result Statistical Significance Interpretation: M-RED beneficiaries Average expenditure (log value) (0.56) Not Primary Indicators of Economic Growth HDDI (-0.19) Likelihood of living under the national poverty line (calculated through PPI) (-1.31) Not Not Secondary Indicators of Economic Growth Income over previous month (log value) Income in an average month (log value) (-0.62) (-1.09) Not Not HDDI in presence of shock (0.01) Not Primary Indicators of Economic Stability Cut off from IGA in presence of shock CSI in presence of shock (0.36) (1.96) Not Nearing Significance Used more adverse coping strategies at endline, relative to comparison households with similar levels of disaster impact Secondary Indicator of Economic Stability Income from previous month in presence of shock (log) (-0.50) Not To better understand these group-level effects, we conducted a cost-benefit analysis (CBA) of the sugarcane activities among Terai communities. This CBA is another way of looking at the question of sustainability did the sugarcane s combined benefits of income and disaster protection exceed the costs of planting and harvesting? If so, the nexus approach is likely to continue even without project funds. We see that across the 21 communities, total sale of sugarcane averaged NRP 283,500 (approximately $2700 USD) in 2015 and NPR 443,600 ($4225 USD) in Less than half of the communities earned more income than was spent on inputs and labor. Many had lower production than anticipated as the farmers learned a new agricultural technique, and a few villages stopped tending sections of the original plantations. However, direct income was only one of the benefits of the nexus intervention it was also intended to protect communities from Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 33

34 disaster effects. When we factor in the value of land protected by the plantations, the large majority of them experienced a net benefit. For 19 of the 21 communities the value of the sugarcane harvest combined with the rental value of the land protected exceeded the cost of inputs and harvesting. Other Resilience Capacities This study examines other program effects beyond direct measures of disaster risk reduction and economic development. While this was not a formal goal of the program, its activities were hypothesized to build other capacities which would help households cope with shocks and stresses without falling into a cycle of decreasing well-being. We tested for changes in social capital, financial inclusion, and market linkages. The disaster risk reduction activities discussed previously also fall within the resilience umbrella, particularly those aimed at building skills and capacity. M-RED clearly increased social cohesion within the Terai communities, likely as a result of the communal cooperation required to manage the plantations. It also built connections to both government actors and private sector agro-vets and sugar processors. In focus groups, farmers reported that sugar mills were offering higher prices for sugarcane, making it more worthwhile for them to invest. They also purchased inputs and learned new skills from agro-vets, as well as using them to access the M-RED-designed micro-insurance scheme created by the government. Although there are reports of communities now taking loans directly from sugar mills, and forming their own loan groups internally, this was not reflected in the household survey. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 34

35 Table 15: Terai Other Resilience Capacity Results Topic Indicator Result Statistical Significance Interpretation: M-RED beneficiaries Market Access Able to buy desired agricultural inputs (-0.54) Not Financial Inclusion Social Capital Migration Loans (0.17) Formal loans 0.255** (-2.88) Micro-insurance (Agriculture/livestock) (1.87) Savings (-0.83) Formal savings (-0.57) Bonding: Trust (1.35) Bonding: Can rely on others for assistance Linking: Connection to local and national government Number of HH members migrating for work 0.272** (2.78) 0.600** (3.03) (-0.86) Not Nearing Significance Not Not Not Not Were 1/4 as likely to take formal loans, relative to the comparison group, from baseline to endline Were times as likely to have micro-insurance relative to the comparison group, at endline Were more confident than the comparison group in being able to rely on community members for assistance (0.3 points on a 1-5-point scale), at endline Were more confident than the comparison group in being able to influence government departments (0.6 points on a 1-5-point scale), at endline Duration of migration (months) (-0.78) Not Impact and Discussion of Results M-RED successfully increased communities preparation for disasters across several fronts. The program built community response capacity in the form of DMC s, early warning systems, and household-level plans and preparations. M-RED also increased physical mitigation measures intended to reduce damage from the acute shocks of flooding and river-cutting reported widely at the program s inception. In the economic realm, it increased sugarcane growth and brought together a broad-spanning network of actors including farmer groups, government, and private sector actors. With their incentives aligned, communities can expect to receive support including loans, seeds, and other inputs from mills and the DADO as they continue to expand their plantations. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 35

36 However, within the three-year duration of the program (and two-year duration of the sugarcane) it was difficult to see broader impacts on households themselves. In a period of drought with little flooding or river-cutting, the program s DRR and resilience capacities didn t translate into lower disaster losses for the treatment. Likewise, there was not a clear improvement in either economic growth or economic stability among the beneficiary households. This is not surprising, since most communities directly reinvested their earnings back into sugarcane, but even at the community level only half of the groups have experienced a net profit to date. These more successful groups largely had easy access to agro-vets and mills, and a higher degree of social cohesion. They prove that while the plantations can be profitable, it may take some time before all farmers develop their expertise and productivity needed to realize boosts to income. This null result within the first two years of a program falls in line with the experience of many other market development and vocational programs, where no major results were found until after the two-year mark 11. A USAID meta-review found that in many cases, programs exclusively targeting market development have actually decided not to conduct an evaluation until a longer period has elapsed. 12 Overall, this initial trial has not given us a sufficient opportunity to confirm whether M-RED s nexus activities can provide a sustainable and income-earning source of protection. Communities have not suffered an economic loss while starting to adopt them, which is an achievement on its own, and over a longer time period we expect that they will begin to earn a profit. M-RED has also prepared communities to mitigate the effect of river-based disasters, but this hasn t yet paid off during the recent drought. Beyond this, the increased linkages between communities, the government, and the private sector have aligned all three groups incentives to continue with the nexus activities into the future. Further studies may be able to track the nexus communities to assess their profitability and sustainability over a longer time-period. 11 Card, David, et. al. July What Works? A Meta-Analysis of Recent Active Labor Market Program Evaluations. IZA DP, No USAID, May Assessing the Effectiveness of Economic Growth Programs. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 36

37 4.2 Nepal Hilly Region: Fodder Intervention Purpose and Activities Fodder Nexus Activity The Hilly region of Nepal is characterized by rugged and often fragile topography. Road access is limited in many areas, and often made worse by frequent landslides. Community discussions in the planning phase identified Figure 11: Improved Breed Dairy Cows in the Nepal Hilly Region landslides as the most significant threat, though flooding and drought were also mentioned as notable concerns. M- RED s solution in this case took two stages. First, households were encouraged to grow fodder grass on fragile hillsides to prevent landslides. The program then developed the region s dairy industry, where farmers used the fodder to increase the fat content and quantity of their cows milk. This created an economic incentive to continue the nexus activity, albeit an incentive one step removed from the nexus activity itself. We test whether community members experienced the dual benefits of the program: reduced disaster loss and enhanced economic growth and stability. As with the sugarcane intervention in the Terai, this would promise a chance for long-term sustainability by providing beneficiaries with a source of disaster protection that also earned them income, rather than costing them. Even if the total level of income did not increase, a more stable income source that stayed constant in spite of disasters would still be preferable to a fragile one. Our analysis also includes M-RED s other activities: general disaster management training for DMC s and community members, and constructing mitigation works. To build the dairy industry, the program also connected communities with agro-vets for technical training and inputs, milk producers as purchasers, and provided subsidies for improved-breed highproductivity milk cows. Figure 12 below reiterates the Theory of Change specifically for the Hilly context and presents a brief summary of our findings. Areas where M-RED achieved the desired results are highlighted in red, and areas where we found no effect are colored grey. Overall, M-RED bolstered DRR capabilities, led households to plant fodder on hillsides, and increased milk production. These outputs led to the program beneficiaries experiencing fewer losses to disasters, but had no effect on economic growth or stability. The fodder intervention did not enhance communities other resilience capacities, unlike the communally-run sugarcane in the Terai. Overall, the analysis shows that the fodder nexus intervention is likely to be sustainable, since communities have experienced a clear reduction in disaster impact without presenting an economic loss to participating households. These findings are expanded on after the Theory of Change. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 37

38 Figure 12: M-RED Nepal Hilly Region - Theory of Change Program Activities Outputs Outcomes Sustainability DRR Outputs: Mitigation Capacity Building Early Warning Systems Fodder Improved-breed cows Disaster management training Links to milk processors and agro-vets Disaster Response Other Capacities: Social Capital Use of Financial Instruments Migration ED Outputs: DRR Outcomes: Loss of land and property Coping strategies ED Outcomes: Economic stability Economic growth Communities perceive benefits of activities to outweigh costs Planting fodder on Marginal Land Milk Production Results Disaster Risk Reduction M-RED had a strong effect on the DRR outputs treatment households were 3.4 times as likely to have structural mitigation protecting their land as the comparison group. They were also 3.1 times as likely to use bio-engineering, such as fodder plantations, and 3.7 times as likely to use advanced planting techniques to protect their fields. The program also increased the prevalence and activity of DMCs and spread the use of early warning systems in treatment areas, though this did not spread to include disaster response plans. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 38

39 Table 16: Hilly DRR Output Results Topic Indicator Result Mitigation Techniques DMC Activity Use structural mitigation to protect land Use bioengineering to protect land Use improved agricultural techniques to protect land 3.407** (3.01) 3.096** (2.64) 3.728* (2.33) DMC active 317.2*** (7.39) Frequency of DMC meetings in past year Community has disaster response plan 13.74*** (10.39) (0.39) Statistical Significance Not Interpretation: M-RED beneficiaries 3.4 times as likely as comparison households to protect their fields with structural mitigation (gabion walls, fences), at endline 3.1 times as likely as comparison households to protect their fields with bioengineering (fodder or other plantations), at endline 3.7 times as likely as comparison households to protect their fields by using planting techniques like intercropping, at endline Vastly more likely than comparison households to say their community had an active DMC at endline Reported their DMC s meeting 14 more times per year than comparison households, at endline Disaster Response Community has EWS HH knows how to respond in disaster 12.34*** (4.69) (-0.07) Not 12 times as likely as comparison households to say their community had an early warning system at endline HH has formal response plan (-1.46) Not We would expect these enhanced DRR outputs to translate into lower disaster losses for M-RED beneficiaries in the Hills. This connection was proven during our analysis: the program beneficiaries were only a third as likely as comparison households to lose land or agricultural inputs to a disaster, and the value of losses over the previous year was 80% less than in comparison communities. Clearly, in the Hilly context M-RED s DRR components protected communities, and greatly reduced their disaster losses. This matches our expectations; the fodder nexus was intended to reduce landslides as well as water-based hazards, and would have been applicable even in the presence of a drought. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 39

40 Table 17: Hilly DRR Outcome Results Topic Indicator Result Lost land or agricultural inputs to disaster over previous year 0.332* (-2.17) Statistical Significance Interpretation: M-RED beneficiaries 1/3 as likely to lose lands or agricultural inputs to a disaster over the previous year, relative to comparison groups with similar losses of land at baseline Loss of Property to Disasters Lost property or assets to disaster over previous year Total value of losses to disaster (log) (-0.78) (-1.99) Not Nearing Significance Had total disaster losses valued at 80% less than the comparison group, at endline Value of losses to last disaster (log) (-1.72) Nearing Significance Had disaster losses to the most recent disaster valued at 75% less at endline, relative to comparison households with similar disaster losses at baseline Economic Development M-RED s second branch is ED, where it is intended to cause both economic growth and stability. We expect that increased access to fodder, subsidies for improved dairy cows, and improved technical skills learned through agrovets would increase households earnings. Moreover, the program would open up more stable IGA s for beneficiaries, reducing the risk that their economic status would be hindered by disasters. First, we look at whether the program achieved its ED outputs. In the Hilly region, M-RED clearly increased the likelihood of planting on marginalized land, and the amount of fodder that farmers planted. This result is encouraging, as fodder was not intended to be a stand-alone income source, but households could use it to feed dairy cows while protecting themselves from landslides. Additionally, fodder, combined with subsidies for improved breed cows, increased the level of milk production among the treatment group. Beneficiary households produced an extra 544 NPR worth of milk each month, relative to the comparison group. There was no clear effect on the value of all dairy produced, which in this context is influenced by cream and ghee (clarified butter) production in several comparison villages. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 40

41 Table 18: Hilly Economic Development Output Results Topic Indicator Result Statistical Significance Interpretation: M-RED beneficiaries Household grows fodder on marginal land 4.644** (2.73) 4.6 times as likely to grow fodder on marginal land and hillsides at endline, relative to comparison households Fodder and Dairy Number of fodder grass slips planted on hillsides 167.1* (2.47) Planted 167 more slips of fodder on hillsides than comparison households at endline Value of All Dairy Production (-0.47) Not Value of Milk Production (1.76) Nearing Significance Produced 544 NPR more milk over the previous month, relative to comparison households with a similar level of milk production at baseline However, this increase in milk production didn t translate into increased economic growth or greater stability for the treatment group. As with the sugarcane intervention, there may be several reasons for this result. First, the development of the local dairy sector had been a slower process than that of sugarcane, and many communities only began purchasing improved-breed cows in the last year of the program. There were also fewer milk collectors in the beginning of the program, so the ability to actually transport their milk production to a buyer was inconsistent. Both of these factors changed by the endline, but we wouldn t expect to see any substantial change in economic well-being after a period of one year only. Since the fodder and dairy intervention was conducted at the household level, there was no opportunity for us to explore community-level profit in the Hilly region. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 41

42 Table 19: Hilly Economic Development Outcome Results Topic Indicator Result Statistical Significance Interpretation: M-RED beneficiaries Average expenditure (log value) (0.63) Not Primary Indicators of Economic Growth HDDI (0.29) Likelihood of living under the national poverty line (calculated through PPI) (0.18) Not Not Secondary Indicators of Economic Growth Income over previous month (log value) Income in an average month (log value) (0.51) (0.07) Not Not Primary Indicators of Economic Stability HDDI in presence of shock Cut off from IGA in presence of shock (0.45) (-0.42) Not Not CSI in presence of shock (-0.25) Not Secondary Indicator of Economic Stability Income from previous month in presence of shock (log) (0.79) Not Other Resilience Capacities This study examines other program effects beyond direct measures of DRR and ED. While this was not a formal goal of the program, its broad-reaching activities were hypothesized to build capacities which would allow households to cope with shocks and stresses without falling into a cycle of decreasing well-being. Like the Terai context, we tested for changes in social capital, financial inclusion, and market linkages. M-RED clearly increased use of micro-insurance, which was generally a required component of purchasing the improved-breed cows. Beyond this, it did not have a substantial effect on other components of financial inclusion, nor did it increase the treatment group s social capital. This is surprising, especially given qualitative reports that Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 42

43 women s participation in the dairy activities had also led to a greater presence and voice in both community activities and the household decision-making process. However, this study was not designed to test for changes in gender dynamics, and further research is recommended to better understand these effects. It is less surprising that, compared to the group-managed sugarcane in the Terai, the individually-managed dairy production in the Hilly region would not foster cohesion among the treatment communities. One interesting revelation was that M-RED households had slightly higher rates of migration for labor. This was not an explicit target of the program, in either direction, but could be researched further for future programming. Table 20: Hilly Resilience Capacity Results Topic Indicator Result Statistical Significance Interpretation: M-RED beneficiaries Market Access Able to buy desired agricultural inputs (0.08) Not Loans (-1.37) Not Formal loans (0.15) Not Financial Inclusion Micro-insurance (Agriculture/lives tock) * (2.28) 18 times as likely to have micro-insurance as comparison households at endline Savings (-0.10) Not Formal savings (0.00) Not Bonding: Trust (-0.56) Not Social Capital Bonding: Can rely on others for assistance Linking: Connection to local and national government (-0.52) (0.20) Not Not Migration Number of HH members migrating for work 0.137* (2.09) Had 0.14 more people per household migrate for work than in comparison households, at endline Duration of migration (months) 1.523* (2.16) Had household members migrate for 1.5 more months over the previous year than comparison households, at endline Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 43

44 4.2.3 Impact and Discussion of Results The fodder nexus enhanced communities DRR capabilities, including their use of physical mitigation and their ability to respond after a disaster. These capabilities successfully reduced losses to disasters among the beneficiary households in the Hilly region. This result stands in contrast to our findings in the Terai, where treatment villages did not have lower disaster losses because of the misalignment between the type of disasters targeted by the program and the actual disasters experienced. In the Hills, mitigation efforts were targeted at both floods and landslides, and paid off even during the regional drought. On the economic front, M-RED increased the rate of fodder planting, as well as the value of milk produced. However, as with the Terai, this did not lead directly to overall economic growth or increased stability during the program period. This is not entirely surprising, since households had to invest in new breeds of cow and expend their labor planting and harvesting fodder. These costs didn t harm them in the short term, but the beneficiaries also did not experience economic growth by the endline in line with the two-plus years that many marketdevelopment programs see as the minimum for real effects to emerge. M-RED continued to develop the dairy supply chain through the end of the program period, and farmers expected to see a strong demand for their milk production and more reliable storage and transportation options in the future. Overall, although we don t see a large economic gain by the end of the program, the nexus activity nevertheless seemed to combine economic incentives with disaster mitigation. Households earned more from milk, at the least, and did not suffer a broader cost to investing in the nexus activity. They also saw a clear pay-off in terms of disaster mitigation. Even if their economic status did not grow beyond their starting point, this disaster protection means they were still better-off continuing the nexus activity than not strongly encouraging sustainability. Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 44

45 4.3 Timor-Leste Purpose and Activities Timor-Leste Nexus Activities The disaster risks of Timor-Leste are more diverse than in Nepal: communities reported landslide, flood, and drought as concerns, but fires, wind, and earthquakes were also mentioned across all three districts. Climate Figure 13: Terraced Banana and Legume Plantation in Timor-Leste change and reductions in soil fertility are also considered by technical experts to be a growing challenge, and drought as a result of climate change was cited by farmers. To address these, M-RED used a series of nexus interventions, including: Banana for consumption and sale, planted on erosion prone slopes and riverbanks Legumes for consumption and sale, planted on erosion prone and marginal (poor fertility) lands Agro Forestry trees including fruit trees for short-term consumption and sale, and hardwood trees for long term sale; planted with land management techniques on erosion prone slopes Beyond these, M-RED also conducted in DMC capacity building and direct mitigation. However, in Timor-Leste DMCs were a pre-existing government entity sitting at the Suco level, a tier above the communities themselves. DMC members were also appointed by the government, primarily from among village heads. As in Nepal, we tested for M-RED s impact in both reducing disaster loss and developing economic well-being. By directly providing an economic incentive for DRR activities, we would expect the nexus interventions to provide a sustainable longterm answer to the many disasters faced by the island. Figure 15 reiterates the Theory of Change specifically for Timor-Leste and presents a brief summary of our findings. Areas where M-RED achieved the desired results are highlighted in purple, and areas where we found no effect are colored grey. Overall, M-RED increased communities preparedness for disasters through the use of advanced planting techniques, DMC capacity-building, and dissemination of response plans. These outputs paid off with lower disaster losses among the beneficiaries, while contributing to economic stability. The treatment group was able to continue IGA s in spite of disasters, and had higher levels of food security and nutrition. The program s disaster mitigation components were effective because, unlike Nepal s Terai, the weather and disaster profile during implementation matched the risks identified at its start. Figure 14 shows the weather trends identified by the program team, which were supported by community discussions. Figure 14: Weather Trends in Timor-Leste Managing Risk through Economic Development Phase I: Impact Evaluation CAUSAL DESIGN 45