Climate Change Adaptation and Advocacy Project in Nepal Project Effectiveness Review

Transcription

1 Climate Change Adaptation and Advocacy Project in Nepal Project Effectiveness Review Summary Report Photo: Women from Dolakot village construct a pond to irrigate their land, Oxfam GB Oxfam GB Adaptation and Risk Reduction Outcome Indicator April, 2013 Acknowledgements We would like to thank the Oxfam Nepal and IDeS team for being so supportive during the exercise. Particular thanks to Prabin Man Singh, Shanta Upadhyaya, Karuna Amatya and Soyesh Lakhey. Photo: Oxfam GB, Women from Dolakot village construct a pond to irrigate their land

2 Executive summary Under Oxfam Great Britain s (OGB) Global Performance Framework (GPF), sufficiently mature projects are being randomly selected each year and their effectiveness rigorously assessed. Nepal s Climate Change Adaptation and Advocacy project was randomly selected for an Effectiveness Review under the adaptation and risk reduction thematic area. The project aims to increase climate change resilience among target groups in Dadeldhura district of western Nepal, through creation of livelihood options and incorporation of climate change adaptation practices in district- and national-level plans and programmes. The community-level activities carried out included awareness-raising on climate change, provision of high-yielding and drought-tolerant varieties of cereal crop seeds, introduction of other vegetable seeds to encourage crop diversification, together with the implementation of various water resource conservation schemes. These project activities were implemented between 2009 and 2012 in seven communities in the Alital Village Development Committee (VDC) area in Dadeldhura, by a local partner organisation Integrated Development Society (IDeS). To assess the effectiveness of this project, a quasi-experimental impact evaluation was implemented. This involved carrying out surveys with households in the seven communities supported by the project, as well as with households in eight nearby comparison communities. In all, surveys were carried out with 437 households. At the analysis stage, the statistical tools of propensity-score matching and multivariable regression were used to control for demographic and baseline differences between the intervention and comparison groups. The effectiveness of the project in affecting 30 resilience characteristics was assessed through this process. These characteristics fall under five interrelated dimensions: livelihood viability, innovation potential, access to contingency resources and support, integrity of the natural and built environment, and social response capability. Composite indices were developed to aggregate the data associated with the 30 characteristics, following the Alkire- Foster method used by the Oxford Poverty and Human Development Initiative (OPHI) to measure multi-dimensional poverty. One of these indices, in particular, referred to as the Alkire-Foster Resilience Index informs Oxfam GB s global outcome indicator for its adaptation and risk reduction thematic area: % of households demonstrating greater ability to minimise risk from shocks and adapt to emerging trends and uncertainty (as measured by the Alkire-Foster Resilience Index). Following analysis of the data, there is evidence that the project positively affected several important characteristics assumed important for promoting resilience among the intervention population. In particular, even after controlling for measured differences between the intervention and comparison households, the former performed between 29 and 32 percentage points and points better than the latter on Oxfam GB s global ARR indicator and the Alkire-Foster Resilience Index, respectively. Strong performance in relation to the global indicator shows that in total, 86 per cent of surveyed intervention households demonstrate greater ability to reduce risk and adapt to emerging trends and uncertainty than the typical comparison household. While this effectiveness review generated positive results, it also identified opportunities for reflection and learning. Oxfam in general and the Nepal country team and partners in particular are encouraged to consider the following: 1

3 Consider further research to evaluate the effects of advocacy efforts connected to this project. Keep monitoring progress of the supported villages, and consider whether it is appropriate to scale-up the project interventions to the wider area. Explore options for strengthening the support to existing livelihood practices by considering greater investment in the improved cereal seed and water resource management components of the project. Introduction and purpose Oxfam GB has put in place a Global Performance Framework (GPF) as part of its effort to better understand and communicate its effectiveness, as well as enhance learning across the organisation. As part of this framework, modest samples of mature projects (e.g. those closing during a given financial year) associated with each thematic indicator area are being randomly selected each year and rigorously evaluated. One key focus is on the extent they have promoted change in relation to relevant OGB global outcome indicators. One of the projects randomly selected for the effectiveness review under the adaptation and risk reduction thematic area in 2012/13 is Nepal s Climate Change Adaptation and Advocacy Project. The effectiveness review that took place in Alital VDC in Dadeldhura district in December 2012, intended to evaluate the success of the Climate Change Adaptation and Advocacy Project in promoting resilience to climatic shocks among supported households. Although the project started in two districts Dadeldhura and Nawalparasi, the activities in the latter were phased out during 2011 due to various programmatic and technical reasons. Therefore the focus of the review was on the seven communities in Dadeldhura supported by the project through to its completion in March The project activities were implemented by a local partner organisation Integrated Development Society (IDeS). Figure 1: Location of Project Effectiveness Review 2

4 The project s overall objective was to increase climate change resilience among target groups through the creation of livelihood options and incorporation of climate change adaptation practices in district and national level plans and programmes. The activities undertaken to achieve the first part of the objective included: Awareness raising on climate change: including posters depicting impact of climate change on people s livelihoods, development of training materials related to various adaptation measures, and two-day training events on community-based adaptation to climate change. Exposure visits were also organised between the communities to see where community-based adaptation schemes had been demonstrated successfully. Introduction of high-yielding-variety seeds of major cereal crops tolerant to local climatic conditions. Introduction of other vegetable seeds to encourage crop diversification. Implementing various water resource conservation schemes, including construction of check dams, tree planting, rehabilitation of springs, construction of household level rainwater harvesting tanks, and construction of irrigation ponds. The second part of the objective was to be achieved through building the advocacy capacity of key climate action network and alliances, to ensure mainstreaming of community-based adaptation practices into national and district level plans and programmes. This component of the project was not limited to the supported communities in Alital VDC, and thus it is not formally covered by the results of this effectiveness review. Evaluation approach The Climate Change Adaptation and Advocacy Project, implemented in Alital VDC, attempted to strengthen the resilience of households living in seven communities in the central part of the VDC area (see Figure 1 above). From a rigorous impact evaluation perspective, the best way to evaluate such an intervention would have been to restrict the project s implementation to randomly selected geographical areas, leaving others sites for comparative purposes, i.e. as controls. This impact evaluation design is known as a clustered randomised control trial. If it was successfully implemented, the impact of the project could have been assessed by directly comparing the households of the intervention and control sites in relation to relevant measures of outcome. This is specifically because the randomisation process would have made the households in the two sites comparable in every way, save their participation in the project. However, Oxfam and its district partners did not implement the project in randomly selected geographic areas; the communities were purposively chosen. An alternative impact assessment design was consequently pursued. This design is referred to as a quasiexperiment because it attempts to mimic what a randomised control trial does by purposively identifying a comparison group that is similar to the intervention group, and then statistically controlling for any measured differences between the two. To implement the design, communities in Alital VDC, both where the project was implemented and not implemented were mapped out. A total of 437 households were interviewed, including 173 from the seven communities that had participated in the project, and a further 264 from eight communities selected as appropriate for comparison. To reduce bias, propensity-score matching (PSM) and multivariable regression were used in the statistical comparison of these two groups. 3

5 Intervention logic of the project Figure 2 shows a simple characterisation of the theory of change behind the project activities. The primary aim of the project was to strengthen the capacity of communities and households to manage the risks associated with climatic shocks, such as drought, flood, water scarcity and crop failure. The community-based training contributes to this through increasing the awareness of community members to climate-change hazards and providing a forum to discuss adaptation interventions that can be applied at a community level (e.g. check-dam construction, waterpoint rationing, etc.). The project is also providing high yielding variety seeds of major cereal crops suitable to the local climatic conditions, with the intention to increase crop yield, especially when and where rainfall is erratic. Introduction of these seeds was expected to improve food security of the targeted families, and therefore their resilience to drought events. Figure 2: Project theory of change (simplified) The introduction of vegetable seed for example, peas, tomatoes and cauliflower to the supported communities was intended to encourage a more diversified livelihood base for farmers, who have traditionally relied on subsistence crops, including wheat, maize and paddy. As well as reducing the risk of food insecurity in the event of cereal crop failure, it was envisaged that that there would also be the opportunity for increased income gained from the sale of these high-value vegetables. The introduction of both improved cereal and vegetable seeds was complemented by a number of water schemes, such as construction of check-dams, irrigation ponds, rehabilitation of water springs, and provision of household rainwater-harvesting tanks. The overarching aim of these various activities was to reduce current levels of water shortage, which are particularly exacerbated during the dry months (February May). As well as providing more secure availability of safe drinking water, it was also intended to permit longer growing seasons in the supported communities. 4

6 Measuring resilience Within Oxfam GB, efforts are being undertaken to develop an approach to measuring the resilience of households to shocks and stress and their ability to adapt to change. This approach involves capturing data on various household and community characteristics falling under the five interrelated dimensions presented below. Figure 3: Oxfam GB s conceptual framework for understanding and measuring resilience Dimensions affecting the ability of households and communities to minimise risks from shocks and adapt to emerging trends and uncertainty Extent livelihood strategies can thrive in spite of shocks, stresses, and uncertainty Ability to take appropriate risks and positively adjust to change Access to back up resources and appropriate assistance in times of crisis Health of local ecosystems, soundness of natural resource management practices, and robustness of essential physical infrastructure Extent formal & informal institutions are able to reduce risk, support positive adaptation, and ensure equitable access to essential services in times of shock/stress. First, if we think about what a household would need in order to cope with current and future shocks, stresses, and uncertainly, a viable livelihood is likely one of them. If a shock happens, for instance, a household dependent on just one precarious livelihood activity will likely be more negatively affected than another that has one or more less sensitive alternatives to fall back on, all other things being equal. In addition, households that are on the margins of survival are less likely to be resilient than their relatively more wealthy counterparts. Where longer-term climatic trend prediction information exists, it is also important to assess how viable current livelihood strategies would be given the range of likely future climatic scenarios. Innovation potential is different and hence separate. It is focused on a household s ability to positively adjust to change, whether anticipated or not. We can hypothesise that such potential is dependent on factors such as the knowledge and attitudes of relevant household members themselves, their ability to take risks, and their access to weather prediction, market information and relevant technology and resources. Moreover, there will likely be times when even households with the most resilient and adaptive livelihood strategies will find it tough to get by. Access to contingency resources and external support e.g. savings, food and seed reserves, social protection, kin and nonkin support networks, emergency services, etc. are, therefore, likely to be critical in supporting households in coping with shocks and positively adjust to change. 5

7 It is further recognised that healthy ecosystems are better able to cope/adjust to climatic shocks/change than those that are relatively more degraded. We may reasonably assume again, with all other things being equal that households whose livelihoods are dependent on healthier ecosystems will be in a better position to adjust to climatic shocks/change than those that are not. The presence of appropriate infrastructure (e.g. pit latrines and roads) that is resilient to shocks and stresses (e.g. flooding) is equally important; if critical infrastructure no longer functions or collapses in times of shocks and stress, the livelihoods and/or health of community members can be negatively affected. In most situations it is necessary to look beyond the household level when examining resilience and adaptive capacity. Indeed, it is reasonable to assume that households are better able to successfully adjust to climatic shocks or change when they are part of larger coordinated efforts at the community level and beyond. The social and institutional capability dimension, in particular, is concerned with the effectiveness of informal and formal institutions in reducing risk, supporting positive adaptation, and ensuring equitable access to essential services in times of shock/stress. In the absence of this capability, we can assume that community-level duty bearers will be less effective in fulfilling their responsibilities in supporting community members to reduce risk and/or successfully adapt. Specific characteristics believed to influence both resilience and adaptation fall under each of the five dimensions. However, no one size fits all ; that is, many of the characteristics appropriate for a particular population may not be so for another. As such, each particular suite of characteristics needs to be appropriately specified given the nature of the population in question and the hazards and change processes to which it is likely to be subjected. Application of the Resilience Framework To make use of the framework described above in measuring resilience, an approach was used which is adapted from that employed by the Oxford Policy and Human Development Initiative (OPHI) and other organisations in the measurement of complex constructs such as poverty and women s empowerment. Firstly, under each of the five dimensions of resilience, specific characteristics were identified which are believed to be important to household resilience in the particular context in Dadeldhura district. The full list of the 30 characteristics used is shown in Figure 4 below. Some of these characteristics such as participation in community-level droughtpreparedness planning are closely linked to the project interventions. Other characteristics were not directly linked to the objectives of the project, but were included in order to provide an overall view of resilience among the surveyed population. For each characteristic of resilience listed in Figure 4, a benchmark was defined, based on what it means for a household to be faring reasonably well in relation to the characteristic in question. For example, a household was defined to score positively in terms of livelihood diversification if household members were engaged in two or more livelihoods activities, with more than half of household income deriving from activities judged to be reasonably tolerant to drought. The survey data was then compared to these definitions, to determine whether each household scored positively in terms of each of the characteristics. Weights were assigned to the characteristics under each of the dimensions, as shown in Figure 4. For example, the characteristics of livelihood viability together accounted for 30 per cent of the total weight in calculating overall measures of resilience. The overall index of resilience was then defined as the proportion of weighted characteristics in which each household scored positively. 6

8 The Oxfam GB global indicator for resilience was defined to be positive for any household that scored positively in more of the characteristics than a typical household in the area. For these purposes, the typical household was taken to be the median household surveyed in the comparison communities. Figure 4: Characteristics and dimensional weights used in the Effectiveness Review Summary results table The following summary table provides a snapshot of the key findings of the effectiveness review. A short narrative description related to each outcome then follows to unpack each key finding. A separate report is also available that provides a more detailed and technical description of the evaluation design, process and results. Table 1 summarises the extent to which there is evidence that the project realised its targeted outcomes in the form of a simple five-point traffic light system. The key below illustrates what the various traffic lights represent. Evidence supporting large impact Evidence supporting more modest impact Evidence of large impact, but only for specific sub-groups/measures Evidence of modest impact, but only for specific sub-groups/measures No evidence of impact 7

9 Summary results table Dimension Outcome/indicator Cut-off a household scores positively if: % supported households above cut-off (unadjusted) Overall resilience G Overall Resilience Index global outcome indicator Household has an AF Resilience score above the median score for comparator households Evidence of impact Large/ modest impact Directly targeted by project Linked to project logic 86% Yes L Yes Yes Livelihood viability (30% weighting) Innovation potential (20% weighting) G Household wealth status Household food security Household dietary diversity Livelihood diversification Livelihood risk males in household Livelihood risk females in household Crop portfolio Attitudes towards new livelihood practices Awareness of climate change Innovation practice Access to credit Access to state innovative support It owns >=3 small assets Or >=2 big assets Or 2 small assets + 1 big asset. 79% No No Yes It reports having had to cut the size of meals, eat fewer meals, or reduce food consumed by adults in household >= 3 times in past week and reports no incidence of having to borrow food, going to sleep hungry, or going through a whole day with no food. 87% No No Yes It consumed in the past 7 days a carbohydrate source >=7 times; a protein source >=3 times; and any vegetable source >=3 times. 87% Yes M No Yes It engages in >=2 livelihood activities with >= 50% dependency on activities assumed to be significantly drought tolerant. 46% No Yes Yes Male household members are involved in at least one livelihood activity which is assumed to be significantly drought tolerant. 90% Yes M No No Female household members are involved in at least one livelihood activity which is assumed to be significantly drought tolerant 83% No No No It cultivated >=3 crop types, including at least one drought-resistant crop. 88% Yes M Yes Yes Respondent either does not agree at all or agrees only to a small extent with 5 out of the 6 negatively phrased statements (Likert scale). 67% No Yes Yes Respondent either does not agree at all or agrees only to a small extent with 5 out of the 6 negatively phrased statements (Likert scale). 93% No Yes Yes Respondent reports having tried out or experimented with at least one new activity over the past 2 years. 68% Yes L Yes Yes Respondent reports that household took out loan in last 2 years Or could borrow at least 10,000 rupees in the event it was needed from a money lender, non-local family members, savings group, or bank/credit institution. Respondent reports having had accessed state extension support in new techniques in the last two years and reports at least finding the support moderately helpful. 89% No No No 21% Yes M No Yes 8

10 Dimension Outcome / indicator Cut-off a household scores positively if: % supported households above cut-off (unadjusted) Access to contingency resources and support (20% weighting) G Integrity of the natural and built environment (15% weighting) G Social and institutional capability (15% weighting) G Group participation Social connectivity Perceptions of local government emergency support Savings Evidence of impact Large/ modest impact Directly targeted by project Linked to project logic Respondent reports having accessed state extension support in new techniques in the last two years and reports finding the support at least moderately helpful. 84% Yes L Yes Yes Respondent reports being an active participant in at least 2 groups with medium involvement in decision making in at least one. 87% Yes L No No Respondent agrees at least to a medium extent with 5 out of the 6 positively phrased statements. 22% Yes M No No Respondent states that there is enough savings to enable them to survive for at least 14 days in the event of a drought. 68% Yes M No Yes Remittances or formal It reports receipt of transfer money from outside community and/or earnings someone in the home has a formal job. 62% Yes M No No Fertility of local soils It reports no negative change in fertility of farm plot. 31% Yes M No Yes Extent of soil erosion It does not report experiencing severe erosion. 97% Yes L No Yes Access to irrigation for It reports having access to irrigation facilities or experienced only a small farming portion of its crops being lost during the 2011 drought. 60% Yes M Yes Yes Access to safe drinking It did not report experiencing serious difficulties accessing drinking water water year round for domestic use. 43% No Yes Yes Extent of vegetative It reports having at least 5% coverage of agricultural land with trees/nonagricultural vegetation. cover in farm plot 78% Yes M No Yes Extent farming activities It reports having experienced only of small portion of its crops being lost affected by drought during the 2011 drought. 55% Yes L Yes Yes Awareness of drought It is at least partly aware of the contents of the plan. preparedness plan 38% Yes L Yes Yes Participation in drought It has participated at least one meeting in past 12 months. prep. meetings 30% Yes L Yes Yes Receipt of drought It had received such information in past 12 months. prep. information 36% Yes L Yes Yes Awareness of It is aware of at least 2 community level initiatives having taken place in community level past 3 years. drought risk reduction 90% Yes L Yes Yes initiatives Water resource dispute experience Aware of local leaders are undertaking action Level of confidence in effectiveness of local leaders/institutions It does not report being involved in any disputes in past 2 years. 81% No No No It is at least partly aware that community leaders/institutions are doing something on the adaptation front. 30% Yes L No No Respondent agrees at least to a medium extent with 5 out of the 6 positively phrased statements. 24% Yes M No No 9

11 Impact assessment findings Overall results G As described above, the indices of resilience are defined from 30 different characteristics, all of which were assessed by means of a survey at household level. The proportion of characteristics in which each household scored positively was calculated: this measure is known as the base resilience index. Interviewed households scored positively, on average, on 67 per cent of characteristics, and the comparison households in 55 per cent of characteristics. The various statistical models which are used to control for baseline and demographic differences all concur that this represents a significant positive difference between the supported and comparison households. Under the Alkire-Foster method described above, a household was defined as resilient overall if it scored positively on more than two-thirds of the characteristics. Overall, 48 per cent of the supported households reached this benchmark, compared to 16 per cent of comparison households. Again, this represents a highly significant positive difference. The Oxfam GB global indicator is defined to be positive for any household that has a resilience index score greater than the median of the comparison group. In this way, the global indicator reflects whether a household scores positively in more characteristics than a typical household in the comparison group. On this measure, 86 per cent of supported households scored positively, compared to 50 per cent of comparison households a statistically significant difference. In essence, this shows that 86 per cent of surveyed intervention households demonstrate greater ability to reduce risk and adapt to emerging trends and uncertainty (as measured by this index). While the overall resilience index provides an overview, it is essential to know the situation of the population in relation to each of the characteristics, and which characteristics are driving the difference between the intervention and comparison households. The following sections therefore consider the various characteristics and the contributions that each makes towards households resilience. Dimension 1: Livelihood viability The first dimension included in the resilience index is that of livelihood viability. To what extent is there evidence that households in the intervention villages possess livelihoods that are more resilient to shocks than the comparison households? On average, the intervention households scored positively on 80 per cent of the characteristics of livelihood viability, while the comparison households scored positively on 74 per cent. This difference is statistically significant across three out of the five statistical procedures with at least a 95 per cent level of confidence. There is therefore modest evidence that the project has enhanced the livelihood viability of the intervention population. Figure 5 presents the percentage of households scoring positively on each of the indicators in the livelihood viability dimension. The indicators have been ranked for the intervention population from most deprived to least deprived. As is evident, both the general intervention and comparison populations are most deprived in relation to livelihood diversification. For the remaining indicators, it is interesting to note that the majority of households scored positively, both in the intervention and comparison villages. 10

12 100% Figure 5: Percentage of HHs scoring positively on characteristics of the livelihood viability dimension 80% 60% 40% 20% 0% Livelihood diversification Asset wealth Drought-tolerant activities - female Intervention population Food security Dietary diversity Crop portfolio Drought-tolerant activities - male Comparison population Differences between the intervention and comparison households are also apparent from the graph. The intervention households clearly perform better in relation to the comparison households for several of the characteristics. Turning our attention to the characteristic with the lowest proportion of households scoring positively i.e. livelihood diversification, the analysis finds that there are no significant differences between the intervention and comparison households. As a reminder, the cut-offs for determining whether a household receives a positive score on a particular characteristic are detailed in the summary results table. When examining the underlying data (not shown here) on the total number of livelihood activities that each household engages in, there is a significant difference: the supported households engage (on average) in at least 0.2 more livelihood activities than comparison households. This indicates that there is modest evidence that the project has successfully effected an increase in the number of different livelihood activities. However, there is no evidence of a difference between the intervention and comparison households in terms of livelihood diversification into more drought tolerant activities. In terms of the other indicators in this domain, the analysis shows that overall a high proportion of households, both in the intervention and comparison villages, scored positively. However, there are some significant differences between the populations. For example, 87 per cent of supported households scored positively on the dietary diversity characteristic, compared to 77 per cent of comparison households. This difference is statistically significant; indicating that there is evidence the project has increased the diversity of food types consumed by supported households. There is also a more modest difference in the range of crops cultivated by intervention and comparison households. As one of the key project interventions was provision of improved cereal seeds and vegetable seeds, it is interesting to compare the proportion of households cultivating various crops in the intervention and comparison villages. Figure 6 illustrates these differences. The most significant difference between the intervention and comparison households is the proportion of households growing peas. Over half of supported households reported growing peas, compared to one-third of comparison households. Statistically significant differences are also evident for other vegetables, including potatoes and cauliflower. Interestingly, a significantly smaller proportion of households in the intervention villages reported cultivating traditional rice. 11

13 This appears to have been replaced by the cultivation of improved varieties of rice and wheat provided by the project. While the differences in the proportion of intervention and comparison households growing these improved varieties is significant, it is worth noting that at the time of survey, only about 10 per cent of households were engaged in doing so. Figure 6: Percentage of HHs cultivating various crops 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Intervention Comparison The other characteristic in which there is a significant difference is the number of droughttolerant activities in which the male household members are involved. For a household to score positively in this measure, male household members must be involved in at least one livelihood activity that is assumed to be significantly drought-tolerant for example, casual labour that is not affected by climatic shocks, or being involved in irrigated crop cultivation. Overall 90 per cent of supported households score positively for this indicator, compared to 82 per cent of comparison households. In terms of women s involvement in drought-tolerant activities, the small positive difference between intervention and comparison households is not statistically significant, so it is not clear that this represents a real difference between the groups The remaining two characteristics of livelihood viability are indicators of a household s wealth status as measured by asset ownership, and its experience of food security. For both of these indicators there is no clear difference between households in the project and comparison villages. However, it should be noted that the proportion of households scoring positively for food security implies that in the four weeks preceding the household interviews, there was little experience of food insecurity. This is a limitation of the tool, and future iterations plan to try and capture experience of food insecurity over a greater time period. This should allow more meaningful analysis of potential project impact on a household s experience of food security. Supporting analyses (not shown here), including assessing changes in a household s wealth status between 2009 and 2012, confirm that there is no significant difference between the intervention and comparison households in asset wealth. Unfortunately this implies that there is no evidence of a change in household wealth. 12

14 Dimension 2: Innovation potential G Data were collected on five characteristics falling under the innovation potential dimension. On average, the supported households scored positively on two-thirds of the five characteristics, compared to 57 per cent for comparison households. This difference is highly statistically significant, indicating a positive project effect on the characteristics that make up the innovation dimension. Figure 7 presents the percentage of households scoring positively on each of the indicators in the innovation potential dimension. As is evident, there are clear differences between the proportions of households scoring positively on the different characteristics. Interestingly, households scored most positively on awareness of climate change issues and access to credit. 100% Figure 7: Percentage of HHs scoring positively on characteristics of the innovation potential dimension 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Access to state innovative support Attitudes towards new livelihood practices Innovation practice Access to credit Awareness of climate change Intervention population Comparison population The access to state innovative support characteristic was the indicator in which households scored least positively. However, the statistical procedures estimate a significant difference between the proportion of intervention and comparison households above the cut-off, i.e. they reported having received support from government extension programmes and found the support helpful. Assuming there are no geographical differences in how state support is made available to households, this may indicate that the project has successfully affected the way in which households engage with this type of support. However, an alternative explanation may be that respondents were confused, and that in this instance they were actually referring to support received from Oxfam, rather than from state extension workers. Further follow up will be required to confirm. Interestingly, the proportion of households scoring positively for awareness of climate change issues was very high 93 per cent of intervention households, and 97 per cent of comparison households. However, there were some concerns regarding how accurately this characteristic was measured, both due to a lack of consistency of responses to the six Likert-scale questions related to this indicator, and reported difficulties from the fieldwork regarding translation of these particular questions. While the results show significant positive differences in favour of the comparison households, these should therefore be treated with a degree of caution. 13

15 A high proportion of households also scored positively on the access to credit measure, with 89 per cent of intervention households, and 86 per cent of comparison households reporting that they would be able to access a modest-sized loan if required. However, the difference between the groups is not significant. It is interesting to note that while the proportion of households scoring positively for their attitudes towards trying new livelihood practices is very similar between the intervention and comparison households, there is a highly significant difference in the proportion of households scoring positively in actually practising innovative activities. Over two-thirds of supported households scored positively i.e. they tried at least one new activity (e.g. cultivating new crops), in the previous two years compared to 22 per cent of comparison households. This clearly implies a strongly successful project effect on innovative practice in supported households. Such a strong effect warrants further analysis. Figure 8 presents the proportion of households who have tried a variety of activities for the first time in the two years leading up to the data collection. These are the same activities reported on under the Innovation practice indicator shown in Figure 7. As is evident from the graph, just over 40 per cent of supported households said they had begun harvesting rainwater, just over 30 per cent had started cultivating a new crop, and 18 per cent had purchased a new type of seed. For each of these activities the difference in the proportion of households in the intervention and comparison villages practising each is highly significant. While the proportions of intervention households who have started selling crops in a different market and began using irrigation for crops is smaller, significant differences exist between the intervention and comparison households. 50% 40% Figure 8: Percentage of HHs implementing activities for the first time Intervention Comparison 30% 20% 10% 0% Harvesting rainwater Cultivated a Tilled soil in Purchased new crop a new way a new type of seed Used fertiliser (natural or artificial) Started a new business Started to sell crops in a different market Began rearing livestock Began using irrigation for crops In summary, this highlights that the project has successfully affected the practice of more innovative activities in the intervention villages. 14

16 Dimension 3: Access to contingency resources and support G Data were collected on five characteristics falling under the access to contingency resources and support dimension. On average, the supported households scored positively on almost two-thirds of the five characteristics, compared to 49 per cent for comparison households. This difference is highly statistically significant, again indicating a positive project effect on the characteristics that make up this dimension. Figure 9 presents the percentage of households scoring positively on each of the indicators in the access to contingency resources and support dimension. 100% Figure 9: Percentage of HHs scoring positively on characteristics of the contingency resources and support dimension 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Local gov. emergency support Remittances or formal earnings Savings Group participation Social connectivity Intervention population Comparison population Figure 9 clearly shows that more of the supported households reported participating widely in community groups than did the corresponding comparison households. The difference is highly statistically significant, indicating strong evidence of a positive project effect on this indicator. Particularly interesting is the analysis of the type of groups in which households are participating. Figure 10 illustrates the proportion of households that participate in the various groups appropriate to the context. 15

17 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Figure 10: Percentage of HHs participating in various community groups Intervention Comparison It is important to consider the group memberships that the project was particularly trying to affect. Perhaps unsurprisingly there is a clear and significant difference in the proportion of households participating in agricultural producer groups, as this was an aspect deliberately targeted by the project. More interesting are the significant differences that exist in terms of participation in forest user groups, savings co-operatives and VDC/ward committees. This analysis indicates strong evidence that the project has affected participation in important groups that are not necessarily directly connected to the project s activities. The other characteristic in this dimension where there is a highly significant difference is in relation to social connectivity. The measurement of this characteristic looks at how respondents perceive the strength of their social support system. Overall, 87 per cent of supported households scored positively for this measure, compared to 72 per cent of comparison households. This indicates a positive project effect on supported household s perception of their social support systems. Households scored least positively in how they perceived the efficacy of the local government support system in the event of a disaster. While this characteristic may not be explicitly linked to the project, it is still interesting to note that three of the four adjustment methods estimate a significant difference of 8 per cent in the proportion of households scoring positively. In order to measure the savings characteristic, households were not asked directly for the monetary value of their savings, but were instead asked the question, if they had a crisis and had to live on their cash savings without other income sources, how long they would be able to do so? Just over two-thirds of the intervention households scored positively on this measure, meaning that they could live from their savings for more than 14 days. There is a statistically significant difference among the supported households compared to the comparison households (50 per cent). It is interesting to note that while there was no overall difference between the intervention and comparison households in asset ownership (see Dimension 1), this result indicates a greater availability of current cash resources in the supported households. The final characteristic analysed in this dimension is the availability within the household of remittances or formal employment. Approximately 55 per cent of the survey respondents reported receiving remittances from relatives outside the community during the 12 months prior to the survey, and 11 per cent reported that a household member was receiving regular 16

18 earnings. While these are important factors in determining the household s level of resilience, they cannot plausibly have been affected by the Climate Change Adaptation and Advocacy project. It is also worth noting that there were significant differences in the proportion of households receiving remittances at baseline. The difference reported for this characteristic is likely a result of the regression and PSM models not fully controlling for the baseline differences. Dimension 4: Integrity of the natural and built environment G Six characteristics were identified for the integrity of the natural and built environment dimension. On average, the supported households scored positively on almost two-thirds of the five characteristics, compared to 49 per cent for comparison households. This difference is again highly statistically significant. Figure 11 presents the percentage of households scoring positively on each of the indicators in the integrity of the natural and built environment dimension. 100% Figure 11: Percentage of HHs scoring positively on characteristics of the integrity of the natural environment dimension 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Fertility of local soils Access to safe drinking water year round Loss of crops due to drought Access to water for farming Extent of vegetation cover Extent of soil erosion Intervention population Comparison population In terms of the household s perception of how their agricultural land is affected by erosion, almost all intervention households scored positively for this measure i.e. they responded that that their land was not severely affected by erosion, compared to 87 per cent of households in the comparison group. This difference is highly statistically significant, although to be confident that this is a genuine effect of the project, further research is required to assess the level to which interventions that could plausibly affect this indicator such as tree planting and conservation farming were carried out in the supported communities. In follow up to this, it is interesting to note that there is also a significant difference in the proportion of households reporting that there is at least 5 per cent tree or other nonagricultural vegetation coverage in their agricultural land. As this information was not collected 17

19 for the baseline period, it is difficult to be sure that the differences can be attributed to project activity, rather than to existing differences in the ecological or topographic characteristics of the intervention and comparison villages. However, it should be noted that the project did implement interventions to encourage households to use a conservation approach to farming, e.g. minimal tilling. As for the first characteristic, further follow up research is recommended. Additionally, 31 per cent of supported households scored positively in the soil fertility measure, compared to 27 per cent of comparison households. This difference is estimated to be significant by three of the four models although only at the 90 per cent confidence level. As for the first two characteristics, further research is recommended to confirm these differences; however, it is positive to note that the findings from these three indicators are consistent and therefore corroborate the theory that the project has successfully affected these outcomes. In terms of a household s access to water for farming, just over 60 per cent of intervention households reported either irrigating their land in the last year, or would be able to water at least part of their land in the event of a drought. The difference between the intervention and comparison households is estimated to be significant by two of the four estimation methods. This therefore indicates there is modest evidence that the project has successfully affected the way in which households are able to access water for agriculture. Finally in this dimension, we look at a very important characteristic in terms of this project s aims and objectives. For a household to score positively on the loss of crops due to drought characteristic, they had to report losing less than half of their harvest due to the drought which occurred in Overall, 55 per cent of supported households lost less than half of their crops, compared to 35 per cent of households in the comparison villages. In reversing this measure, it is sobering to consider that 45 per cent of intervention households and 65 per cent of comparison households lost more than half of their harvest due to the drought. These differences are however, highly statistically significant. This provides strong evidence that the project has had a positive effect on this important measure. It is interesting to note that evidence of project impact is weakest in the characteristic that considers the availability of safe drinking water. For a household to score positively in this measure, they had to respond that they had access to a year-round safe drinking water source, within a 30 minute round trip. Overall, 43 per cent of supported households scored positively, compared to 36 per cent of comparison households. Unfortunately, none of the estimation methods consider this difference to be significant, indicating there is no evidence of a positive project effect in this regard. Dimension 5: Social and institutional capability G The final dimension of resilience considered in this effectiveness review is the capability of institutions in the community. On average, the supported households scored positively on just less than half of the seven characteristics that make up this dimension, compared to 24 per cent for comparison households. This difference is highly statistically significant, indicating a strongly positive project-effect on the characteristics that make up this dimension. The remainder of this subsection will explore what particular characteristics are driving this difference. Figure 12 presents the percentage of households scoring positively on each of the indicators in the social and institutional capability dimension. 18

20 100% 90% Figure 12: Percentage of HHs scoring positively on characteristics of the social and institutional capability dimension Intervention population Comparison population 80% 70% 60% 50% 40% 30% 20% 10% 0% Level of confidence in capacity of local leaders Participation in drought preparedness meetings Awareness that local leaders are undertaking action Receipt of drought preparedness information Awareness of drought preparedness plan Disputes over natural resources Community drought preparedness activities Households scored most positively on aspects related to their awareness of community-level drought preparedness activities, whereas households scored least positively in their level of confidence in the capacity of local leaders to provide leadership and support in times of crisis. Given the focus of the project under review, it is particularly interesting to compare the proportion of households who are aware of community-level drought preparedness activities, with those who have participated in preparedness meetings, received preparedness information, or are aware of any preparedness plan. While the supported households did not score particularly well on these latter characteristics, they are clearly better off in these regards than households in the comparison villages. It is also worth pointing out that the majority of all the interviewed households said that they had not been involved in a dispute over access to water in the previous two years, hence resulting in strong performance among both groups on this measure. In Figure 12, we see that approximately a third of intervention households score positively in terms of their awareness of a community drought preparedness plan, their participation in drought preparedness meetings, and their receipt of drought preparedness information. While these proportions are relatively low, the differences between the intervention and comparison households are highly statistically significant. Similar differences emerge in terms of respondent s awareness that local leaders are undertaking action. It will be important to clarify whether the relatively low proportion of households scoring positively is a result of only a subset of villages preparing drought mitigation plans or holding preparedness meetings. From an initial analysis of the data, it appears that this may be the case, with either none, or very few households scoring positively in three of the supported villages Dandakharda, Tekundanda and Dolakot. As the sample sizes are very small at a village level, we cannot say with statistical certainty that the differences between the villages are significant, however this does warrant further investigation. It is interesting to note that while scores are relatively low for aspects related to the more formal structures of community-level drought-preparedness planning (see above), over 90 per cent of supported households are aware of community-level activities that are intended to mitigate some of the effects of drought. This compares to less than half of households in comparison villages a difference which is again highly statistically significant. Figure 13 provides a breakdown of the particular activities for which respondents were asked to report on in their community. With the exception of water rationing, the differences between the intervention and comparison households for each of the activities is statistically significant. 19