Evaluation Climate Change Adaptation and Advocacy Project in Nepal Project Effectiveness Review Full Technical Report

Transcription

1 Evaluation Climate Change Adaptation and Advocacy Project in Nepal Project Effectiveness Review Full Technical Report Oxfam GB Adaptation and Risk Reduction Outcome indicator April, 2013 Acknowledgements We would like to thank the Oxfam Nepal and IDeS team for being so supportive during the exercise. Particular thanks to Prabin Man Singh, Shanta Upadhyaya, Karuna Amatya and Soyesh Lakhey. Photo: Oxfam GB, Women from Dolakot village construct a pond to irrigate their land.

2 Table of Contents Executive summary Introduction and purpose Intervention logic of project The ARR Outcome Indicator and its conceptual underpinnings Introducing the ARR Outcome Indicator The particular ARR characteristics used in the Nepal Effectiveness Review Constructing the resilience indices The overall resilience measures Impact assessment design Limitations in pursuing the gold standard Alternative evaluation design pursued Intervention and comparison villages surveyed Methods of data collection and analysis Data collection Data analysis Main problems and constraints encountered Results General characteristics Differences between the intervention and comparison households on the outcome measures Results on the Resilience Index Decomposition of Resilience Index Dimension 1: Livelihood viability Dimension 2: Innovation potential Dimension 3: Access to contingency resources and support Dimension 4: Integrity of the natural and built environment Dimension 5: Social and institutional capability Further analysis Conclusions and learning considerations Conclusions Programme learning considerations Appendix 1: Cut-offs and weights used for each characteristic Appendix 2: Covariate balance following propensity score matching procedures... 52

3 Executive summary Under Oxfam Great Britain s (OGB) Global Performance Framework (GPF), sufficiently mature projects are being randomly selected each year and their effectiveness rigorously assessed. Nepal s Climate Change Adaptation and Advocacy project was randomly selected for an Effectiveness Review under the adaptation and risk reduction thematic area. The project aims to increase climate change resilience among target groups in Dadeldhura district of western Nepal, through creation of livelihood options and incorporation of climate change adaptation practices in district- and national-level plans and programmes. The community-level activities carried out included awareness-raising on climate change, provision of high-yielding and drought-tolerant varieties of cereal crop seeds, introduction of other vegetable seeds to encourage crop diversification, together with the implementation of various water resource conservation schemes. These project activities were implemented between 2009 and 2012 in seven communities in the Alital Village Development Committee (VDC) area in Dadeldhura, by a local partner organisation Integrated Development Society (IDeS). To assess the effectiveness of this project, a quasi-experimental impact evaluation was implemented. This involved carrying out surveys with households in the seven communities supported by the project, as well as with households in eight nearby comparison communities. In all, surveys were carried out with 437 households. At the analysis stage, the statistical tools of propensity-score matching and multivariable regression were used to control for demographic and baseline differences between the intervention and comparison groups. The effectiveness of the project in affecting 30 resilience characteristics was assessed through this process. These characteristics fall under five interrelated dimensions: livelihood viability, innovation potential, access to contingency resources and support, integrity of the natural and built environment, and social response capability. Composite indices were developed to aggregate the data associated with the 30 characteristics, following the Alkire-Foster method used by the Oxford Poverty and Human Development Initiative (OPHI) to measure multidimensional poverty. One of these indices, in particular, referred to as the Alkire-Foster Resilience Index informs Oxfam GB s global outcome indicator for its adaptation and risk reduction thematic area: % of households demonstrating greater ability to minimise risk from shocks and adapt to emerging trends and uncertainty (as measured by the Alkire-Foster Resilience Index). Following analysis of the data, there is evidence that the project positively affected several important characteristics assumed important for promoting resilience among the intervention population. In particular, even after controlling for measured differences between the intervention and comparison households, the former performed between 29 and 32 percentage points and points better than the latter on Oxfam GB s global ARR indicator and the Alkire-Foster Resilience Index, respectively. Strong performance in relation to the global indicator shows that in total, 86 per cent of surveyed intervention households demonstrate greater ability to reduce risk and adapt to emerging trends and uncertainty than the typical comparison household. While this effectiveness review generated positive results, it also identified opportunities for reflection and learning. Oxfam in general and the Nepal country team and partners in particular are encouraged to consider the following: Consider further research to evaluate the effects of advocacy efforts connected to this project. Keep monitoring progress of the supported villages, and consider whether it is appropriate to scale-up the project interventions to the wider area. Explore options for strengthening the support to existing livelihood practices by considering greater investment in the improved cereal seed and water resource management components of the project. 1

4 1 Introduction and purpose This report documents the findings of a project effectiveness review, focusing on outcomes related to risk reduction and adaptation to climate change Oxfam GB has put in place a Global Performance Framework (GPF) as part of its effort to better understand and communicate its effectiveness, as well as enhance learning across the organisation. This framework requires project/programme teams to annually report output data across six thematic indicator areas. In addition, modest samples of mature projects (e.g. those closing during a given financial year) under each thematic indicator area are being randomly selected each year and rigorously evaluated. One key focus is on the extent they have promoted change in relation to relevant OGB global outcome indicators. The global outcome indicator for the adaptation and risk reduction thematic area is defined as the percentage of households demonstrating greater ability to minimise risk from shocks and adapt to emerging trends and uncertainty, compared to a typical comparison household. This indicator is explained in more detail in Section 3 below. The effectiveness review that took place in Alital VDC in Dadeldhura district in December 2012, intended to evaluate the success of the Climate Change Adaptation and Advocacy Project in promoting resilience to climatic shocks among supported households. Although the project started in two districts Dadeldhura and Nawalparasi, the activities in the latter were phased out during 2011 due to various programmatic and technical reasons. Therefore the focus of the review was on the seven communities in Dadeldhura supported by the project through to its completion in March The project activities were implemented by a local partner organisation Integrated Development Society (IDeS). The project s overall objective was to increase climate change resilience among target communities through the creation of livelihood options and incorporation of climate change adaptation practices in district- and nationallevel plans and programmes. The activities undertaken to achieve the first part of the objective included: Awareness raising on climate change: including posters depicting impact of climate change on people s livelihoods, development of training materials related to various adaptation measures, and twoday training events on community-based adaptation to climate change. Exposure visits were also organised between the communities to see where community-based adaptation schemes had been demonstrated successfully. Introduction of high-yielding-variety seeds of major cereal crops tolerant to local climatic conditions. Introduction of other vegetable seeds to encourage crop diversification. Implementing various water resource conservation schemes, including construction of check dams, tree planting, rehabilitation of springs, construction of household level rainwater harvesting tanks, and construction of irrigation ponds. The second part of the objective was to be achieved through building the advocacy capacity of key climate action network and alliances, to ensure mainstreaming of community-based adaptation practices into national and district level plans and programmes. This component of the project was not limited to the supported communities in Alital VDC, and thus it is not formally 2

5 covered by the results of this effectiveness review. Figure 1.1: Location of project implementation 2013 Google This report presents the findings of the project effectiveness review. Section 2 begins by reviewing the intervention logic of the project, and Section 3 follows by introducing the framework for measuring resilience that was adopted. Section 4 then goes on to describe the evaluation design, while Section 5 describes how this design was implemented. Section 6 thereafter presents the results of the data analysis, including the descriptive statistics on the population surveyed and the differences in outcome measures between the intervention and comparison groups. Section 7 concludes the document with a summary of the findings and some programme learning considerations. 3

6 2 Intervention logic of project Figure 2.1 shows a simple diagram of the theory of change behind the project activities. The project s overall objective was to strengthen the capacity of communities and households to manage the risks associated with climatic shocks, such as drought, flood, water scarcity and crop failure. The community-based training contributes to this through increasing the awareness of community members to climate-change hazards and providing a forum to discuss adaptation interventions that can be applied at a community level (e.g. check-dam construction, water-point rationing, etc.). The project is also providing high yielding variety seeds of major cereal crops suitable to the local climatic conditions, with the intention to increase crop yield, especially when and where rainfall is erratic. Introduction of these seeds was expected to improve food security of the targeted families, and therefore their resilience to drought events. Figure 2.1: Project theory of change (simplified) The introduction of vegetable seed for example, peas, tomatoes and cauliflower to the supported communities was intended to encourage a more diversified livelihood base for farmers, who have traditionally relied on subsistence crops, including wheat, maize and paddy. As well as reducing the risk of food insecurity in the event of cereal crop failure, it was envisaged that that there would also be the opportunity for increased income gained from the sale of these high-value vegetables. The introduction of both improved cereal and vegetable seeds was complemented by a number of water schemes, such as construction of check-dams, irrigation ponds, rehabilitation of water springs, and provision of household rainwater-harvesting tanks. The overarching aim of these various activities was to reduce current levels of water shortage, which are particularly exacerbated during the dry months (February May). As well as providing more secure availability of safe drinking water, it was also intended to permit longer growing seasons in the supported communities. 4

7 3 The ARR Outcome Indicator and its conceptual underpinnings 3.1 Introducing the ARR Outcome Indicator As part of Oxfam GB s (OGB) Global Performance Framework, efforts are being undertaken to develop an innovative approach to measuring the resilience of households to shocks and stress and their ability to adapt to change. This approach involves capturing data on various household and community characteristics falling under the five interrelated dimensions presented in Figure 3.1. Following the Alkire-Foster method used in the measurement of multidimensional poverty, 1 a binary cut-off is defined for each characteristic. A household is considered to be faring well in relation to the characteristic if it is above this cut-off and not well if below. Weighted indices, described further in Section 6, are then developed from these binary indicators. These indices can be used as continuous outcome measures in statistical analysis. Alternatively, binary outcome variables can be created by defining cut-off points for the index, with 1 specified for households that have surpassed this threshold and 0 for those below it. For OGB s global Adaptation and Risk Reduction (ARR) outcome indicator, the binary version of this indicator is defined as follows: Proportion of targeted households demonstrating greater ability to minimise risk from shocks and adapt to emerging trends and uncertainty The term greater ability appears in the wording of the indicator because of how it is computed in practice. Specifically, a household is coded with 1 if it is above the median of the comparison group in relation to the Alkire-Foster Resilience Index and 0 if otherwise. Thus, households demonstrating greater ability are those that are above the typical household of the comparison group in relation to this index. The characteristic approach assumes that households that are better able to cope with shocks and adapt to change possess particular attributes. One reason why measuring concepts such as resilience and adaptive capacity is complicated is because we can only really assess whether a system has successfully coped or adapted after the fact. 2 In other words, we would have to wait until after a disaster has struck and/or climatic change has taken place in order to assess the effectiveness of the intervention in question. The characteristic approach attempts to get around this issue by hypothesising that there are particular characteristics of households (and even communities, organisations, governments, etc.) that affect how well they are able to cope with shocks and positively adapt to change. A limitation, of course, is that we do not know for certain how relevant these characteristics actually are; rather, we assume they are important based on common sense, theory, and/or field experience. However, there is nothing preventing them from being informed by stronger empirical evidence and/or community consultation. It is further recommended that they be continuously updated, as the body of research on the determinants of resilience and adaptive capacity grows. 1 Alkire, S. and Foster, J. (2011) Counting and multidimensional poverty measurement. Journal of Public Economics, 95: Dodman, D., Ayers, J. and Huq, S. (2009) Building Resilience, Chapter 5, in World Watch Institute (ed.), 2009 State of the World: Into a Warming World, Washington D.C: World Watch Institute, pp

8 The characteristics are context specific but informed by a framework comprising of five distinct dimensions The characteristics that inform the ARR indicator fall under the five dimensions presented in Figure 3.1. First, if we think about what a household would need in order to cope with current and future shocks, stresses, and uncertainly, a viable livelihood is likely to be one of them. If a shock happens, for instance, a household dependent on just one precarious livelihood activity is likely to be more negatively affected than another that has one or more less sensitive alternatives to fall back on, all other things being equal. In addition, households that are on the margins of survival are less likely to be resilient than their relatively more wealthy counterparts. Where longer-term climatic trend prediction information exists, it is also important to assess how viable current livelihood strategies would be given the range of likely future climatic scenarios. FIGURE 3.1: Dimensions affecting the ability of households and communities to minimise risks from shocks and adapt to emerging trends and uncertainty Extent livelihood strategies can thrive in spite of shocks, stresses, and uncertainty Ability to take appropriate risks and positively adjust to change Access to back up resources and appropriate assistance in times of crisis Health of local ecosystems, soundness of natural resource management practices, and robustness of essential physical infrastructure Extent formal & informal institutions are able to reduce risk, support positive adaptation, and ensure equitable access to essential services in times of shock/stress. Innovation potential is different and hence separate. It is focused on a household s ability to positively adjust to change, whether anticipated or not. We can hypothesise that such potential is dependent on factors such as the knowledge and attitudes of relevant household members themselves, their ability to take risks, and their access to weather prediction, market information and relevant technology and resources. Moreover, there will likely be times when even households with the most resilient and adaptive livelihood strategies will find it tough to get by. Access to contingency resources and external support e.g. savings, food and seed reserves, social protection, kin and non-kin support networks, emergency services, etc. are, therefore, likely to be critical in supporting households in coping with shocks and positively adjust to change. 6

9 It is further recognised that healthy ecosystems are better able to cope/adjust to climatic shocks/change than those that are relatively more degraded. 3 We may reasonably assume again, with all other things being equal that households whose livelihoods are dependent on healthier ecosystems will be in a better position to adjust to climatic shocks/change than those that are not. The presence of appropriate infrastructure (e.g. pit latrines and roads) that is resilient to shocks and stresses (e.g. flooding) is equally important; if critical infrastructure no longer functions or collapses in times of shocks and stress, the livelihoods and/or health of community members can be negatively affected. Efforts must be undertaken to specify specific characteristics relevant to the context in question In most, if not all cases, it is necessary to look beyond the household level when examining resilience and adaptive capacity. Indeed, it is reasonable to assume that households are likely better able to successfully adjust to climatic shocks/change when they are part of larger coordinated efforts at the community level and beyond. The social and institutional capability dimension, in particular, is concerned with the effectiveness of informal and formal institutions in reducing risk, supporting positive adaptation, and ensuring equitable access to essential services in times of shock/stress. In the absence of this capability, we can assume that community-level duty bearers will be less effective in fulfilling their responsibilities in supporting community members to reduce risk and/or successfully adapt. Specific characteristics believed to influence both resilience and adaptation fall under each of the five dimensions. However, no one size fits all ; that is, many of the characteristics appropriate for a particular population (e.g. slum dwellers in Mumbai, India) may not be so for another (e.g. Bolivian shifting cultivationists). As such, each particular suite of characteristics needs to be appropriately specified given the nature of the population in question and the hazards and change processes to which it is likely to be subjected. 3.2 The particular ARR characteristics used in the Nepal Effectiveness Review As mentioned above, there is no one generic set of resilience characteristics that are applicable to all contexts. Given this, efforts were made to specify characteristics relevant to the Dadeldhura climatic and agricultural context. These characteristics are presented in Table 3.1 by dimension, along with a summary rationale for including each. Data was collected on a total of 30 characteristics under the five dimensions As can be seen in the table, data were collected on a total of 30 characteristics. Seven characteristics were defined for the livelihood viability dimension. Several of these characteristics relate to the ability of households to meet their basic needs. Those on the margins of survival are assumed to be in a worse position to cope with drought than are their more wealthy counterparts. The levels of livelihood and crop diversification are also assumed to be important, so that the household in question has something to fall back on in times of stress. The gender risk differential characteristic was further added to examine whether women s livelihood activities are more at risk than are those of men. Finally, a farming household is also assumed more likely to cope better with drought if its crop portfolio is sufficiently diverse. 3 Ibid 7

10 Table 3.1: Specific ARR characteristics used in the Nepal Effectiveness Review Dimension Characteristic Rationale for Inclusion Livelihood viability Household wealth status Household food security Household dietary diversity Livelihood diversification Livelihood risk males in HH Poor households assumed to be more at risk Food insecure HHs assumed to have less viable livelihoods HHs with poorer nutrition assumed to be more at risk HHs with more diverse livelihoods assumed to be at less risk HHs dependent on climate related activities = more risk Innovation potential Access to contingency resources and support Integrity of the natural and built environment Social and institutional capability Livelihood risk females in HH Crop portfolio Attitudes towards new livelihood practices Awareness of climate change Innovation practice Access to credit Access to state innovative support Group participation Social connectivity Perceptions of local government emergency support Savings Remittances or formal earnings Fertility of local soils Extent of soil erosion Access to irrigation for farming Access to safe drinking-water year round Extent of vegetative cover in farm plot Extent farming activities affected by drought Awareness of drought preparedness plan Participation in drought prep. meetings Receipt of drought prep. information Awareness of community level drought risk reduction initiatives Water resource dispute experience Awareness that local leaders are undertaking action Level of confidence in effectiveness of local leaders /institutions HHs dependent on climate related activities = more risk HHs with a greater crop portfolio assumed to be at less risk HHs less open to new practices are less likely to innovate HHs with more awareness in better position to adapt Direct indicator that HH is innovative Enables HH to access resources to support innovation Sustainable access to such support conducive for innovation More opportunities for support in times of crisis More opportunities for support in times of crisis Level of confidence of HHs assumed related to what will actually happen in times of crisis The more savings a HH has, the more it can cope in crises Better access to remittances = better coping in crises High fertility increases productivity High levels of soil erosion decrease productivity Enables yields to be maintained despite rainfall variability More difficulties in access makes it more difficult to cope Greater cover increases groundwater recharge Greater impact of drought on farming = more risk Indicates planning is taking place + public participation Indicates planning is taking place + public participation Indicates that community institutions are fulfilling roles Indicates that community institutions are fulfilling roles Levels of conflict reflect capacity to address disputes Indicates that community institutions are fulfilling roles Level of confidence of HHs assumed related to the effectiveness of the actions of local leaders and institutions 8

11 Five characteristics were defined for the innovative potential dimension. It is assumed that households are more likely to positively adapt to change if they are open to modifying their livelihood practices, are aware that climate change is happening, and have good access to credit and innovation support. In addition, they are assumed more likely to innovate in the future if they have done something new and innovative in the past. As implied by the resilience framework presented in Subsection 3.1, there will be times when even households with significantly viable livelihoods and internal adaptive capacity will find it difficult to cope with serious shocks (e.g. a severe drought). Consequently, having access to both local and external resources and support during such events is clearly advantageous. Five characteristics are defined under the Access to Contingency Resources and Support Dimension. Two of these relate to things directly possessed or received by the household in question: savings and remittances/formal earnings. Being strongly connected to social networks within the community is further assumed to be important, hence the group participation and social connectivity characteristics. However, there will often be times when the state (or, in its absence, other external actors) will need to intervene. Given this, household perceptions on how well they would be supported by local government in the event of a serious crisis were also solicited. Six characteristics were defined for the integrity of the natural and built environment dimension. Given the context, access to water in times of drought is clearly a critically important characteristic. However, given that vast numbers of households are dependent on agriculture, the extent of soil erosion, soil fertility, vegetation cover, and access to irrigation facilitates were also deemed important. This brings us to the social and institutional capability dimension, where data were collected on seven characteristics. The characteristics are intended to measure the strength of community-level institutions to both reduce risk and support adaptation. Hence, such capacity is assumed to be high when community members are significantly aware of and participate in relevant disaster risk reduction and adaption processes, experience minimal conflict over natural resources, and are confident in the capacity of local leaders and institutions. 3.3 Constructing the resilience indices Following the Alkire-Foster 4 method, binary cut-offs were defined for each of the 30 characteristics. A household was coded as being non-deprived if it can be considered as faring reasonably well in relation to the characteristic in question. The particular cut-offs used for each characteristic are presented in Appendix 1. There is inevitably a degree of arbitrariness in defining such cut-offs. However, the results presented in Section 6 also include some alternative measures, which act as a check on the robustness of the results obtained from applying the cut-offs. Each of the dimensions presented in Table 3.1 was then weighted in order to calculate the overall resilience measures. Figure 3.2 presents the higher-level dimensional weights that were used to construct the resilience indices. 4 See 9

12 Figure 3.2: Dimensional Weights Used The 30 characteristics used to create the indices were not weighted equally Integrity of Natural & Built Environment 15% Social and Institutional Capability 15% Access to Contingency Resources & Support 20% Livelihood Viability 30% Innovation Potential 20% It is readily apparent that the dimensions are not weighted equally, something that requires justification. The context in which the project is being implemented in is one in which there are recurrent droughts, floods and other climatic shocks that negatively affect the livelihoods of the targeted population. Consequently, developing strong livelihoods that can still thrive is such a context is absolutely critical, hence, justifying the greater weight given to the livelihood viability dimension. At the same time, climate change is a reality and there is a need for both adaptation and access to resources and support during times of intense stress. Consequently, the innovation potential and access to contingency resources and support dimensions are each weighted at 20 per cent. The integrity of the natural environment in the context is important, but directly relates to securing viable livelihoods. As such, it is strongly connected to the livelihood viability dimension, resulting in it being given less weight. Finally, the ability of the targeted households to cope effectively with stress and adapt to emerging trends and uncertainty is assumed to be more influenced by their own characteristics and efforts, rather than those of local leaders and institutions. Therefore less weight was given to the social and institutional capability dimension. 3.4 The overall resilience measures The first measure of overall resilience that was used to derive the results detailed in Section 6.2 below is the proportion of characteristics the household scored positively on. Furthermore, a household was defined as having positive resilience overall if it met the cut-off for positive resilience in at least two-thirds of these characteristics. A resilience index was created which takes a value of 1 if the household reaches that benchmark for overall resilience and otherwise is equal to the proportion of characteristics the household scored positively on. Finally, the Oxfam GB global indicator for resilience is based on whether each household is doing better in terms of overall resilience than a typical household in the area. This is defined by comparing each household s resilience index with the median of the comparison group. In particular, the global indicator takes the value of 1 if the resilience index is greater than the median of the comparison group and zero otherwise. 10

13 In summary, the three key measures of overall resilience analysed in Section 6.2 are: The base resilience index: the proportion of characteristics for which the household reaches the cut-off for positive resilience. The Alkire-Foster (AF) resilience index: whether the household reaches the cut-off in at least two-thirds of the characteristics, and otherwise equal to the proportion of characteristics for which they do reach the cut-off. The global indicator, based on whether or not the AF resilience index is greater than the median of the comparison group. 4 Impact assessment design 4.1 Limitations in pursuing the gold standard A social programme s net effect is typically defined as the average gain participants realise in outcome (e.g. improved household food security) from their participation. In other words: The Effectiveness Review attempted to ascertain what would have happened in the intervention villages had the project never been implemented Impact = average post-programme outcome of participants minus what the average post-programme outcome of these same participants would have been had they never participated This formula seems straightforward enough. However, directly obtaining data on the latter part of the equation commonly referred to as the counterfactual is logically impossible. This is because a person, household, community, etc. cannot simultaneously both participate and not participate in a programme. The counterfactual state can therefore never be observed directly; it can only be estimated. The randomised experiment is regarded by many as the most credible way of estimating the counterfactual, particularly when the number of units (e.g. people, households, or, in some cases, communities) that are being targeted is large. The random assignment of a sufficiently large number of such units to intervention and control groups should ensure that the statistical attributes of the two resulting groups are similar in terms of their a) pre-programmes outcomes (e.g. both groups have the same average incomes); and b) observed characteristics (e.g. education levels) and unobserved characteristics (e.g. motivation) relevant to the outcome variables of interest. In other words, randomisation works to ensure that the potential outcomes of both groups are the same. As a result provided that threats, such differential attrition and intervention spillover, are minimal any observed outcome differences observed at follow-up between the groups can be attributed to the programme. However, implementing an ideal impact assessment design like this is only possible if it is integrated into the programme design from the start, since it requires the introduction of some random element that influences participation. To evaluate an ongoing or completed programme as in this Effectiveness Review or one where randomisation is judged to be impractical, it is therefore necessary to apply alternative techniques to approximate the counterfactual as closely as possible. 11

14 4.2 Alternative evaluation design pursued There are several evaluation designs when the comparison group is nonequivalent that can particularly when certain assumptions are made identify reasonably precise intervention effects. One solution is offered by matching. Find units in an external comparison group that possess the same characteristics, e.g. ethnicity, age and sex, relevant to the outcome variable, as those of the intervention group and match them on the bases of these characteristics. If matching is done properly in this way, the observed characteristics of the matched comparison group will be identical to those of the intervention group. The problem, however, with conventional matching methods is that, with large numbers of characteristics on which to match, it is difficult to find comparators with similar combinations of characteristics for each of the units in the intervention group. The end result, typically, is that only a few units from the intervention and comparison groups get matched up. This not only significantly reduces the size of the sample but also limits the extent the findings can be generalised to all programme participants. (This is referred to as the curse of dimensionality in the literature.) In an attempt to mitigate bias, two statistical procedures were used propensity score matching and multi-variable regression Fortunately, matching on the basis of the propensity score the conditional probability of being assigned to the programme group, given particular background variables or observable characteristics offers a way out. The way propensity score matching (PSM) works is a follows: Units from both the intervention and comparison groups are pooled. A statistical probability model is estimated, typically through logit or probit regression. This is used to estimate programme participation probabilities for all units in the pooled sample. Intervention and comparison units are then matched within certain ranges of their conditional probability scores. Tests are further carried out to assess whether the distributions of characteristics are similar in both groups after matching. If not, the matching bandwidth or calliper is repeatedly narrowed until the observed characteristics of the groups are statistically similar. Provided that a) the dataset in question is rich and of good quality; b) the groups possess many units with common characteristics (i.e. there is a large area of common support); and c) there are no unobserved differences relevant to the outcome lurking among the groups, PSM is capable of identifying unbiased intervention effects. Multivariable regression is another approach that is also used to control for measured differences between intervention and comparison groups. It operates differently from PSM in that it seeks to isolate the variation in the outcome variable explained by being in the intervention group net of other explanatory variables (key factors that explain variability in outcome) included in the model. The validity of both PSM and multivariable regression are founded heavily on the selection on observables assumption, and, therefore, treatment effect estimates can be biased if unmeasured (or improperly measured) but relevant differences exist between the groups. 5 5 One of the MVR procedures that was used attempted to control for possible unobserved differences between the groups. This is the Heckman Selection Model or 2-step Estimator. Here, efforts are made to directly control for the part of the error term associated with the participation equation that is correlated with both participation and non-participation. The effectiveness of this method, however, depends, in part, on how well the drivers of participation are modelled. 12

15 Both PSM and multivariable regression were used to analyse the data collected for this effectiveness review, and efforts were made to capture key explanatory variables believed to be relevant in terms of the assessed outcomes, e.g. sex and age of household head, educations levels, etc. (see Section 6 below). While no baseline data were available, efforts were made, as explained below, to reconstruct it through respondent recall. This method does have limitations, e.g. memory failure, confusion between time periods, etc. However, for data that can be sensibly recalled, e.g. ownership of particular household assets, it can serve to enhance the validity of a cross-sectional impact evaluation design. The reconstructed baseline data were used in two ways. First, several of the variables included in the PSM and regression procedures were baseline variables constructed from recalled baseline data. One variable, for example, was related to the respondents wealth status at baseline derived through the construction of a household wealth index based on asset ownership and other wealth indicators. This was done in an attempt to control for baseline wealth differences between the intervention and comparison groups. The second way the reconstructed baseline data were used was to derive pseudo difference-in-difference (double difference) intervention effect estimates. With longitudinal or panel data, this is implemented by subtracting each unit s baseline measure of outcome from its end line measure of outcome (i.e. end line outcome status minus baseline outcome status). The intention here is to control for time invariant differences between the groups. Bearing in mind the limitations associated recalled baseline data, using PSM and/or regression and the double difference approaches together is considered to be a strong quasi-experimental impact evaluation design. 4.3 Intervention and comparison villages surveyed A key factor in ensuring the validity of any non-randomised impact evaluation design is to use an appropriate comparison group. This is particularly true for an ex-post, cross-sectional evaluation design. Comparators that differ in relevant baseline characteristics and/or are subjected to different external events and influences will likely result in misleading conclusions about programme impact. Identifying a plausible comparison group is therefore critically important and is, generally speaking, not an easy task in nonexperimental work. The evaluation design made a comparison between households in the intervention villages versus households in matched comparison villages In this case, the project activities under review had been implemented in communities from a VDC area that was specifically chosen as being particularly vulnerable or particularly in need of support in building riskreduction capacity. In order to ensure a plausible comparison, similar villages from neighbouring areas had to be selected. This was done in conjunction with local field staff, who have significant experience in working across Alital VDC. Field staff confirmed that communities in the implementation areas are highly homogenous in the risks they face. In all cases, field staff were therefore able to identify nearby communities that were believed to have similar characteristics to the communities where the project was implemented, and so could be suitable for comparison. 13

16 5 Methods of data collection and analysis 5.1 Data collection A household questionnaire was developed by Oxfam staff and translated by the Consultant to capture data on various outcome and intervention exposure measures associated with the measurements of resilience discussed in Section 3. Demographic data and recalled baseline data were also collected, to statistically control for differences between the supported and comparison households that could not plausibly be affected by the project. The questionnaire was pre-tested first by Oxfam local staff and then by the enumerators during a practice exercise and revised accordingly. A team of 16 enumerators was locally recruited from Dadeldhura district. These enumerators participated in a two-day training workshop, which was led by the Consultant with support from Oxfam staff. The second day of the workshop involved a practical exercise, carrying out the questionnaire in a community in Alital VDC. Following this exercise, the performance of each of the enumerators was reviewed individually before their appointments were confirmed. The enumerator team was divided into three groups and mobilised to different areas in the VDC to ensure smooth movement from one community to another. The movement plan was created in consultation with the field supervisor to ensure that completed surveys were collected and reviewed at the end of every day. Feedback was provided regularly to all enumerators regarding their performance. A total of 437 households were interviewed 173 from intervention villages, and 264 from comparison villages Communities were informed in advance through a local contact person of the enumerators arriving to survey households. Using village household lists, respondents from both intervention and comparison villages were selected randomly and mobilised appropriately. No major problems were encountered during the fieldwork except for in one case where a local leader of one of the selected villages refused to allow just one village in the Ward to be surveyed. Asladi-Dobato was one of the selected comparison communities with a total of 62 households. The coordinator of the local Citizen s Ward Forum requested that all 200 households in the Ward be surveyed or none at all. After attempts to explain the purpose of the survey, and the impracticality of surveying all 200 households failed to convince the local leader, two other communities were selected instead Chaud, with a total of 49 households, and Chiraan, with an estimated households. These have comparable background characteristics to project communities with regard to their socioeconomic status, risk profile, market access, and agricultural practices. Table 5.1 shows the numbers of households interviewed in each community in the survey. A total of 437 households were interviewed, of which 173 were in project communities and the remaining 264 in comparison communities. The work of the enumerators was closely monitored and scrutinised by the Consultant and Oxfam staff. Oxfam s Oxford-based adviser spent the first two days of the survey monitoring the interviews, reviewing the completed questionnaires, and providing detailed feedback to the consultant and enumerators to ensure that the appropriate quality standards were met. 14

17 Table 5.1: Intervention and comparison villages and sample sizes Intervention Villages Matched Comparison Villages Village Name HHs interviewed Village Name HHs interviewed Jharkanda/Makwanpur 49 Bhandi/Aaakhali 60 Nauli 24 Lamijule 29 Chain 28 Dayani 20 Tekundanda 21 Chaud 42 Dandakharkha 11 Kantipur 19 Dolakot 13 Chaudala 28 Melganja 27 Mamur 23 Chiraan 43 Totals Data analysis Data-entry tools were developed in Adobe Acrobat Pro. Date-entry from the completed questionnaires was done in the Oxfam office in Dadeldhura by a team of four temporary staff, managed by the Consultant. Data analysis was performed in Stata by staff from OGB s office in Oxford. The results of this analysis are presented in Section 6. Most of the analyses involved group mean comparisons using t-tests, as well as PSM with the psmatch2 module and various multivariable regression approaches. Kernel and nearest neighbour matching without replacement were used to implement PSM. Variables used in the matching process were identified by using backwards stepwise regression to identify those variables correlated with being in an intervention village at p-values of 0.20 or less. Covariate balance was checked following the implementation of each matching procedure, and efforts were made to ensure that the covariates were balanced across groups at p-values greater than Bootstrapped standard errors enabled the generation of confidence intervals to facilitate statistical hypothesis testing. (See Appendix 2 for further details.) All the covariates presented in Table 6.1 were included in the various regression approaches undertaken, i.e. regression with robust standard errors (to address issues of heteroscedasticity), robust regression (to reduce the influence of outliers), and regression with control functions (to attempt to control for relevant unobserved differences between the intervention and comparison groups). 5.3 Main problems and constraints encountered Despite some of the logistical difficulties related to operating in fairly remote parts of Dadeldhura district, the data collection process was completed successfully. However, some factors were encountered which affect the analysis process and the interpretation of results presented in Section 6: Some significant differences baseline and demographic differences between intervention and comparison groups 15

18 Some significant baseline and demographic differences were found between the intervention and comparison populations As presented in Section 6.1 below, there are some systematic differences between the intervention and comparison groups in terms of the baseline and demographic characteristics reported in the survey. In the analysis of the outcome measures, both PSM and regression procedures were used to control for these differences to the greatest extent possible. However, in the case of the former, some of the supported households were dropped because of the absence of appropriate comparison households. In particular, eight of the 173 intervention households were dropped using the PSM kernel model, and eight were dropped using the no-replacement model. This means that the estimates of differences in outcome characteristics between the various treatment groups only apply to those intervention households that were not dropped; that is, they do not represent the surveyed population as a whole. Measurement challenges using the Likert scales Efforts were made to measure five of the 30 resilience characteristics using six-item, four-point Likert scales. These characteristics include: 1) attitudes towards new livelihood practices; 2) awareness of climate change; 3) social connectivity; 4) perceived effectiveness of local government emergency support; and 5) perceived effectiveness of local leaders and institutions in supporting drought preparedness. The statements for each of the scales were mixed at random and placed in two sections of the household questionnaire. At the analysis stage, the extent to which the respondents agreed with each was subsequently analysed. Ideally, there should be a high degree of internal consistency with respect to the level of agreement the respondents had with the six statements developed for each of the five characteristics. In other words, each respondent should have agreed with each statement associated with each characteristic-specific set in a similar way, given that these statements are intended to measure the same underlying latent construct. The degree of such internal consistency is often measured using Cronbach s alpha statistic. Unfortunately, for the awareness of climate change characteristic the alpha statistic was low (0.60). This means that the respondents were not responding to the items related to awareness of climate change in a significantly consistent fashion. Consequently, there is reason to be suspicious of how accurately this characteristic was measured. That being said, the other four characteristics measured using the Likert scales show high degrees of internal consistency. 16

19 6 Results 6.1 General characteristics Table 6.1 presents statistics for various household characteristics obtained through the administration of the questionnaires to the respondents from both the intervention and comparison villages. The stars indicate differences between the groups that are statistically significant at a 90 per cent confidence level or greater. Table 6.1: Descriptive statistics for intervention and comparison respondents Intervention Comparison Difference t-statistic mean mean Respondent HH head Respondent female Distance to district centre (hrs) * 1.66 Distance to nearest market (mins) ** 2.14 HH female headed ** Head has secondary education Adult has secondary education HH elderly headed Age of head of HH Head of HH is productive Household size Number of adults in HH Number of children in HH Number of young children in HH Number of dependents Number of unproductive adults Number of productive adults HH has only one adult HH grew crops at baseline HH reared livestock at baseline *** 2.80 HH ran off-farm IGA at baseline ** 2.10 HH did casual labour at baseline ** 2.25 HH did formal work at baseline Household received remittances at baseline ** 2.09 HH wealth index at baseline (PCA) % income from farming at baseline *** % income from rearing livestock at baseline % income from IGA at baseline * 1.79 % income from casual labour at baseline % income from formal labour at baseline % income from community at baseline % income from remittances at baseline * 1.77 Land area used for crops at baseline Observations * p<0.1, ** p<0.05, *** p<0.01 As is evident, there are some significant differences between the intervention and comparison groups. In particular: Households in the project communities are on average further from the district centre and nearest market. Households in the comparison communities are significantly more likely to be female-headed than households in the project communities. 17