SNV WASH Project in Ethiopia

Transcription

1 SNV WASH Project in Ethiopia Final impact evaluation Commissioned by SNV Netherlands October 2015 Project number 2529 Aidenvironment Barentszplein NJ Amsterdam The Netherlands + 31 (0) info@aidenvironment.org

2 Contents Abbreviations and definitions Executive Summary iv v 1. Introduction 1 2. Methodology Baseline study Overview methodology Survey methods, sampling and analyes User and facility level: households and schools Water schemes Capacity building efforts Analysis of result pathways Limitations of the methodology Results Households Access to safe water Access to improved sanitation Incidence of diarrhea and analysis Access to information and training on WASH Additional analysis on explaining factors Schools Access to safe water in schools Access to improved sanitation in schools Functionality of WASH club and analysis WASH related school management and analysis School attendance and analysis Water schemes Functionality of water schemes Performance of the WASH committees Preventive and Operational capacity training by SNV Comparison with baseline situation Capacity building Capacity building interviews and focus group discussions per woreda Interviews at regional level Interviews at the Hawassa TVETC Interviews at UNICEF Hawassah branch Conclusions from the regional level interviews Analysis and conclusions Summary of main findings Comparison of woredas School level findings Analysis of impact pathways Impact pathway 1. Household access to improved sanitation Impact pathway 2. Schools access to water and improved sanitation 40 Project number 2529 ii

3 4.2.3 Impact pathway 3. Improved management of water schemes Impact pathway 4. Capacity building of institutions responsible for WASH Evaluation questions 45 Annex 1: Theory of change (reconstructed by the evaluation team) 51 Annex 2: Evaluation Questions (extract from ToR) 53 Project number 2529 iii

4 Abbreviations and definitions AKVO CLTSH FGD GDL GLOWS MDG M&E MHM MIS NGO O&M POM SNNPRS SNV SPSS TTLM TVETC UAP UNICEF VIP WASH Akvo (organization in the Netherlands) Community Led Total Sanitation and Hygiene Focus Group discussion guided distance learning Guided Learning on water and Sanitation Millennium Development Goals Monitoring & Evaluation Menstrual Hygiene Management Management Information Systems Non Governmental Organisation Operations and Maintenance Preventive Operations and Maintenance Southern Nations, Nationalities and People Regional State Netherlands Development Organisation Statistical Packages for Social Scientists Teaching Training learning Materials preparation Technical vocational education training center Universal Action plan United Nations Children s Fund ventilated improved pit water sanitation and hygiene Definitions Improved sanitation facility: one that hygienically separates human excreta from human contact. (WHO definition). The improved sanitation facilities include flush/pour flush (to piped sewer system, septic tank, pit latrine), ventilated improved pit (VIP) latrine, pit latrine with slab, and composting toilet. Unimproved sanitation facilities include: public or shared latrine, open pit latrine or bucket latrine Secure sanitation facility: sanitation facility with secured privacy (in schools) Traditional sanitation facility: open field defecation or traditional latrine facilities that are pit latrines without a slab. Project number 2529 iv

5 Executive Summary Background In the period of 2007 to 2012, SNV implemented a WASH project in 6 woredas (districts) of the SNNPRS (Southern Nations, Nationalities and People Regional State) in Ethiopia. The project aimed at increasing access to water supply, sanitation services, and improved sanitary practice in the communities and at schools. The project introduced the CLTSH (Community led Total Sanitation and Hygiene) approach in the 6 woredas. The project chose a model of close collaboration with, and capacity building of, the local government offices, focused at strategic planning, improved service delivery, community involvement, monitoring and evaluation. Aidenvironment carried out an impact evaluation of this project in 2015, with the purpose to assess the continuing impacts of the SNV WASH project, especially the improved access to improved water and sanitation for the target groups as well as the effects on the capacities of the government functionaries in the WASH sector. Methodology The impact evaluation focused at household level, school level, water schemes and capacities of institutions within targeted woredas and regional level. Mixed methods were used including structured and semi-structured surveys, in-depth interviews and focus group discussions. The household and schools surveys focused on access to safe water, access to improved sanitation and information and training on WASH. The surveys were translated in local language and use was made of digital data collection. In total 1450 household interviews were conducted, selected from 58 kebeles (rural municipalities) in the 6 woredas. In total 50 schools were selected and 100 school interviews (50 pupils and 50 teachers) were conducted, equally distributed over the 6 woredas. Survey data were analysed using SPSS, descriptive statistics and inferential analysis. In total 50 water schemes were visited, with surveys to determine functionality. To evaluate capacity building effects of relevant staff (within the WASH, education and health sectors) at woreda level, the 5C (5 capabilities) approach of evaluating capacity building was used as a basis for interviews, using a balance score card system specifically developed for this study. The study did not include control groups for households, schools or water schemes. The study did include two non-targeted woredas, as control groups, to make a comparison in capacities between staff within targeted and non-targeted woredas. To establish the main changes in time, the study compared the current values with the baseline values (from 2007). This posed some challenges because the baseline data and the approach used were not well defined. Also, a reliable midterm measurement survey was missing. To draw conclusions on the contribution by the SNV project, the quantitative and qualitative results were used to assess whether defined impact pathways (based on the project s theory of change) could explain the changes in time, including the relative contribution by the SNV project and other (external) factors. Findings In the analysis of access to water, in line with the SNV baseline survey we used three criteria: a) the type of main water source and its safety (by interview and observation), b) the travel distance or travel time to the safe main water source being less than 1.5 km or travel time not being more than 30 minutes, and c) the access of clean water storage. When taking into account these three criteria, access to water for households in the project area has declined from 53% in 2008 to 46.4% in Currently travel time was more than 30 minutes for 40.7% of the respondents, which is still better than the average of 62% for rural areas in Ethiopia, according to the Ethiopia Demographic and Health survey of In the analysis of access to improved sanitation, we adopted four criteria, in line with the baseline survey: a) the presence of an improved latrine facility; b) cleanliness of the latrine, c) regular use of the latrine, and d) cleanliness (no feces remains) in the compound. Project number 2529 v

6 When taking into account these criteria, we found that an increase of access to improved sanitation within households from 27% to 41.4%. Looking at the various factors that influence access to sanitation (according to the impact pathway), we conclude there is a significant relation and plausible contribution by the SNV project to the improvement in access to improved sanitation. There is also a significant correlation between access to safe water and access to improved sanitation for households, suggesting that access to safe water influences improved sanitation. The decline in the incidence of diarrhoea is highly significant (from 52% to 17%), and can be partly explained by the improved access to improved sanitation. Concerning the schools, access to water has significantly improved (from 7% to 22%, or 52% when water sources beyond the school premises are also considered). In the schools 94% have a latrine facility, but access to improved sanitation declines to 28% if we also take into account aspects of cleanliness and safety. This can be explained by limited access to water within the school compounds or limited management by teachers of WASH within schools (only 26% scores very good). Most student-based WASH clubs (72%) are functional. We cannot conclude there is a significant relation by the SNV project to access to improved sanitation in schools but a contribution by the SNV project is plausible based on evidence from interviews and focus group discussions. To determine functionality of water schemes, three criteria were used: water discharge, water quality and waiting time. It was found that for the majority (72%) of water schemes in the project area functionality has improved. The SNV project has contributed to the improved functionality of water schemes in the project area, by training on operations & maintenance and supporting local WASH committees. For instance, the time it takes before a reported failure is repaired has considerably reduced. However, for 70.6% of the water scheme users, waiting time at the water scheme was more than 30 minutes (with a maximum of 5 hours), reducing their functionality. We conclude there is a significant contribution by the SNV project to the improved functionality of water schemes, based on evidence from interviews and focus group discussions This also means that access to water would probably have been lower (than 46.4%, see above) if SNV had not contributed to improved functionality of water schemes. The remaining low functionality of water schemes is mainly due to low volumes of water from existing water schemes in relation to high and increasing water demand, with underlying causes of relatively few new water schemes being constructed or low water volume per water scheme. Opinions of woreda staff are positive with respect to the services being provided by the SNV project. At woreda level, strategic planning, service delivery, community involvement and monitoring have much improved. The targeted woreda s have better capacities for WASH strategic planning and WASH service delivery than control woreda s. The model of training-on-the job (GLOWS) has been very effective in terms of improving service delivery. The response rate to repairs has increased. Use of the Management Information System (MIS) has failed due to lack of support and continuity at regional level. We conclude there are positive changes in capacities of woreda level staff to manage WASH related aspects and plausible evidence for a significant contribution by the SNV project. With respect to sustainability of the results, most capacity building results will sustain, such as the integrated strategic planning approach (education, health and WASH) and the improved service delivery. Also, most local water schemes are well maintained with WASH committees (88%), including fees being charged. Among the targeted woreda s, there is some evidence that the targeted woreda s are better able to access funds from donors for investment objectives, presumably because of their improved planning and funding requests. Nevertheless, funds for investments in new water schemes and large-scale maintenance of existing water schemes remain insufficient. The GLOWS approach is reported to be costly and still not formally recognised, and therefore uncertain to be continued. Altogether, there are insufficient recurrent financial inputs to the WASH sector in the Project number 2529 vi

7 region, e.g. through fees or taxes. From an environmental angle, it is worrisome that water boreholes have to go deeper and yet cannot supply sufficient water for the whole population in the region. This may lead to depletion of underground water resources. More attention could be given to rainwater capture and/or underground storage. There may also be need for collaboration and integration with natural resources and watershed management for improving vegetation coverage around water sources so as to improve water recharge. Thus, there are remaining challenges, which are mainly related to external factors, including the need for institutional support to use the MIS and continue the use of GLOWS, additional hardware especially on new water schemes, operations and maintenance of large water schemes to increase water discharge, and improved motivation of staff and improved skills on specific tasks. Conclusions and recommendations The approach taken by SNV in the WASH sector is innovative by focusing on capacity building of local public institutions, on awareness raising and change of practices, involving training institutes and networking. This focus by SNV on the software was part of a programme partnership with UNICEF that focused on the hardware. Overall, this evaluation shows that the SNV approach has generated significant results, but the hardware component now seems to constitute the main constraint. The contribution by SNV is positive in relation to change of behaviour on sanitation and hygiene at household level, improved maintenance of water schemes, build up of capacities at woreda level, among the communities, WASH committees and WASH clubs at schools. Also, there is increasing private sector involvement in the WASH sector. Although the funding from national sources to the WASH sector has improved for the 6 targeted woreda s, funds for investments in new water schemes and large-scale maintenance of existing water schemes remain insufficient. Considering the remaining challenges, there is a potential role for SNV to play as follows: Scaling the successes achieved in the sanitation sector, by capitalizing on the significant correlation between access to improved sanitation for households and the support and training provided; Strengthen the WASH monitoring and evaluation system within SNV and with local partners, including proper establishment of a baseline measurement, and including a technology to monitor water discharge of water systems and waiting time; Strengthen a joint learning and evaluation system and assure continuity in follow-up or refresher training, to assure further improvement and/or to avoid fall-back; Take the lead in proactively engaging in advocacy and lobby to narrow the existing gap on demand for safe water, by lobbying for budget increases to develop additional water facilities and developing more sustainable solutions ; Further strengthen private sector engagement in the WASH sector; Pay more attention to rainwater capture, underground water recharge and storage and watershed management, to avoid water depletion and to increase access to water from sustainable sources. Project number 2529 vii

8 1. Introduction The SNV WASH project In the period of 2007 to 2012, SNV implemented a WASH project in 6 woredas (districts) of the SNNPRS (Southern Nations, Nationalities and People Regional State) in Ethiopia. The project aimed at increasing access to water supply, sanitation services, and improved hygiene practice in the communities and at schools. The project introduced the CLTSH (Community led Total Sanitation and Hygiene) approach in the selected woreda s. To assure sustainability of its efforts, the project chose a model of close collaboration with, and capacity building of, the local government offices. Capacity development focused at community needs assessment, planning and introducing a Management Information Systems (MIS). It supported the woreda s to develop their WASH strategic plan aligned with the UAP (Universal Access Plan) at national level. Another element of the project was to improve Operations and Maintenance (O&M) of selected water schemes. To do so, within each woreda the technicians of the WASH team were mobilised and trained on O&M aspects. In each woreda, water schemes were supported on O&M through the local WASH committees (committee responsible for O&M of the water scheme) to properly manage user fees by users, keep the water scheme clean, install a fence and establish effective drainage. At schools the approach of multi-stakeholder inquiry was applied. This a form of action research to raise awareness and then make a plan to improve WASH performance. To do so, SNV selected schools in each woreda. SNV also introduced the MHM (Menstrual Hygiene Management) for girls at schools (introduction of low-cost pads). Within the selected schools, SNV also introduced the establishment of WASH clubs and stimulated these clubs to come together and develop an action plan. SNV also worked with the local vocational college (TVETC) to improve their curriculum and develop practical skills of their graduates in operating and maintaining water and sanitation facilities. This was later on expanded with an approach of guided distance learning (GDL), which was formulated as GLOWS (Guided Learning on water and Sanitation). Based on the above project headlines, the evaluation team reconstructed the theory of change (see Annex 1) and a number of impact pathways (see chapter 4). The theory of change distinguishes between a range of software or capacity building activities, and assumes that either the hardware is already available or that resources for hardware are made available by other partners. The partnership with UNICEF would assure that hardware (water schemes) could be provided where this was deemed necessary and useful. The functionality of water schemes is an important element of the theory of change. The assumption was also that upscaling would take place as a result of increased awareness, increased finance allocation for WASH hardware provision by the district, and improved capacities within woredas to manage the existing water schemes. This impact evaluation study Aidenvironment has been commissioned the assignment to carry out an impact evaluation of this project. The purpose of the evaluation documented in this report is to assess the continuing impacts of the above SNV WASH project. It will include assessing the effects on the capacity and ways of working of the government functionaries in the WASH sector as well as any lasting effects on improved access to improved water and sanitation for the target groups in these woredas. The findings will be used both for accountability to the donors and other stakeholders, and for organisational learning to improve SNV s practice in WASH and of working with local governments in particular. Project number

9 The scope of the evaluation can be characterized as follows: focus on the WASH project in 6 woreda s of the SNNRPS region in Ethiopia. focus on the OECD/DAC criteria impact, effectiveness and sustainability. cover the period from the start of the project until now. The effort will be to assess the changes that the project brought and whether those changes have been sustained until now or not. focus on the 6 project woreda s in the SNNP region and comparison areas/ woredas selected for the comparison of assessment in the SNNP region. A set of research questions was formulated that form the basis for this evaluation (see Annex 2). This report An inception report has been delivered in February To do so, Aidenvironment studied the available documentation and had a 3-day workshop in Ethiopia (28-30 January), together with the local consultants and relevant SNV staff, to prepare the impact study. The fieldwork was carried out in March-April. Data were received in May and this draft report was prepared in June Following feedback by the Steering Committee a final version of this report was prepared in August This report is organized by the following chapters: Chapter 2: methodology followed for the field data collection and analysis of the data Chapter 3: results of the data collection and analyses, for household survey, school survey, water scheme survey and capacity building analysis separately Chapter 4: conclusions, validation of impact pathways and responses to the main evaluation questions. There are a series of appendices in a separate document that provide all the basic data. Project number

10 2. Methodology 2.1 Baseline study According to the ToR the SNV WASH project conducted in 2007 a baseline study as well as in 2011 a midterm evaluation study. During the inception phase we reviewed the quality of these studies. Baseline study In 2007 an extensive baseline study was executed. The main characteristics of the baseline study executed in 2007 are the following: Baseline survey undertaken in the selected 6 woredas, the Kebele and major urban centres are the basic unit for the survey in each Woreda The basic units for data collection and observations are the households, the institutions, Kebele WaSH/Water scheme committees, Woreda WaSH team and Town WaSH Teams. The data were collected by local capacity builders, Woreda WASH teams, Kebele baseline survey teams and interviewers at the Woreda, Kebele, institutions and water source levels. Both quantitative and qualitative data were collected by interviewing of household members and informants from institutions based on a structured questionnaire and also by observations. 50 households were sampled randomly as respondents per Kebele. For all schools, Health centers, Markets and drinking Water sources surveys were done. A team of 3 interviewers each interviewing 7-8 households in a day. A Kebele baseline survey team consisting of 1 man and 1 woman was trained at each Kebele In each Woreda and Kebele, a focus group discussion was held with the Woreda WaSH team and Kebele WASH/water scheme committees to collect relevant WaSH information. During verification of the baseline data collected in 2007 several woredas found the data did not correspond to their perception of the reality of the field. Especially, access to sanitation data were found to be too positive. This was assumed to be mainly the result of the fact that many households construct latrines but do not use them, or have constructed very simple latrines because this was a requirement to be able to receive food aid. These latrines were not used and therefore cannot be qualified as access to sanitation. Apparently enumerators too easily provided a positive qualification of access to sanitation if only a simple latrine had been produced. Therefore, in 2008 a verification survey was carried out, with the following approach: Adopt both observation and interviewing of households and institutions in the Kebeles. In total 58 kebeles were selected from the 6 woredas (see in below table) Out of 50 households interviewed in the previous survey, 25 were selected randomly and reinterviewed. Each data collector interviewed 5 units (households and or institutions) per day. Female teachers collected the data and fewer questions were asked. Female teachers were preferred because they were presumed to be more credible and therefore unlikely to lie about the WaSH status and they could easily win the confidence of the household members and therefore be allowed to observe cleanliness in the households and water storage containers. The teachers signed an agreement with SNV to provide accurate data. A Woreda employee not associated with the WaSH activities supervised the data validation process. The verification process generated new and more reliable data, leading to corrections on the previous set of baseline data. The impact evaluation team does not fully understand how the validated data were combined with the initial survey data to result in a final value, which is the reason why we took as the baseline the data from the verification survey (see relevant tables in chapter 3). Project number

11 As verified during the inception phase, according to SNV the verified access to water baseline data have taken into account: distance to the water points (at most 1.5 km) cleanliness of the water storage containers As verified during the inception phase, according to SNV the verified access to sanitation data have taken into account: use of the latrines (if a latrine is not used, it falls in category of open defecation) cleanliness of the latrines (if latrines are unclean it would assume open defecation). absence of faeces in the compound. Midterm evaluation 2011 During the inception phase it was found that SNV did not carry out a midterm evaluation survey. However, surveys have also been carried out by the Water bureau of the SNNPRS and consultants have made use of these data in case study and other reports for SNV. For instance, a report for SNV entitled report on six SNV supported woreda s in SNNPRS makes use of the 2008/09 and 2010/11 inventory of water access by the SNNPRS (mentioning that these are unpublished data). However, there are different reasons why these data cannot be used: we do not have access to the survey sheets and do not know which questions were asked and how the data had been interpreted and analysed; the data for 2011 vary in almost every available document; moreover, the 2011 data in the formal report of this survey (which only appeared in 2014) are all different from the data mentioned in the case study reports and show outliers that cannot be correct (this is even admitted in the report). There is also available a Demographic and Health Survey (DHS) from However, this survey does not provide data at woreda level. We did make use of the DHS survey questionnaire to compile our own questionnaire. 2.2 Overview methodology The evaluation focused at four different levels of evaluation: household user level; facilities (school) level (schools), water schemes level and local government capacities level. In terms of facilities, only schools were assessed, because at the level of markets and health facilities no significant effects are expected. A variety of methods was used to collect data to assess the impact of SNV s WASH-project in Ethiopia: structured and semi-structured surveys, in-depth interviews and focus group discussions (FGD). Table 1 provides an overview of the methods used at the three evaluation levels. It should be emphasized that the study did not include control groups for households, schools or water schemes. On the contrary, the study did include control groups at woreda level. This was done according to the ToR and is understandable for two main reasons. First, as direct effects of the SNV WASH project we expect results at the level of local woreda agencies, reason why controls are included. According to the theory of change capacity building will result in effects at household, school and water scheme levels. However, most likely these effects will expand beyond the focus woredas, also in view of the fact that other NGOs may have interventions that directly target households, schools and water schemes. Thus, controls at woreda level are in line with the fact that we want to test the specific SNV WASH theory of change. The second reason is pragmatic, being the high costs that would be related to control groups at household, school and water scheme levels. Project number

12 Table 1. Overview of methodology User level (households) and facilities (schools) Type of method Before-after comparison: comparing baseline study data with impact data Mixed methods: quantitative survey and qualitative FGD Surveys based on baseline survey, for comparison Contribution analysis based on impact pathways Water schemes In-depth interviews and FGDs Interview scheme based on questionnaire baseline study Contribution analysis based on impact pathways Institutional capacity building efforts Control group approach: selection of 2 Woredas for comparison In-depth interviews and FGDs Interview scheme based on questionnaire baseline study Contribution analysis based on impact pathways For each of the 4 levels mentioned in table 1, we will describe briefly the methodology implemented. 2.3 Survey methods, sampling and analyes User and facility level: households and schools Survey construction For household and school level, separate surveys were constructed: a structured survey for household level interviews and a semi-structured survey for school level interviews. Both surveys have been developed in collaboration with SNV Ethiopia and local consultants. The household surveys focused on 3 topics: access to safe water, access to sanitation and awareness, received support and information. The school level surveys has separate sections for students and teachers, covering as topics: access to safe water, access to improved sanitation, membership of WASH-club, management of WASH-facility (teachers only) and school attendance (teachers only). Note that these surveys were designed as based upon the baseline survey formats in order to allow for before (baseline)-after comparisons. Both surveys can be found in appendix 1. Both surveys have been translated into local language and have been uploaded in the AKVO tool to allow for digital data collection. Data collection The surveys for households and schools were conducted in the period between March 15 th and April 7 th, The household and school surveys were conducted by a team of 10 local enumerators, in collaboration with and supervised by the 2 local consultants. All local enumerators and consultants were trained extensively beforehand on the understanding of the questionnaire, the use of the AKVO tool, and the ethical criteria to be respected. The enumerators and consultants also received training in using the AKVO tool, a tool for data collection and analysis. A survey coordinator was present during data-collection to support the team, to ensure sampling procedures were followed and to perform quality checks on data collected. As four different local languages are spoken in the 6 surveyed woredas, local translators were deployed to translate the local language to Amharic. These local translators were selected from each surveyed village, taking into account a) their ability to speak the local language, b) their ability to speak the Amharic language and c) neutrality and prevention of bias by making sure the translators did not work for the kebele administration and/ or WASH-sector. Project number

13 Households: sampling In total 1450 Household interviews were conducted using the AKVO -tool and included in the analysis. 1 Due to technical reasons the data of 24 interviews were not saved into the tool and could therefore not be included in the analysis. These 24 interviews are from different villages, so did not create a bias. In total, data of 1426 interviews are available for analysis. The 1426 households were randomly selected from 58 kebeles (rural municipalities), situated in 6 woreda s (districts). To ensure comparability, the woreda s and kebeles that were selected for the survey are identical to those selected in the baseline study. Selection of villages within the kebeles was made at random: for each of the 58 kebeles a list of villages was made and, using the random numbers-method, 1 village per kebele was selected. Likewise, the selection of households within the selected villages was made at random: per village a list of households was prepared and the random numbers method was used to select 30 households per village. Note that at average there are about 10 villages per kebele. The number of households per kebele generally varies between 40 and 70. Schools: sampling In total 50 schools were selected and 100 school interviews (50 pupil interviews and 50 teacher interviews) were conducted, using the AKVO tool. For 2 woreda s a list of SNV-targeted schools was available. In these woreda s we selected all targeted schools for the survey (8 schools in Misrak Badewacho and 5 schools in Kedida Gamella). The selection of the remaining 37 schools in the other 4 woreda s took place by randomly selecting schools from the villages where the household surveys were carried out. School characteristics (such as first cycle primary, second cycle primary, secondary, preparatory) were included in the sampling procedure, to ensure different types of schools were included in the sample. Data Cleaning and Analysis The data submitted to the server for storage were accessed on AKVO dashboard and inspected by the survey coordinator for quality check on daily basis throughout the data collection period. At the end of the data collection for the quantitative household and school surveys, the 1426 data forms for the household survey and 100 data forms for school survey submitted to the dashboard were exported into EXCEL for making ready to import into SPSS (Statistical Packages for Social Scientists). The three EXCEL data sets (one for household survey, and two for teachers and students surveys in the school) were converted into SPSS files for cleaning, processing and analysis. Then the household, student and teacher surveys were used for generating quantitative insights which were interpreted and analysed on apparent correlations between parameters. In analyzing the quantitative data in SPSS, descriptive statistics and inferential analysis were carried out. Descriptive statistics, like percentages and ratios, were calculated for describing the key statistical results. Chi-square and correlation tests were done for assessing association between selected dependent variables (which are incidence of diarrheal diseases) and the independent/explanatory variables like access to safe drinking water and improved sanitation. Note that in the analysis of criteria for access to water, in line with the SNV baseline survey we introduced three criteria: a) an observation is made of the safety of the main water source, b) travel distance to the safe main water source is less than 1.5 km, and c) people have the possibility of clean water storage. We also looked at waiting time because this could be a more reliable indicator than distance to the nearest water source. 1 To calculate the number of household interviews, the following formula for determining sample size was used: n = ((z2 x p (1-p)) /d2, where z2 stands for confidence level of 90%, p stands for prevalence of WASH indicators, d stands for acceptable difference/ margin of error of 5%. Project number

14 In the analysis of access to improved sanitation, we also adopted four criteria, in line with the baseline survey, as follows: a) observation that there is an improved latrine facility; b) cleanliness of the latrine, c) regular use of the latrine, and d) cleanliness (no feces remains) in the compound. Note that an improved latrine facility for MDG monitoring is defined as one that hygienically separates human excreta from human contact. The improved sanitation facilities include flush/pour flush (to piped sewer system, septic tank, pit latrine), ventilated improved pit (VIP) latrine, pit latrine with slab, and composting toilet. This compares with open field defecation or traditional latrine facilities that are pit latrines without a slab Water schemes For assessing impact on the water schemes, focus group discussions (FGD) were held. The checklist of questions for the FGDs is based on the baseline survey and has been constructed in collaboration with SNV Ethiopia and the local consultants. The FGD checklist can be found in appendix 1.4. In total 50 Water schemes have been visited, consisting of boreholes, shallow wells, hand dug wells and springs. Of the 50 water schemes that were surveyed, 25 were constructed before 2007 (when the project started) and 25 were constructed after Also, SNV support that was provided to these water schemes was a selection criterion, so that half of the old and new water schemes received maintenance support from SNV and the other half did not. Thus we have 4 categories: 12 constructed before 2007 and no SNV operation and maintenance support received 13 constructed before 2007 and SNV operation and maintenance support received 12 constructed in or after 2007 and no SNV operation and maintenance support received 12 constructed in or after 2007 and SNV operation and maintenance support received The focus group discussions were carried out with users and WASH committees (WASHCO s). Based on the questionnaire results, the water schemes and WASH committees were assessed and classified in terms of their current functionality (of water schemes) and operations and maintenance performance of the WASH committees. Functionality of water schemes To determine functionality, we developed 3 criteria to assess water schemes: water discharge, water quality and waiting time. We defined three levels of water scheme functionality, as follows. Functionality level of water schemes Good Partly Poor Criteria Discharge volume = good or low / less during dry season, and Water quality= good or moderate / lower during rainy season, and Waiting time = <30 min Discharge volume = low / less during dry season, and Water quality = moderate or lower during rainy season, or Waiting time = >30 min Discharge volume = not functional, or Water quality = low Please note that in the above classification of functionality we may deal with cumulative (and) criteria or with facultative (or) criteria. The criterion of waiting time more than 30 minutes was introduced Project number

15 because of the very high proportion of users that have to wait long to be able to collect water. In the literature, a waiting time of more than 30 minutes is commonly considered as being too long. 2 Performance of the WASH committees To determine performance, WASH committees were assessed on 7 criteria: presence of WASH committee, satisfaction of users about WASH committee, charge of fees, fees used for Operation & Monitoring (O&M) and satisfaction of fee use, incidence of conflict and resolution of conflicts. Using these criteria, we defined 2 levels of performance, as follows. Performance of WASH committee Good Poor Criteria Presence of WASH committee Satisfied with WASH committee, or partly satisfied Fees charged Fees used for O&M Satisfied with use of fee No incidence of conflict or resolution of conflicts Absence of WASH committee Users not satisfied with WASH committee Conflicts existent and/or not solved Capacity building efforts Survey construction The checklist of questions for evaluating capacity building has been developed in collaboration with SNV Ethiopia and the local consultants. As a basis we took the capacity building interventions that took place during the SNV project period, and the 5C approach of evaluating capacity building (as also being adopted by SNV). The survey checklists can be found in appendix 1.4. Separate surveys were designed for the 4 different levels where interviews were held. To assess the impact of capacity building efforts, focus group discussions (FGD) and in-depth interviews were performed at 4 levels. Level Method Number Woredas (8 woredas, of which 6 targeted and 2 control woredas) - Individual in-depth interview - FGD WASH team Regional authorities Individual in-depth interviews 3 4 TVETC Individual in-depth interviews 1-2 UNICEF Individual in-depth interviews per district - 1 per district Sampling method In total 8 woredas were visited, including the 6 SNV-targeted woredas and 2 control woredas. The 2 control woredas (Baloso Bombe and Angecha) were selected on the basis of the following criteria: presence of water schemes a bit comparable to the target woredas; near / not too far from the target woredas (for reasons of logistics and comparison); no past interventions by SNV, no major interventions by other NGOs. 2 Queuing time at a water source is no more than 30 minutes. See : Project number

16 Appendix 2.2 provides names of consulted respondents, through individual interviews or FGDs. Scoring capacity building In order to classify the wealth of information generated on capacity building by using the surveys, we developed a balance score system to assess capacity building efforts on 7 aspects, in line with the 5C approach of evaluating capacity building efforts as being used and promoted by SNV: Strategic planning of WASH sector Finance for WASH sector WASH service delivery Gender aspects in WASH Communication and private sector participation in WASH setor WASH database and monitoring and evaluation system Partnerships for support to the WASH sector. The balance score card makes an assessment of the information for each of the above 7 aspects, as indicated in the below assessment matrix. Strategic planning Financing WASH delivery Gender and youth aspects Current situation score (1= lowest score) There is no strategic plan (1) Strategic plan follows the Growth and Transformation Plan (GTP), but little or no special attention for WASH (2) There is a strategic plan with special attention for WASH (3) Low priority (awareness) and no adequate budget for WASH and high dependency on donors (1) There is high priority and budget for WASH but not yet sufficient, dependency on donors (2) There is high priority and adequate budget for WASH ( 3) No or limited WASH services are provided, or are not of good quality (1) WASH services are provided but do not fully meet demand client (2) WASH services are provided in terms of coverage and utilization, are adequate in terms of coverage and quality needs of client (3) Limited or no involvement in WASH and no specific service delivery for women, youth and vulnerable groups(1) Good involvement of women,youth and vulnerable groups, OR specific service delivery is provided ( 2) What has changed score (a=lowest) No improved focus on WASH in strategic plan (a) Improved design of strategic plan but no improved result (yet) or lessons learned (b) Improved design and implementation of strategic plan, possibly leading to improved WASH services (c) No improvement of budget and / or priority (a) Improved priority (awareness) but no increased budget (b) Increased budget and improved priority for WASH (c) No change or no improvement in service delivery capacities (a) Improved capacity of staff but does not translate into improved services (b) Improved capacity of staff to deliver WASH services and improved services delivered (in terms coverage and quality) (c) No change in gender, youth and vulnerable social groups inclusion, or situation has become worse (a) Improved gender, youth and vulnerable group inclusion OR improved gender, youth, vulnerable group focus / priority (b) Project number

17 Community and private sector participation Data base, monitoring and evaluation Partnerships Good involvement of women, youth vulnerable groups AND specific WASH service delivery is provided for women and youth (3) No or little participation of community in WASH (1) Communities participate in WASH but there is not sufficient capacity (fees, skills, human) (2) Community (and private sector if applicable) participates in WASH, expresses their own demands for support(3) There is no management information system (MIS) or data base on WASH and no resources for WASH monitoring (1) There is a MIS or data base on WASH but it is not used or not up-to-date (2) There is a MIS and up to date data base on WASH (3) No partners active in the field that contribute to WASH strategy (1) Some partners active in the field that contribute to WASH strategy (2) Many partners active in the field that contribute to WASH strategy (3) Improved inclusion of gender aspects in WASH activities and service delivery ( c) No improvement in community participation, in terms of water payment and maintenance (a) Improved community participation, but sill insufficient to meet maintenance needs (b) Improved participation community, enough to meet maintenance needs (c) No improvement or change in WASH data base and monitoring (a) Introduction or involvement of staff in M&E activities or MIS, but no follow-up (b) Improved involvement of staff in WASH M&E activities or MIS, adequate resources (c) Partners have left the area (a) The level of support has not changed (b) Level of support for WASH strategy has increased (c) 2.4 Analysis of result pathways The results of the three levels of evaluation were analysed and interpreted to assess whether the identified impact pathways could be validated or not. To do so we combined the quantitative data (of the household and school surveys and the water schemes) and compared these with the qualitative data and scores on capacity building based on the FGDs and individual interviews, to draw conclusions whether impact pathways have been realized or not. Combining quantitative and qualitative scores is a challenge when using a mixed approach. In our study, we have given scores to the capacity building results according to the 5C approach (see previous section) in order to be able to match the scores with quantitative results per woreda. However, we found the variation between woredas to be minimal. Also, we looked at specific qualitative information to support or disqualify quantitative data. 2.5 Limitations of the methodology The above methodology has been worked out as part of the inception report and submitted to the Steering Committee that was assigned for this impact evaluation (with representatives of the donor, SNV and an impact evaluation research institute). It has been approved with some minor adjustments that were incorporated in the final version. In spite of the rigorous method, the impact evaluation met the following limitations: Project number

18 One is the fact that the impact evaluation results were supposed to be compared with baseline data. The baseline data appeared to be difficult to interpret, in spite of a verification survey of which the final results are more reliable. Also, the criteria for assessment of access to water and access to improved sanitation are not always clear. Also, a midterm assessment of the project results was considered too unreliable. As a result, uncertainties remain with respect to the comparison between baseline and impact data, and reliable trends were not always easy to establish. In the design of the impact evaluation method we used as much as possible the same criteria as had been used during the baseline survey. However, as noted above, in some cases the exact criteria were not celar, e.g. to determine access to improved sanitation. Also, some adjustments had to be made in order to stay in line with national (Ethiopia) and international standards. Therefore, the methodology was not exactly the same as had been applied during the baseline survey. There are no controls at household or school level. However, we believe that having controls at this level would have been costly and there were no baseline data for control woredas (reason why it was not part of the ToR). The matching of quantitative and qualitative data remains a challenge but we believe to have developed an approach to generate conclusions that make best use of both sources of information. Ideally, one would either carry out a qualitative survey and then test resulting insights through quantitative surveys, or the other way round carry out a quantitative survey and validate the findings by a qualitative survey. We have done neither because the quantitative and qualitative surveys were done in parallel (at the same time). Thus, for instance, we did not validate through focus group discussions the quantitative finding that WASH clubs or WASH management capacity at school did not make a difference in access to sanitation at schools. Lastly, it remains difficult to fully attribute observed changes or differences to the SNV WASH project, due to the fact that there are many intervening NGOs as well as government services. Over the last few years, both have been also been active in the WASH sector. However, it has been possible to draw firm conclusions on the contribution by SNV to the observed changes, which in most cases are very positive. Project number

19 3. Results 3.1 Households Access to safe water Access to safe water is first of all based on the statement by the respondent if there is access to a water source. We then included in the analysis of access to safe water the following three criteria, in line with the SNV baseline survey: a) an observation is made to judge the safety of the main water source, b) travel distance to the safe main water source is less than 1.5 km, and c) an observation is made if people have the possibility of clean water storage. When just considering the presence of a water source (looking neither at distance nor at the possibility of safe storage) 88% of the respondents have access to safe water. When using the above criteria only 46% of the respondents have access to safe water (Table 2). These respondents have access to water that is safe, relatively nearby (under 1.5 km distance) and that can be stored safely. Almost 54% of the respondents do not have access to safe water. When looking at access to safe water without taking into account the possibility of safe storage, 53% of the respondents report having access to safe water. Travel distance to collect water from safe main source is the variable which has the highest influence on household s access to safe drinking water. Among households who collected water from safe (protected) sources, close to one-half (47%) travelled over 1.5 km for getting water from these safe main sources, which cannot be considered as access to safe water. We found that for 40.7% of the respondents travel time to a water scheme was more than 30 minutes, while for 70.6% of the respondents waiting time at the water scheme was more than 30 minutes. Table 2: Access to safe water in the 6 woredas together, according to three criteria. Access to safe water Access to safe main water source (no criteria) Access to safe source (with first two criteria) Access to safe water (with all three criteria) Count (%) Count (%) Count (%) Yes No Total It is interesting to observe that the proportion of households with a source of water at more than 1.5 km distance (47%) is comparable to the proportion of households that have to travel more than 30 minutes (40.7%). When looking at woreda-level, results show that access to clean water, varies among woredas. Highest is 70% in Misrak Badewacho having access to safe water, compared to the lowest score of 17% in Demboya. Results for these and other woredas are summarized in Table 3, with indication of the baseline values and the conclusions with respect to trends. Project number