Final Evaluation Report

Transcription

1 Final Evaluation Report Project #1: Introduction of innovative and affordable Self-Supply technologies for sustainable improvements in water supply and sanitation in Kenema and Western Rural Districts Project Number: SLE-1014/P 4087 Project duration: Project #2: Development of a strategy to accelerate household investment in improved water supply (Self Supply) Project Number: WF004 Project duration: Project Holder #1 Deutsche Welthungerhilfe e.v. Project Holder #2 WaterAid Co-financed by Department for International Development Implemented by WASH Facility Sierra Leone Evaluation carried out on behalf of Deutsche Welthungerhilfe e.v. Friedrich-Ebert-Straße 1 D Bonn, Germany Evaluator Dipl.-Ing. (FH) Mario Gelhard, MSc. (Independent Consultant) Dauner Straße Köln, Germany info@gelhard-consulting.de Cologne, February 6, 2014

2 Acknowledgements and Disclaimer I would like to thank all people involved in the evaluation. My thanks go to the Welthungerhilfe team in Bonn and in Freetown for the good organisation of the mission, the sharing of relevant information and for answering my questions patiently and for taking the findings as a learning exercise. Special thanks go to the Welthungerhilfe country representative Mr. Jochen Moninger, the project manager Mr. Ensah Bunduka and the Monitoring & Evaluation officer Mr. Juma Bah who supported the evaluation in any conceivable way and accompanied me in my field visits. I am indebted to the country manager of WaterAid, Mr. Christian Kekurah who contributed a great part to the successful evaluation mission. Moreover I want to thank Mr. Will Tillett from the WASH Facility Sierra Leone for sharing his knowledge on self-supply and the insights of the two projects with me. At the same time I want to express my thanks and respect to the local communities, the local authorities and all the others who shared their time with me in discussions and interviews. The results of the evaluation and the report are the sole responsibility of the evaluator and can in no way be taken to reflect the views of Welthungerhilfe. Cologne, Freetown, February

3 Contents Summary Brief description of the project and framework conditions Relevance Effectiveness Efficiency Outcomes and impacts Sustainability Most important recommendations General conclusions and "Lessons Main Learnt" 1 3 text Preliminary Remarks Reason for -and aim of the evaluation Evaluation process Evaluation methods Limitations Description of the project 1 6 3

4 1.1 Brief project description Framework conditions Brief description of the target group Relevance Relevance to core problems of the target group Relevance to the objectives of Welthungerhilfe Relevance to the objectives of the partner country Effectiveness Achievement of project purpose Project preparation Quality of target group, actors and situational analyses Extent to which self-help approaches and structures are taken into account Project planning Quality of the Project Planning Matrix (PPM) Analysis of the results framework Realism of project purpose

5 3.3.4 Appropriateness of staff, material and financial planning Quality of the analysis of project holder Organizational structure and focal points of work Staff, equipment and logistics Project steering and financial administration Steering by the project office and cooperation with the project holder Quality of the project execution Presentation and evaluation of the project outputs and activities Involvement of the target group and project cooperation Overall evaluation of the execution Occurrence of assumptions and risks Efficiency Cost/benefit ratio for the overall project Quality of the internal project M&E system Direct and indirect, short, medium and long-term OUTCOMES and IMPACTS 4 8 5

6 5.1 Use of Outputs / Outcomes Economic outcomes Socio-cultural outcomes Impacts Economic impacts Socio-cultural impacts Environmental impacts Contribution of the project to the Millennium Development Goals Sustainability Organizational and institutional/political The status of self-supply in Sierra Leone and the way forward Policy and strategy status The appropriateness of the EMAS technology to support selfsupply Results of TAF assessment on the EMAS technology in the Western Rural and Kenema districts Barriers to scaling up decentralised technologies in water and sanitation provision Specific recommendations for the scale up

7 8. Conclusions and Recommendations Project-specific conclusions Project-specific 2 9. General recommendations... 6 conclusions Important lessons learnt Positive examples from the project (good practice) References Annexes 6 6 7

8 List of abbreviations and acronyms A4P ASI BMZ BU CLTS DIFD DWO EC ECOWAS FAO GDP GIZ GoSL HDI HH HoP HQ INGO lpcd MoEWR MoHS MoWR NGO O&M ODF PPM RWH SALWACO SSL TAF TG ToR UNICEF VIP WASH WD WHH WHO Agenda for Prosperity Adam Smith International Federal Ministry for Economic Cooperation and Development Business Units Community Led Total Sanitation Department for International Development District Water Office European Commission Economic Community of West African States Food and Agriculture Organisation of the United Nations Gross Domestic Product Deutsche Gesellschaft für international Zusammenarbeit GmbH Government of Sierra Leone Human Development Index Household Head of project Headquarters International Non-Governmental Organisation litres per capita and day Ministry of Energy and Water Resources Ministry of Health and Sanitation Ministry of Water Resources Non-governmental organisation Operation & Maintenance Open defecation free Project Planning Matrix Rain Water Harvesting Sierra Leone Water Company Leone Technology Applicability Framework Target group Terms of Reference United Nations International Children s Emergency Fund Ventilated Improved Pit Water, Sanitation and Hygiene Water Directorate Welthungerhilfe / German Agro Action World Health Organization 8

9 List of Tables Table 1-1: Projects overview Table 1-2: The different target groups of Welthungerhilfe SLE Table 1-3: Calculation of beneficiaries related to the outputs of project SLE Table 1-4: Calculation of beneficiaries related to outputs of project WF Table 3-1: Outputs of WF Table 3-2: Quantitative achievements of SLE Table 3-3: Quantitative achievements of WF Table 3-4: Evaluation of the main measures of SLE 1014 (Welthungerhilfe) Table 3-5: Evaluation of the main measures of WF004 (WaterAid) Table 3-6: Assumptions & risks defined in the project proposal of SLE Table 3-7: Assumptions & risks defined in the project proposal of WF Table 4-1: Total project budget of SLE-1014 (status November 2013) Table 4-2: Total project budget of WF004 (status November2013) Table 4-3: Measures for internal project monitoring & evaluation Table 5-1: The projects contribution to the MDG Table 6-1: Welthungerhilfe sustainability criteria and evaluation Table 7-1: Approaches and classification to communal, government borne water supply Table 7-2: Major barriers to scaling up self-supply Table 7-3: Specific recommendations for scaling up self-supply in Sierra Leone Table 8-1: Project specific recommendations Table 9-1: Important lessons learnt Table 9-2: Good practice examples from the project List of Boxes Box 7-1: Six principal objectives relating to WASH are detailed in the A4P List of Figures Figure 3-1: Low-cost components installed at private HH in Kenema Figure 3-2: Low-cost components installed at private HH in Western Rural Area Figure 3-3: Intervention sectors Welthungerhilfe Sierra Leone Figure 3-4: Organisational structure of SLE Figure 5-1: Private installations in nine districts Figure 5-2: Private installations per quarter in nine districts Figure 7-1: Graphical profile showing the results of the TAF scoring

10 SUMMARY 1. Brief description of the project and framework conditions The DFID sponsored WASH Facility Sierra Leone managed by Adam Smith International (ASI) has invested into management support of the Ministry of Water Resources and its implementing institutions GUMA Water Company and SALWACO targeting the revision of policies and regulations. In addition the WASH Facility has supported technically and financially two pilot projects on slef-supply acceleration implemented by the two INGOs Welthungerhilfe and Water Aid. Both projects have started in July 2012 with a time frame of 20 months and come to an end February 28, Welthungerhilfe: The project focused on the promotion of low-cost technologies which are appropriate to support WASH self-supply efforts of individuals in urban, peri-urban and rural settlements. The project targeted the qualification of 80 technicians and practitioners, including advanced training of 10 of the 80 as master trainers on EMAS technology, the implementation of 30 demonstration sites using range of EMAS WASH technology for marketing, the increase of demand for self-supply WASH schemes due to public awareness raising and community outreach, the establishment of 15 start-up businesses entering the market and improved sector learning competence and strengthened capacity of stakeholders with an allocated budget of 263,095. WaterAid: The project seeks to develop a set of practical guidelines for accelerating private investment in water self-supply and improving the standards of supplies financed by users themselves to complement community supplies aiming at increasing demand for supply construction or incremental up-grading to provide a safe, convenient water supply. The project was launched on October 12, 2012 in Kenema but actual work begun in August 2012 with an allocated budget of 318, Relevance Traditional investment in the WASH sector in Sierra Leone is still primarily in the hands of the government and/or NGOs and is implemented as community projects (e.g. community had dug wells and installed hand pumps). People s willingness to pay for the maintenance and repair of the communal services and to voluntarily manage the water supply systems is limited through the lack of ownership. The peoples perception still is that water supply should be provided for free either by the government or INGOs. Private investment into improved individually used WASH systems is still low key and at its beginning without any real recognition and support from the government. According to the Progress on Water and Sanitation and Drinking Water Update 2013, the national rate of people using improved water sources is at 57% and for the use of sanitation facilities the picture becomes even more challenging since only 13% of the population nationwide is using improved facilities. The strategy document Rural water supply and small towns (2013) introduces self-supply as 10

11 complementary to conventional communal supply, which is generally government or NGO funded and which forms the backbone of rural water supply. 3. Effectiveness Welthungerhilfe: The targeted outputs were mainly achieved at the time of the evaluation (January 2014). Some outputs were overachieved and some are slightly behind schedule. 25 out of 30 demonstration sites were established and the project management should consider ceasing the implementation of the last five and rather spending time and resources into monitoring and quality management activities. The trained technicians were all involved in setting up the demonstration sites and did the installations at the private households. The owners who invested in self-supply technology are all involved in the maintenance and repair of the supply structures and show a great sense of ownership. They are willing to provide information on frequently asked questions from the interested public and refer to the technicians for a quotation or further detailed information. WaterAid: Four out of the five targeted outputs were not achieved at the time of the evaluation. This was mainly due to an overambitious proposal and the overwhelmed organisational structure of WaterAid in Sierra Leone. The project management applied for a non-cost extension until June 2014 to finalise the commenced activities. 4. Efficiency The two projects did not follow the classical approach of a WASH project which is delivering WASH services to a defined number of beneficiaries but rather on the promotion and piloting of the selfsupply concept to increase coverage and WASH service levels. Welthungerhilfe: The project aimed at targeting 5, individuals. This results in a cost/benefit ratio of the overall project of per beneficiary. The project was able to generate significant funding (on top of the initial funding) with initiatives resulting directly from project activities, e.g. IFAD contract. Overall the project delivered a good value for the money invested. WaterAid: The project aimed at targeting 4,315 direct beneficiaries. This results in a cost/benefit ratio of the overall project of per beneficiary. Around 50% of the total budget was allocated for the development of a) study on the potential of accelerating self-supply in Kenema and Pujehun b) comprehensive training needs assessment including a self-supply training package 11

12 c) study of the financing of self-supply initiatives as well as the development of a self-supply acceleration package At the date of the evaluation only one study (output 1) was completed. The other outputs are still under revision. 5. Outcomes and impacts Welthungerhilfe: In this project two groups of direct beneficiaries are targeted where an economic benefit can be assessed. 1. Private households willing to learn about the concept and the technology and willing to explain it to anyone interested serving as demonstration sites: For these households the benefits were more on the side of enhancing convenience by increasing the service levels. Therefore neither an economic outcome nor an economic impact could be assessed by the project. 2. Technicians who will be trained on the EMAS technology and the self-supply approach: 75 technicians and practitioners received training on several components of the EMAS technology. Out of this trained technicians the best 10 were chosen to be further promoted to master trainers. 11 business units were established comprising 5-10 trained technicians. The project was able to conduct private installations beyond the Western Rural Area and Kenema Districts (project locations) additionally to seven other districts. WaterAid: The project is fare behind schedule in achieving the proposed outcomes. Only one out of five outputs has been accomplished at the time of the evaluation making a sound evaluation of the outcomes and impacts nearly impossible. 6. Sustainability The project of Welthungerhilfe has convinced and motivated the beneficiaries to participate actively in the programme execution and has engaged people in self-development efforts creating a good sense of ownership. The project was able to expand the implementation of privately funded installations beyond the two districts targeted in the proposal to additional seven districts in Sierra Leone. This supports to believe that there is potential for up-scaling the self-supply concept in Sierra Leone. 12

13 7. Most important recommendations Corresponding to Table 8-1: Project specific recommendations on page 58 the project specific recommendations are: No. Recommendations WHH Directed to WaterAid WASH Facility Priority 2.2 Relevance to objectives of Welthungerhilfe Integrate the self-supply concept in other sectors of 1 Welthungerhilfe Sierra Leone such agro-forestry, natural resource management and food security and nutrition x medium 3.1 Achievement of project purpose 2 Assess the number of private installations to correlate them to the demand. x medium 3 Monitor the performance of the business units for at least 6 months. x medium Quality of the project planning matrix (Logframe) 4 Don t mix outputs and indicators. Develop consistent Logframe x medium Analysis of the results framework Reconsider the execution of output 2. Reduce the 5 amount of piloting activities to a number which is manageable with the personal resources of Water Aid in Sierra Leone x high Presentation and evaluation of the project outputs and activities 6 Don t put more efforts in establishing the last 5 demonstration sites x high 7 Don t put more efforts in establishing the missing business units x high 8 Review EMAS quality standards and develop checklists for supporting monitoring activities x high 9 Make the reports available to the sector x high Occurrence of assumptions and risks 10 The rating of the risks in the project should be realistic x medium 5.2 Impacts 11 Allow for a consolidation phase in-between the pilot phase and the strategy development x high 8. General conclusions and "Lessons Learnt" Corresponding to Table 9-1 on page 63 the lessons learnt are: 1. The underlying approach of executing the steps of piloting > strategy development > up-scaling consecutively, as intended by the WASH Facility, is inexpedient because it does not allow for consolidation of the results of the pilot phase. 2. To create a perceptible impact on the WASH sector in terms of self-supply in Sierra Leone the efforts of NGOs have to be bundled and well-coordinated. 13

14 3. Low-cost technologies and private supplies are not necessarily associated with lower levels of water quality. All private HH interviewed drink the water from their improved wells right away, however HHWTS concepts have to be strengthened in the self-supply concept. 4. Self-supply strengthens ownership and hence the lifespan of WASH infrastructure 5. Addressing the needs and capacities of the poorest households with self-supply initiatives is the greatest barrier to overcome in order to ensure they are not neglected. MAIN TEXT 0. Preliminary Remarks 0.1 Reason for -and aim of the evaluation In achieving common (development) goals, impact orientation plays an increasingly important role in international development cooperation. To reliably prove the effective and efficient use of the provided funds and their contribution towards the development process in the partner countries, Welthungerhilfe is called to conduct independent impact oriented evaluations at the end of each project. The primary aim of the independent evaluation is to analyse the status of the self-supply Sector in Sierra Leone to formulate recommendations on whether and how to roll-out the WASH Self-Supply approach in Sierra Leone by analysing the impacts of the pilot project Development of a strategy to accelerate household investment in improved water supply (Self Supply) implemented by Water Aid and the pilot project Introduction of innovative and affordable Self-Supply technologies for sustainable improvements in water supply and sanitation in Kenema and Western Rural Districts implemented by Welthungerhilfe, and to derive lessons learnt for the WASH sector This includes the analysis and the assessment of the current WASH policy framework in Sierra Leone the perceptions about the pilot projects by the local national and communal administration or line agencies the applied EMAS technology and whether it is a good and applicable technology to supporting WASH self-supply in Sierra Leone barriers for scaling up self-supply roles and responsibilities of stakeholders in scaling up self-supply in Sierra Leone The foundations for the evaluation are provided in the concept paper "Outcomes and Impact-Oriented Evaluation of the Work of Welthungerhilfe". The evaluation is based on DAC 1 criteria (efficiency, effectiveness, relevance, impact and sustainability) and the standards of the DAC and DeGEval 2. 1 Development Assistance Committee 2 Gesellschaft für Evaluation e.v. 14

15 0.2 Evaluation process The preparation for the evaluation started in the end of November 2013 with the study of the relevant project documents and an initial briefing by the desk officer and the staff of the internal audit and evaluation unit at Welthungerhilfe headquarters in Bonn on December 13, The visit to Sierra Leone took place from 6 th until 19 th January 2014 without any incident, except that the departure to Freetown was delayed by one day, due to cancelation of the flight from Brussels to Freetown on the 5 th January The initial briefing in Sierra Leone was carried out by Mr. Jochen Moninger and complemented by the national project coordinator Mr Ensah Budunka. The necessary information on the WaterAid project was provided by Mr. Christian Kekurah, the WaterAid Program Manager in Sierra Leone. The evaluator in cooperation with Welthungerhilfe and WaterAid developed an evaluation time- and work schedule which was constantly adapted in the course of the mission (kindly refer to Annex 2). During the field visits (kindly refer to Annex 3 and 4 for a list of the visited sites and institutions) the evaluator was accompanied by Welthungerhilfe staff and partly by Mr Christian Kekurah from WaterAid. During the whole course of the evaluation no other person was involved in the evaluation. Two debriefings were held in Sierra Leone, one on January 16 at the Welthungerhilfe office in Freetown and another at the Civil Services Training College in front of staffs of the Water Directorate. After returning to Germany, a third debriefing was held with the project manager and the country representative of Welthungerhilfe and the project manager of WaterAid via Skype. This was necessary because both the head of project on Welthunerhilfe and WaterAids side could not be present during the debriefings held in Freetown. Further a draft report was written and submitted to the internal audit and evaluation unit at Welthungerhilfe Bonn on February 6 th A concluding discussion took place on February 20 at the Welthungerhilfe HQ, where the evaluation report was officially acknowledged. 0.3 Evaluation methods Based on the terms of reference (kindly refer to Annex 1) the evaluation took into account the five DAC criteria mentioned above. The evaluation was conducted in three the districts Western Rural Area, Kenama and Pujehun and was complemented with stakeholder meetings and site visits in the capital Freetown. During the evaluation the following methods were applied: Project documentation and literature review and assessment Field visits to demonstration and private sites implemented during the project Transect walks across the project sites and observation of framework conditions and structures set up by the project Semi-structured interviews and focus group discussions with target beneficiaries, producers and providers of the WASH technology, national and local government representatives Application of the Technology Applicability Framework (TAF) 15

16 0.4 Limitations Due to the fact that the mission was conducted before the implementation of the project was finally completed it was only partially possible to measure the impact of the implemented hard- and software components of the project at the time of the evaluation mission. 1. Description of the project 1.1 Brief project description The DFID sponsored WASH Facility managed by Adam Smith International (ASI) has invested into management support of the Ministry of Water Resources and its implementing institutions GUMA Water Company and SALWACO targeting the revision of policies and regulations in the water sector. In addition the WASH Facility has supported technically and financially two pilot projects on accelerating self-supply implemented by the two INGOs Welthungerhilfe and Water Aid. Both projects have started in July 2012 with a time frame of 20 months and come to an end February 28, Welthungerhilfe: The project focused on the promotion of low-cost technologies appropriate to support WASH selfsupply efforts of individuals in urban, peri-urban and rural settlements 3. The self-supply approach has been practiced over the last decade in several African countries and has potential to create significant impacts in water supply as proven by examples of Ghana, Mali, Uganda or Ethiopia. Self-supply encourages the incremental improvement of household and community supplies through user investment in water treatment, supply construction and up-grading, including small rainwater harvesting and groundwater systems as well as sanitation services such as VIP latrines. It is a concept which complements conventional water and sanitation services implemented by the government, enabling self-help improvement of supplies where no protected supply is available, or where consumers feel they can support higher levels of service than are presently provided by the public sector[1]. The latter, however, is not equally sustainable everywhere, and is inadequately funded to reach MDG target coverage in Sierra Leone. The project targeted at qualifying 80 technicians and practitioners, including advanced training of 10 of the 80 as master trainers on EMAS technology, the implementation of 30 demonstration sites using range of EMAS WASH technology for marketing, the increase of demand for self-supply WASH schemes due to public awareness raising and community outreach, the establishment of 15 start-up businesses 3 The project piloted a range of low-cost technologies (built from local materials) developed by the INGO EMAS International. EMAS International has successfully introduced the technology in Latin America, Africa and Asia. The main objective of EMAS range of technology is to empower the user and to provide a major innovation in the sector by introducing low cost schemes where local technicians in close collaboration with households produce their own WASH hard ware, install them and conduct repair services. These comprise e.g. self-made hand pumps, upgrading of traditional wells, micro irrigation systems, storage tanks, shower cubicles, kitchen sink, rain water harvesting systems (rain capture, sediment removal and storage), and low-cost VIP latrine design. 16

17 entering the market and improved sector learning competence and strengthened capacity of stakeholders. The project targets around 5,930 4 individuals in Western Rural Area and Nongowa chiefdom in Kenema District. The implementation is being carried out between July 9, 2012 and February 28, 2014 with an allocated budget of 263,095 ( 218,996.62). (Kindly refer to Table 4-1 for a budget breakdown). It has to be mentioned that this project is not following the classical approach of a WASH project which is delivering WASH services to a defined number of beneficiaries but rather the promotion and piloting of the self-supply concept to increase coverage and WASH service levels. WaterAid: The project seeks to develop a set of practical guidelines for accelerating private investment in water self-supply and improving the standards of supplies financed by users themselves to complement community supplies. Such investment could help fill some of the gaps in public financing in the medium term, increasing coverage especially in peri-urban and more remote rural areas. The project aims at increasing demand for supply construction or incremental up-grading of water supply structures to provide a safe and convenient water supply. A study of potential will first be carried out to learn from public and private investment that already exists and the interests of unserved communities. Based on the findings, a model will then be tested to develop complementarities between public and private supplies and to address the constraints which are inhibiting private investment. Piloting the creation of an enabling environment with councils and private sector in two districts will promote and facilitate private investment in good quality rainwater harvesting (RWH), shallow well construction and up-grading and household water treatment and storage (HWTS) to reach national minimum levels of service in steps affordable to the end-user. The approach at community level will partly be a water supply extension to CLTS, with similar incremental steps, requiring capacity building among and promotion by, all stakeholders. The capacity building aspect will be done by conducting a comprehensive assessment on existing training needs and available training materials for artisans and developing guidelines, manuals and IEC materials as a package for conducting trainings for well diggers, masons and other artisans used for up-grading and construction. Project activities will be confined to two districts, Kenema and Pujehun, and to a maximum of three chiefdoms within both district [2]. The project targets around 4,315 5 individuals. The project was launched on October 12, 2012 in Kenema but actual work begun in August 2012 with an allocated budget of 264,735 ( 318,000). (Kindly refer to Table 4-1 for a budget breakdown). 4 Numbers as specified in the project proposal 5 Numbers as specified in the project proposal 17

18 Table 1-1: Projects overview Welthungerhilfe Country/Region: Sierra Leone, Freetown, Western Rural Area Peninsula District-Ward C, Kenema District: Nongowa Chiefdom Project title: Introduction of innovative and affordable Self-Supply technologies for sustainable improvements in water supply and sanitation in Kenema and Western Rural Districts Project no: SLE Project holder: Welthungerhilfe Approved budget: Own (WHH) funds: Co-financer: Project period: (as stated in the proposal) WaterAid Country/Region: DFID/WASH Facility: (basic funding) (amendment and project extension) EMAS International: Sierra Leone, Pujehun and Kenema Project title: Development of a strategy to accelerate household investment in improved water supply (Self Supply). Project no: WF 004 Project holder: WaterAid Approved budget: 264, ( 318,000.00) Co-financer (line): Project period: (as stated in the proposal) DFID/WASH Facility: 200,000 (basic funding) 64, (amendment and project extension) Framework conditions Sierra Leone Sierra Leone is remarkable as one of the most water rich countries in West Africa, yet also one of the poorest, where water scarcity and grossly inadequate sanitation threaten to compromise the gains of a hard won peace. After a ten year civil war which put most of its water supply and sanitation infrastructure out of service, Sierra Leone has achieved a fragile stability, and a growing economy, yet the poverty indicators are not encouraging. Sierra Leone ranks very low on most human development indicators placing it at an overall 176 th position out of a total of 177 countries on the Human Development Index, 2006, surpassed only by Niger [3]. 18

19 The climate is tropical and there are two distinct seasons determining the agricultural cycle: the rainy season from May to November, and a dry season from December to April. The average temperature is 26 C and varies from around 26 C to 36 C during the year. Average rainfall is highest at the coast with 3,000 5,000 mm and decreases moving inland to 2,000-2,500mm. Western Area Rural District The Western Area Rural District is one of the 14 Districts of Sierra Leone. It is located mostly around the peninsula, in the Western Area of Sierra Leone. The Western Area Rural District has an estimated population of 197,098. The district capital and largest city is Waterloo[4]. Pujehun District Pujehun District lies in the Southern Province of Sierra Leone. Its capital is the town of Pujehun. As of 2010, the district has an estimated population of 292,543. The district of Pujehun occupies a total space of 4,105 km² and comprises twelve chiefdoms[5]. Kenema District Kenema District is a district in the Eastern Province of Sierra Leone. Its capital and largest city is Kenema, which is also the third largest city in Sierra Leone. Kenema District has an area of 6,053 km² and comprises sixteen chiefdoms[6]. 1.3 Brief description of the target group Welthungerhilfe chose as target region for the project the Western Rural Area and Nongowa chiefdom in Kenema. The target groups can be categorised in: Table 1-2: The different target groups of Welthungerhilfe SLE Direct beneficiaries 1. Private households willing to learn about the concept and the technology and willing to explain it to anyone interested serving as demonstration sites 2. Technicians who will be trained on the EMAS technology and the self-supply approach. The selection of the participants was done based to a rating and scoring related to the needed competencies such as welding and hardware installation experience and other related mechanical skills Indirect beneficiaries 1. Individuals targeted in the outreach activities who might become potential buyers if the technology 2. Households implementing the technology through the trained technicians in the run of the project 3. WASH sector professionals targeted in regular stakeholder meetings The first group of direct beneficiaries are private households which able and willing to invest in lowcost technology to gradually improve their water and sanitation situation. In Kenema 12 demonstration sites were established. 8 out of the 12 sites were placed at homes of section chiefs. The demonstration sites were all labelled with signboards indicating basic information on the project and a pricelist with the range of products and contact numbers. 19

20 In the Western Rural District 13 demonstration sites were established following the same approach as in Kenema. Homes of chiefs, businessman and -women and compounds of NGOs were the places of choice. The second group comprise technicians and artisans willing to undergo professional training on components of the EMAS technology and later be able to establish private start up service businesses with hardware shops and spare parts stocks. Various stakeholder meetings were held in both districts to determine selection criteria for the trainees. Criteria included experience in construction work e.g. carpentry, masonry, plumbing, welding and other related crafts. Application forms were developed which had to be submitted to the district councils for drawing up a shortlist (please refer to Annex. Announcements on the training program were broadcasted in several radio stations in the project locations. Interviews were held with potential technicians at the District Councils. Table 1-3: Calculation of beneficiaries related to the outputs of project SLE Output Number of beneficiaries Category 1: Qualification of 80 technicians and 80 beneficiaries (of which 10 are Direct beneficiaries practitioners, including advanced training of 10 of the 80 as master trainers on EMAS technology trained to a higher level) 2: 30 demonstration sites using range 30 demonstration sites x 50 direct Direct beneficiaries of EMAS WASH technology are available and functional consumers = 1,500 beneficiaries 3: Increased demand for Self-Supply 18 outreaches each 150 Indirect beneficiaries WASH schemes due to public awareness raising and community outreach participants: 2,700 beneficiaries 4: 15 start-up businesses are established during the project implementation each of the 15 start-up business shall sell at least 5 systems benefiting an average of 20 persons Indirect beneficiaries (Start-up businesses are already included in the beneficiaries of output 1) at least= 15x5x20=1,500 persons 5: Improved Sector Learning approx. 150 sector professionals Indirect beneficiaries Competence and Strengthened Capacity Building of Stakeholders Total 5,930 WaterAids project sites are in three chiefdoms in Pujehun and Kenema districts where supply coverage is low, well-digging, rainwater harvesting or household water treatment and storage is feasible, and chiefs are keen to support self-supply initiatives. As already mentioned in paragraph 1.1 the project is not beneficiary driven, since it is about developing of a strategy to accelerate household investment in water supply to support the self-supply approach. However the calculation of the beneficiaries seems not to be consistent. For example basis for the calculation of the beneficiaries of the well-upgrading activities is a household size of 5 persons, 20

21 whereas a household size of 10 persons was taken for the calculation of the beneficiaries of the improved household water and treatment activities. Table 1-4: Calculation of beneficiaries related to outputs of project WF004 Output Number of beneficiaries Category 2: 100 unprotected water sources upgraded and are being used as improved source for drinking water by December 2013 (piloting of self-supply) 15 households x 5 pers/hh x 50 upgraded wells= 3,750 Improved household water treatment: 50hh x 10pers/hh=500 Direct beneficiaries Trained artisans=45 Stakeholders Training (ToT)= 20 Total 4, Relevance 2.1 Relevance to core problems of the target group Traditional investment in the WASH sector in Sierra Leone is still primarily in the hands of the government and/or NGOs and is implemented as community projects (e.g. community had dug wells and installed hand pumps). People s willingness to pay for the maintenance and repair of the communal services and to voluntarily manage the water supply systems is limited through the lack of ownership. The peoples perception still is that water supply should be provided for free either by the government or INGOs. Private investment into improved individually used WASH systems is still low key and at its beginning without any real recognition and support of the government. The electricity sector, however, gives evidence that Sierra Leoneans are ready to invest in individual self-supply systems in form of private generators or solar systems. This investment is judged to add an important value of comfort to the daily life. Water supply is even more important than electricity to the daily life and as the consumer realizes an enhanced comfort in daily life through time savings for water fetching or cleaner and more comfortable latrines, individual households will invest in their water and sanitation system and foster direct private investment in the WASH sector, rather than just relying on government/ngo led community WASH services. The subsequent spread of the range of technologies will lead to greater accessibility to water and sanitation and therefore a healthier nation. More job opportunities are created as the scheme scales up[7]. 2.2 Relevance to the objectives of Welthungerhilfe As formulated in the Welthungerhilfe s Country Program 2013 for Sierra Leone the objective is to combine the core sectors, agro-forestry and agriculture, with other sectors, such as natural resource management, WASH and eco-system services, to support a comprehensive view on food security and enhanced nutrition[8]. In that Welthungerhilfe s annual planning 2014 urges to apply its work towards self-help, poverty orientation, target group orientation, participation, sustainability and broad-based impact oriented work in Sierra Leone. The organization s goal in Sierra Leone is to improve the lives of disadvantaged sections of the population and to contribute to the sustainable management and protection of natural resources. The pilot project SLE supports the self-help approach and has 21

22 potential for further integration into the other sectors such as agro-forestry, natural resources management and food security and nutrition since they are all holding WASH services as a component. [Rec1] 2.3 Relevance to the objectives of the partner country The Sierra Leone Vision 2025, the Poverty Reduction Strategy Paper (PRSP) and the Millennium Development Goals (MDGs) aim at achieving a high quality livelihood for the people of Sierra Leone, attaining good governance through the rule of law and developing a strong and competitive economy. In the Poverty Reduction Strategy Paper (PRSP) water is stated to be one of the most important agents to enable Sierra Leone to achieve its vision for social and economic development including poverty eradication, and attaining water and food security. Core objectives are [9]: provide safe drinking water and sanitation facilities for deprived communities in rural and urban areas rehabilitate and reconstruct existing water facilities provide adequate liquid and solid waste disposal facilities in urban areas and increase community awareness of good hygiene practices relating to collection and storage of drinking water, use and maintenance of sanitary facilities and other environmental sanitation issues In the immediate short term, Sierra Leone is also committed to achieving its 2015 MDG targets of 74% for improved water supply and 66% for sanitation coverage respectively [9]. According to the Progress on Water and Sanitation and Drinking Water Update 2013, the national rate of people using improved water sources is at 57% (84% urban, 40% rural) and for the use of sanitation facilities the picture becomes even more challenging since only 13% of the population nationwide is using improved facilities disaggregated in 39% in the urban parts and only 7% in rural environments [10]. Besides the objective that stakeholders participate in the management of water resources and in the planning, construction, ownership, operation and maintenance of community based domestic water supply schemes in rural areas [9] a strategy document Rural water supply and small towns (2013) introduces self-supply as complementary to conventional communal supply, which is generally government or NGO funded and which forms the backbone of rural water supply [11]. 3. Effectiveness 3.1 Achievement of project purpose Welthungerhilfe: The purpose of the project is that new low-cost technologies for supporting WASH self-supply are introduced in Kenema and Western Rural Districts in Sierra Leone. In doing so Welthungerhilfe focused more on the introduction of innovative and affordable WASH self- supply technologies, here the EMAS technology. The formulation water self-supply should be turned into WASH self-supply since the new technologies comprise not only water supply technologies but sanitation and hygiene components as well. 22

23 Outcome / Purpose: A new low cost technology for water self-supply is available and increasingly utilised by customers in targeted districts Outcome Indicator 1: 12 out of 15 start-up businesses report at the end of the project increasing demand for self-supply WASH installations and services Comments evaluator: Focus group discussions were held with representatives of 11 business units, 6 in the Western Rural Area and 5 in Kenema. All the business units report a steadily increasing demand for low-cost WASH installations, without being able to prove their statements with numbers. They judge the increasing demand according to requests they receive either via phone or in personal contact. In how fare the number of installations correlate with the increasing requests needs to be proved. Below there are two charts (Figure 3-1and Figure 3-2) showing the installations of single low-cos technology in both project areas Kenema and Western Rural. For the area of Kenema a total of 118 components of low-cost technology installations were carried out whereas in the Western Rural Area a total of 54 components of low-cost technology were installed. Although the demand in the Western Rural Area was reported to be higher than that in Kenema, the numbers of installations show the contrary. According to the project manager this is attributed largely to the better commitment of technicians in Kenema than in Freetown. Efforts will be made to empower the technicians in Freetown. Exchange visits will be organised to promote knowledge transfer between the two districts. Nevertheless the database on the private installations shows inconsistencies in the installation dates. In order to proof that the number installations are following the increasing demand, this database should be updated. [Rec2] Figure 3-1: Low-cost components installed at private HH in Kenema 23

24 Figure 3-2: Low-cost components installed at private HH in Western Rural Area Outcome Indicator 2: 12 out of 15 start-up businesses state at the end of the project that they will continue with their business operations Comments evaluator: All of the business units are very confident that the business with the EMAS technology will continue and even increase. At the time of the evaluation all of them report to have ongoing orders from private customers. Whether or not this situation will prevail after some six months time has to be assessed a posteriori. [Rec3] Water Aid: The purpose of the WaterAid project was to develop a tested and replicable approach which facilitates investment by individuals or communities in improving access to, reliability and/or safety of their own water supplies in Kenema and Pujehun Districts. Outcome Indicator 1: Investment by individuals or communities in improving water supply in pilot areas Outcome Indicator 2: Government acceptance of a well-defined approach fit for scaling up Comments evaluator: Both indicators are neither measurable nor time bound and therefore impossible to assess in how fare it was met. Output 4 is to deliver a study of the financing and a financing guideline of self-supply initiatives, being one result on the way reaching the outcome. The study has been conducted and the 24

25 report written but was rejected twice by the WASH Facility due to inept performance. The other outputs and their indicators will be presented in chapter Project preparation Quality of target group, actors and situational analyses Welthungerhilfe: As target region for the project the Western Rural Area and Nongowa Chiefdom in Kenema District have been chosen. Reasons are that both areas can be classified as peri-urban areas with sufficient capital availability on consumer site to invest in their water and sanitation services which is meant to be ideal for the project purpose. Besides this the specific identification of the project areas and the target groups was done in consultation with district authorities, coordination with other WASH stakeholders, especially WaterAid, to avoid overlap and maximise complementarities. The selection of the demonstration sites at both locations followed a deliberate strategy. The idea behind it was to place the demonstration units where public awareness can be created, e.g. close to market places, at homes of important people such as chiefs or other well-known members of the communities. The selection can be viewed controversial because chiefs are per se privileged due to their status in the communities and businesspeople as well have the means for the installation and cannot be taken as a representative target group in a scale up phase. According to the proposal the demonstration sites were chosen according to consumer size, willingness, needs and pro-poor orientation. A pro poor orientation could not be observed since the project missed to install demonstration units in poor communities or households. Welthungerhilfe implemented the project together with EMAS International who already successfully introduced parts of the low-cost technology in Latin America, Africa and Asia. The director of EMAS came to Sierra Leone for a master training course organised from June 1 st to 16 th WaterAid WF004: The project targeted Pujehun and Kenema districts. The following partners were involved in the project: HELP SL (Hands Empowering the Less Privileged in Sierra Leone) HELP is a local NGO established 1996 and at that time supporting displaced people from Liberia and Sierra Leone. With their HQ in Bo, their approach is entrepreneurial and market based and the take the view that charity only perpetuates poverty by taking the initiative away from the poor. Their work concentrates on four (4) sectors: - Human Security comprising WASH in schools (in cooperation with UNICEF) WASH in communities > CLTS Post ODF (with WaterAid) - Food Security & Youth Development Vocational training 25

26 Skills and training for youth - Women s Empowerment and Advancement Registration of marriages Property rights - Micro Finance Village Savings and Loans Association (VSLA) In this project HELP and The Pujehun Technical Team worked closely together to implement the action research in Pujehun district. MOPADA (Movement towards Peace and Development Agency of Sierra Leone) MOPADA SL is a national non-governmental organization, established in 2006 with its head office in the city of Bo. Their work concentrates on the following themes: 1. Women and Youths empowerment 2. Human Security Promotion/Enhancement Community sensitisation on safe water and sanitation (CLTS) 3. Livelihood and Educational support 4. Child sponsorship 5. Just and Democratic Governance 6. Human Resource and Organizational development MOPADA worked closely with the Kenema Technical Team and was mainly responsible for raising awareness and advocacy on the self-supply concept in communities <150 people. Pujehun Technical Team (PTT) The PTT consist of a representative of the district council, the district health management team and the water supply division. They collaborate with WaterAid since 2010 when they had a project on CLTS. Their role mainly concentrated on the identification and selection of artisans which were to be trained in both the WaterAid and the Welthungerhilfe projects. Kenema Technical Team (KTT) The KTT is similarly composed as the PTT and besides the identification and selection of artisans for training; they identified communities eligible for self-supply WASHnet Sierra Leone WASH Network of Sierra Leone is a network of more than 95 civil society organizations aiming the joining existing platforms in addressing the critical barriers to achieving universal and sustainable sanitation and drinking water for all. WASH-Net s approach is to enhance and facilitate a multi-stakeholder process actor across the sector, engender and promote knowledge, identify opportunities for intervention and foster learning and joint action. Their designated roles in the project: - Leading on all advocacy campaigns 26

27 - Visiting communities to discuss practical issues of self-supply acceleration and alternatives in providing supplies to low density dispersed populations - Advocacy for increased government support for self-supply including promotion and marketing - Providing inputs to the preparation of the draft strategy document - Review of the strategy document in collaboration with other sector stakeholders such as MoWR, UNICEF, GOAL, WHH All the partner organisations have long lasting experiences in their field of activities and seem to be well chosen. The selection of the envisaged 15 demonstration sites for well upgrading followed the same deliberate pattern as those of Welthungerhilfe (see above). Additionally WaterAid introduced some attributes communities need to have to be eligible to benefit [2]: small communities with no protected water source ones with significant marginal populations to community/public supplies, frequently using unprotected sources those which became ODF in minimum time (strong community spirit to improve well-being) those which have demand for water for productive use those with high interest to solve their water supply problems and who have already dug a local well and have demonstrated initiative upgrading their traditional water sources those whose only supply is surface water a) more than minutes away b) close to houses Both projects maintain good relationships to the relevant Ministries on national as well as on district level Extent to which self-help approaches and structures are taken into account The nature of the self-supply approach is that the households plan, implement, repair and maintain their own WASH infrastructure with advice from experts but without any external financial contribution. The simplicity of the piloted EMAS technology allows for an easy repair and maintenance. The interviews with the individuals who have implemented the technology show that most of the repairs and maintenance was done by the owners or a well-known technician from the community. 27

28 3.3 Project planning Quality of the Project Planning Matrix (PPM) The application for the WASH Facility funding required a detailed Logframe as it is commonly applied in project planning. Welthungerhilfe: The impact of the intervention is well formulated and the definitions of the outcomes with the relevant outputs are set up in a logical sequence. The indicators are SMART (specific, measurable, achievable, relevant and time bound) and allow for a result-based project management and steering. Obviously the project planning is not gender sensitive. Neither the outputs nor the indicators refer to gender issues. It remains unclear whether and how the project intended to address gender specific needs. WaterAid WF004: The impact of the project as well as the outcome and the intended outputs are well formulated and follow a logical sequence. However the indicators in total, especially that for output 1, are more of an output nature and not measurable. For example indicator 1.4: Attitude of stakeholders towards the concept of self-supply acceleration assessed and known : this is rather an outcome than an indicator. The formulation of the indicators does not allow for a result-based project management and steering. Like the Logframe of the Welthungerhilfe project, neither the outputs nor the indicators refer to gender issues. It remains unclear whether and how the project intended to address gender specific needs. [Rec4] Analysis of the results framework Welthungerhilfe: The five outputs (results) are realistic, specifically defined and achievable and allow for proving the outcome hypothesis that A new low-cost technology for water self-supply is available and increasingly utilised by customers in the target regions. With the definition of milestones, the Logframe additionally allowed for tracking results on a quarterly basis. WaterAid WF004: The five outputs (results) are specifically defined and generally achievable and allow for achieving the outcome of a tested and replicable approach which facilitates investment by individuals or communities in improving access to ( ) their own water supplies. (With regard to the limited staffing of WaterAid in Sierra Leone the achievement of all the outputs has shown to be a challenge (refer to chapter 3.6.1)) 28

29 Table 3-1: Outputs of WF004 Output 1. A study on the potential of accelerating self-supply in Kenema and Pujehun conducted by end of December unprotected water sources upgraded and are being used as improved source for drinking water by December 2013 (piloting of self-supply) 3. By end of November 2012, a comprehensive training needs assessment done and a self-supply training package produced. 4. By end of December 2012, a study of the financing of self-supply initiatives conducted and a financing guideline for self-supply initiative developed. 5. By end of December 2013, a complete Self Supply Acceleration package developed and accepted nationally Against the background that the outcome of the project should be a replicable approach facilitating household investment in self-supply, so to speak a software component of self-supply acceleration, output 2 is overambitious. For the understanding of the evaluator the improvement of unprotected water sources, the empowerment of households in HHWTS, the training of (additional) artisans and the establishment of demonstration sites are all important measures but with limited contribution to the project s desired outcome. Talking into account the organisational structure (no in-country support) of WaterAid in Sierra Leone, the amount of piloting activities should be drastically reduced in order to safeguard the quality execution of the other outputs. [Rec5] There should have been a better coordination with the Welthungerhilfe project which concentrated on building entrepreneurial capacity and piloting low-cost solutions for supporting self-supply Realism of project purpose Welthungerhilfe: The project purpose to introduce innovative and affordable self-supply technologies for sustainable improvements in water supply and sanitation in Kenema and Western Rural Districts is realistic and generally achievable. The project duration of 20 months although is too short and made especially the proper achievement of output 2 (kindly refer to Table 3-2), a challenging task. The digging and drilling of wells could only be made during the dry season where the water table is lowest to secure a sustainable yield. The project started in July 2012 in the rainy season and had only one dry season (December 2012 April 2013) available to implement the wells which was the most time consuming work. This led to the fact that the implementation pressure grew in the course of the project which resulted in hasty implementation, leaving quality aspects sometimes neglected. WaterAid WF004: The project purpose to develop a strategy to accelerate household investment in improved water supply (self-supply) is realistic and generally achievable. Again the project duration is too short and the outputs with their related activities are seemed to be overambitious (refer to chapter 3.3.2) Appropriateness of staff, material and financial planning Welthungerhilfe: The personnel involved in the implementation of the project were found to be appropriate. Initially there was a logistician employed but in the course of the project execution it was found that the 29

30 technicians are able to take care of the supplies as well. The project manager reacted to this in shifting the allocated budget to enhance the technical support on the site. The same was true of the employment of a watchman at a demonstration site found unnecessary and the budget was also shifted into financing a store keeper. Beneficial for the project was that Welthungerhilfe maintains a field office in Kenema from which the operations were carried out in the district. One demonstration site was located on the Welthungerhilfe compound and technicians are always around to inform interested visitors. WaterAid WF004: Water Aid does not maintain field offices which made the execution and supervision of the project work a challenging task for the program manager. The fact that WaterAid does not have a pronounced organisational structure in Sierra Leone and hence no direct support to the program manager leaves the impression that the manager was at some stage over-challenged with the project implementation and management. 3.4 Quality of the analysis of project holder Organizational structure and focal points of work Welthungerhilfe: Welthungerhilfe is the project holder and implemented the project with the partner EMAS International a German NGO registered in Munich, promoting the EMAS technology. Welthungerhilfe started its engagement in Sierra Leone in Since then, around 20 projects with a total amount of approximately 35 Mio. (co-financed by EC, BMZ, GIZ, DfID UK, ECOWAS Renewable Energy Facility (ECREEE), University of Wuppertal, FAO) were put into action. At the beginning the interventions in Sierra Leone focused on relief (e.g. school feeding) and rehabilitation of social infrastructure, farms and the distribution of tools and seeds. The new country strategy still focuses on rural development, but also highlights the necessary shift towards more sustainable and long term development and capacity measures in sectors like agriculture, food security, employment, natural resource management, WASH and income generation targeting poor and vulnerable rural as well as urban population. Welthungerhilfe is currently implementing 11 development projects with a total amount of 16 Mio. and has around 8 international and 100 national staff. The Regional Office for West Africa Coast, responsible for Liberia and Sierra Leone, is located in Monrovia. The country liaison office in Freetown is complemented by project offices in Tombo, Bo, Kenema and Pujehun [8]. 30

31 Figure 3-3: Intervention sectors Welthungerhilfe Sierra Leone Source: Welthungerhilfe By combining our core sectors, agro-forestry and agriculture, with other sectors, such as natural resource management, WASH and eco-system services Welthungerhilfe Sierra Leone supports a comprehensive view on food security and enhanced nutrition. The project SLE 1014 was managed efficiently. A national project coordinator was fully employed by the project. Several supporting staff was partly involved in administration issues. The organisational structure is presented in Figure 3-4. Figure 3-4: Organisational structure of SLE 1014 Source: Welthungerhilfe 31

32 WaterAid: WaterAid helps to build the capacity of the water, sanitation and hygiene sector to improve access to these basic services. In doing so, they work in local communities to raise awareness of the importance of hygiene and basic sanitation provision. WaterAid is a promoter of Community-led Total Sanitation (CLTS). Their focus is on reaching the poorest and most marginalised people with safe water and sanitation provision. The project team in Sierra Leone consist of a team leader who is responsible for the liaison between WaterAid and the donor, signatory on behalf of WaterAid to donor and consultants, approval of activity budgets, a program manager holding the overall project coordination and management including narrative reporting. The program officer supports the partners in project implementation in the field and the finance manager is responsible for the financial reporting and processing of request for funds for project implementation. Finally a logistics & admin officer supports the logistics and the financial administration of the project. WaterAid is a well-known player in the WASH sector not only in Sierra Leone. The organisation does neither maintain a country office in Freetown nor field offices in the targeted districts which made the timely execution of the project a challenging task. For future projects in Sierra Leone WaterAid needs to rethink their institutional setup in the country and augment their in country support to the project manager Staff, equipment and logistics On project and program level Welthungerhilfe operates with core staff and fixed-term employees whose contracts terminate with the project period they were assigned for. Although Welthungerhilfe has no other options of employment, since the nature of third party funded project work does not allow for permanent contracts, it still remains a challenge for those concerned since they have no solid basis for their future planning. The staffs carrying out SLE 1014 was qualified and highly motivated. The purchase of the material was done by the technicians and supervised by Welthungerhilfe staff. Reportedly there were no problems with the logistics since the supplies are all available on the local market. WaterAid: Due to the lack of supporting staff the program manager relied in the execution of the project completely on external local consultants with limited knowledge on self-supply and limited capacity in scientific writing. 32

33 3.4.3 Project steering and financial administration Welthungerhilfe: The quality of the project steering is good. The quarterly progress reports which had to be delivered to the WASH Facility, allows for a well-informed progress tracking. The experiences project manager reacted in an adequate manner to up-coming challenges. Initially it was planned to start with the demonstration sites (well digging and drilling) right away from the start but due to the fact that the project started in the rainy season (July 2012), the management decided to commence the training activities in the first place and start with the drilling and digging in the dry season when the water table is at its lowest. The financial administration of the project was challenged by the leaving of the financial administrator to another employer at the end of Until the deployment of her successor the financial management was done by the country coordinator. WaterAid: As mentioned before, due to the lack of an organisational structure in the country WaterAid mainly dealt with local partners. According to the manager this resulted in late and incomplete reporting on the activities and the finances as well. 3.5 Steering by the project office and cooperation with the project holder The country office in Freetown supported the project manager in all issues. A good communication is maintained either via phone or through personal contacts. There were no major problems reported by the staff on project level. 3.6 Quality of the project execution Presentation and evaluation of the project outputs and activities Welthungerhilfe: Reportedly there were no major obstructions in the project implementation. However some hindrances were reported: Administrative: The leaving of the financial administrator hampered the proper administration but the country coordinator closed this gap until a new administrator was employed. Supplies: Constraints in transporting pipes which are normally 6m of lengths. The pipes had to be cut and later reassembled again Environment: Rough terrains made it a challenging task to reach some of the project locations Project execution Due to the fact that the project started in the rainy season (July 2012) the technical implementation of the project was restricted to only one dry season. It is impossible to dig latrines or drill wells in the rainy season because of the high water tables. 33

34 The targeted outputs were not completely reached (but the major part was) at the time of the evaluation (January 2014), but taken into consideration that the project period ends at the end of February 2014, it must be assumed that the results will be reached by then. Most of the indicators selected against which the results and impacts should be measured could be assessed at the time of the evaluation, but need to be re-assessed sometime after the project s completion. For an assessment of the project outputs kindly refer to Table 3-2. Table 3-2: Quantitative achievements of SLE

35 Output 1: Qualification of 80 technicians and practitioners, including advanced training of 10 of the 80 as master trainers on EMAS technology Verifiable indicators Achievement Comments of the evaluator (until December 2013) 80 technicians and practitioners are trained on EMAS technology 10 master trainers are trained on EMAS technology 80% of number of technicians who are trained pass the post training test 75 technicians on the complete range of products, 100 farmers on rain water harvesting and 60 private households on O&M practices 25 master technicians (second training) 50 passed (62.5% of total and 98% of first two quarter) Overall above target The training of the 100 farmers on RWHS was in the course of the installations done under the IFAD contract. (Please refer to chapter 5.1.1) Above target, because 15 more technicians received training on manual drilling Slightly behind target. It is not clear why the trainees did not pass the test and what will be the consequence of not passing. Output 2: 30 demonstration sites using range of EMAS WASH technology are available and functional Verifiable indicators Achievement Comments of the evaluator (until December 2013) 30 demonstration sites established (15 in Freetown Rural and 15 in Kenema Nongowa chiefdom) 25 completed (Kenema-11 and Western Area Rural 14) According to the project manager the remaining 5 demo sites will be installed until February. The evaluator is of the opinion that no further efforts should be put in setting up the last 5 demo sites. The money should rather be used for paying somebody for monitoring and counselling the performance of the technicians and assessing the self-supply development after project termination. [Rec 6] Output 3: Increased demand for Self-Supply WASH schemes due to public awareness raising and community outreach Verifiable indicators Achievement (until December 2013) Comments of the evaluator 35

36 12 out of 15 start-up businesses (output 4) report at the end of the project increasing demand for Self-Supply WASH installations and services 12 out of 15 start-up business units state at the end of the project that they will continue with their business operations (due to profit making) 13 business units are functional and report increasing demand Output 4: 15 start-up businesses are established Verifiable indicators Achievement (until December 2013) A minimum number of business units established and businesses are registered at supplied with kits, 2 business units authorities and have adequate registration 50% complete and tools and equipment available tools supplied 75% of installations which are tested pass quality specification on water quality and further defined standards The projects registers a minimum increase in demand of the technology by 10% quarterly The indicator is the same as for the outcome (kindly refer to chapter 3.1). This indicator is difficult to measure since it would require a proper documentation by the business units on their enquiries. Therefore only qualitative statement possible. Two units, the Tombo Technicians Association and Mountains Technicians Association however need to finalise their registration protocols. 10 The indicator is the same as for the outcome (kindly refer to chapter 3.1). This indicator is difficult to measure since it would require a proper documentation by the business units on their enquiries and installed hardware. Therefore only qualitative statement possible. 100% sampled systems pass quality standards 300 registered potential customers, 119 total private installations systems (paid by private resources) Comments of the evaluator Although behind target, there should not be put more effort on establishing the missing business units. It is conceivable that after some time, e.g. 6 months, not all the 15 units would be operating anyway. [Rec 7] The implementation quality is widely differing from one site to the other. Quality standards or requirements need to be reviewed and gaps closed. [Rec 8] In Kenema 33 private installations were done in the first quarter of 2013 (Jan-Mar). From April to June another 35 installations were implemented which is an increase of around 6% (refer to Figure 5-2). In the Western Rural area there is as well a positive trend in installations (refer to Figure 3-2). Besides the two main project areas another 26 installations were done in Bombali, Port Loko, Kailahun, Moyamba and Bo. Output 5: Improved Sector Learning Competence and Strengthened Capacity Building of Stakeholders Verifiable indicators Achievement (until December 2013) Comments of the evaluator 36

37 At quarterly stakeholder meetings important actors as the Council, Water Directorates, DHMT and locally based NGOs do participate at district level At national stakeholder meetings sector actors as MoEWR, MoHS, UNICEF, etc. are interested and proof their interest by active participation in workshops At least 3 NGOs are integrating EMAS schemes and the selfsupply approach in their implementation work 4 There should be further participation in those meetings 3 There should be further participation in those meetings 6 NGOS and 1 Donor Agency (IFAD) Green Scenery Sierra Leone UPA(urban and Peri -urban agriculture) Liberia COVEP(community vulnerable empowerment program, Liberia) and Water Aid Sierra Leone MOPADA and HELP Sierra Leone Kambui North Forest Management association Above target WaterAid: Only one of the five targeted outputs was achieved until January The project is fare behind schedule and an amendment for a non-cost extension until June 2014 was submitted to the WASH Facility. For an assessment of the project outputs kindly refer to Table

38 Table 3-3: Quantitative achievements of WF004 38

39 Output 1: A study on the potential of accelerating self-supply in Kenema and Pujehun conducted by end of December 2012 Verifiable indicators Achievement Comments of the evaluator (until June 2013) Existing self-supply options and appropriate upgrading mechanism identified by end of quarter 2 Existing options and upgrading mechanisms identified. It was agreed that with key stakeholders to pursue the various forms of the rope pump technology while upgrading will focus on using burnt clay bricks to lower cost. Water quality of existing unprotected sources tested/ analysed and documented by end of quarter 2 Capacity needs of existing users of unprotected sources identified for upgrading by end of quarter 2 Attitude of stakeholders towards the concept of self-supply acceleration assessed and known by end of quarter 2 Unprotected water sources were tested during the baseline survey Capacity needs of existing water users of unprotected sources identified during the baseline survey Assessment of the attitude of key stakeholders towards concept of self-supply also formed a key component t of the baseline survey as questionnaires were designed to elicit information in that regard. It is said that paper is available in the publications section of WaterAid s website: Right now the paper can t be downlo.aded. [Rec 9] Output 2: 100 unprotected water sources upgraded and are being used as improved source for drinking water by December 2013 (piloting of self-supply) Verifiable indicators Achievement Comments of the evaluator (until December 2013) 50 ground water sources improved and providing improved quality of drinking water to 3750 people by end of month 20 Only one upgrade has been done in Bandama. Start of the rest of the pilot activities February 2014 This indicator is overambitious since the piloting activities have not yet started. The number of should be decreased in order to safeguard the quality execution of the other outputs. [Rec 5] 50 household are empowered and are improving the quality of their drinking water through household water treatment and/or RWH by end of month 20 This has not been done. Start of the pilot activities February 2014 For the part of household water treatment WaterAid procured 50 Tulip Water Filters, a candle-type water filter which uses gravity siphon pressure to force water through a highquality ceramic filter element. Procurement and life-cycle costs are unclear but the fact that these filters have to be completely imported does not support the idea of self-supply and is therefore of limited sustainability. 39

40 45 Artisans trained as masons, plumbers, well diggers, slab makers, etc. and passed quality inspection on work after two months of completing course. 15 demonstration sites (inclusive of the 50 improved ground water sources) established for training of water points up-graders are operating by end of month artisans have received training to undertake the piloting. 45 was an unmanageable number and the cost of training far outweighed the funds provided in the project budget. This has not been done. Start of the pilot activities February Strategic sites identified The distinction between the artisans trained by Welthungerhilfe and those by WaterAid is not clear. This indicator is overambitious since the piloting activities have not yet started. The number of should be decreased in order to safeguard the quality execution of the other outputs. [Rec 5] Community management structures Management structures formed (WASH committees) set up in 100% of intervention areas by end of - month 20 Output 3: By end of November 2012, a comprehensive training needs assessment done and a self-supply training package produced. Verifiable indicators Achievement Comments of the evaluator (until December 2013) Training needs of at least 80% of target communities and small scale private practitioners assessed and documented by end of quarter 2 It is said that a new manual was finalised No manual available upon request, therefore impossible to judge on the usefulness of the reports. At least 5 existing training materials Completed reviewed and used in the production of self-supply training manual by end of quarter 2 - One synthesised training manual of existing fragmented training materials produced and being used by partners by end of quarter 2 Completed but under final review as the first review recommended that it be aligned with the contents of the local artisans training on rope pumps The manual produced above is allencompassing including various themes. So this activity was considered a duplication No manual available upon request, therefore impossible to judge on the usefulness of the reports. At least one standardised Not available upon request, comprehensive training package therefore impossible to judge produced for self-supply initiatives on the usefulness of the by end of quarter 2 reports. Output 4: By end of December 2012, a study of the financing of self-supply initiatives conducted and a financing guideline for self-supply initiative developed Verifiable indicators Achievement (until December 2013) Study of financing mechanism conducted considering existing financing options by end of quarter 2 First draft of the study report was updated based on the comments of WaterAids technical support unit (TSU) and the Rural Water Supply Network (RWSN). The study is currently under final revision Comments of the evaluator Not possible to judge on the usefulness of the reports. Guidelines for financial options to maximise effective investment in water supply are produced and in use by key actors by end of quarter 2 Output 5: By end of December 2013, a complete Self Supply Acceleration package developed and accepted nationally Verifiable indicators Achievement (until December 2013) Comments of the evaluator 40

41 At least 6 advocacy meetings held with key stakeholders at national and district levels by end of month 20 Acceleration package finalised and agreed with government by end of month 20 At least 80% of key stakeholders in the WASH sector participate in national self-supply acceleration validation workshop by end of month 20 A report on Self supply Acceleration Action Research with findings and recommendation for integration into the national rural water supply strategy developed and disseminated to 80% of stakeholders by end of month 20 At least 5 strategic review meetings held with self-supply related partners and key stakeholders by end of month 20 3 district level meetings and 1 national meeting has been held Not finalised yet. This will be developed after piloting Not finalised and stage unclear Not finalised yet. This report will only be produced after the project is completed Not finalised and stage unclear No minutes available Not possible to judge on the usefulness of the reports. Not possible to judge on the usefulness of the reports Involvement of the target group and project cooperation The trained technicians were all involved in setting up the demonstration sites and did the installations at the private households. The owners (private households) who invested in self-supply technology are all involved in the maintenance and repair of the supply structures and show a great sense of ownership. They are willing to provide information on frequently asked questions from the interested public and refer to the technicians for a quotation or further detailed information Overall evaluation of the execution Based on the result matrix given in the project proposal and further information from the project management and the stakeholders, the following details can be provided: Welthungerhilfe: Table 3-4: Evaluation of the main measures of SLE 1014 (Welthungerhilfe) Main measures Overall evaluation Project steering and The project management was done professionally. The project management manager showed great commitment in the exercise of his duties. Project monitoring and Monitoring and quality control of the installations at private reporting households became a challenge in the course of the project because some technicians perceived this as policing of their work and were suspected to hold back information. The management reacted to this in intensifying clarification on the purpose of monitoring and the benefits of quality control. The quarterly reports are well structured and allow for a systematic tracking of the activities and outputs. 41

42 Selection of locations for demonstration sites Selection of private homes for pilot installations The demonstration sites were established at strategically favourable locations (close to market places with public business, at major roads with a good traffic volume) The selection of private homes followed a deliberate consideration. Besides the installation at homes of better off people, some installations should have been sponsored to rural and less well-off people to as well observe their handling of the low-cost technology. Quality of installations The implementation quality was varying from site to site. It was reported that the technicians wanted to install the water supply and sanitation systems in the most reasonable way to not scare away customers with high prices. Nevertheless minimum quality requirements need to be met regardless of the improvements made. [Rec 8] Training of the artisans The technical training was done to the extent that the artisans were able to build and install the WASH systems. Reportedly more training on welding and business administration is needed. Establishment if business units The idea of bringing together technicians in the business units first of all makes sense, since some of the works can only be done in a team (e.g. drilling of wells). However it has to be monitored and assessed whether the BU still exist after some time after project termination because some attempts of splitting-off of some artisans from the BU were reported. [Rec3] WaterAid: Table 3-5: Evaluation of the main measures of WF004 (WaterAid) Main measures Overall evaluation Project steering and management The project manager has his problems with steering and managing the project properly. This is proven by the delay in achieving the targeted outputs. Project monitoring and reporting The quarterly reports follow the standards given by the WASH Facility and allow for a systematic tracking of the activities and outputs. Study on the potential of accelerating The study is currently under revision. No evaluation possible self-supply in Kenema and Pujehun Upgrading of unprotected water Not yet started, no evaluation possible sources Development of training needs According to the WaterAid manager the needs assessment and the assessment and a self-supply training training package were developed. No statements on their quality package possible due to unavailability of the report Study of the financing of self-supply The study is currently under revision. No evaluation possible initiatives and a financing guideline for self-supply 42

43 3.6.4 Occurrence of assumptions and risks The exposition of assumptions and risks was done in the project proposal. Welthungerhilfe: Table 3-6: Assumptions & risks defined in the project proposal of SLE-1014 Risks to Project: Project partners disagree on project implementation strategy Local Government/ population hesitates to accept the self-supply concept Unstable political situation in Sierra Leone Low demand/ low willingness to pay for technology by consumers Probability (low, medium, high) Impact on Project (low, medium, severe) Strategy to minimise/mitigate risk Low Severe Detailed MOU S at beginning of project, continuous partnership cooperation and communication Low Severe Appropriate marketing and communication strategy, proficiency in implementation Low Severe Assumption of stable political conditions even in the case of regime change Low Severe Intensive education program, good marketing approach, preparation of arguments; potentially establishment of links with micro credits Failing of business models Low Severe Intensive training on business administration and conduction; good screening of business concepts Non-adoption/authorisation to implement low cost technologies by authorities and communities Poor construction quality by the established businesses and condemnation/poor reputation of the technologies Alternative technologies are preferred by consumer Low Severe Intensive discussion of advantages and offering of examples Low Severe Intensive training courses with excellent modules Low Severe good district level coordination The general rating of a low probability of the risks occurrences seems to be unrealistic. For example the probability of the risk of a poor construction quality could at least be rated as medium, since one does not know whether the training contents and methods are reaching every individual trained and the definition of quality might be subjective. [Rec 10] 43

44 WaterAid: Table 3-7: Assumptions & risks defined in the project proposal of WF004 Risks to Project: Local authorities do not support the concept of private investment or private sector does not see support services to household investment as offering a viable return Local partners not having the right skills and experience in implementing self-supply projects and having low capacity Household response is low because of historical donor dependency UNICEF providing funds and leading on specific aspect/ ouput of the project which will feed into the achievement of other outputs risk of delay Sector professionals assumptions on water quality outweigh end-user views on convenience and flexibility of use Electoral period and election campaign may interrupt project activities Other projects do not become established to develop low cost technology production. Other actors reluctant to take on and scale up self-supply concept Probability (low, medium, high) Impact on Project(low, medium, severe) Strategy to minimise/mitigate risk Medium Severe Involvement of authorities in studies, and full discussion of results and their implications and of international experience. Exchange visits to successful community/ household investment in SL medium medium RWSN and WaterAid Technical Support Unit (TSU) will provide advice; support and technical backstopping and mostly technical work will be done by qualified and competent consultant with relevant experience in self-supply/ research work. High Medium Expect slow start-up but speeding up as natural leaders show the Yes we can mentality is possible, affordable and replicable (based on CLTS experience and other countries with similar problem) Low Severe UNICEF is adequately informed and participates in the conceptualisation of the self-supply acceleration. WaterAid and UNICEF will have a signed MoU and proposal with detailed work plan stipulating time for execution of work and an efficient system for time deliver. Low Medium Ensure good water quality monitoring by government labs and joint assessment of performance, user satisfaction and cost of up-graded supplies and conventionally improved ones, linked to assessment of any alternatives for hard to reach groups. Medium Low Ensure activities are scaled down during peak of election campaigns. Schedule activities so that they don t coincide with major campaign activities Medium Medium Encourage potential organisations to apply to WASH facility or other funding Medium High Ensure effective participation of many as possible stakeholders in discussions and meetings, strong documentation and advocacy. The promotion of self-supply ensure that the process is community driven and encourage local initiatives and local enterprising The general rating of the probability of the risks occurrences are realistic. 44

45 4. Efficiency 4.1 Cost/benefit ratio for the overall project Welthungerhilfe: The basis of the analysis is the budget PVL Aufstockung II as from According to this the amount of 263,096 will be spent until project end. The project aimed at targeting 5, individuals. This results in a cost/benefit ratio of the overall project of Again it must be mentioned that this project is not beneficiary oriented, following the classical approach of a WASH project which is delivering WASH services to a defined number of beneficiaries but rather the promotion and piloting of the self-supply concept to increase coverage and service levels of WASH services. The cost/benefit ratio of the overall project is evaluated on the basis of the percentage relationships of the budget lines given in Table 4-1. Table 4-1: Total project budget of SLE-1014 (status November 2013) No Cost category Investment costs Operational costs 12 Personnel 14 Evaluation Public relations Material and activities Total [ ] DfID (WASH Facility) 6,505 ( 5,246) 91,846 (74,069) 73,944 (59,632) 18,600 (15,000) 2,169 (1,749) 36,406 (29,360) 229,470 (185,056) Expenditure WHH 5,500 (4,435) 7,460 (6,016) 1,666 (1,343) 14,626 (11,795) EMAS - 8,000 (6,452) 11,000 (8,871) ,000 15,323 Total expenditure 12,005 (9,682) 107,306 (86,537) 86,610 (69,847) 18,600 (15,000) 2,169 (1,750) 36,406 (29,360) 263,096 (212,175) Total expenditure 4.56% 40.79% 32.92% 7.07% 0.82% 13.84% [%] 87.22% 5.56% 7.22% 100% (Exchange rate 1GBP = 1,24 Euro) Around 14% of the budget was allocated directly to the beneficiaries, whereas around 33% of the budget was allocated for personnel and around 41% for carrying out the operations. 100% In addition to the above outlined budget, Welthungerhilfe won a contract for the installation of 20 rain water harvesting systems including training of 100 farmers from the International Fund for Agricultural Development (IFAD). The contract covered a budget of US$ 48,000. WaterAid: The basis of the analysis is the ASI-Budget per Partner as from According to this plan the amount of 264,735 will be spent until project end. 45

46 The project aimed at targeting 4,315 direct beneficiaries. This results in a cost/benefit ratio of the overall project of Again it must be mentioned that this project is not beneficiary oriented, following the classical approach of a WASH project which is delivering WASH services to a defined number of beneficiaries but rather the promotion and piloting of the self-supply concept to increase coverage and service levels of WASH services. The cost/benefit ratio of the overall project is evaluated on the basis of the percentage relationships of the budget lines given in Table 4-2. Table 4-2: Total project budget of WF004 (status November2013) Cost category group DfID (WASH Facility) Expenditure WaterAid WASH-Net Total expenditure Total expenditure Water Sanitation & Hygiene (Studies) 83,860 40,300 3, , % Personnel 34,290 21,435-55, % Procurement/Asset Purchase 7, , % Transportation 35, , % Communication 4, , % Office & Administration 35, , % Total [ ] 200,000 61,735 3, , % [%] 75.56% 23.32% 1.13% 100% Around 50% of the total budget was allocated for the development of a) study on the potential of accelerating self-supply in Kenema and Pujehun b) comprehensive training needs assessment including a self-supply training package c) study of the financing of self-supply initiatives as well as the development of a self-supply acceleration package At the date of the evaluation only one study (study a) was completed and provided to the sector. The other outputs are still under revision. 46

47 4.2 Quality of the internal project M&E system Both projects used various instruments for their internal project monitoring and evaluation. Table 4-3: Measures for internal project monitoring & evaluation Welthungerhilfe WaterAid The Project Planning Matrix (logical framework): This is generally the principle basis for internal and external impact related monitoring, using the defined hierarchical structure and indicators. The Project Progress Reports (PPR): The PPR had to be submitted to the WASH Facility on a quarterly basis presenting the progress and achievements of the project. Workshops and discussions: Protocols of stakeholder discussions on self-supply approach, quality management and business development. There is a need for further training on business administration Meetings to coordinate project activities in the districts: There are monthly sector coordination meetings but due to time constraints (implementation pressure because of the short project period) the project manager rarely participates in these meetings. The meetings are perceived as an additional burden to the existing workload. Site visits In the Welthungerhilfe project a Monitoring and Evaluation Officer was deployed. He visited all the construction sites and monitored the progress and the implementation quality. The Welthungerhilfe project manager was present at all sites during the implementation. Frequent site visits were conducted by the WASH Facility manager. Financial monitoring At the field level the financial administrator and the cashier are handling the financial issues. The cashier is responsible for handling any cash transactions following approval by the project management. Welthungerhilfe uses the software WINPACCS for all its financial reporting. Through WINPACCS one is able to view total budget, expenditures as at the end of the reporting period and balances per each budget header and budget line. All project expenditures are booked into the system. An end of period report is generated on a monthly basis. Financial audits were conducted by the WASH Facility with no objections reported. Internal evaluation: An internal evaluation focusing on the technology (EMAS) was carried out in November Site visits Field visits were conducted to the demonstration site on the Welthungerhilfe compound. Monthly monitoring visits by the program officer to the operational areas to ascertain progress of implementation according to the plan. Financial monitoring Financial audits were conducted by the WASH Facility with no objections reported. Funds are requested based on agreed quarterly plans linked to the project budget and activity plans. These funds are then released to the partners of the relevant staff who is responsible for implementing the activity. Once the activity is completed, the funds are retired as an advance which is reviewed by the finance department and then sent to the Team Leader for sign off. The partners are expected to retire the funds on a quarterly basis using a template provided by the finance department which they have been trained on. 47

48 5. Direct and indirect, short, medium and long-term OUTCOMES and IMPACTS 5.1 Use of Outputs / Outcomes The evaluated projects are pilot projects on accelerating self-supply in Sierra Leone. Self-supply, although practised through rainwater harvesting already but not known as self-supply, is a new concept in Sierra Leone. The aim of the projects were not to serve a certain number of beneficiaries with water or sanitation provision or train as many people as possible on hygiene standards, as it is the case in classical WASH projects were the beneficiaries are in the centre of all activities. The projects are rather focusing on market based approaches and are sector-, not beneficiary driven with sector learning being the focus of the pilot projects Economic outcomes Welthungerhilfe: In this project two groups of direct beneficiaries are targeted where an economic benefit can be assessed: Private households willing to learn about the concept and the technology and willing to explain it to anyone interested serving as demonstration sites The households have learned about the concept of self-supply and the supporting low-cost technologies. The project did not aim at enhancing the economic situation of the households, because the households chosen for the piloting of the EMAS technology were households able to raise the upfront investment for the implementation. For these households the benefits were more on the side of enhancing convenience by increasing their service levels. Therefore neither an economic outcome nor an economic impact could be assessed. Technicians who will be trained on the EMAS technology and the self-supply approach 75 technicians and practitioners received training on several components of the EMAS technology. Out of this trained technicians the best 10 were chosen to be further promoted to master trainers. 11 business units were established comprising 5-10 trained technicians. Reportedly the technicians are all working in installing the EMAS components and are able to make a living. (This is a qualitative statement since no one interviewed was able to proof this with numbers). The economic situation of the target group has improved although the main intention in training them was to have technicians available for implementation. It is remarkable that the technicians were able to conduct private installations beyond the Western Rural Area and Kenema Districts (project locations) additionally to seven other districts. Figure 5-1 shows the number of installations in nine districts of Sierra Leone, whereas the orange shaded columns represent those districts which were not directly targeted by the project. This was mainly because the project was able to win a contract with The International Fund for Agricultural Development (IFAD) over the installation of 20 rain water harvesting structures for microirrigation for 20 farming groups in Kenema, Kailahun, Kono and Koinadugu Districts (5 systems and five farmers group per district). Part of this contract is the capacity development of 100 farmers in 48

49 construction of rain water harvesting structures. IFAD plans to scale up rain water harvesting as a climate change adaptation strategy for irrigation purposes in inland valley swamps. Figure 5-1: Private installations in nine districts Figure 5-2 shows the increase of the private installations over a period of 18 months. It shows a steep increase during the months of the dry season from December 2012 to April Expectedly the numbers of installations decrease with the incipient rainy season in July. Figure 5-2: Private installations per quarter in nine districts 49

50 WaterAid: The project is fare behind schedule in achieving the proposed outcomes. Only one out of five outputs has been accomplished at the time of the evaluation. Likewise the Welthungerhilfe project the WaterAid part did not focus on enhancing the economic situation of the target groups Socio-cultural outcomes The indirect beneficiaries namely the private customers who invested in upgrading their service levels by contracting the (EMAS) technicians show a great sense of ownership of their water and sanitation systems. It is obvious that they want to protect their investment from unwanted tampering with the water supply systems or the use of the latrines. This is quite a strong outcome since such kind of ownership is what WASH systems need to safeguard lasting operations and what can be rarely observed with communal subsidised systems. 5.2 Impacts As mentioned already in chapter 0.4 it is too early to assess the impact of the projects. The impact formulated by the Welthungerhilfe project that the self-supply approach becomes widely accepted in Sierra Leone is not measurable at this stage of the project. The underlying approach of executing the steps of piloting > strategy development > up-scaling consecutively, as intended by the WASH Facility, is inexpedient because it does not allow for consolidation of the results of the pilot phase. The consolidation phase would allow for monitoring user satisfaction with the new technology and the further development of the established business units as well the performances of the individual technicians in their jobs. It would show whether the approach of establishing business units was worthwhile or whether the units would disintegrate after some time and each individual technician would work on his/her own. The consolidation phase would also show the shortcomings of the pilot phase in terms of approach. For example the pilot phase did not target the rural small communities for whom the self-supply concept -supported by affordable low-cost technology- would be a good opportunity for improving their WASH service levels. Both the results of the pilot phase and the consolidation phase build the basis of a strategy for up scaling self-supply. [Rec 11] Economic impacts Refer to paragraph Socio-cultural impacts Refer to paragraph Environmental impacts Through the provision of affordable latrines and further sanitation marketing there is a good case to believe that ODF communities will increase. 50

51 5.3 Contribution of the project to the Millennium Development Goals The projects contribute more or less to all of the eight Millennium Development Goals. Table 5-1: The projects contribution to the MDG MDG Goal Contribution Households with an improved and advanced water supply might use their water for many purposes. The might engage in small holder businesses to additionally generate family income. School attendance will increase due to a better water availability. Pupils do not have to walk long distances to fetch water for their school day. The daily fetching of water is traditionally the duty of women. Water provision close to the house offers women the opportunity to engage themselves in other productive use of their time and hence fosters their empowerment through changing gender relations within the household. Children with homes more than 500 m from their water supply had much higher rates of diarrhoea compared with children from houses with their own supply. WASH facilities closer to the homes reduce the burden of water carrying during pregnancy and allow for a better hygiene. Easier availability of water for hygiene/hand washing which reduce opportunistic diseases and respiratory infections for HIV/AIDS and others Self-Supply offers technologies and management systems, spreads abstraction of groundwater among diffuse points and reduces concentration of livestock around water points. It may lead to increased abstraction but this tends to be naturally limited by its effect on the water table. Access to drinking water can be generally at household level. Safety is ensured by household water treatment. Cooperation with the private sector bears the potential for the introduction of new technologies and knowledge. Growth in demand for well-diggers, masons, blacksmiths and carpenters expands (rural) job opportunities. Adapted from [12] 51

52 6. Sustainability For assessing on the sustainability of the WASH structures the definition of sustainability given in the Welthungerhilfe Orientation Framework Water, Sanitation and Hygiene from February 2013 [13] is used. Table 6-1 provides the sustainability criteria. Table 6-1: Welthungerhilfe sustainability criteria and evaluation Sustainability criteria Evaluators comments It functions and its being used Right now there is no long-term data on the functioning of the low cost technology in Sierra Leone but a study from the Rural Water Supply Network suggests that e.g. the EMAS manual water pumps are shown to have a high rate of functionality as used at the household level in the studied contexts in Bolivia [14]. It is able to deliver an appropriate level of Reportedly the benefits of the EMAS technology are benefits that meet user needs, priorities and foremost the increase in service level at household level. expected service levels People move from an unprotected water source to a gravity fed water supply into their houses. The toilet seats are appreciated since they easy the situation for the aged and disabled. It continues over a prolonged period of time which goes beyond the life-cycle of the equipment Its life cycle costs are covered at local level through user fees, or alternative financial mechanisms It can be operated and maintained at local level with limited but feasible, external post-construction support It does not negatively affect the environment The parts e.g. of the EMAS pump prone to wear and tear can be replace easily and for only little costs which increases the life cycle of the technology. The self-supply concept entails that the individual household is able to carry the life cycle costs of the technology. Supporting mechanisms such as microfinance or other traditional saving systems need to be utilised for accelerating self-supply. Repairs and maintenance can be done locally either by the owner him/herself or a nearby technician. However a constant check on the implementation quality is of crucial importance. No indications 6.1 Organizational and institutional/political To make self-supply a complementary service delivery model for small rural communities the embedding into the National Water and Sanitation Policy is an important step showing the government s commitment in serving small rural communities. 52

53 7. The status of self-supply in Sierra Leone and the way forward This section is added to the given structure of the report. This was done because the evaluator found it unrewarding and confusing to integrate the answers to the orientation questions provided in the terms of reference (kindly refer to Annex 1), into the predefined setup of the evaluation report. It is for the sake of a better understanding of the findings on whether or not the self-supply approach is deemed to be promising for up-scaling. 7.1 Policy and strategy status There are three main policies which influence WASH in Sierra Leone, the Agenda for Prosperity (GoSL, 2013), the National Water and Sanitation Policy (GoSL, 2010), and the National Environmental Health Policy (GoSL, 2000) and the Decentralisation Policy [15]. Agenda for Prosperity In the Agenda for Prosperity (A4P) there are five focus areas, one of which is Water and Sanitation (GoSL, 2013). The A4P emphasizes the important role of having reliable, affordable access to good sanitation and safe drinking water, making reference to the burden of disease associated with poor water and sanitation. Strategic WASH priorities set out in the A4P include: Improve access to integrated WASH services and develop its information system Scale-up and promote the Community Led Total Sanitation programme Improve household storage and treatment of drinking water Box 7-1: Six principal objectives relating to WASH are detailed in the A4P 1. Improved capacity of key government institutions to implement WASH 2. Improved water resource management at national, district and local levels 3. Improved coordination and harmonization of the delivery approaches of state and non-state actors, and partners 4. Enactment of a new legal and regulatory framework to implement the National Water and Sanitation Policy 5. Development and operationalization of WASH sector investment plans 6. Increase capacity for monitoring and evaluation of data collection systems, analysis and feedback to improve sector wide information and decision making Source: [9] National Water and Sanitation Policy (NWSP) In 2010 the National Water and Sanitation Policy (NWSP) for Sierra Leone was published by the Ministry of Energy and Water Resources (MoEWR). The policy promotes integrated water resource management, to ensure that water does not become a constraint to economic development [15]. In 2013 the MoEWR was split into the Ministry for Energy and the Ministry of Water Resources which is a clear indication of the importance the management of water resources is given for the future development of Sierra Leone. 53

54 In regard to self-supply some indications can be found in the specific objectives for rural water supply that the communities are to be encouraged to invest in and manage their own water supply and sanitation services. E.g. [9]: To ensure that communities participate in the planning, construction, ownership, operation and maintenance of their water supply schemes To promote participation of the private sector in the delivery of water supply related goods and services To enable local community ownership of rural water supply schemes To increase communities participation in financing their water supply programs Although a step in the direction of acknowledging self-supply as service delivery approach it is not yet formally recognised as such. Some statements of government officials on self-supply are: There is basic knowledge existing on self-supply but the challenge is how to capacitate the communities to enhance/improve on the systems Self-Supply is a good and viable option which has been practised for several years in SL. Now the challenge is to enhance the knowledge and put it to the next level Not only the communities need to be sensitised but the Ministry incl. the local councils as well Monitoring, done by the engineers on district level, mostly concerns to conventional water supply. Self-Supply initiatives are not monitored yet This clearly shows that there is little knowledge on how to address the issue of self-supply. Table 7-1 shows the distinct approaches of government-borne water supply in Sierra Leone. For the rural areas it shows that there is a gap in serving communities smaller than 150 persons. Table 7-1: Approaches and classification to communal, government borne water supply Urban Peri-urban Rural (small towns and villages) Population > 20,000 Population between 5,000 Communities of 150 5,000 Full reticulation and customer 20,000 people connections Reticulation and limited or 250m fetching distance 100 lpcd full-house connection people per water point 60lpcd (dug or drilled wells) 20 lpcd Source: [9] The Rural Water Supply and Small Towns Strategy Paper issued by the Ministry of Water Resources in 2013 suggests self-supply as a decentralised approach for households and small communities > 150 people, where there is little incentive for state utilities to invest (large sums) of money to extend water supply coverage [11]. 54

55 7.2 The appropriateness of the EMAS technology to support self-supply EMAS is the acronym for Escuela Móvil de Agua y Saneamiento (Mobile School for Water and Sanitation), in Bolivia, whose director, Wolfgang Eloy Buchner, developed the EMAS pump in the 1990 s. EMAS is not only the name of the mobile school for water and sanitation, but also a whole technical and social concept of water and sanitation which includes rain water harvesting, solar water heaters, wind power, hydraulic rams, water treatment, small tanks and sinks, a variety of hand and foot pumps, and ferro cement tanks. The aim of the technologies and systems is to achieve the necessary supply of drinkable water, and water for micro irrigation in rural and sub urban areas [16]. The EMAS water and sanitation technology, labelled as low-cost technology was developed to encourage families to incrementally improve their household infrastructure and hence add value to their household living conditions and lifestyles. EMAS implements its strategy primarily through the training of local independent technicians and typically partners with other organisations and local/national governments for implementation, and pro-motes the same strategy through trainings and assessment trips [14]. The applicability of the chosen technology to support self-supply is a main part of the evaluation. It is done with the application of the Technology Applicability Framework Results of TAF assessment on the EMAS technology in the Western Rural and Kenema districts Limitations of the method According to the ToR (refer to Annex 1) the evaluation approach was guided by the Technology Applicability Framework (TAF), a decision support tool on the applicability, scalability and sustainability of a specific WASH technology to provide lasting services in a specific context and on the readiness for its introduction, developed within the WASHtech project. For a comprehensive description of the method kindly refer to The TAF is designed to assess a single WASH technology which is or will be used to provide WASH services in a district or region. Each assessment has to undergo a four stepped procedure on which at the end only one technology is being assessed. This approach would have by far exceeded the given time scale for the evaluation mission, since the intention of the evaluation is to formulate recommendations on whether and how the concept of selfsupply can be accelerated in Sierra Leone rather than looking at each technology implemented by the project. The evaluator utilised the 18 TAF indicators and adapted and limited the questions in a way to come to meaningful conclusions on the potential of accelerating the concept of self-supply with support of the EMAS technology. (Kindly refer to Annex 7 for the questions adapted and developed for the 18 indicators). Due to the limitations of applying the TAF mentioned above, the scoring was not done in a participatory way, e.g. as proposed by the TAF guidelines in a stakeholder workshop, it was rather done by the evaluator on the basis of interviews and group discussions held in the course of the evaluation. Therefore the results solely represent the findings of the evaluator and can serve only as a rough analysis on the scalability of the technology to support self-supply in Sierra Leone. 55

56 Scoring Figure 7-1: Graphical profile showing the results of the TAF scoring Interpretation Social [Ind.1]: From the users and buyers side the indicator was scored positively. The demand might not be obvious at first but once the people are aware of the benefits they want to have it for themselves, or those who have already installed some components don t want to miss them anymore. The willingness of the HH is there to pay for their systems and cover maintenance costs. Additional values can be added in upgrading their systems increasing the service levels. Most of the HH chosen in the project have more than only one component of the technology. E.g. those having an EMAS pump simultaneously installed, or are planning to do so, an overhead tank to distribute water per gravity to the house or an outside shower. [Ind.2]: From the perspective of the service providers of the technology the indicator was scored yellow meaning that the technicians are all trained on the technical requirements of the technology but lack knowledge in business administration. This can be overcome with additional training and capacity building activities. [Rec 14] [Ind.3]: Critically seen are the capacity and the willingness of the government to promote a change of the perception of people that water should be provided for free and to convince to take own initiative to improve the WASH service levels. 56