Terms of Reference. Final Project Evaluation: Integrated River Basin Management at Koshi River Safeguarding the lives of millions

Transcription

1 Terms of Reference Final Project Evaluation: Integrated River Basin Management at Koshi River Safeguarding the lives of millions 1. BACKGROUND TO THE FINAL EVALUATION 1.1. Programme background The project was initiated due to the recognition that Koshi basin is major water system providing well-being and development for millions of people in the region. At the same time primary water users are exhibiting stress signals (mainly local communities), as a result of environmental catastrophes (flooding of 2008, which displaced tens of thousands of people), large scale infrastructure development plans (mainly hydropower), and climate change (melting of glaciers and climatic variation). At the same time, the Government of Nepal (GoN) has developed National Water Plan (NWP) in 2005, which stresses strongly the need for Integrated River Basin Management, in order to decrease the increasing amount of inter-sectoral, inter-regional and riparian conflicts related to river systems (NWP Nepal, 2005). National development priorities for the GoN as outlined in the current planning documents (e.g. included in the draft National Development Strategy Paper, NDSP 2009; Trade, Private Sector Development and Investment Opportunities discussion paper, 2009, and Transformation of Nepal s Economy Agenda for Growth and Social Equity discussion paper, 2009) are economic growth and dramatic increased investments in energy, agriculture and rural development sectors. These are very typical priorities for any developing country, but the government s emphasis on ensuring social equity and the role of civil society envisioned to achieve these ambitious plans single out Nepal from other developing countries. It is often contested that economic growth and social equity are difficult to match, while the fact is that they shouldn t be mutually exclusive. This project was initiated based on the ground realities (rampant poverty, unsustainable land use practices and lack of capacity at the local level) combined with national priorities. During the scoping exercise WWF found that even though there were necessary plans and policy framework in place, their implementation is limited, partly due to the political instability, partly because of government s limited resources and capacity (financial and human) to deliver services. WWF Nepal initiated a stakeholder engagement process, reaching from local communities to the highest level of government, which was culminated in February 2009 with an inauguration of pilot IRBM initiative, jointly implemented by WWF Nepal and Water and Energy Commission Secretariat (WECS), the GON and financially supported by the Ministry for Foreign Affairs of Finland (MFA). 1

2 From a long-term perspective, this project is designed to act as a showcase to demonstrate integrated river basin management approach and help replicating successes in other parts of the country. However, since the basin and the water systems it contributes to are of trans-boundary nature, the objective is to develop a multi-country basin initiative, as practiced in other parts of the world as well. This is one of the ways to implement the approach in the appropriate scale, which is required by freshwater systems like the one in question. While this project focused on one of the major sub basins of Koshi basin, Indrawati sub-basin, the long term plan is to upscale this IRBM project to cross-border basin wide initiative. The strategy is focused on stepwise approach and demonstration, which are the main characteristics of this project Integrated River Basin Management at Koshi River Project Objectives The trans-boundary Koshi basin provides freshwater for livelihoods and ecosystem services for millions of people in China, Nepal and India. Due to pressure originating from anthropogenic sources and climate change, Koshi is under threats that compromise the stability of the system. The overall development objective of the project is By 2020, water and related resources of Koshi basin are best managed (in the face of climate change and development scenario) to promote socioeconomic development for the benefit of all the local people, especially the poorest and marginalized while maintaining the ecological balance. The immediate objectives include: 1) By 2016, institutional structures are functional having broad-based partner support to implement democratic IWRM approach, which empowers local communities and marginalized groups to participate in decision-making, planning and implementation 2) By 2016, local communities in target areas are empowered and capacitated to manage natural resources sustainably and to reduce their vulnerability against the impact of climate change through rights-based approach, complemented by optimized economic benefits from water and other natural resources As in the first phase, the project followed the same approach that uses three E-pillars of IWRM as the foundations: Social Equity Economic Efficiency Environmental Sustainability At the catchment level, the project has been implemented through the 9 Integrated Resource Management Committees (IRMCs), the legally registered institutions that were established and institutionalized during the first phase. The IRMCs are responsible for planning and implementation 2

3 of the IWRM activities at catchment levels with the technical and financial support from project office. The project proposed to benefit directly at least 4,000 households and 20,000 individuals across the project intervention areas in Indrawati. Indigenous people (IPs), Marginalized Janajatis and disadvantaged groups, primarily women will be project's prime focus. IRMCs, local water users, Ecoclubs, water smart communities, are the direct beneficiaries. The indirect beneficiaries were identified to be primarily the stakeholders indirectly associated with IRMCs and different user groups (water, irrigation, farmers, forest and agriculture). The Koshi project is implemented by Water and Energy Commission Secretariat (WECS), Government of Nepal and WWF Nepal. The project has had two phases: phase I and phase II with extension to 2016 because of earthquake in Nepal in Both phases are funded by the MFA (85 %) and WWF Finland (15 %). Budget for phase I was and for phase II RATIONALE, PURPOSE AND OBJECTIVES OF THE EVALUATION The project has been running now for six years with one year extension due to the earthquake. This evaluation should analyze if project has achieved the goals and results that were determined before the start of the project. Therefore this evaluation will analyze the success of the project by reflecting on the objectives and results of the project, originally set in the project logframe and document. Furthermore thorough analysis is needed to be able to plan the future steps for WWF on water sector. The Koshi project was included in a wider audit of MFA funded projects in Nepal which was conducted by KPMG in May This audit was focused on the administration and management of the project, and therefore it didn t capture the content and achievements of the project. Therefore it was decided that a full final evaluation on the project should be conducted. The purpose of this evaluation is to: Analyze the performance of the project to date by reflecting on the objectives and results of the project, set in the project logframe and document. Analyze to what extent the project supports the cross-cutting objectives that are outlined in the Development Policy Program of Finland (see section 4.1) Analyze how well the project supports and is linked and incorporated to the wider conservation programmes of WWF Nepal or projects and programmes by other development partners Present the lessons learned from the project Analyze the partnership and cooperation between the partners 3

4 Reflect achievements and performance of this project on similar water sector initiatives in Nepal Reflect the role of WWF Nepal in water sector in Nepal and possible added value of WWF implementing such projects Provide recommendations for future role of WWF Nepal on water sector Provide recommendations on up scaling IRBM approach in Nepal with potential funding opportunities for WWF Provide recommendation on how WWF water sector support could cooperate and strengthen Finnish bilateral cooperation in Nepal on water sector Provide summary status of the IRMCs and recommend strategies for their long term sustainability 3. SCOPE OF THE FINAL EVALUATION: STAKEHOLDERS AND GEOGRAPHICAL AREA The evaluation will cover the project operating time span of: to The evaluation is expected to be future oriented strategic evaluation but the past performance and especially the lessons learned shall be assessed to the extent it is necessary to achieve the project objectives. The evaluation team is expected to concentrate especially on issues of impact/influence and sustainability, coherence, compatibility and their interlinkages. Crosscutting issues shall be paid due attention where considered of particular importance (for example. gender etc.). The same concerns the issues of Finnish value added in supporting the forestry and water sector and WWF s role as implementing entity. The evaluation team shall pay attention to compliance and coherence and complementarity of the policies and practices among the partners and with the policies of the partner countries or other donors in the sector. Certain aspects of water will be reviewed through representative examples. For example watershed management in forestry, biodiversity, conservation and agriculture interventions. The existing material i.e. project documents, appraisals, mid-term reviews, progress reports, steering committee decisions, completion and other evaluations shall be utilized to the extent possible. An in-depth study of documentation is an important aspect of the evaluation and an adequate time shall be reserved for it. The supporting field visits and interviews shall be arranged so that maximum benefit can be achieved from the time allocation for them. The central findings and conclusions shall be synthesized in the main report. Stakeholders that will be included in the evaluation are: WWF Nepal WWF Finland (in Finland) WECS 4

5 Embassy of Finland, Nepal Integrated Resource Management Committees (IRMCs) Sample of beneficiary communities District Soil, Forest, Agriculture offices The geographical area of the evaluation will cover whole project area from where the evaluation team can propose suitable sample size for consultations. Due to time and resources restrictions, the evaluation team is not expected to cover all the projects but to take a representative sample to ensure proper analysis. 4. ISSUES TO BE ADDRESSED AND EVALUATION QUESTIONS 4.1. Evaluation questions The evaluation team is expected to draft the specific evaluation questions which should cover the following aspects. The evaluation team should, however, use their professional experience and knowledge by adding questions deemed necessary. Analyze the performance of the project to date by reflecting on the objectives and results of the project, originally set in the project logframe and document. Analyze WWF s role on water sector Analyze the performance of the project compared to the other similar initiatives. Identify the added value and possible challenges on WWF on IRBM approach implementer in Nepal and region 4.2. Alignment with the Finland s Development Policy Program objectives The project will be evaluated based on the Evaluation Manual published by MFA Finland in 2014 ( ure=en-us) The evaluation should reflect on how the project has promoted the priority areas of the Finland s Development Policy Programme, which are: o a democratic and accountable society that promotes human rights, o an inclusive green economy that promotes employment, o sustainable management of natural resources and environmental protection, o human development. In addition the evaluation should clearly present how the cross-cutting objectives have been incorporated to the project design and implementation and how the project has promoted these objectives which are: o gender equality, o reduction of inequality and o climate sustainability. 5

6 4.3. Evaluation criteria The evaluation team should incorporate aspects related to the following criteria in the evaluation (specific question examples can be found in the evaluation manual): Relevance Are the results, purpose and overall objectives of the project in line with the needs and aspirations of the beneficiaries, and with the policy environment of the intervention? Is the project consistent with the needs and priorities of its target group and the policies of the partner country and donor agencies? Has the situation changed since the approval of the intervention (e.g. programme/project) document? To what extent the approach of Finland to support the sector has been/is relevant from the point of view of the most urgent needs of the partner country s water sector challenges? Lessons learned? And role of WWF as implementing entity? Efficiency How well the various activities have transformed the available resources in to the intended outputs. Can the cost of the intervention be justified by the results? What lessons can be learned from transforming the available resources into results? Effectiveness How well the results achieved have promoted the purpose of the project. Has the project achieved its objectives or will it do so in the future? What are the particular points of interest for lessons learned? Impact Has there been a change towards the achievement of the overall objective(s) as a consequence of the project? Both intended and the unintended impacts are reviewed. What are the overall effects of the intervention, intended and unintended, long term and short term, positive and negative? Sustainability The degree to which the benefits produced by the project continue after the external support has come to an end. Will the benefits produced by the project be maintained after the termination of external support? What kind of positive features of or limitations for sustainability can be identified? To what extent the cross-cutting issues have played a role in promotion of sustainability in a positive or negative sense? What are the most important lessons to be learned for future? Coherence Whether and how the goals that are central to Finland s development cooperation policy or the partner country s development policy are taken into account in the planning and implementation. How have they affected the intervention? 6

7 5. METHODOLOGY Multiple methodologies should be used, both quantitative and qualitative (for instance meeting documents and participant lists and personal interviews). Validation of results must be done through multiple sources. No single statements should be taken as a general outcome. However, the detailed methodology will be proposed by the team conducting the evaluation. A careful review of existing documentation is of particular importance and should be given a due attention to in the first preparatory phase. A well formulated approach and methodology combined with a realistic time schedule and evaluation matrix form a solid basis for the field visits. During the field visit all modern interactive methods shall be taken into use to extract the information needed to complement and countercheck the evidence found in the documentation. Debriefing sessions with relevant stakeholders shall be arranged and also the WWF consulted on the observations and to identify the possible gaps and errors in information. Complementary interviews and e-questionnaires can be organized in the final stages of report drafting. Possible methods that can be used for data collection and analysis are: o Desk study covering project work plans, project reports, project documents, Finnish Development Policy program and other materials o Identify sample sites and catchments and conduct direct field observation o Personal and group interviews with relevant stakeholders and key informants o Other data/information collection tools such as surveys and questionnaires o The methodology should ensure that the evaluation process: o is participatory and inclusive involving different stakeholders including the final beneficiaries o takes into consideration gender and human rights aspects o respects local traditional and cultural norms 6. THE EVALUATION PROCESS, SCHEDULE and REPORTING The time period of the evaluation will be 3 months, effective from the day the contract is signed. The evaluation should start in September It is expected that the evaluation team will Finalize the evaluation methodology and prepare an inception report which will be commented by WWF before starting the evaluation activities Evaluate relevant sources of information through desk reviews and literature studies Carry out field visits, observations and surveys Interact/discuss with relevant stakeholders Prepare draft report with evaluation findings and recommendations Receive and incorporate feedbacks from stakeholders before finalizing reports Organize a feedback session for WWF Finland and WWF Nepal, including WECS and Embassy of Finland to receive comments and test the results (skype meeting or similar) Prepare final report of the evaluation in English and submit it to WWF Nepal. Consultant should submit 2 copies each (hard and soft copies) of the final report Inception report and the final workplan should be submitted to WWF Nepal after 3 weeks the contract has been signed and the final report by final report by 15 of December

8 The final report should be: o maximum of 50 pages + annexes o compact and include only relevant information o answer to the main evaluation questions o reflect on the cross-cutting objectives o include pictures o include findings and clear table of recommendation o List of interviewed stakeholders Publication of the report shall be responsibility of WWF Nepal and WWF Finland. WWF Nepal will hold copy right of the report and the consultant cannot publish part or full report without prior approval from WWF Nepal. 7. EXPERTISE REQUIRED The evaluation team should comprise of experts (1-3 members) that have understanding of Nepal as the operational environment and/or expertise on the region and at least one of the experts should be from Nepal and at least one of the team members should be a woman. One of the consultant should act as a team leader and WWF Nepal will sign the contract with the one company or lead consultant only. The team should have strong expertise in conducting evaluations. In addition the team should have expertise in the following thematic concepts: Evaluation experience preferably on MFA funded projects IRBM and water sector in general Environment and livelihoods o socio-economic aspects o governance/institutional aspect o climate change and adaptation o Sustainable livelihoods The cross-cutting themes: Human rights, gender equality and climate sustainability. In addition: o The evaluation team should have knowledge on Finnish Development Policy Programme and its objectives o The evaluation team should use the Evaluation Manual (MFA Finland 2014) as the guidelines for the evaluation o The evaluation team should have ability to speak and understand Nepali language. 8. BUDGET, PAYMENT AND TAXES The total available budget for this evaluation is 30,000 euros, which cannot be exceeded. Consultant will take of all relevant taxes, travel arrangements etc. 50 % will be paid after receiving inception report and detailed workplan. The consultant will be contracted by WWF Nepal. 8

9 9. MANDATE The evaluation team is entitled and expected to discuss matters relevant to this evaluation with pertinent persons and organizations. However, it is not authorized to make any commitments on the behalf of WWF Nepal or WWF Finland. 10. SELECTION PROCESS WWF Nepal with other stakeholders will select the evaluation team based on the proposals from the interested consultants. The expression of the interest, detail workplan and the CVs of the evaluation team should be submitted to WWF Nepal by Friday, 19 August 2016 by 17: ANNEXES. 1. Link to the evaluation manual. 2. Link to the OECD/DAC and EU quality standards for evaluations 3. Annual report of the project, year 2015 (separate document) 4. Project document of Phase II (separate document) 9

10 10

11 Evaluation Manual: re=en-us OECD/DAC and EU quality standards for evaluations: CROSS-CUTTING OBJECTIVES Has adequate and appropriately disaggregated baseline been data available on the cross-cutting objectives for planning, implementation, monitoring and evaluation? Have clear objectives and indicators been defined for the cross-cutting objectives? Have cross-cutting objectives been integrated into programme activities and/or are specific targeted measures been planned? Have adequate resources and expertise been allocated for implementation that promoted crosscutting objectives? Have potential negative effects on cross-cutting objectives been identified? Have measures been planned to minimize them? Are cross-cutting objectives systematically and explicitly integrated into programme monitoring, reporting and evaluation? Does the programme planning, implementation and decision-making directly involve people, representatives and expertise related / relevant to cross-cutting objectives? If cross-cutting objectives have not been addressed in the programme, has this been explicitly justified? ENVIRONMENT, CLIMATE CHANGE, DISASTER RISKS Relevance Does the programme address the environmental issues identified in the problem analysis? If not, are there good reasons for giving priority to other problems? Does the programme incorporate environmental objectives addressing its potential environmental impacts? Effectiveness Does the programme comply with its environmental requirements and objectives? Does the environmental protection measures produce the expected results? Efficiency Is the programme making efficient use of environmentally damaging means? Are adequate resources or efforts invested in mitigating/optimising environmental impacts? Is the programme handicapped by under-estimated environmental constraints? Sustainability Are programme outcomes threatened by environmental degradation (or resource impoverishment) or disasters? Impact 11

12 What are the programme s contributions to sustainable development? Can these be improved? What are the social and environmental effects external to programme objectives? What are the impacts compared to those predicted by the EIA? 12

13 RELEVANCE Relevance refers to the extent to which the objectives of the programme are consistent with beneficiaries' requirements, country needs, global priorities and partners' and Finland's policies. Problems, needs Are the objectives and achievements of the programme still consistent with the needs and priorities of the stakeholders, including the final beneficiaries? Whose poverty and inequality is the programme focused to reduce? Whose sustainable development is it promoting? Policy priorities Are the objectives and achievements of the programme consistent with the policies of the partner country? Are the objectives and achievements of the programme consistent with global goals, commitments and principles? Are the objectives of the programme consistent with Finland s development policy (incl. regional and country-specific priorities, sectoral and thematic priorities)? Are the objectives consistent with poverty reduction and promotion of sustainable development achieving the cross-cutting objectives, ie. human rights based equality and vulnerability reduction objectives environmental sustainability, climate change mitigation and adaptation, disaster risk reduction Has the situation changed since the approval of the programme?] EFFICIENCY The efficiency of a programme is defined by how well the various activities transformed the available resources into the intended results in terms of quantity, quality and timeliness. Comparison should be made against what was planned. Value for money How well have the activities transformed the available resources into the intended outputs/results, in terms of quantity, quality and time? Can the costs of the programme be justified by the results? Have the contributions by the partner country and the donor(s) been provided as planned? Quality of technical assistance? DEVELOPMENT EFFECTIVENESS Effectiveness describes if the results have furthered the achievement of the purpose of the programme, or are expected to do so in the future. The evaluation will be made against the related indicators. Achievement of immediate benefits 13

14 Is the quality and quantity of the produced results and outputs in accordance with the plans, how are the results/outputs applied by the beneficiaries and other intended stakeholders? To what extent has the programme achieved its purpose or will it do so in the future? Have behavioral patterns changed as planned in the stakeholder institutions or groups at various levels? How far have the changed institutional arrangements and characteristics produced the planned services and improvements? Have the planned benefits been delivered and received, as perceived by all key stakeholders? Are the results/outputs and the programme purpose making a contribution towards reducing poverty and inequality, and promoting sustainable development? DEVELOPMENT IMPACT Impact describes how the programme has succeeded in the attainment of its overall objective, i.e. targeted impact for its beneficiaries. The evaluation will be made against the related indicators. Achievement of wider benefits Has progress been made towards achieving the overall objective(s) of the programme? How is the programme intended to reduce the poverty of the intended primary beneficiaries? Through which transmission channels (prices, employment, transfers, access, authority, and assets) are the programme impacts supposed to be transmitted to the lives of the poor women and men? What is the overall poverty, inequality and sustainability impact of the programme, intended and unintended, long term and short term, positive and negative? Do the indicators for the overall objective show that the intended changes are starting to take place? In whose lives are the poverty, inequality and sustainability impacts starting to make a difference? SUSTAINABILITY Sustainability can be described as the degree to which the benefits produced by the programme continue after the external support has come to an end. Likely continuation of the achieved benefits What are the possible factors that enhance or inhibit sustainability, including ownership/commitment, economic/financial, institutional, technical, socio-cultural and environmental sustainability aspects? Will the benefits produced by the programme be maintained after the termination of external support? Who will take over the responsibility of financing the activities, or have they become selfsustaining? Is there an exit strategy and gradual handing over plan in place and will it ensure sustainability? PROGRAMME MANAGEMENT AND ADMINISTRATIVE ARRANGEMENTS Sound management 14

15 Quality of the day-to-day management? Are possible problems in implementation adequately addressed? What is the quality of work planning, monitoring and reporting incl. use of indicators, resource and personnel management, financial management, cooperation and communication between stakeholders? Is adequate baseline data available and utilised in monitoring and reporting, disaggregated by gender, age group and other relevant categories? Is the balance of power and responsibilities between the various stakeholders appropriate? Have important assumptions been identified? Are risks appropriately managed, including flexible adaptation to unforeseen situations? AID EFFECTIVENESS = EFFECTIVENESS OF AID MANAGEMENT AND DELIVERY Ownership Does the partner institution have a development strategy? Is it operational, results-oriented, and with clear strategic priorities? Has the strategy been developed through a broad consultative process involving relevant stakeholders, such as parliaments, local governments, civil society organizations, private sector? Is the programme finance included in the partner institution s medium term expenditure framework and reflected in the annual budget? Does the partner institution exercise leadership in coordinating aid? Does it have an active and leading role in decision making concerning planning and implementation and in joint decision making structures? Alignment with partner strategies and priorities Is the programme based on the partner s development strategies and periodic reviews of progress? Is conditionality drawn from the partner s development strategy? Is there a limited set of conditions that are mutually agreed, jointly assessed, made public, and results-based? Use of country s own institutions and systems This includes public financial management (i) budget execution procedures eg. funds included in budget, deposited and disbursed through country treasury system, ii) financial reporting procedures and iii) national audit procedures), monitoring, reporting and evaluation, social and environmental assessments Does the partner institution provide reliable assessments of performance, transparency and accountability of country systems, and undertake necessary reforms? Have the country systems been jointly assessed by using mutually agreed diagnostic tools? Does the programme use country systems fully or partially? How have risks of using country systems been identified and mitigated? If country systems are not used partially or at all, is the rationale transparently stated? Is the position regularly reviewed? Is capacity development provided to strengthen country systems? Is staff guidance and internal incentives provided in the MFA to encourage the use of country systems? 15

16 Is the programme day-to-day management accountable to the partner s implementing agency (rather than to the donor)? Are Terms of Reference of externally appointed staff determined by the country implementing agency (rather than by the donor)? Is most of the professional staff appointed by the implementing agency? Is the salary structure of national staff similar to that of the civil service personnel? Capacity development Does the partner institution have a capacity development strategy? Is the capacity development support provided by the programme based on the partner s capacity development strategy? Is technical assistance jointly selected and managed? Does the programme promote use of local and regional resources, including through South- South cooperation? Use of common arrangements Does the programme use common donor arrangements for planning, budgeting and financial management (e.g. joint financial arrangements), procurement, monitoring, evaluating and/or reporting? Are the field missions of the programme conducted as joint missions? Predictability of aid Does Finland provide full and timely information on annual commitments and actual disbursements? In which format / through which mechanism? Has the partner institution integrated this information into its own planning, ie. budget estimates and accounting systems? Have disbursements been made according to the jointly agreed schedule? If not, what are the reasons for this? Does Finland provide regular and timely information on three- to five-year forward expenditure? In which format / through which mechanism? Has the partner institution integrated this information into its own medium-term planning and macroeconomic frameworks? Complementarity Does the programme improve complementarity and division of labour among donors? How has donor coordination been carried out in initial identification and planning? How have comparative advantages between donors been assessed? Harmonized approach to environmental assessments and cross-cutting objectives Are common approaches and procedures been applied for environmental assessments and promotion of cross-cutting objectives? Managing for results Has the partner institution established results-oriented reporting and assessment frameworks with a manageable number of indicators? Does the programme rely on these frameworks? Have the monitoring and reporting requirements of the programme been harmonized with other donors, e.g. by using joint formats for periodic reporting? 16

17 Does the programme develop capacity and demand for results-based management? Mutual accountability Does the partner institution reinforce participatory approaches in formulation, implementation and assessing progress of its development strategies? Does Finland provide timely, transparent and comprehensive information on aid flows? Are both Finland and the partner institution held accountable for commitments in the programme? Are the risks for corruption/possible instances of corruption addressed in the programme implementation and monitoring framework? COHERENCE Do contradictions with other policies prevent implementation and the achievement of the programme s overall objectives? 17