(FY ) TERMS OF REFERENCE DRAFT [DATE]

Transcription

1 WWF Deutschland Evaluation of the project Increasing the resilience of forest ecosystems against climate change in the South Caucasus countries through forest transformation (FY ) TERMS OF REFERENCE DRAFT [DATE] Project Name(s) Project Location(s) Project Reference Number(s) Names of Project Executants (WWF Office, name of project/programme manager) Project Duration (from start year) Period to Be Evaluated Project Budget Sources and Amounts (for period to be evaluated) Names of Implementing Partners (if relevant) Increasing the resilience of forest ecosystems against climate change in the South Caucasus Countries through forest transformation Armenia, Azerbaijan, Georgia DCI/ENV/2009/10/19 WWF Germany through WWF Caucasus Programme Office, WWF Armenia and WWF Azerbaijan (including one-year no-cost extension) Total project duration EUR 2.3 Mio. National Forest Authorities PROJECT BACKGROUND AND OVERVIEW In line with its overall strategy in the Caucasus, and paying tribute to the importance and sensitivity of forest there, WWF had started a series of projects to improve forest legislation and to increase close to nature silvicultural practice in the three Caucasus countries in An early EU FLEGT project (administered through World Bank and implemented in a cooperation with IUCN) focused on forest legislation and forest law enforcement. A second FLEGT project which receives additional support from the Austrian Development Agency was later granted and is now ongoing with an increased scope, covering also sustainable forest management and good forest governance. Later projects concentrated on reforestation of deforested and degraded sites and contributed to an increased awareness in the region on the importance of forests for different public services (water storage, erosion control, micro climate etc.). In light of the potential climatic changes in the Caucasus, stress tolerance and resilience of forest became increasingly important, especially Soviet-time forest monoculture plantations proved problematic in this respect. This lead to the development of the above mentioned project. The overall objective of the project is to increase the resilience of forest ecosystems in the Southern Caucasus against climate change impacts and to improve biodiversity and livelihoods of local populations. The overall objective addresses the overarching threat of climate change to biodiversity and to forest ecosystem services which support the livelihoods of rural communities. Those services include protection of soils and water supply and quality, and timber and non-timber forest products. The Objectively Verifiable Indicator for this objective is: By 2015 (two years after the proposed action s completion), the governments of Armenia, Azerbaijan and Georgia will have adopted and started to implement policies that will make forests and the services they provide highly resilient to climate change 1

2 The following four results were envisaged within the project. Result 1 - Selected forest stands vulnerable to climate change have been transformed into highly resilient "close to nature" forest stands. At six pilot sites (two in each of the three target countries), the action will demonstrate a variety of silvicultural measures for transforming vulnerable forest stands into stands which will be able to withstand the expected impacts of climate change. Result 2 - Silvicultural guidelines for the transformation of monoculture stands into more resilient stands are elaborated, published in three languages and made available for relevant officials and experts. Result 3 - The capacities of forest administration experts to develop silvicultural strategies to transform monoculture stands into stable, site-adapted forests are increased. Result 4 - The awareness of local communities about the importance of forest rehabilitation with regard to mitigating negative biotic and abiotic impact of climate change is improved. The project currently is in its fourth year of implementation (after approval of a one-year no-cost extension). The purpose of the no-cost extension was to gain additional time for maintenance measures with respect to the selected forest stands and to consolidate activities on the ground. At the time of the planned evaluation, the project will be phasing-out. The learnings from this project are expected to contribute to the overall forest program of WWF in the Caucasus, especially to the second EU FLEGT project. EVALUATION PURPOSE AND USE, OBJECTIVES, AND SCOPE The scope of the evaluation is the above mentioned project. An ex-post project evaluation was planned for within the design of the project and is defined as an activity in the logical framework of the project. As this is not a mid-term evaluation, and the work on improving silvicultural capacity in the Southern Caucasus of WWF will only be continued in slightly different settings, the focus of the evaluation is not to improve project performance. On the one side, the evaluation shall give an external account on the overall performance of the project and lessons learned that can be drawn for future project work; on the other side make additional suggestions on how to increase the sustainability of the project. The auditor shall also be task to compare his/her findings with respect to the project with earlier ROM Reports of the EU. Findings, conclusions and recommendations will be used for future similar interventions. The final evaluation report will be submitted by WWF to the donor (European Delegation in Georgia). A Management Response to the evaluation will be prepared by the International Project Manager. EVALUATION CRITERIA AND GUIDING QUESTIONS Criterion 1: Relevance and Quality of Design RQ1. Focal conservation targets and related goals: Is there a clear and relevant definition of ultimate conservation success in terms of improved status of conservation targets? Are conservation targets clearly defined and justified and related goals SMART? RQ2. Relevance to context, priorities of stakeholders, and objectives: Has the project/ programme focused on and does it remain relevant to issues of highest priority? RQ3. Suitability of strategic approach: Has the project/programme taken and will it continue to take the best, most efficient strategic approach? RQ4. Coherence and sufficiency of project portfolio: Is the project a logical, necessary and integral element in the overall forest program of WWF CauPO? RQ5. Relevance to WWF priorities: Does the project make a clearly aligned and meaningful contribution to attaining WWF s 2020 and 2050 goals, as outlined in the GPF? RQ6. Adherence to WWF social policies. How well has the social context been understood by the project team? 2

3 Criterion 2: Efficiency Efic1. Financial & Administrative Resources: Is actual spend in line with the budget? Are there improvements to be made in financial planning and resourcing? Efic2. Use of Time: Have there thorough, well founded work plans been implemented according to plan, monitored, and adapted as necessary? Efic3. Human Resources: Were human resources appropriate, adequate, efficiently organized and operating effectively? Efic4. Resource use: Is the project/programme delivering value for money in that costs are reasonable given the outputs and outcomes generated? Criterion 3: Effectiveness Efct1. Planned result verses Achievement: Focusing on stated objectives, desired outcomes, and intermediate results (as opposed to delivery of activities and outputs), what has and has not been achieved (both intended and unintended)? Efct2. Significance of Progress: What is the significance/strategic importance of the progress or any lack thereof made to date? To what extent have targeted key factors drivers, opportunities, threats been affected to the degree they need to be to achieve the stated goals? Efct3. Coordination & Communication: To what extent has coordination/communication been effective within and between the implementation team, stakeholders, partners and participants and external donors? Efct4. Improving Effectiveness: What lessons can be taken and applied to improve effectiveness in the coming years? Criterion 4: Impact This evaluation is not about a rigorous impact assessment and the following questions do not need to be analysed with a relevant methodological approach. Rather do we ask the evaluator to collect evidence for lasting positive impact. Imp1. Evidence of Change: To what extent has the project attained its stated vision and goals, in terms of outcomes effecting positive change in biodiversity quality, ecosystem services and, in turn if relevant, human wellbeing? Discuss observed impacts at all appropriate scales local, (pilote sites), national (policy & capacity), regional, and present evidence? Criterion 5: Sustainability Sust1. Evidence for Sustainability: Is there evidence that the following key ingredients are being established or exist to the extent necessary to ensure the desired long-term positive impacts of the project or programme? o Necessary policy support measures. o Adequate socio-cultural integration, including no negative impact on affect groups and/or on benefits realized by them, as well as ensuring necessary motivation, support, and leadership by relevant individuals and groups. o Adequate institutional and organisational capacity and clear distribution of responsibilities among those organisations or individuals necessary to ensure continuity of project activities or impacts. o Technical and economic viability and financial sustainability. o Technology (if applicable) that is appropriate to existing conditions and capacity. 3

4 Sust2. Risk and Mitigation: What external factors could have a high or medium likelihood of undoing or undermining the future sustainability of project/programme positive impacts? Is the project adequately anticipating and taking measures to ensure resilience to these? Sust3. Exit Phase Out Plan: To what extend has the project team a clear understanding what needs to be done in order to hand over the responsibility for the pilot sites? Based upon existing plans and observations made during the evaluation, what are the key strategic options for the future of the project/programme (e.g. exit, scale down, replicate, scale-up, continue business-as-usual, major changes to approach)? Criterion 6: Adaptive Capacity AC1. Applying Good Practice: Did the team examine good practice lessons from other conservation/ development experiences and consider these experiences in the project/programme design? AC2. Monitoring of status: Did the project/programme establish a baseline status of conservation targets and key contextual factors? Is there ongoing systematic monitoring of these? AC3. Monitoring of efficiency, effectiveness, impact: o Did the project track intermediate results that clearly lay out anticipated cause-effect relationships and enable definition of appropriate indicators? o Is there ongoing, systematic, rigorous monitoring of output delivery, outcome attainment, and impact measurement, with plausible attribution to WWF s actions? o Is monitoring information being used to support regular adaptation of the strategic approach? o Are lessons documented and shared in a manner that is promoting learning by the project/programme team and the broader organisation? Is this documentation adequate given the pilot character of activities. AC4. Learning: Identify any exceptional experiences that should be highlighted regarding what worked and didn t work (e.g. case-studies, stories, good practices)? AC5. Risk Assessment: How often were the original risks and assumptions revisited during the intervention cycle? Were the risks assessed adequately enough and were external assumptions identified realistically? METHODOLOGY CONSIDERATIONS The evaluators should apply a mixture of: - Desk analysis of existing documentation - Field visit to the project sites - Interviews and focused group discussions with key The methods shall be mixed in such a way to ensure that significant qualitative and quantitative data are gathered as evidence for further analysis and development of recommendations. The following documents will be made available to the evaluator: 1) Project proposal (including logframe, budget and Time plan) 2) Project reports 3) All project deliveries (studies etc.) 4) EU ROM Reports 5) Maps etc. The documentation will be made available through WWF Germany as main point of contact of the evaluator. WWF Germany will also provide a briefing and expects a debriefing before and after the mission. 4

5 Further, the evaluator shall have field interviews with the following people: 1) Regional and international project manager 2) National project managers in all three countries. 3) Local stakeholders in vicinity of pilot sides 4) The EC delegations or any person suggested by the EC Evaluators have to adhere to the principles for ensuring quality evaluations presented in Box 1 of these guidelines. The evaluation mission must make shure that the following principles for quality evaluations are adhered to: Useful: Evaluations must be tailored to respond to the questions and key information needs of those most likely to utilize the evaluation results. Independent: For an evaluation to be impartial and therefore more likely to be objective, respected, and accepted, it must be free from bias in findings, analysis, and conclusions. Inclusive: Evaluators and evaluation managers (those overseeing the evaluation process) must design and conduct the process with an eye to promoting project/program team self analysis, learning, and constructive participation. Timely: Evaluation should be timed to inform key decision-making processes, such as planning a second program phase. Respectful. Evaluations and evaluators must respect the project/programme team and their key stakeholders and supporters. Credible: To be viewed as credible, evaluations must be rigorous, impartial, and conducted by a well-qualified evaluator. Transparent: Findings must be readily available to all stakeholders, relevant stakeholders should have the opportunity to comment on the draft evaluation products. Ethical: Evaluations must adhere to relevant professional and ethical guidelines, be undertaken with integrity and honesty, and be respectful of human rights, differences in culture, customs, and practices of all stakeholders. In applying for this task, the evaluator is requested to prepare an initial evaluation plan (max 3 pages). This plan should contain a brief presentation of approach with the proposed data collection methods and data sources to be used for answering each evaluation question. The plan should also contain a timeline of key dates and a proposed price. Expected outputs The following are expected outputs of the evaluation: a) Extensive debriefing to WWF and the EC with presentation of most relevant findings. b) Draft evaluation report circulated simultaneously to WWF and EC Delegation c) Final evaluation report (not more than 25 pages, excl. annexes) The evaluation report shall be written following the structure given above. The evaluation report should include an overall executive summary (2 pages), individual summaries of the findings for each of the three countries, the presentation of the findings in detail for the Chinese, Mongolian and Russian part respectively and relevant supporting annexes. PROFILE OF EVALUATOR(S) AND WWF SUPPORTING RESPONSIBILITIES Evaluators: The Consultant should be a trained forester (or related field) with proven practical experience in forest management. She/he should have experience with the evaluation of international aid projects. Knowledge of the project region (Armenia, Azerbaijan and Georgia) is an asset. Scope of the consultancy: up to 12 days in the region, travel days and up to six days desk-work in preparation and finalization of the consultancy in home-country. 5

6 WWF Support: Uli Gräbener, Director Monitoring and Evaluation of WWF Germany, will coordinate the selection process and will commission the evaluation. Further he shall be closely involved in the process. He will especially give guidance on the methodology to be applied. Matthias Lichtenberger, International Project Manager, WWF Germany, will coordinate the evaluation and will act as first point of contact to the consultant. He will organize a briefing meeting (most likely virtual) and provide the mentioned documents as well as further information required to carry out the evaluation. He will be supported by the National Project Coordinators. EVALUATION PROCESS, DELIVERABLES, AND TIMELINE Major Evaluation Task/Output Dates or Deadline Who is Responsible Evaluation Terms of Reference finalised Mid July Uli Gräbener, WWF D (UG) Tender documents prepared and consulted with EC Delegation Tender open Deadline for submission of application documents End of July End July End August Evaluator selected Early September EC Delegation Evaluator(s) Contracted Early September UG Evaluation information request sent to relevant sources Early September UG UG & JM (WWF D tender expert) UG with support of ML Sources provide requested information September WWF CauPO and partners Evaluator reviews project information September & October Evaluator Project/programme team arranges for evaluator s visit, including interviews, site visits, and logistics September & October CauPO. Evaluator visits the region End of October, beginning of November Evaluation Team, working with evaluated project staff. Evaluation Team briefs those relevant on preliminary findings. Evaluation report drafted and circulated to relevant staff. Project/programme team review report findings Evaluation report finalised and approved by person/people who commissioned the evaluation. Presentation of evaluation results to Evaluation Manager, evaluated programme, and relevant Network staff. Management response developed by programme leadership (see Annex B, Table D template). 6- to 12-month check-in on progress on management response. 1-2-year check-in on progress on management response. 0,5 day at end of region visit CauPO Early December 2-week review and comment period End of December Within a month of finalising report. An in depth response within 1 month of receiving the report. Evaluator UG Evaluator. UG reviews and gives final approval of report. Evaluation Team Evaluation Manager and evaluated programme 6 to 12 months post-report. Evaluation Manager 1-2 year post report on the management response. Evaluation Manager BUDGET, FUNDING, AND PAYMENT TERMS Consultants shall prepare a budget as part of their offer including daily rates, travel allowances, visa costs and international travels. In-country transport will be provided by the Project. 6

7 Annex 1: Evaluation Reports To support more systematic recording of evaluation findings to advance WWF s broader organisational learning, all evaluators should follow, to the extent possible, the evaluation report structure below and complete the following table (Part B), to be attached to the evaluation report. Part A - Report Table of Contents Template The following provides a basic outline for an evaluation report. While this should be easily applied to evaluations of simpler projects or programmes, adaptation will be needed to ensure reports of more complex programmes (e.g. Country Offices, multi-country regions, eco-regions, Network Initiatives) are well organised, easy to read and navigate, and not too lengthy. Title Page Report title, project or programme title, and contract number (if appropriate), Date of report, Authors and their affiliation, Locator map (if appropriate) Executive Summary (between 2 to 4 pages) Principal findings and recommendations, organised by the six core evaluation criteria Summary of lessons learned Acknowledgements Table of Contents List of Acronyms and Abbreviations Body of the report (no more than 25 pages) A. Introduction (max 3 pages) - Concise presentation of the project/programme characteristics - Purpose, objectives, and intended use of the evaluation (reference and attach the ToR as an annex) - Evaluation methodology and rationale for approach (reference and attach as annexes the mission itinerary; names of key informants; a list of consulted documents; and any synthesis tables containing project/programme information used in the exercise) - Composition of the evaluation team, including any specific roles of team members B. Project/Programme Overview (max 5 pages) - Concise summary of the project or programme s history, evolution, purpose, objectives, and strategies to achieve conservation goals (attach theory of change including conceptual model, results chain or logical framework and project monitoring system as annexes) - Essential characteristics: context, underlying rationale, stakeholders and beneficiaries - Summarise WWF s main interest in this project or programme C. Evaluation Findings (3-5 pages) - Findings organised by each of the six core evaluation criteria, including sufficient but concise rationale. - Tables, graphics, and other figures to help convey key findings D. Recommendations (3-5pages) - Recommendation organised each of the six core evaluation criteria, including sufficient but concise rationale recommendations should be specific, actionable and numbered. - Project/programme performance rating tables to provide a quick summary of performance and to facilitate comparison with other projects/programmes (see the Summary Table Part B, below). E. Overall Lessons Learned (max 3 pages) - Lessons learned regarding what worked, what didn t work, and why - Lessons learned with wider relevance, that can be generalised beyond the project F. Conclusions - General summation of key findings and recommendations Annexes Terms of Reference Evaluation methodology detail Itinerary with key informants Documents consulted Project/programme theory of change/ logical framework/ conceptual model/ list of primary goals and objectives Specific project/programme and monitoring data, as appropriate Summary tables of progress towards outputs, objectives, and goals Maps Recommendations summary table 7

8 Annex 2: Evaluation Summary Table scoring against core evaluation criteria Evaluators are to assign the project/programme a Rating and Score for each criterion as follows: o Very Good/4: The project/programme embodies the description of strong performance provided below to a very good extent. o Good/3: The project/programme embodies the description of strong performance provided below to a good extent. o Fair/2: The project/programme embodies the description of strong performance provided below to a fair extent. o Poor/1: The project/programme embodies the description of strong performance provided below to a poor extent. o N/A: The criterion was not assessed (in the Justification, explain why). o D/I: The criterion was considered but data were insufficient to assign a rating or score (in the Justification, elaborate). Evaluators are also to provide a brief justification for the rating and score assigned. Identify most notable strengths to build upon as well as highest priority issues or obstacles to overcome. Note that this table should not be a comprehensive summary of findings and recommendations, but an overview only. A more comprehensive presentation should be captured in the evaluation report and the management response document. Even if the report itself contains sensitive information, the table should be completed in a manner that can be readily shared with any internal WWF audience. Rating/Score Relevance Quality of Design Efficiency Effectiveness Impact Sustainability Adaptive Management Description of Strong Performance The project/programme addresses the necessary factors in the specific programme context to bring about positive changes in conservation targets biodiversity and/or footprint issues (i.e. species, ecosystems, ecological processes, including associated ecosystem services supporting human wellbeing). 1.The project/programme has rigorously applied key design tools (e.g. the WWF PPMS). 2. The project/programme is hitting the right 'pressure points' to meet necessary and sufficient conditions for success 1. Most/all programme activities have been delivered with efficient use of human & financial resources and with strong value for money. 2. Governance and management systems are appropriate, sufficient, and operate efficiently. 1. Most/all intended outcomes stated objectives/intermediate results regarding key threats and other factors affecting project/programme targets were attained. 2. There is strong evidence indicating that changes can be attributed wholly or largely to the WWF project or programme 1. Most/all goals stated desired changes in the status of species, ecosystems, and ecological processes were realised. 2. Evidence indicates that perceived changes can be attributed wholly or largely to the WWF project or programme. 1. Most or all factors for ensuring sustainability of results/impacts are being or have been established. 2. Scaling up mechanisms have been put in place with risks and assumptions re-assessed and addressed. 1. Project/programme results (outputs, outcomes, impacts) are qualitatively and quantitatively demonstrated through regular collection and analysis of monitoring data. 2. The project/programme team uses these findings, as well as those from related projects/ efforts, to strengthen its work and performance 3. Learning is documented and shared for project/programme and organisational learning Evaluator Rating/ Score Evaluator Brief Justification 8