US$M): Sector Board : Environment Cofinancing (US$M (US$M US$M): US$M): ICR Review Coordinator : Nelson

Transcription

1 Public Disclosure Authorized IEG ICR Review Independent Evaluation Group Report Number : ICRR Project Data: Date Posted : 10/28/2014 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Country: Turkey Project ID : P Project Name : Anatolia Watershed Rehabilitation Project Appraisal Actual Project Costs (US$M US$M): L/C Number: L4741 Loan/Credit (US$M US$M): Sector Board : Environment Cofinancing (US$M US$M): Prepared by : J. W. Van Holst Pellekaan Cofinanciers : Local Communities Board Approval Date : 06/01/2004 Closing Date : 06/30/ /30/2012 Sector(s): General agriculture fishing and forestry sector (50%); General water sanitation and flood protection sector (15%); Irrigation and drainage (13%); Agro-industry (13%); Central government administration (9%) Theme(s): Other environment and natural resources management (25% - P); Other rural development (25% - P); Pollution management and environmental health (24% - P); Environmental policies and institutions (13% - S); Regional integration (13% - S) Reviewed by : John R. Eriksson 2. Project Objectives and Components: ICR Review Coordinator : Christopher David Nelson Group: IEGPS1 a. Objectives: The Anatolia Watershed Rehabilitation Project (AWRP) was financed by an IBRD loan and a grant from the Global Environment Fund (GEF). The Project Appraisal Document (PAD) for the AWRP stated that the project development objective (PDO) was "to support sustainable natural resource management practices in 28 microcatchments in Anatolia and Turkey s Black Sea Region and thereby raise incomes of communities affected by resource degradation" (paragraph 4). The IBRD Loan Agreement for the project also stated that the project objective was "to support sustainable natural resource management practices in about 28 micro-catchments in Anatolia and Turkey s Black Sea Region and thereby raise incomes of communities affected by resource degradation" (Schedule 2). The objective in the IBRD Loan Agreement will be used for the purpose of this Review's rating of the project's achievements. The project objective in the GEF Grant Agreement was the same as in the Loan Agreement. The PAD stated that the global environmental objective (GEO), financed by the GEF which funded parts of the AWRP, was "to introduce farming practices which would reduce the discharge of agricultural nutrients into surface and ground water in watersheds draining into the Black Sea in four provinces." (page 3 and Annex 1). Neither the GEF Grant Agreement nor the IBRD Loan Agreement refer to this GEO. b.were the project objectives/key associated outcome targets revised during implementation? No c. Components: Component 1: Rehabilitation of Degraded Natural Resources (appraisal cost US$23.50 million; actual cost

2 US$22.33 million). This component was to provide support for the planning and implementation of activities to be implemented by village communities under the direction of the Ministry of Environment and Forestry (MEF) and the Ministry of Agriculture and Rural Affairs (MARA). The component s primary objective was to protect degraded areas from further degradation, erosion and pollution. Rehabilitation interventions were focused around four sub-components (rehabilitation of forest land, rangelands and agricultural land, as well as environment friendly agricultural practices) in 28 micro-catchments in 6 provinces. Environment friendly agricultural practices were aimed at reducing nutrient discharge into the water bodies in the lower watersheds of four participating Black Sea provinces. Component 2: Income Raising Activities (appraisal cost US$17.57 million; actual cost US$4.67 million). Under this component target communities would be offered a menu of activities designed to raise household incomes in return for participation in conservation activities supported under Component 1. Income generating activities were designed to provide participating communities with the incentives to undertake conservation efforts even if they incurred short or medium term costs (e.g. short term closure of rangelands, closure of forest land ) or if benefits could only be reaped in the long run (e.g. afforestation). The activities varied in accordance with agro -ecological and socio-economic conditions in each village, as well as with farmers resources and needs. The approach was flexible so as to be able to respond to the needs of the villagers and flexibility was maintained during project implementation. The main income generating activities to be financed by the project included : (a) small scale irrigation, including creation of small irrigated perimeters and farm ponds, which was subsequently cancelled resulting in the substantial reduction in this component's cost; (b) investments in livestock improvement, greenhouses and small -scale freshwater fisheries; and (c) farm and crop enterprise diversification (e.g. rain fed and irrigated horticulture, irrigated forage crops, vegetable production, planting trees on field boundaries, agricultural processing and beekeeping ). As explained below the irrigation activity was cancelled and an activity supplying solar heating systems for hot water in farm households was added. These two latter activities were implemented by two Directorates in MARA Component 3: Strengthening Policy and Regulatory Capacity towards Meeting EU Standards (appraisal cost US$0.28 million; actual cost US$0.22 million). This component funded support for (a) the application of the EU Nitrates Directive; (b) development and promotion of Code of Good Agricultural Practices; and (c) institutional support for organic farming. Component 4 : Awareness Raising, Capacity Building and Replication Strategy (appraisal cost US$1.06 million; actual cost US$0.43 million). This component was aimed at supporting (a) public awareness in micro-catchment development among target beneficiaries and other stakeholders; and (b) public awareness, capacity building an d future replication in other Black Sea provinces. Component 5: Project Management and Support Services (appraisal cost US$2.50 million; actual cost US$1.01 million). This component included sub-components addressing (a) project administration; (b) support services such as technical assistance; monitoring and evaluation; and (c) funds for applied research and technology dissemination. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Costs : Actual project costs funded by the IBRD loan were $ million which was 75.8 percent of appraisal estimate of $20 million. One-fifth of the loan was cancelled after the institution charged with implementing the irrigation component was disbanded by the Government and an alternative arrangement for implementing the component was not found. Other reasons for lower costs were reductions in expenditures in Component 1. The actual cost of activities funded by the GEF grant was $ 6.98 million which was almost exactly the same as the appraisal estimate. It is noted that Annex 1 of the ICR contains an error because the appraisal estimate for the IBRD loan in top half of Table (a) "Project Costs by Component" includes the GEF grant, which appears again in the bottom half of Table (a) showing only the costs of the GEF funded activities. Financing : At appraisal it was anticipated that in-kind contributions from local communities of $ 7.70 million would supplement for activities financed by the IBRD Loan, and $ 0.80 million would supplement the GEF Grant. In the event there was no information about contributions from local communities because, as stated in the ICR, the Government made no estimates of the value of these contributions during project implementation Borrower Contributions : The Government's contribution to IBRD financing was $ 4.49 million (61 percent of the appraisal estimate of $7.30 million) due mainly to the cancellation of the irrigation component. At appraisal the Government made a commitment of $2.20 million to augment the GEF grant but actually contributed $ 2.14 million (97 percent of the commitment at appraisal). Dates: The date of effectiveness (January 4, 2005) was about five and a half months after approval although it was 2 weeks earlier than planned. The closing dates were not amended for either the IBRD Loan or the GEF Grant.

3 3. Relevance of Objectives & Design: a. Relevance of Objectives: Substantial. AWRP's objectives continue to be relevant to Turkey's economic and development "characterized by a relatively low, but rapidly increasing environmental footprint " (see World Bank, "Turkey Green Growth Policy Paper: Towards a Greener Economy", 2013). The ICR notes that the strategic objectives of Turkey's National Rural Development Strategy ( ) were Protection and Improvement of the Rural Environment. The two strategies were: (a) Improvement of Environment friendly Agricultural Practices (integrated agricultural basin programs, expanding organic agriculture and good agricultural practices, implementing and diversifying environment friendly production methods, monitoring environmental pollution arising from agricultural activities, developing agricultural land and pastures); and (b) Protecting Forest Ecosystems and Sustainable Utilization of Forest Resources (accelerating forest cadastre works, alleviating the pressure of animal husbandry on the forests, protecting biodiversity of ecosystems and sustainable utilization of biological resources, forestation and soil preservation works ). The Government of Turkey gives priority to integrating sustainable and equitable resource management and environmental protection (such as completion of the National Basin Management Strategy ) as the basis for future investments in watershed management. In addition Turkey is a member of the Black Sea Commission and a signatory to the Convention on the Protection of the Black Sea from Pollution and is obliged to implement these international agreements. AWRP and GEF objectives were relevant to these strategies and agreements. The Country Partnership Strategy for FY12-15 aims to contribute to Turkey s goal of fast, sustainable and inclusive growth that respects the environment (paragraph iv). It has three main strategic objectives, namely : (i) enhanced competitiveness and employment; (ii) improved equity and public services; and (iii) deepened sustainable development - to contribute to strengthened environmental management and adaptation to climate change. Hence during implementation and at exit AWRP was substantially relevant to Government policy as well as to Bank and GEF assistance strategies. b. Relevance of Design: Substantial Annex 1 in the PAD (Project Design Summary) was not a summary of the project's design. Instead it was a partial results matrix containing summaries of the PDO and the GEO (as stated in the PAD) with some key performance indicators. It did not provide causal chains of activities ending in a final outcomes because it failed to define intermediate outcomes to measure the project's progressive performance. By the time of the mid-term review in 2008, however, the results matrix in the ISR was improved by establishing baselines, final and intermediate outcome indicators, and targets. The project design called for activities to take place in 28 micro-catchments (MCs) in six provinces. Resource management practices were to be implemented to reduce the rate of soil and vegetative degradation and thereby raise incomes of relevant communities (PDO) and reduce the discharge of nutrients and other agricultural pollutants in catchment areas that drain into the Black Sea (GEO in the PAD). AWRP's design drew on the successful model and lessons learned in the Eastern Anatolia Watershed Rehabilitation Project (EAWRP) as well as the agricultural pollution control projects funded under the Black Sea -Danube Strategic Partnership Program for Nutrient Reduction in other countries such as Romania, Poland an d Moldova. Community decisions on priority resource management interventions and their development "on a rolling basis" (ICR, paragraph 23) were a feature of the project's design allowing knowledge to be collected from one round of MC activities to the next. The ICR asserted that this design was justified for AWRP. Responsibilities of each ministry/directorate were clearly spelled out so that there was no ambiguity about who does what during implementation. Micro-catchment Implementation teams (MCITs) were established with relevant staff from all seven provincial directorates of MEF and MARA to assist communities develop MC plans. According to the ICR, establishing the MCIT was a useful institutional design arrangement because communities had limited knowledge and skills for collective action on natural resource rehabilitation, environment friendly agricultural practices and animal waste management (paragraph 25). The GEF-financed agricultural pollution control and animal waste management practices activities to reduce nutrient levels in soil and water bodies in the MCs was also a relevant design because it met obligations under the EU Nitrates and Water Framework Directives (relevant to Turkey's plans for accession to EU membership ) and referred to in Annex 1 of the PAD. While the project's original results matrix was weak it was later improved considerably. The project's design was

4 substantially consistent with its objectives of improving natural resource management and raising incomes of communities affected by resource degradation. 4. Achievement of Objectives (Efficacy): This Review will assess whether the project's development objective was achieved and whether the achievements were attributable to the project. Based on the IBRD Loan and GEF Grant Agreements two sub-objectives were defined for improving natural resource management practices in 28 micro-catchments in Anatolia and Turkey s Black Sea Region, namely to (a) raise incomes of communities affected by the project's actions to address resource degradation; and (b) reduce the discharge of agricultural pollutants into surface and ground water in watersheds draining into the Black Sea in four provinces. The extent to which they were achieved will be assessed separately. (a) Raise incomes of communities affected by resource degradation - Modest Outputs Increased Vegetative Cover. The project aimed at increasing vegetative cover in the 28 micro-catchments by 20% above the baseline area of about 34,000 ha of degraded forestland, rangeland, and agricultural land by midterm, and by 50% by the close of the project (PAD, Annex 1) - Soil conservation, afforestation, vegetation improvement, fallow reduction, rehabilitation through participatory planting, gallery plantation, and rangeland rehabilitation works, resulted in the rehabilitation of about 58,000 ha of degraded lands and increased vegetative cover by 70% in micro-catchments. - There was no mention in the ICR of "control" areas where soil conservation and other practices were not adopted that could assist in determining whether the increase in vegetative cover in the 28 targeted MCs was attributable to the project. Improved Soil Fertility : The project's aim was to increase soil fertility on sloping lands in project MCs by 10% above the baseline at midterm and by 20% at project closure (PAD, Annex 1). Increased productivity of three crops was used as a proxy to measure improvements in soil fertility. The target increases for yields were 44% for sainfoin, 27% for chickpeas, and 25% for alfalfa by the project's close (ICR, Annex 2, para 3)). - At the project's close yield increases were 182% for sainfoin, 89% for chickpeas, and 18% for alfalfa, thereby exceeding two of the three targets and getting close to the third. - The Data Sheet in the ICR mentions that "crop productivity of sainfoin, chickpeas and alfalfa were used as proxies for measuring improved soil fertility with baseline and targets set when MC plans were prepared. It was noted that several external factors contributed to improvements in soil fertility and that there is no evidence in the ICR that the increases in yields were all attributable to the project. Farm Income Diversification. The project aimed at providing training and materials to qualify farmers to diversify farming activities and thereby increase farm income. The baseline was a total of 2,674 farmers (60%) trained in the MCs of Sivas, Çorum, Kayseri and Amasya provinces. The training equipped farmers with skills in new agriculture based income generation/diversification activities (PAD, Annex 1, page 33). Outcomes - The ICR states that 83% of the farmers were trained in the production of high value crops and improved livestock management by the end of 2010, and that "the achievement rate is expected to reach some 142% by the end of 2012" (page 23) because farmers outside the target MCs participated in the training. Increased Household Incomes. The project aimed to increase household incomes in participating MC communities by 10 percent above an unspecified baseline at midterm and by 40% at project closure (PAD, Annex 1, page 32). - The mid-term review provided evidence from a household survey in 2007/2008 that in five MCs in five provinces average household incomes were 14.8% above household incomes in five non -project MCs in the same five provinces. - The ICR also stated that "The project aimed at increasing household incomes in participating MC communities starting from a modest baseline of average HH income estimated at TL 4,200, where the underlying figures were derived from surveys conducted in project MCs before project implementation started. A target was set to increase HH incomes in the participating MC communities by 10% above baseline at midterm and by 40% at project closure. These targets were far exceeded. It was estimated that the average HH income was raised by 53% as revealed by the beneficiary surveys completed in project MCs" (Annex 2, paragraph 6)

5 - However, the ICR contains no information on household incomes in the MCs with and without the project based on the final household survey to support the claim of a 53% net increase in income above the income achieved in households not in the project areas. The task team explained that the survey data were not available from either the Government or the Bank's supervision files. - The ICR Data Sheet cautions that "several external factors contributed to such increases in income levels." However, it concludes that "it can be safely assumed that project interventions targeted towards this outcome were significant contributors to improvements in household incomes in the project area " (page viii). - Nevertheless, attribution remained uncertain because the second household survey quoted in the mid-term review showed that income generated from crops, livestock or forestry (the main enterprises supported by AWRP) in the five project MCs accounted for only 43% of income for households in the project MCs and 55% of household income in the non-project MCs (Aide Memoire for Supervision Mission, Annex 4). - In addition, Government comments on the draft ICR also cast doubt on the attribution of increased income in the project MCs to the impact of the project and noted the absence of baselines (ICR, Annex 7). On balance, based on the net incomes in households in project MCs at the mid -term, the achievement of this sub-objective was modest (b) Reduce the discharge of agricultural pollutants into surface and ground water in watersheds draining into the Black Sea in four provinces - Substantial Outputs Reducing Pollutant Discharge in Water Bodies : This activity's objective was to develop and test a package of farm investments and farming practices to reduce pollutant discharge into water bodies in the four Black Sea Provinces. The project funded training and demonstration programs to show that environment friendly practices, such as crop rotation, crop nutrient management, animal waste management, and the use of organic matter can reduce nutrient discharge in water bodies (PAD, Annex 1). - Packages of appropriate practices were successfully tested in pilot areas and 3,500 persons were trained in their use through 200 training programs (ICR, Annex 2, paragraph 4). Manure Handling : The target was to promote adoption of improved manure handling and storage facilities among 55% - 60% of farmers in areas where such practices would be piloted (PAD, Annex 1). - The Data Sheet shows that 60% of farmers in areas where such practices were piloted adopted improved manure handling/storage practices (ICR, page viii). Nitrates Directive Legislation : The project aimed at development and promotion of a legal framework consistent with the EU Nitrates Directive which aims to protect water quality by promoting good farming practices and prevent nitrates from polluting ground and surface waters. No such legal framework existed. - The Official Gazette published a regulation on February 18, 2004 (soon after project approval) which seeks to protect ground water and surface water run off against pollution from agricultural sources. However, the Government is still searching for ways to enforce it effectively. The regulation is a critical requirement for Turkey's compliance with the EU Nitrates and Water Framework Directives and hence are important to its EU accession plans. Development and Promotion of a Code for Good Agricultural Practices (GAP GAP): The project developed the GAP as required under the EU Nitrates Directive. When the project started, virtually no farmers were known to be using environment friendly agricultural practices (PAD, Annex 1). - The ICR mentions no target for the adoption of the code. Nevertheless, at project closure GAP implementation was ongoing in 14 MCs in the four Black Sea provinces (Annex 2, paragraph 9). Some 90% of farmers in these provinces were reported to have adopted nutrient management practices (Annex 2, paragraph 4). Capacity Building for Water Quality Monitoring : The project provided necessary training and equipment to strengthen capacity of laboratories for soil and water quality monitoring. A detailed laboratory handbook was developed to harmonize analytical procedures by all laboratories. - According to the ICR the laboratory in Samsun is EU -accredited and the National Reference Laboratory in

6 MEF in Ankara is well equipped with a mobile laboratory for water quality analysis. Water is analyzed for ph, temperature, dissolved oxygen, turbidity, suspended solids, nitrates and phosphates, and fecal coliform. Sampling period is monthly for surface water and quarterly for ground water (Annex 2, paragraph 12). Replication Strategy for Pollutant Discharge : A comprehensive replication strategy was put in place and targets were set for future nutrient reduction on the basis of achievements of activities in pilot MCs. Outcomes - An international consultant was hired to evaluate the results of activities in ten selected watersheds and prepare a report was to establish a replication strategy for the whole country (ICR Annex 2, paragraph 15, but no further information about this report is available in the ICR. Improved Farming Practices. Farming practices to reduce the discharge of agricultural pollutants into surface and ground water in watersheds draining into the Black Sea in four provinces were introduced. - Farming practices such as improved manure handling to reduce the discharge of agricultural nutrients were introduced in MCs by 60 percent of farmers in target provinces bordering the Black Sea (ICR, Annex 2, paragraph 5). Discharge of agricultural pollutants into surface and ground water - Various institutional and regulatory mechanisms were put in place supporting a formal water quality monitoring program (PAD, Annex 1) - These mechanisms were mainstreamed into Ministry operations to monitor water quality continuously. Some 140 stations in 14 MCs were equipped with 57 groundwater and 83 surface water measuring devices as part of a water quality monitoring program (ICR, Annex 2, paragraphs 11 and 12). - The ICR provided no evidence that improved farming practices reduced the discharge of agricultural pollutants into surface and ground water. The reason was that the effect of improved farming practices is slow and it will therefore take some years before the water quality monitoring system will identify any measurable impact of improved farming practices in MCs on reduced agricultural pollutants in watersheds draining into the Black Sea, The achievement of this sub-objective was substantial. 5. Efficiency: Modest The PAD undertook a "partial and mostly indicative" analysis (ICR, Annex 3, paragraph 8) of efficiency based on the analysis of benefits from reduced soil loss, improved soil quality, reduced flood control costs, fallow reduction benefits, small scale irrigation, and the additional incomes from increased livestock production. The result was an ERR of 18.1% which decreased to 14.4% if costs increased by 20% and benefits decreased by 20%. The economic analysis in the ICR focused only on the costs and benefits of the first component (rehabilitation of degraded natural resources) of the IBRD-financed part of the project because there were inadequate data on which to undertake a detailed evaluation of the efficiency with which income changes were achieved. The actual cost was $14.01 million for Component 1 (rehabilitation of degraded natural resources - representing 71 percent of the IBRD-financed part of the project). Contributions to the project's cost for this component made by GEF ($8.32 million) were not included because the focus was on the IBRD-financed part of the project. The benefits for Component 1 were generated by savings in erosion induced soil loss. yield increases due to improved agricultural land, and reduced flood control costs. The ICR explained that "a plan was made to revisit these three benefit streams". Benefits were savings in erosion induced soil loss, yield increases, and reduced flood control costs - which were all regarded as benefits from erosion control. However, during the preparation of the ICR it became clear that there were no reliable data available for the analysis of benefits from the AWRP It was therefore decided to use data for flood control benefits from the appraisal for the Coruh Watershed Rehabilitation Project as the basis for estimating the flood control benefits in AWRP (ICR, Annex 3, paragraph 28). Benefits from reduced soil loss and higher crop yields were revised by updating the project's own figures, using undefined "external escalation factors" (ICR, Annex 3, paragraph 26). In the event, the flood control benefits accounted for 82 percent of the estimated benefits.

7 While the ICR is clear on the fact that adjustments were made to assess the project's benefits, these adjustments involved many assumptions and the ICR describes the logic of flood control estimates as "crude" (Annex 3, paragraph 28). It was, however, not clear from the ICR whether the physical terrain of the AWRP and the Coruh project in eastern Turkey (in a more mountainous region than the AWRP project area ) were similar. Hence estimated flood control benefits for the Coruh project were unlikely to be a sound basis for estimating flood control benefits in AWRP. At any rate the estimated "base case" partial ERR was calculated as 14 percent. In summary the partial analysis of efficiency in the ICR is based largely on the appraisal report of the Coruh project with numerous adjustments because, as the task team agreed during discussions on this Review, the terrain in the Coruh project was not similar to the terrain in AWRP. Moreover, the analysis was focused on only one component in AWRP. There was also no evaluation of household income increases in communities in AWRP because, as already discussed in Section 4, the data were not available to do so. A substitute analysis of efficiency based on assumptions would not have been credible. This Review concluded that the partial assessment of efficiency, while inventive, had significant shortcomings in terms of its coverage and relevance to AWRP. Notwithstanding the task team's claim that the project's administrative efficiency was high, overall efficiency is rated modest.. a. If available, enter the Economic Rate of Return (ERR ERR)/Financial Rate of Return (FRR FRR) at appraisal and the re-estimated estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 18.1% 84% ICR estimate Yes 14% 49% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The relevance of the project's objective and design was substantial. The efficacy of the project's core sub-objective of raising incomes of communities affected by resource degradation was rated as modest and the sub-objective of reducing the discharge of agricultural pollutants into surface and ground water in watersheds draining into the Black Sea was rated as substantial. Efficiency was also rated as modest, Overall the project's achievement had significant shortcomings and its outcome was therefore rated as moderately unsatisfactory. a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR states that on the basis of its judgment of efficacy and efficiency "there is a high likelihood of ongoing support for project outcomes beyond the life of the project ". It also provides some positive reasons for sustainability such as continuous commitment to appropriate economic, financial and sector policies on the part of the government. In addition, it is stated that educated beneficiaries, low-cost technologies, and enhanced institutional capacity and coordination mechanisms among ministries and directorates at the national and provincial level are likely to remain (paragraph 68). However at the start of the project "household incomes in the project area were significantly below the average rural household income in Turkey and many households remained largely in the subsistence economy " (ICR, paragraph 10). It is therefore not clear whether the alleged 53 percent increase in income achieved by farmers as a result of the project will be enough for those farmers to pay for continued conservation efforts when there are inevitably some maintenance costs as well as other demands at the household level on additional income. The ICR states that "EU payments can be received by farmers only if they keep their land in good condition" (paragraph 68). However there are no assurances in the ICR that farmers will keep their land "in good condition" or that EU payments will continue. Hence, for a number of reasons, there are moderate risks regarding the sustainability of farmers' efforts to continue best practices in future. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry:

8 The team built on the successful East Anatolia Watershed Rehabilitation Project (EAWRP) and reflected the lessons learned from that project in this project's design. It linked agricultural pollution control activities to natural resource management which was an innovative approach sought by the Government and welcomed by local beneficiaries. According to the ICR there was good cooperation with all counterparts which resulted in smooth project preparation. Regular stakeholder consultations ensured that there was agreement on project design, scope and activities. The appraisal team ensured that communities in the targeted micro -catchments were actively involved in identifying interventions for the MC plans. A comprehensive M&E plan was outlined in the PAD. Five MC plans were developed during preparation to ensure that the project could start implementation quickly after effectiveness. On the other hand the results matrix was weak and this led to some extent to inadequate monitoring and evaluation. Quality-at at-entry Rating : Satisfactory b. Quality of supervision: According to the ICR supervision missions provided thorough reports on progress towards key performance indicators and action reports were agreed upon in detail with counterparts. Aide Memoires and Implementation Status and Results Reports (ISRs) were closely tracked. The supervision team maintained a regular and constructive dialogue with the Project Management Unit (PMU), central and local government agencies, as well as the MC communities. Its pro-active approach in managing and resolving issues that arose during implementation helped to keep project activities on schedule. However, the ICR noted that on many occasions the Government was advised that M&E was inadequate but in the event the data needed for an assessment of the project's efficacy and efficiency were not available. This was a significant shortcoming in supervision performance. While Bank's supervision cannot be held responsible for the Government's inaction it was surprising that the M&E weaknesses were not raised to a higher level of management in the Bank. On balance there were significant shortcomings in supervision performance. Quality of Supervision Rating : Moderately Unsatisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The ICR noted that "the government largely prepared the project " (paragraph 74). Relevant staff from MEF and MARA, both at the central and local level, worked cooperatively throughout ensuring coordination to execute their specific responsibilities. The provincial governments worked closely with the communities to ensure that their voices were adequately represented in the MC planning and implementation process. According to the ICR the Government actively promoted awareness of potential project benefits not only in the project MCs but also at the national level to foster increased adoption and replication of project activities. Timely counterpart financing contributed to satisfactory disbursement performance. Issues and challenges that arose during implementation (such as dissolution of the General Directorate of Rural Services ) were effectively addressed by the Government. However, irrespective of these positive aspects of the Government's role during project preparation and implementation, the Government did not successfully ensure adequate M&E. This weakness emerged even though "The Bank team sensitized the implementing agencies to the importance and urgency of establishing a quality baseline... and followed up closely with the Government" (ICR, paragraph 33). There is no information in the ICR on how the Government responded to this dialogue with the Bank's supervision mission. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: With two ministries and seven agencies involved with project implementation in the Project Management Group (PMG) and the PMU worked effectively in coordinating their work. Regular meetings were organized to address project issues jointly and in a timely manner. The ICR states that the Operations Unit of the PMU undertook its day-to-day management of project activities efficiently, ensuring that there were no significant delays or lags with procurement and disbursement. Weaknesses in the capacity of procurement staff were addressed through regular training provided by the Bank Country Office staff. The ICR states that detailed annual progress reports were made available to the Bank in a timely manner. According to the ICR the Micro-catchment Implementation Teams performed satisfactorily, working closely with the local beneficiaries advising them on the development of the MC plans and the menu of activities applicable to each MC s needs.

9 However, despite the apparent regular and substantial reporting on progress, M&E performance fell short of acceptable performance since there were delays in the establishment of a sound baseline for some project interventions, such as water and soil quality monitoring. Also, volumes of data (including two household surveys ) were collected but there was little or no evaluation of the data from the second and final survey. As a result compilations of data for a rigorous evaluation of the p roject's performance on income generation and efficiency were not available when the project closed. Overall, despite the PMU's eventual adequate fiduciary management, there were significant shortcomings in its performance on M&E which had important implications for an assessment of the project's achievements. Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The PAD stated that: "Overall project Monitoring and Evaluation will be the responsibility of the Project Management Group which will be supported in this task by the Operation Unit s M&E staff. Building on the M&E system of the EAWRP, the M&E unit will develop and implement an M&E system which will include both routine monitoring and evaluation and special-purpose M&E focused on impact assessment. The M&E system will include targeted annual performance objectives and monitoring indicators using the Key Performance Indicators in Annex 1 as a basis. A baseline survey against which project performance targets can be measured will be carried out in each MC during the detailed MC development planning phase." This plan left a lot of decisions to be made after effectiveness and the results were not as the PAD intended. The narrative on M&E in the ICR suggested that there were significant shortcomings in establishing a robust M&E design that built on the M&E system managed by the EAWP. For example the ICR states that "while extensive data and background information was collected during the preparation of the MC plans, there were delays with compiling, aggregating and analyzing it in a way that would establish a sound baseline for the MC plans and subsequently allow evaluation of outcomes of selected investments " (paragraph 32). In the event it was not clear whether adequate baselines were established. b. M&E Implementation: As noted in the assessment of Government Performance, the ICR states that the Bank team sensitized the implementing agencies to the importance and urgency of establishing a quality baseline during its supervision missions. The PMU subsequently hired a consultant to develop a computer program for consolidating all project monitoring information and aggregating the data in a meaningful way which would allow for measurable, systematic reporting on project achievements. According to the ICR, although the implementing agencies diligently tracked outputs achieved under each project component, it was agreed that "an assessment of project outcomes would be undertaken over the last three months of project implementation and the findings shared with the Bank team by September 15, 2012." However, this date was two and a half months after the closing date. At the time of the ICR, the Bank team was still awaiting the findings of this assessment (paragraph 34). This was a severe shortcoming which made it impossible to adequately assess the project's achievements and efficiency. c. M&E Utilization: The ICR states that "Overall, MC plans performed well due in part to the rolling basis of MC plan development. The lessons learned from preceding plans were fed into the preparation and implementation of subsequent plans which improved the quality of the MC plans and increased the likelihood of their success " (paragraph 35). However, this achievement in M&E utilization at the MC level was not reflected at the project level. M&E Quality Rating : Negligible 11. Other Issues

10 a. Safeguards: The safeguards triggered under the project included : Environmental Assessment (O.P. 4.01), Forestry (O.P. 4.36), Pest Management (O.P. 4.09) and Safety of Dams (O.P. 4.37). No major safeguard issues arose during project implementation. The project was classified in Environmental Category B (Partial Assessment) project because although it generally supported environmentally sustainable development, some activities, such as manure platform construction, could have some direct environmental impacts, albeit of insignificant scale and easily mitigated. The norms for mitigating and monitoring such impacts were defined in the PAD which referred to an Environmental Management Plan (EMP) that was based on a Regional Environmental Assessment (REA) disclosed in The REA and Forestry safeguards remained relevant because of the afforestation, rangeland improvement, and manure management initiatives. Compliance with provisions of both safeguards was satisfactory throughout project implementation. b. Fiduciary Compliance: Financial Management. Regular Bank reviews confirmed a satisfactory financial management system during the life of the Project. Satisfactory internal controls and procedures ensured reliable accounting records and safeguarded project resources and assets. Quarterly Financial Management Reports (FMRs) were submitted on a timely basis in agreed content and format. Audit reports were unqualified Counterpart financing was satisfactory during the life of the project. Overall, disbursements were smooth and there were no significant lags. Procurement. The ICR noted that at appraisal, procurement risk was rated high. With increased decentralization of responsibilities for procurement management, lack of strong technical backstopping and quality control in the PMU to provide assistance to decentralized units, and frequent staff changes, overall procurement capacity remained relatively weak. This required the provision of continuous training (and re-training) in the Bank s procurement rules and guidelines. However, with support from the Country Office, major procurement problems and delays were avoided. c. Unintended Impacts (positive or negative): When the irrigation component was eventually cancelled after many attempts to find an institution that would be responsible for implementation failed, the Government and the Bank agreed to introduce solar water heating systems for households which would reduce the demand for wood to heat water. This unexpected component was successfully implemented. d. Other: A positive effect of the project's organization was that its focus on community action generated considerable social capital in villages that resulted in broader benefits for the communities' development programs Ratings: Risk to Development Outcome: Outcome: Satisfactory ICR IEG Review Moderately Unsatisfactory Reason for Disagreement /Comments The relevance of the project's objectives and design was rated substantial. The efficacy of the project development objective of increasing incomes of communities affected by resource degradation in 28 micro-catchments through sustainable natural resource management practices was rated as modest. The project's efficiency was also rated as modest because of unconvincing estimates of benefits. In conclusion, evidence on the project's results had significant shortcomings and outcome was therefore rated as moderately unsatisfactory. Negligible to Low Moderate The sustainability of improved natural resource management will be

11 Bank Performance : Satisfactory Borrower Performance : Satisfactory Quality of ICR : Moderately Unsatisfactory Moderately Unsatisfactory Satisfactory influenced by farmers' capacity to pay and EU payments to support them to keep their land in good condition. However, farmers in the project area are relatively poor and there is also no certainty that EU payments will be sustained. Overall this Review therefore rated the risk to development outcome as moderate Quality at entry is rated as satisfactory. Despite the reported focus by supervision missions on the importance of M&E the lack of adequate high level attention to improving its quality had significant negative implications and resulted in a lack of data compilations available to the Bank for a convincing analysis of the project's achievements in crop productivity, income generation and efficiency. Bank supervision is therefore rated as moderately unsatisfactory. Consistent with the harmonized criteria agreed between IEG and OPCS overall Borrower performance is rated as moderately unsatisfactory. Government performance was rated as moderately satisfactory. However, there were significant weaknesses in the development and management of M&E by the PMU; for example there was little or no evaluation of data from the final household survey. This made it impossible to adequately evaluate the project's achievements and efficiency. Hence the PMU's performance was rated as moderately unsatisfactory. Consistent with the harmonized criteria agreed between IEG and OPCS overall Borrower performance is rated as moderately unsatisfactory. NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR concluded that the lessons from this project were that (a) participatory design is key to promoting project ownership; (b) a flexible approach to project management allows timely adaptations to changing needs and circumstances during implementation; (c) establishing a direct link between natural resource rehabilitation and tangible economic and social benefits is critical for increased uptake of natural resource management activities; and (d) dissemination of information through public awareness programs is crucial for the widespread adoption of new practices and technologies. Lessons (a) and (b) were indeed relevant to this project. However, there was no evidence for lessons (c) and (d) because the project did not generate information on "tangible economic and social benefits" that could be used to

12 "increase uptake of natural resource management activities " or for "dissemination of information through public awareness programs". An additional lesson from the project highlighted by this Review is one that has emerged in many projects in the past, namely that M&E is a crucial aspect of project implementation without which a convincing evaluation of a project's achievements is impossible. 14. Assessment Recommended? Yes No Why? The achievements of this project are based on a few minimal and unvalidated statements in the ICR on yields, household income, and increased government capacity to undertake measurements of ground and surface water quality. There were also uncertainties about the extent to which increased incomes were attributable to the project's investments. In addition this Review considered there were inadequacies in the ICR's assessment of efficiency. At the core of these uncertainties and inadequacies was a weak M&E system which had not delivered compilations of data that could be used in the analysis when the ICR was prepared. A Project Performance Assessment Report (PPAR) would provide the opportunity to use the data that were allegedly to become available after the project closed to revisit the rating in this Review. 15. Comments on Quality of ICR: On the whole the ICR was concise, clear and frank. It was particularly candid about the challenges faced in assessing the project's efficiency caused by the absence of comprehensive information on the project's results. This was in turn due to the inadequate M&E implementation about which th e ICR discussion could have been more transparent. a.quality of ICR Rating : Satisfactory