L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF Feb ,053,851.80

Transcription

1 Public Disclosure Authorized Independent Evaluation Group (IEG) 1. Project Data Report Number : ICRR Public Disclosure Authorized Public Disclosure Authorized Project ID P Country Madagascar Project Name MG-BVPI PHRD Rice Practice Area(Lead) Agriculture L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF Feb ,053, Bank Approval Date 27-Mar-2014 Closing Date (Actual) 28-Feb-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 12,660, ,660, Revised Commitment 12,053, ,053, Actual 12,053, ,053, Prepared by Reviewed by ICR Review Coordinator Group Hassan Wally John R. Eriksson Christopher David Nelson IEGSD (Unit 4) Public Disclosure Authorized 2. Project Objectives and Components a. Objectives The Project Development Objective (PDO) in the Project Appraisal Document (PAD, p. 5) was identical to that in the Grant Agreement (p. 7) and aimed to: "increase rice productivity in selected irrigation sites and their surrounding watersheds." b. Were the project objectives/key associated outcome targets revised during implementation? Page 1 of 17

2 PHEVALUNDERTAKENLBL Independent Evaluation Group (IEG) Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? No d. Components The PDO was supported by five components: 1. Development of Commercial Agriculture (appraisal estimate: US$1.96 million, actual cost: US$1.18 million). This component aimed to lay the foundations for improved market access and sustainable intensification and diversification of irrigated and rainfed rice production systems in the watersheds targeted by the Project. It included two sub-components: 1.1. Support to Agricultural Services. This aimed to support the Government in improving access to markets and in the development of commercial agriculture value chains in the following areas: (a) support to the development of commercial agricultural supply chains, including market research and surveys, thematic studies, research and development of technologies for agricultural intensification, support to beneficiaries in accessing financing, and promoting partnerships; (b) strengthening capacities of eligible beneficiaries (according to the project team, beneficiaries must be endorsed by a Water User Association (WUA) or another related project group); (c) support to the Agricultural Service Centers (ASC); and (d) support and advisory services for the implementation of technologies for agricultural intensification and diversification, including implementation of agro-ecological and agro-forestry techniques Provision of Demand-based Sub-project Matching Grants. This aimed to support farmers, farmer organizations, and other private stakeholders to carry out activities to improve production and marketing along the rice value chain. The activities could include, for example, support to productive investments through: the introduction, dissemination, and on-farm development of new agricultural production techniques; applied research; awareness raising and demonstration campaigns; or the rehabilitation or development of quality rice seed production. It thereby would address one of the biggest constraints believed to be facing smallholder farmers and other private entrepreneurs, i.e. the perceived high risks associated with new technologies. 2. Irrigation Development (appraisal estimate US$4.93 million; actual cost: US$5.73 million). This component aimed to lay the foundations for improved management, maintenance, and sustainability of irrigation services provision in selected small to large-scale irrigation schemes through targeted rehabilitation of irrigation infrastructure, capacity strengthening of stakeholders and clarification of roles and responsibilities, and establishment of an appropriate incentive framework. It included two subcomponents: 2.1. Support to Irrigation Development. This aimed to support the Ministry of Agriculture (MOA), Water User Associations (WUAs), and the Regions and Communes within the Project areas concerned with: (a) the preparation of an Scheme Development Plan for Irrigation, if not yet available, and Planning Contract (CPs), including Transfer Agreements of newly rehabilitated irrigation schemes to WUAs; (b) the implementation of the relevant Performance Contracts, including by supporting stakeholders during implementation of the Performance Contracts, through establishment and capacity strengthening of WUAs, Page 2 of 17

3 setting up of sustainable O&M systems, annual evaluation of performance indicators, conduct of user satisfaction surveys, and (c) the carrying out of studies on O&M costs and cyclone-related damages Irrigation Investments. These aimed to support MOA, WUAs, and the Regions and Communes within the Project areas concerned with: (a) the preparation of technical design studies, including preparation of bidding documents; (b) the rehabilitation of irrigation and related infrastructure works agreed upon in the Performance Contracts; and (c) the supervision of the works financed under (2.1.) above, and; (d) development and dissemination of low-cost irrigation technologies. 3. Watershed Development (appraisal estimate: US$1.53 million; actual cost: US$1.10 million). This component aimed to lay the foundations for sustainable management of watersheds including irrigated and rain-fed agriculture, and for improved productivity of natural resources. It included two sub-components: 3.1. Planning and Capacity Building for Sustainable Watershed Development. This aimed to support: (a) the preparation of participatory plans for managing sub-watersheds; (b) the introduction and/or strengthening of communication and negotiation platforms; (c) training and capacity strengthening of sustainable land management (SLM) groups; and (d) improvement of land tenure security Sustainable Investments in Watersheds. This aimed to support: (a) construction of strategic antierosion infrastructure works through, among others, biological methods and technologies; and (b) provision of sub-project Matching Grants to sub-project Beneficiaries for carrying out demand-driven improvements of the vegetation cover, including pastures and reforestation. 4. Support for Rice Research and Development (appraisal estimate US$2.60 million; actual cost: US3.00 million). This component aimed to strengthen Madagascar s rice research system and strengthen the links with extension. It included three sub-components: 4.1. Improving Soil Chemical Analysis Capacity at the National Center for Applied Research in Rural Development (FOFIFA). This aimed to provide support to the National Soil Laboratory, to provide soil mapping services and soil management advice to selected National Program for Irrigation Development and Watershed Management (PN-BVPI) areas Strengthening Basic Seed Production Capacity in Rice. This aimed to provide support to the national Rice Germplasm Center under FOFIFA and several associated regional research stations and seed multiplication farms in the areas targeted by the Irrigation Development and Watershed Management (BVPI) Developing Rice Research and Extension Linkages. This would provide support to researchers from FOFIFA and/or universities to conduct on-farm research, as well as to deliver extension services in selected BVPI sites. 5. Project Management (appraisal estimate: US$1.64 million; actual cost: US$1.04 million). This component aimed to manage and use resources in accordance with the Project s objectives and procedures and to evaluate its results. It included two sub-components: 5.1. Management of the Project. this aimed to provide support to the MOA and the Regional Directorate for Rural Development (DRDR) to enable them to effectively carry out the overall Project planning, quality oversight, procurement, financial management, and monitoring of Project activities, including through the provision of TA, training, equipment, evaluation studies and incremental operating costs to the MOA and DRDRs. Page 3 of 17

4 5.2. Monitoring and Evaluation. This aimed to support to the MOA and DRDRs in the Project areas in the management of information systems and the preparation of procedures for data collection and reporting to carry out: (a) regular project monitoring and completion evaluations of Project output and impact indicators; and (b) surveys and participatory assessments, including those on regular user satisfaction. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. Total Project cost was estimated to be US$15.05 million, including physical and price contingencies. Actual cost reported by the ICR (p. 2) was US$12.05 million (80% of the appraisal estimate). The difference stemmed from lower than expected disbursements under components 1, 3 and 5 where disbursements were 60%, 72% and 63% of the appraisal estimates, respectively. Financing. The project was financed through a US$12.66 million Grant from the Japanese Policy and Human Resources Development Fund (PHRD). Actual amount disbursed was US$12.05 million (ICR, p. 2). In a further communication, the project team explained that it was not possible to extend the closing date of the PHRD Grant; and given the limited availability (six months to disburse 50% of the Grant), it was not possible to disburse the full amount. Borrower Contribution. The borrower was expected to contribute US$0.1 million and local beneficiaries were expected to contribute US$2.29 million. According to the ICR (p. 2) no funds were contributed by borrower nor the local beneficiaries. In a further communication, the project team explained that there were in-kind contributions from beneficiaries. Dates. The project was approved on the March 27, 2014 and became effective about five and half months later on September 15, The project closed as expected on February 28, It was restructured once through a Level 2 restructuring on June 20, 2017, when the amount disbursed was US$4.66 million. The restructuring came on the heels of severe natural weather events that impacted the project areas during the 2015 and 2016 growing seasons (Chedza Cyclone in mid-2015, drought in late 2016, and Enawo Cyclone in early 2017). According to the ICR (p. 8, para 13) "these weather related shocks caused damages to the irrigation infrastructure, threatening the livelihoods of farmers who depend on irrigated rice." Through the level 2 restructuring, funds were reallocated totaling US$1.71 million from components1, 3, 4 and 5 to component 2 (Irrigation Development) to expand the area provided with irrigation and drainage services from 2,089 ha to 5,390 ha; and to fund a Community Seed Banks operation to increase the availability of certified rice seeds in the Project areas. Also, the Results Framework was modified (see section 9 for more details). The Mid-term Review was conducted in June 2016 which was slightly earlier than the expected timeframe stated in the PAD (p. 46): "a mid-term review of the Project is envisioned at the end of the second year of implementation." 3. Relevance of Objectives Page 4 of 17

5 Rationale Agriculture continues to underpin the economy in Madagascar where 70 to 80% of the population work in the agricultural sector; and it represents the principal livelihood source for the vast majority of Malagasy households. This importance also explains the strong correlation between the performance of the agricultural sector and the poverty level in the country. Rice is the main staple crop and is grown by about 85% of agricultural households. It is a vital commodity in the Malagasy diet and essential for food security. It represents nearly 70% percent of agricultural production and accounts for 48% of total calorie consumption. Key constraints facing rice production included: outdated production technology, inefficient use of land and water resources, and lack of appropriate institutions to ensure sustainability (PAD, p. 2). At project appraisal, objectives were in line with Government's National Strategy for the Development of Rice which included three main objectives: to contribute to the food security of all regions of the country; to contribute to the improvement of economic growth; and to improve the income and the welfare of the actors along the value chain. Objectives were also in line with the Bank's Interim Strategy Note (ISN, 2012) for Madagascar. In particular, the ISN Pillar 2 called for improved support for productivity in agriculture through bringing 9,000 additional ha of rice under irrigated agriculture and supporting 35,000 additional producers supported through 850 income-generating sub-projects (ISN, p. 35). At project completion, objectives were highly relevant to the Government priorities for the agriculture sector. Objectives were in line with the National Strategy for the Development of Rice which called for doubling the output of paddy rice to reach 7.7 million tons by Objectives were also in line with the Bank's FY17 FY21 Country Partnership Framework (CPF) for Madagascar. The CPF (p. 13) called for improving agricultural productivity through: "ensuring land tenure security to increase incentives for agricultural investment and improved access to credit; promoting the use of fertilizers (and other inputs), irrigation, and mechanization to increase yields and agricultural output; and strengthening the links between agricultural production and nutrition to improve the health status of rural households." The PDO statement was focused and clear, however it lacked a connection to higher level objectives. Based on the above information, relevance of objectives is rated high. Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective PDO: to increase rice productivity in selected irrigation sites and their surrounding watersheds. Rationale Outputs Page 5 of 17

6 1. Development of Commercial Agriculture By the end of the project 2, 283 ha of land were cultivated with improved technologies and/or inputs provided through the sub-projects. This represents 89.9% of the revised target of 2,540 ha. The underachievement was mainly due to: beneficiaries walking away from150 sub-projects in Boeny after their contribution reached 80% of the cost of the technology package, irrigation works in Boeny and Itasy experienced long delays (some works were still ongoing by the project closure); and increased prices of agricultural inputs (NPK and Di Ammonium Phosphate) during the implementation of the activity (ICR, p. 29). 6,810 farmers (52% of these were women) adopted improved agricultural technology compared to a revised target of 7,250. Women had high participation rates in the activities of the commercial Agriculture where 3,570 female beneficiaries compared to a revised target of 3,630 adopted improved technologies promoted by the project. By project completion, the value of additional crop production in sub-projects supported by the project reached US$0.90 million compared to a formally revised target of US$1.27 million 6 technologies were promoted by the project, compared to a target of 3. The promoted technologies included System of Rice Intensification (SRI), Improved Rice System (SRA), and the use of Urea Super Granules (USG); according to the ICR (Annex 1) these technologies were highly demanded by beneficiaries. However, it is not clear how the implementation of SRI was successful despite the delay in irrigation works. The three other technologies faced a lower level of demand (composting, the use of hybrid rice seeds, and Direct Seeding under Vegetative Cover-SCV). 2. Irrigation Development A total of 6,326 ha benefited from improved irrigation and drainage services: 4,795 in Alaotra, 809 in Boeny, and 722 in Itasy, which was 17% higher than the formally revised target of 5,390 ha. By project completion, 8,349 water users benefited from improved irrigation and drainage services (186.8% of the formally revised target of 4,470), including 846 females (126.3% of the formally revised target of 670). By the project completion, 54 WUAs benefited from trainings and capacity building activities provided by the project. Most of the WUAs existed before the Project, but they were either not functional or without an official status. The project provided institutional, organizational, and technical support in the form of trainings to these WUAs (ICR, Annex 1). By project completion, WUAs collected US$39.78 thousand compared to a target of US$33.60 thousand. However, these were fees for the completed works rather than O&M activities. 3. Watershed Development Page 6 of 17

7 By project completion, three Watershed Management Plans were in place (1 per region, target: 3); and 40 sub-watershed management plans developed and signed (target: 40). 109 Producer Organizations implemented 106 anti-erosion sub-projects (compared to target of 40). 1,183 ha were put under sustainable land management practices, which was slightly above the revised target of 1,150 ha. 4. Support for Rice Research and Development By project completion, 2066 soil samples were tested in project areas compared to a target of Soil sampling activities started in the first quarter of 2017, but since there was a delay in the construction of the new soil laboratory, the laboratory of the National Center for Applied Research in Rural Development (FOFIFA) at Tsimbazaza was mobilized to conduct preliminary analyses on the collected soil samples. Following the receipt of the new soil equipment by FOFIFA, the remaining analyses were conducted on the soil samples metric tons of base rice seed were produced compared to a target of metric tons. This result was due to the acquisition of seed production and analyses equipment, as well as the recruitment and training of 4 FOFIFA seed and soil researchers. In addition to the construction of 5 seed laboratories in four different regions (Antananarivo, Alaotra, Boeny and Kianjasoa). 104 collaborative research or extension sub-projects were completed compared to a formally revised target of 84. Outcome The PDO was expected to be achieved through increased access and adoption of improved technologies, improved irrigation infrastructure and services, and improved land management in the watersheds. By project completion, the direct project beneficiaries reached 23,257, compared to a revised target at restructuring of 20,261 (an increase of 14.8%). However, the project did not meet its target in terms of number of female beneficiaries. At the end of the project, only 35% of the beneficiaries were female, which was 6 percentage points less than the revised target of 41%. As a result of the project support, the average rice yield per hectare in project areas reached 4.15 tons per ha compared to a revised target of 4.26 tons per ha. This represented a 13.4% increase over the baseline value of 3.66 tons per ha and 97% achievement of the target. Yield increments varied by region, with the highest yield (6.98 tons per hectare) recorded in Itasy, followed by Aloatra (5.8 tons per hectare), and the least was in Boeny at 4.08 tones per hectare. According to the ICR (p. 10, para 19) "the yield was negatively affected by many weather shocks the drought affected the beginning of the growing season, followed by flooding right after the transplanting." These results were also confirmed through a quantitative assessment that involved producer organizations who benefitted from the project. The assessment at Page 7 of 17

8 project completion showed that 95% of surveyed organizations reported that high to moderate yield increments were a result of the project's support (ICR, p. 10 para 19). These yield increments were mainly the result of improved access to: improved seeds--34 tons of rice seeds were distributed to farmers located in the irrigated perimeters as part of the rice intensification activities, supplying agricultural inputs to beneficiaries (including small machinery, fertilizers and agrochemicals) and adoption of improved production technologies. The Irrigation development activities experienced significant delays and were only completed towards the end of the implementation period. The completion of the irrigation works combined with better watershed management were expected to result in a "more stable on-farm irrigation water supply, improved land management, and reduced farmer s vulnerability to droughts (ICR, p. 10 para 20 )". Improved watershed management was expected to decrease the rate of siltation in the irrigation canals, which, in turn, would improve the flow of water available to WUAs (ICR, p. 13, para 29). Therefore, it is plausible to assume that the project areas would experience further yield improvements in the future if farmers continue to have access to improved seeds and agricultural inputs in the absence of project support. That said, it was not clear in the ICR what was the impact of dropping the works in four large dams on the availability of irrigation water in the project areas. In a further communication, the project team explained that the works in the four dams were transferred to a sister-project implemented at the same time, and funds were reallocated to other project activities. Overall, efficacy is rated substantial despite the above mentioned shortcomings. The project almost met its outcome targets and most of its intermediate outcomes/outputs and with the completion of irrigation works rice yields were expected to improve further. Rating Substantial PHREVDELTBL PHOVRLEFFRATTBL Rationale There is one objective rated Substantial and thus the overall rating remains the same. Overall Efficacy Rating Substantial 5. Efficiency Economic and Financial Efficiency ex-ante Page 8 of 17

9 There was no comprehensive Economic and Financial Analysis (EFA) carried out at the appraisal stage due to short preparation time and relatively limited preparation budget (ICR, p. 6, para 7). The PAD assessed the project to be economically and financially solid based on: (a) the detailed EFA that had been conducted for the Irrigation Development and Watershed Management Project (IDA/GEF), the predecessor to this Project, that showed an ERR of 14%; (b) findings from the World Bank study Madagascar: The Impact of Public Spending on Perimeters Productivity that showed that past donor investments in the irrigation sector in Madagascar have generated attractive economic returns, significantly improved country welfare, and contributed to poverty reduction; (c) findings from simple farm models to explore the likely returns to investment in irrigated rice development, that used the target values specified in the original Results Framework. The analysis found an ERR of 25.9% over an assumed investment life of 20 years. ex-post The ICR EFA showed that the project generated almost US$37 million in NPV, with an overall ERR of 35%. It used a simple cost-benefit analysis to calculate the Net Present Value (NPV) and the Economic Rate of Return (ERR). This analysis was disaggregated by region (Itasy, Boeny, and Alaotra). The analysis was conducted using mainly project reports, final evaluations of the project, monitoring spreadsheets from the coordination unit, and national databases for crop prices. Assumptions: the analysis assessed incremental net benefits of typical beneficiaries in the three project areas (Itasy, Boeny, and Alaotra) by comparing benefits derived in the without-project (WOP) scenario to those derived in the with-project (WP) situation. The analysis used annual farm gate prices of rice and other crop after converting prices to U.S. dollars using the national exchange rate (US$1 = MGA 3,200). The models were conducted for a period of 20 years, at a discount rate of 10% to reflect the opportunity cost of capital in Madagascar. Project activities started a year after the effectiveness date. According to the ICR (p. 50, para 18) the delay resulted in a loss of about US$5.5 million in NPV and a 11.4% reduction in the ERR. However, there was no sensitivity analysis carried out to examine the impact of different scenarios on the ERR, for example, drought, increase or decrease in rice price, and increase or decrease in the price of inputs. The project disbursed 50% of the PHRD Grant in the last six months of implementation. Therefore, it was not possible to determine whether the irrigation works completed at the time of project closure were adequate and functional. Administrative and Institutional Efficiency Effectiveness was delayed by four months to September 15, 2014 rather than May 28, However, actual implementation started almost a year after effectiveness. The project lacked the technical designs of the irrigation works as they were not prepared at appraisal. This resulted in implementation delays where most of Page 9 of 17

10 the irrigation rehabilitation works were only finished in the last six months of implementation. There were also delays when OP 4.37 related to the safety of dams was triggered at mid-term. This operational policy required forming a panel of experts who intervened for eight months; and eventually the activities were not implemented. There were also delays due to weak procurement capacity at both central and regional levels. The project implementation experienced significant delays, and the actual impact of project investments could not be fully assessed at completion. The lack of an ex-ante EFA made assessing efficiency difficult. It was not possible to compare the pre-project and post-project EFAs before concluding whether the project achieved its anticipated economic benefits or if the investments were justifiable and still relevant. Therefore, efficiency is rated modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 0 Appraisal 0 Not Applicable ICR Estimate * Refers to percent of total project cost for which ERR/FRR was calculated Not Applicable 6. Outcome Relevance of Objectives was rated high. Efficacy was rated substantial despite some shortcomings. The evidence provided in the ICR showed that the project succeeded in increasing rice yields in project areas, however, the increment was slightly below the target. Due to significant implementation delays, the impact of rehabilitated irrigation infrastructure on yields was not yet documented. Efficiency was rated modest due to significant implementation delays; and the lack of an EFA at appraisal made it difficult to conclude whether the project achieved its anticipated economic benefits or if the investments were justifiable and still relevant. With a high rating for relevance of objectives, substantial rating for efficacy and modest rating for efficiency, the overall outcome rating is moderately satisfactory. a. Outcome Rating Moderately Satisfactory Page 10 of 17

11 7. Risk to Development Outcome Risk to Development Outcome is impacted by the following: Project related risks. The project experienced significant implementation delays. For example: the activities related to the development of commercial agriculture were expected to be implemented over a period of three growing cycles, with the project providing a technology package with a decreasing subsidy. However, cycle 1 only took place during the last growing season, and the beneficiaries were consequently not able to benefit from the Project support in the following cycles. Also, the retention rate (number of beneficiaries who move from one cycle to the next) was low, indicating low willingness to pay for the inputs when the subsidy decreases to less than 50% the price of the package. The seed banks were expected to benefit from the revolving fund that was provided by the Project. However, it is difficult to tell whether they were viable or not since they only became operational during the last growing season. In a further communication, the project team explained that the seed banks were sustainable after completion. Finally, the rice yield increments resulted mainly from subsidized project supplied inputs and it is doubtful that this level of support would continue post-completion. O&M of irrigation works. The sustainability of the irrigation infrastructure depends on the ability of WUAs to recover fees from members and carry out O&M effectively. At completion it was not clear whether the WUAs were operational or not. In a further communication, the project team explained that WUAs were in poor communities and did not have accounts. Institutional related risks. The activities under the watershed development were unlikely to be sustainable due to the lack of funding and difficulty to implement and enforce sanctions against violators--since the social contracts that regulate the management and preservation of investments in the watershed were not approved at the districts and courts. Since according to the project team, there was no national law on watershed management although there was progress at the regional level to adopt relevant rules and regulations. Climate related risks. Madagascar continues to be prone to severe weather events that could negatively impact the irrigation infrastructure and limit the availability of irrigation water to farmers. Such events could have a negative impact on rice productivity. 8. Assessment of Bank Performance a. Quality-at-Entry The project was prepared in 4 months with limited resources (less than US$90,000). It was designed in coordination with the Ministry of Agriculture and other development partners operating in Madagascar. Objectives were in line with Madagascar s agriculture agenda as it directly supported its Irrigation Development and Watershed Management strategy, as well as the World Bank s CPF. Design was simple with straight forward activities. This was the third project in a series of similar projects in Madagascar. Design reflected two important lessons from the previous projects in the Page 11 of 17

12 country. First, the need to adopt a multi-faceted yet integrated approach that would contribute to increased productivity and incomes in irrigation schemes and surrounding watersheds; protect natural resources in watersheds; improve the provision of agricultural extension and inputs; and supports the emergence of a private sector. Second, the need to put in place an unambiguous institutional framework that clearly defines the responsibilities of all stakeholders, ensures their full participation, coordinates decisions, and introduces mechanisms that encourage appropriate behavior and ensure respect for commitments made. However, design suffered from a number of shortcomings that negatively impacted implementation. First, the technical designs for the irrigation works were prepared during implementation rather than at appraisal. This caused significant implementation delays to the related activities. Second, some projectpromoted technologies were not suitable for the local farming conditions in Madagascar, for example, the production of compost was not adopted because the primary vegetable material used to produce it was rare and the process was labor-intensive. Also, the use of Direct Seeding under Vegetative Cover (SCV) methodology was not adopted by the beneficiaries because it required three years of training; and hybrid rice seeds were not adopted due to poor quality (ICR, p. 12 para 25). Third, there was no EFA prepared at the appraisal stage which made assessing efficiency difficult. Ten risks were identified at the appraisal stage. Two were rated high, three substantial, three moderate and two were rated low. Discussion in the PAD (p. 15, para 48) focused mainly on coordination risk (rated high) and ways to mitigate it. Mitigation measures for the other nine risks were listed in Annex 4 of the PAD. The ICR (p. 20, para 50) correctly highlighted three additional risk that were not identified at appraisal: first the effect of the rice policies on the competitiveness of smallholders, their incentives and welfare, second the risk associated with adverse climate events that could result in flooding or drought; and third the risk associated with the lack of investment in public goods especially roads. M&E suffered from design and implementation shortcomings (see section 9 for more details). QAE suffered from design shortcomings, overlooked risks and M&E weaknesses. Therefore, QAE is rated moderately unsatisfactory. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision Supervision benefitted from consistency at the task team leadership with only two TTLs-- the last one only joining the project at the final stages. Two implementation support missions were conducted every year. While only one mission was organized in 2016, this was compensated by the presence of the TTL and the project team in the field. Also, interim technical missions were organized to address problems with the project. Impilementation benefitted from regular and transparent communication with the government. International and local consultants advised the project on various issues including: procurement, M&E and financial management among other things. The Mid-term Review was conducted in a timely manner and included several recommendations that amended design weaknesses. The Bank was flexible and responsive through the creation of seed banks after the cyclones experienced in the country. Page 12 of 17

13 However, M&E could have benefitted from closer support through an M&E specialist as a full-time member of the World Bank team to provide support to the Coordinating Unit (ICR, p. 24, para 70). Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The PAD did not include an explicit theory of change. Nonetheless, the ICR (p.7) included a clear theory of change that reflected the connection between project activities, outputs, expected outcomes and long term outcomes. Monitoring was expected to be carried out through the implementing agencies (Ministry of Agriculture/Coordination Unit). Also, a consultancy firm under the GEF-financed Irrigation Development and Watershed Management project would undertake a household survey to generate secondary baseline information for use by the Project in April This baseline would be complemented by a household survey to be carried out toward the end of the Project to evaluate impacts. The Results Framework (RF) included three PDO outcome indicators. However, only one was directly linked to the PDO "average yield per hectare in irrigated rice production in project-intervention irrigation sites." The other two PDO outcome indicators were measuring total project beneficiaries and female beneficiaries. The RF also included 25 intermediate outcome indicators to assess activities under the four main project components. In general, these were linked to the activities and measurable. The RF could have benefitted from the following improvements: breaking up the main outcome indicator into two indicators. The first would measure yield increments that resulted from the commercial agriculture component and the second would measure yield increments in areas that benefited from irrigation works but not the commercial agriculture activities (ICR, p. 19 para 48). Also, the commercial agriculture component could have benefited from an indicator to asses market access by measuring the value of rice sold by the beneficiaries, for example. The RF suffered from some weaknesses. Targets for most indicators were set very low. The ICR (p.21 para 56) attributed this to "the crisis context and limited government capacity at the time of Project preparation." The indicator measuring "Clients who have adopted an improved agriculture technology promoted by the Project" was an output indicator. Finally, the indicator measuring the area provided with new irrigation or drainage services was inappropriately disaggregated to new and improved (ICR, p. 9, para 15). Page 13 of 17

14 b. M&E Implementation The M&E framework was implemented by the project coordination unit. However, an M&E specialist was only hired for a period of 5 months. After the Mid-Term Review (MTR), the Technical Assistant for the Operations (ATOP) took over M&E responsibility. While ATOP had good capacity to collect, analyze and report data, it was burdened with other tasks. After the MTR, the National Planning, Monitoring and Evaluation Department of the Ministry of the Presidency for Agriculture and Livestock, and the Public Debt Department of the Ministry of Finance and Budget monitored the Project s progress closely. This was possible through a monthly follow-up of the progress of the implementation of the activities, accompanied by a verification visit in the field. According to the ICR (p. 22, para. 58) "these supports were beneficial for the Project, because they helped to unblock several bottlenecks in carrying out the activities; the reporting of the results was regularly documented in the Aide Memoires." Two final evaluations were conducted by the Project by its closing date: (a) an evaluation based on beneficiaries perceptions of the Project and (b) a desk review of the Project documents combined with fieldwork. However, a quantitative impact assessment at completion was not done. This would have provided a more accurate assessment of the project s achievements. Changes to the RF after restructuring. Four main changes were introduced to the RF: the targets for 19 indicators were increased, targets for two indicators were reduced, one indicator was dropped because it was not being measured; and the disaggregation of the indicator Area provided with new irrigation or drainage services was removed. These changes were relevant because the original targets were conservative, and most of these targtes were almost achieved by mid-term despite the project's low disbutsment rate (ICR, p. 19 para 48). c. M&E Utilization According to the ICR (p. 22, para. 56) the M&E data helped project management to track progress and identify implementation delays. M&E findings also helped in refining activities after the mid-term review and restructuring when targets on most indicators were increased and funds were reallocated between components. M&E design and implementation suffered from moderate shortcomings. Therefore, M&E quality is rated modest. M&E Quality Rating Modest Page 14 of 17

15 10. Other Issues a. Safeguards The Project was classified at appraisal as a Category B. The initial design triggered three safeguard policies: Environmental Assessment (OP/BP 4.01), Pest Management (OP 4.09), and Involuntary Resettlement (OP/BP 4.12). The borrower prepared an Environmental and Social Management Framework (ESMF), Resettlement Policy Framework (RPF), Pest and Pesticides Management Plan (PPMP), and Dam Safety Manual. These were published prior to appraisal at the World Bank s InfoShop on May 25, Also, a manual on small dams was produced during Project preparation to identify and analyze the design of small dams that could be financed by the Project. At mid-term, OP 4.37 was triggered because some planned dams were classified as large. These activities were eventually suspended due to the significant amount of time required to authorize them through a panel of international dam experts. This would have caused further delays to the project. The ICR did not provide details on the expected impacts and mitigation measures regarding the triggered safeguard policies, but reported that "there were no major safeguard issues during Project implementation, and overall, the Project activities complied with all applicable World Bank policies (p. 23, para 62)." The project benefitted from an environmental and social safeguard specialist within the Coordinating Unit who was assigned to carry out the environmental and social monitoring of the Project. However, the project lacked a specific Grievance Redress Mechanism (GRM). b. Fiduciary Compliance Financial Management. While the project's coordinating unit was slow in addressing the Bank's comments--as it oversaw two projects, financial management reports were of acceptable quality and were delivered on time. However, financial management suffered from weak internal control systems including: insufficient and inadequate supporting documentation for expenditures, low quality of the planning and budget follow-up, and delays in implementing external and internal auditors recommendations. The project provided capacity building support to overcome these weaknesses. The ICR (p. 23, para 64) reported that while there were no losses of project resources or ineligible expenditures recorded, some expenses were questionable because of missing documentation. However, these were resolved during the grace period by the Project. According to the ICR (p. 23, para 64) "both external and internal audit reports did not flag major issues." Procurement. Procurement capacity at the central and regional levels was weak and contributed to long delays in procurement activities. This situation improved at a later stage of the project implementation after the Bank provided training, technical support, and remedial measures. The Bank also carried out annual procurement review missions. The first Post Procurement Review (PPR) conducted at mid-term, in June 2016 reported several procurement issues, including: poor planning of the activities; non-publication of unselected contracts; attribution of awards followed by an amendment 10 days later. The PPR also provided recommendations to improve the performance of procurement. A PPR conducted in 2017 showed that most of Page 15 of 17

16 these recommendations were implemented by the project s procurement team. By project completion, all the contracts were awarded except for one contract; which was cancelled because of an underperforming contractor (ICR, pages 24 & 25, para 65). c. Unintended impacts (Positive or Negative) --- d. Other Ratings Ratings ICR IEG Reason for Disagreements/Comment Outcome Moderately Moderately Satisfactory Satisfactory --- Bank Performance Moderately Moderately Satisfactory Satisfactory --- Quality of M&E Modest Modest --- Quality of ICR Substantial Lessons The ICR included seven lessons. The following are emphasized with some adaptation of language: Projects promoting rice intensification should also focus on improving irrigation infrastructure to ensure adequate amounts of water reach the fields. Without rehabilitation and maintenance of the irrigation infrastructure, water losses increase, which eventually leads to a decrease in yields. The availability of irrigation water to farmers is critical in order for intensification activities (improved inputs and growing practices) to achieve the desired yields. Without proper amounts of water, plants become stressed, and intensification activities fail to achieve the desired results. Productivity-enhancing projects should always include market access activities to encourage surplus production by smallholders. Smallholders respond to market incentives, and in the absence of marketing opportunities (due to remoteness and/or lack of information), it is unlikely that they will perceive a benefit in producing a marketable surplus. Thus, supporting market access is key (by building feeder roads, or creating productive partnerships, among others). An Economic and Financial Analysis (EFA) at the appraisal stage is required to be able to confirm project efficiency at completion. It is always important to compare the pre-project and post-project EFAs Page 16 of 17

17 before concluding whether the project achieved its anticipated economic benefits or if the investments were justifiable and still relevant. 13. Assessment Recommended? Yes Please explain To validate the impact of irrigation works on rice yields in project areas and provide more lessons to guide future operations. The project was implemented in remote areas of Madagascar; and it would be important to know how contractors performed and how the project ensured the quality of the works. 14. Comments on Quality of ICR The ICR is well written. It provides thorough coverage of project activities and candidly reports on most shortcomings. Discussion of outcomes reflects the project achievements and was supported by M&E data, assessment survey and reference to external literature. The ICR also includes well organized Annexes that cover different aspects of the project. The ICR could have improved on the following points: Safeguards: provide more details on activities and mitigation measures; and include a specific compliance statement for each of the triggered policies. Annex 2: it would be helpful to include the title of the team members rather than putting their role as "team member" only. This would help the reader to understand the specialization/field of each member. Language: while the ICR was concise, the text included the term "and so on" in fifteen different areas of the document. A more specific description would have better served the reader. a. Quality of ICR Rating Substantial Page 17 of 17