Evaluation Brief. Comprehensive Evaluation of the Development Results of the African Development Bank Group Background and Context

Transcription

1 An IDEV Corporate Evaluation Independent Development Evaluation African Development Bank From experience to knowledge... From knowledge to action... From action to impact Comprehensive Evaluation of the Development Results of the African Development Bank Group Background and Context The Comprehensive Evaluation of the Development Results (CEDR) of the African Development Bank Group (AfDB, or Bank) aims to provide an independent, credible and evidence-based assessment of development results achieved by the Bank between 2004 and It seeks to answer three questions: (1) Has the Bank achieved its objectives? (2) Has the Bank proposed results-focused strategies and programs? And, (3) Has the Bank emerged as a valued partner at country level? As well as contributing to accountability, it identifies lessons and makes recommendations to inform the implementation of the Bank s new strategic priorities, the High 5s. The scope of the evaluation is all Bank interventions (lending and non-lending) that were approved between 2004 and Since it is impossible to review all interventions exhaustively, the CEDR is based on evaluation studies carried out on a sample size of 14 countries. The countries are: Burundi, Cameroon, Democratic Republic of Congo, Ethiopia, Ghana, Morocco, Mozambique, Nigeria, Senegal, South Africa, Tanzania, Togo, Tunisia, and Zambia. (Figure 1). In all, the 14 countries represent almost 60% of the Bank s lending portfolio, based on approvals during , and broadly match the composition of the Bank s portfolio in terms of regional balance, language, fragility and eligibility to the various windows of Bank financing (Figure 2). For each of these countries, strategies and programs were examined, and all projects that were completed or near completion (169 in total) were reviewed. (See Methodological Details for more information) Evaluation Brief An IDEV Corporate Evaluation Independent Development Evaluation African Development Bank From experience to knowledge... From knowledge to action... From action to impact Comprehensive Evaluation of the Development Results of the African Development Bank Group Synthesis Report October 2016

2 Figure 1: The 14 countries involved in the CEDR Evaluation Relevance Senegal Morocco Togo Ghana Cameroon Democratic Republic of the Congo Nigeria Tunisia Zambia Ethiopia Burundi Tanzania Relevance portrays the extent to which Bank interventions are aligned with the needs of the countries, the needs of the target beneficiaries, and the Bank s policies and strategies. The relevance of Bank interventions was rated moderately satisfactory. Just over half of the country strategies and programs (57%) were found to include interventions that were precisely defined in line with country and beneficiary needs. The evaluation finds that the Bank was able to reach stronger alignment with country needs when it mobilized interests across diverse stakeholders; and was able to adapt by coming up with responsive actions to identified constraints to achieving results in the country. Mozambique Effectiveness South Africa Has the Bank achieved its objectives? The evaluation finds that the Bank delivered development results, though not to its full potential as explained below. Effectiveness reflects the extent to which the Bank interventions led to the expected development outcomes, and how far the target groups were able to benefit from them. Effectiveness was rated moderately unsatisfactory. Only close to 30% of Bank interventions examined had achieved or were likely to achieve their intended outcomes. The same proportion was assessed as likely to achieve less than half of planned outcomes. On the upside, more than half of the projects were assessed as likely Figure 2: The CEDR country sample closely matched the composition of the Bank's portfolio 30 % share of sector in CEDR countries and Bankwide Agriculture Communications Environment Finance Industry & Mining Multi-Sector Power Social Transport Water Sup/San Bankwide 14 CEDR

3 by design to lead to positive benefits for targeted groups (men, women, youth and girls), showing that the Bank can make a difference in the lives of people. Sustainability Sustainability reflects the extent to which benefits continued, or are likely to continue, once Bank interventions were completed. Sustainability was rated moderately unsatisfactory. Less than a third of the projects reviewed had economic and financial mechanisms that were robust enough to maintain the outputs and outcomes beyond the lifetime of the project. Less than half of the projects were found to have technically sound project design and only about 40% of the projects attended to strengthening institutional capacity. Efficiency The efficiency of the Bank in delivering its support, in terms of timeliness and cost, was rated moderately unsatisfactory. Close to half the projects assessed took more than the target 12 months from approval to first disbursement of funds. Nearly one-third of the projects took more than 25% more time than foreseen in the initial plan to implement. Project delays were associated with weak project design, lengthy Bank procedures and complicated arrangements with other development partners. Has the Bank proposed results-focused strategies and programs? The ambitious reform agenda on which the Bank has embarked to transform itself into a results-oriented learning institution has set it in the right direction. The agenda is still to yield its full results, in particular due to the behavioral change issues that were not specifically addressed. Selectivity, the capacity to propose proper responses to country needs, improved over time. However some country strategies failed to select sector-specific objectives that focus Bank efforts in its areas of comparative advantage. Furthermore, strategic selectivity did not always translate into a selective portfolio of projects. Whereas the quality of project-level intervention logic improved over the evaluation period, the focus on outputs remained greater than the focus on outcomes. In some cases the indicators were not appropriate. The quality of project design was also limited by flaws in risk analysis and mitigation strategies. While the quality and frequency of supervision increased over time, the information on project performance was often inadequate. Finally, learning remained weak. Country teams did not fully use the lessons from previous country strategies. No clear pattern of improvement emerged over time, and lessons learned from supervision or other oversight mechanisms were not always fully taken into account. Has the Bank emerged as a valued partner at country level? The Bank had strong relationships with its clients and development partners. However, these relationships were not fully backed by the relevant capacity for broadly positioning the Bank beyond a provider of financing as an influential advisor for policy making. Insufficient communication about opportunities offered by the Bank meant that economic and sector work and technical assistance were not fully leveraged to respond to country needs. The discourse and volume of analytical work by the Bank has increased since 2008, but there is limited evidence that they made tangible contributions. The exception was in fragile situations where the Bank was able to use its brand and relationships to engage in influential policy dialogue. Though most strategies mentioned coordination with other partners, this did not always translate into an alignment of priorities and cooperation at the operational level. Here again, building on long-standing relationships with the government, the Bank played a positive role in fragility and emergency contexts. Leveraging in projects was more ad-hoc than driven by strategic goals set forth in the country strategies. The focus was on cofinancing rather than actively mobilizing additional resources, although positive practices were also encountered in some cases. One example of this was promoting and attracting private sector financing into private-public partnerships.

4 Contextual and internal drivers of Bank performance The Bank s performance was influenced by country conditions. Where leadership, ownership and national capacity to implement existed, interventions were more effective and more sustainable. Where there were preexisting frameworks for country dialogue, the Bank engaged in coordinated partnerships and initiatives to leverage more development resources. But when country conditions were less favorable, the Bank was unable to analyze the contextual constraints and adapt its interventions to become relevant, effective, and sustainable. While having a country office was advantageous, it was not a sufficient condition for the Bank to effectively perform its various roles. Capacity constraints and risk-averse behavior at country level were found to limit the effectiveness of the Bank s presence. Task management and supervision of operations from headquarters did not aid contextual learning. Furthermore, the low flexibility of corporate procedures limited usefulness. In fragile situations however, longstanding partnerships facilitated the Bank s work, despite the challenges of working in settings constrained by capacity or resources. Finally, the quality of design and effective supervision proved to be the most important yet most limiting factors in country portfolio performance. The importance of these two factors was clearly recognized and multiple reforms related to them, albeit not exclusively, were initiated by the Bank. However, there is still room for improvement. Recommendations The evaluation makes the following recommendations to the Bank as it implements its new strategic priorities. Where actions are already ongoing in the Bank, the recommendations aim to feed lessons into the process and help identify key priority issues to tackle. form the basis for framing strategies, programs and projects around key selected outcomes. The Bank should ensure its country presence allows for implementing its strategy. It should review the terms of references of country offices and make sure they have adequate skills and resources. Improving corporate services The Bank should clarify its impact pathways by ensuring corporate strategies are based on a well-designed theory of change shared with stakeholders and partners. The Bank needs to enhance the flexibility of corporate procedures to allow nimble responses to country-specific needs and context. The Bank should strengthen its offering in terms of Economic Sector Work. Whenever it can fill knowledge gaps in specific niches related to its strategies, the Bank should be in a position to propose a relevant combination of analytical work, dialogue and financing instruments to the client country. Enhancing delivery The Bank needs to strengthen its performance and accountability frameworks, processes and culture and align incentives to deliver results. The Bank should give closer attention to the depth and quality of supervision for private sector operations and strengthen the implementation of supervision for public sector operations. Positioning in context The Bank needs to analyse and clarify the strategic roles (project financier, knowledge broker, and advisor) it wishes to take, as well as actively review and pursue partnership opportunities, as relevant to the country context. The Bank should also deepen its understanding of constraints to implementation and sustainability. Together with clarity on roles and possible partnerships, this should

5 Methodological details The CEDR is designed as a synthesis of evaluation studies that were undertaken at country level. The sample of representative countries was selected using a purposive sampling strategy. The objective was to portray a significant share of the Bank s portfolio and reflect its composition in terms of regions, language, eligibility for various sources of Bank financing, and fragility status, insofar as possible. The CEDR synthesis is based on a theory of change that guided the design of its 14 evaluation questions. The theory of change depicts the linkages between Bank activities, outputs, and outcomes and provides a detailed description of impact pathways or how outputs contribute to intended outcomes. It was developed by the evaluation team after a thorough review of relevant documents: Bank policies, operational strategies and guidance documents, evaluations and assessments, and comparable documents from major Development Partners. The synthesis drew on multiple lines of evidence (See CEDR process diagram). For all 14 countries an evaluation of the Bank s Country Strategies and Program was conducted and country performance case studies were undertaken as an integral part of the process. Detailed Project Results Assessments were carried out for 169 projects that were completed or ongoing and close to completion in these countries. A rigorous quality assurance process was established to ensure consistency of evaluations across countries and projects. Ten past independent evaluations and studies were used for triangulating findings from the countries and projects evaluations. Consultations were undertaken with stakeholders for each of the country studies. Overall, close to 1900 persons were interviewed. Of these, 10% were Bank staff and 90% were government counterparts and national stakeholders, including private sector and civil society. The CEDR process was also guided by a panel of Senior Independent Advisors including eminent evaluation and development experts. The Panel provided an independent report on the evaluation and gave the evaluation team credit for delivering credible answers to strategic questions raised. As with any evaluation, the CEDR has its limitations. The main limitation is related to the challenge of assessing results along the impact pathways defined in the theory of change. Project level intervention logics were not always consistent with the theory-defined outcomes. Moreover, when they were, the indicators were not always aligned. Other problems included the absence of baseline data, shortcomings in indicators, uncertainty of data quality and reliability for some projects. Clear assessments guidelines, rigorous quality assurance of data, and careful data analysis processes were effected to mitigate this limitation. Some design challenges also arose. Taking countries as a basis of analysis resulted in project samples that were not statistically representative of the overall Bank s portfolio. Findings were therefore not generalized directly to overall Bank performance from this data assuming that the sample was statistically representative of the entire population but based on theory and taking into account the representativeness of the sample as described above. About IDEV IDEV carries out independent evaluations of Bank operations, policies and strategies, working across projects, sectors, themes, regions, and countries. By conducting independent evaluations and proactively sharing best practice, IDEV ensures that the Bank and its stakeholders learn from past experience and plan and deliver development activities to the highest possible standards. Management Response The evaluation provides a sober assessment of the Bank s performance between 2004 and And while Management does not always share IDEV s conclusions, it broadly subscribes to the recommendations it makes. To this effect, since 2009 Management has launched a range of initiatives aimed at addressing the challenges raised by the evaluation. These initiatives received additional impetus in April 2016 when the Board adopted the new Development and Business Delivery Model (DBDM) with the objective of further improving the effectiveness and efficiency of AfDB s actions. IDEV s evaluation is particularly valuable as the Bank rolls out these new reforms.

6 CEDR process Design Theory of change Evaluation Questions Building blocks Country performance case studies Project level assessments 14 Contextual Factor Reviews 169 Project Results Assessments 14 Country Strategy and Program Evaluation reports Review of Bank Corporate Documents Rigorous Quality Assurance Qualitative Comparative Analysis Portfolio review Synthesis 10 past evaluation studies CEDR Synthesis Report CEDR Synthesis Task Manager: Samer Hachem, Division Manager, IDEV Publication coordinated by Jacqueline Nyagahima, Consultant, IDEV Published: November 2016 Full report available at idev.afdb.org Independent Development Evaluation African Development Bank African Development Bank Group Avenue Joseph Anoma 01 BP 1387, Abidjan 01 Côte d Ivoire Phone: Fax: idevhelpdesk@afdb.org Design: CRÉON Layout: Phoenix Design Aid, Denmark idev.afdb.org