Impact Evaluation in Development Institutions: a Look at the African Development Bank

Transcription

1 Jacob Oduor, Elsa de Morais Sarmento, Laurence Lannes Impact Evaluation in Development Institutions: a Look at the African Development Bank evaluation Matters

2 Recent questions regarding aid effectiveness have intensified the demand for better accountability and evaluation of aid impact. The Paris Declaration on Aid Effectiveness (2005) called for results-based and evidence-based management of development aid. The post-2015 development agenda also focuses on addressing accountability gaps that still remain and on ensuring that the new development commitment framework results in fulfilled promises. 69 In view of the greater emphasis placed on accountability, Impact Evaluation (IE) has become an influential tool in determining the effectiveness of interventions at the micro level, At a more macro level, IE has allowed for greater insights in terms of accounting for public spending and has provided a basis for evidence-based policy making. IE is able to demonstrate what does and does not work and the underlying causes and disconnects, while being simultaneously able to quantify effects. It also provides a sound basis for developing lessons learned regarding project/program success factors and governance aspects, which can be applied to the design of future projects. IE is thus essential within the current global efforts to improve aid effectiveness and public spending whilst increasing accountability. The new development approach of strategy-driven operations and of management for development results have established that operational services are increasingly required to demonstrate impact in order to obtain funds and to be accountable to donors and beneficiaries. There has been a growing emphasis in most multilateral and bilateral development institutions on measuring the impact of interventions. The dual meaning of the term impact in the development discourse, meaning either long-term effects or strictly attributable effects, remains an enduring source of ambiguity and misunderstanding. But the debate on the extent of attribution of outcomes to interventions when results-based management is implemented becomes important for an institution such as the African Development Bank (AfDB) that often is involved in partnerships and co-financing activities, also because outcomes may be affected by more than one independent intervention offered to beneficiaries by different institutions. The state of IE in Africa The past decade has seen a proliferation of IEs worldwide, undertaken on a wide range of development interventions from health to cash transfers, and from gender empowerment to infrastructure initiatives. In several cases, the findings have been highly influential in shaping future projects and in shaping policy development and programming. In recent years, IE has been done rigorously by some of Africa s top scholars, who have not shied away from the big questions relevant to the continent. An estimate of up to 250 completed or on-going impact studies has already been conducted in Sub-Saharan Africa. 1 Demands for rigorous policy evaluation have also been building slowly, but steadily. However, the number of large-scale, randomiszed, controlled trials of government policies in Africa remains limited, though on the rise. 1 Africa Impact Evaluation Network ( A quarterly knowledge publication from Independent Development Evaluation at the African Development Bank Group

3 70 Increased knowledge and awareness of IE initiatives in Africa can enhance the pace and quality of development and boost resource mobilisation through improved transparency and accountability. However, much of the pioneering work to develop rigorous IEs in Africa has been conducted in a limited number of locations and in a reduced number of sectors. The role of multilateral development institutions in supporting these developments, namely in mainstreaming IE in the projects they fund and deploying resources to consolidate these efforts remains, therefore, crucial. Measuring development effectiveness at the AfDB At the AfDB, IE still has a limited scope. Although certain types of financing products or projects, such as rural water and sanitation and infrastructure, have mobilized substantial resources, the Bank has only recently started conducting rigorous research on what actually works and about what can be attributable to its own interventions. Traditionally, projects and programs have included a Monitoring and Evaluation (M&E) component (Box 1). However, change is underway. The AfDB, in its Strategy , reinforced its commitment to improving the governance and accountability of its operations and strengthening the focus on results, through amongst others more Impact Evaluations at the country level [ ] to better capture the results and lessons of programs and projects. This mandate is thus focused on ensuring that assistance is focused on development results and impacts. Along the same lines, the Board of Directors of the Bank, during the discussions for the Thirteenth General Replenishment of the Resources of the African Development Fund (ADF-13) committed to increase the use of evidence from IE in project design. Section 3.24 of the final report notes that as part of its efforts to enhance the quality of design and implementation, Management will enhance its efforts to develop and support a learning-oriented culture across the Bank, including putting a greater emphasis on evaluability and use of self-evaluation. The AfDB Group is using IE selectively where the most valuable insights can be assessed. It also committed to build a limited number of IEs into projects to establish sound baselines and incorporate Box 1 Monitoring and Evaluation at the AfDB So far, AfDB-funded projects have mostly included a M&E component that assesses the achievement of specific targets (e.g. performance indicators in the result-based logical framework). Understandably, project Task Managers concentrate on project design and implementation during the early phases and leave for last decisions how to prepare the ground to enable strong IE. So far, there has been very little incentive to offset the cost and development time allocated to the design of an appropriate project-related information system that enables impact assessments. Thus, the management for development results needs to be better embedded and used more systematically throughout the project-cycle. Past evaluations have also produced limited information about projects/programs intermediary or long-term effects and the net change in outcomes that can be directly attributable to the former, mostly due to the lack of baseline information, systematic data collection throughout the project-cycle that allows the evaluation of final outcomes, and the adequate management information systems. After project completion, the AfDB s Independent Development Evaluation (IDEV) conducts independent evaluations of the performance of selected projects (Review Notes of Project Completion Reports and Extended Supervision Reports for the private sector). It also conducts project performance evaluations for individual or thematic clusters of projects, country-level evaluations, and high-level evaluations for specific sectors and themes. IDEV is currently undertaking a comprehensive evaluation of the development results of the last decade of AfDB operations in Africa and under this umbrella is conducting its first two IEs in the water and sanitation sector in Ethiopia and Tanzania. evaluation Matters

4 the methodology into the design of projects and conduct additional IEs to contribute to learning in areas where knowledge is required. A second commitment has been made by the Bank towards a clear focus on knowledge production as a necessary complement to financial activity. At all levels of work, the focus of the AfDB has now been placed on increased and demonstrated effectiveness of development assistance and on knowledge creation. Recent efforts to develop impact evaluation at the AfDB Since 2014, two operations departments, the Human Development Department (OSHD) and the Water and Sanitation Department (OWAS), with the support of the Research Department (EDRE), have joined forces and initiated two IE studies, one in in Senegal and one in Zambia/Malawi to assess the impact of WASH interventions on health and other social outcomes. OSHD has also embedded IE components into a number of new projects in Congo Brazzaville, Kenya, Uganda, and Malawi to evaluate the impact of several education interventions on the access, quality, and relevance of higher education in those countries. In parallel with these sectoral efforts aimed at demonstrating impact and bringing awareness to other departments regarding the need to embark on IE, a Bank-wide Impact Evaluation Reference Group (IERG) was set up in 2012 and is currently working on an IE strategy for the Bank. The IERG aims at raising the awareness of impact evaluation at the institutional level (Box 2). At the moment, the governance structure for mainstreaming IE at the AfDB comprises two bodies. Combined, they are able to synergise efforts and have been providing the resources to put these initiatives to work: the Impact Evaluation Technical Team, housed at EDRE, which is in charge of IE research and technical support, and the IERG, coordinated by EDRE and comprising the IDEV) and 3 operations departments (OHSD, OWAS, ONRI), responsible for the coordination and operational aspects of IE. In addition, it is envisaged that other IE focal points within the remaining operations departments can join this initiative in the near future. As part of its activities to raise awareness on IE, the IERG invited Professor Paul Gertler, a world-class 71 Box 2 AfDB s Impact Evaluation Reference Group (IERG) The draft Impact Evaluation strategy establishes the AfDB s IERG with the mandate of providing strategic guidance at all stages of the IE process and supporting robust and credible impact evaluation. It will also provide operational guidance for the IE management and act as a sounding board for feedback and implementation on IE in the Bank. The key tasks and responsibilities of the RG include, but are not limited to, the following: Preparation of appropriate base documents including Bank s Impact Evaluation Strategic Plan Coordinating training and learning events on IE Providing guidelines for selecting projects for IE Assisting in prioritising and determining the strategic focus of IE in the Bank Identifying relevant topics/themes within the selected strategic focus that IE should deal with and look into the proposals from Operations Departments for IE Assisting in the identification and selection of sources of financing, trust funds and other available sources Overseeing the implementation of directives from Operational Departments in implementing IE Approving the evaluation dissemination strategy and provide guidance on the use of evaluation findings Contributing to the preparation of an annual report on IE in the Bank Developing and promoting new approaches to IE in the major fields of Bank interventions that will stimulate evidence-based policies, such as systematic reviews and real-world impact evaluations. A quarterly knowledge publication from Independent Development Evaluation at the African Development Bank Group

5 72 expert on impact evaluation from the University of California, Berkeley on December 9, 2013, to speak in the first Bank-wide outreach event on measuring results and evaluating the impact of development operations. More than 75 Bank staff attended this unique event providing an opportunity to enhance their understanding of IE challenges, familiarise themselves with strategies to identify when and how to conduct IEs, understand the difference between IE and cost-benefit analyses, and learn how to compare results with its counterfactual. This outreach event was followed by a four-day technical training run by the Regional Centre for Learning on Evaluation and Results based at the University of Witwatersrand in South Africa for a selected group of 25 task managers from IDEV and EDRE. What s next? In line with the AfDB s Strategy, IE should be increasingly used in the Bank. The AfDB s IE agenda is to be advanced by prioritising interventions with the potential to inform on what works and what doesn t, in order to increase the Bank s effectiveness and accountability while strengthening its governance structure and enlarging the impact of interventions with the ultimate aim of improving its contribution to poverty alleviation and inequality reduction. The proposed intervention areas are thus in line with the AfDB s ten-year strategy, which includes infrastructure, regional integration, private sector development, skills and technology, governance and accountability as the main pillars. Closing the gap and staying in the loop Results-based management has generated much enthusiasm as it provides a strong link between development interventions and outcomes. In this context, IE has been put under the spotlight. The need to assess whether development outcomes are generated by particular interventions, their costeffectiveness and how the impacts are generated remains and has been emphasised during the last years. So far, lack of emphasis on IE at the AfDB has made it difficult to conclude whether aid results has created incentives for poverty reduction or has gone further by having an effective additional impact in fighting poverty. The changing landscape towards more results-based approaches to aid and the evaluation of the former has a number of serious implications, namely for financing. There might still be a scarcity of resources and budget to carry IE on a broad scale, thus the challenge is to prioritize while remaining highly strategic. In concomitance with sister institutions, the AfDB needs to move towards best-practices to quickly close the IE gap, while keeping itself in the loop on new methodological developments. The pace of methodological innovations has accelerated, as new qualitative and quantitative evaluation frameworks are constantly being developed and tested. It is now recognized that there is no one-size-fits-all conceptual and methodological approach and no universal solutions. Closing the gap and aligning to the highest standards means accepting that we move beyond the conventional before versus after analysis to develop and implement a diverse portfolio of experimental and quasi-experimental evaluation designs that address the issue of attribution while being aligned with internationally accepted best-practices. The AfDB keeps on pursuing these goals and working towards diversifying its methodological approaches and knowledge products in the field of IE, notably through pilot operations and specialised partnerships. evaluation Matters

6 Profile of the Authors 73 Jacob Oduor joined the African Development Bank in 2012 as Principal Research Economist in the Development Research Department. He holds a PhD in Economics from the University of Bielefeld in Germany with specializations in Econometrics and Macroeconomics. Before joining the Bank he was Director of Research at the Centre for Research on Financial Markets and Policy at the Kenya Bankers Association (KBA). Prior to that, Jacob was a Senior Research Analyst at the Kenya Institute of Public Policy Research and Analysis (KIPPRA) in Kenya where he had worked since He has also served as a lecturer in the Department of Econometrics and Statistics at Kenyatta University, Kenya where he taught for seven years. Elsa de Morais Sarmento is an applied economist and evaluator. She has lectured for over a decade in economics and econometrics and has occupied such positions as lecturer at the University of Aveiro (Portugal), researcher at NOVAFRICA (Nova School of Business and Economics), CEP (LSE/ UK) and the European Parliament, and evaluator at the African Development Bank, Word Bank, UN agencies, Organisation of Eastern Caribbean States, and the European Commission. She also acted as policy adviser at the House of Commons (UK), for several governments in Africa and supra-national institutions in Europe, and was appointed to several Director positions at the Research Office of the Portuguese Ministry of Economy. Laurence Lannes is a Principal Health Economist in the African Development Bank. Prior to joining the Bank, she worked in the field of development, notably in France, in the Africa Region of the World Bank Group and in South East Asia. Her areas of expertise include impact evaluation, ehealth, pharmaceutical sector and health financing. She recently completed her PhD the at London School of Economics on health service delivery performance in Rwanda using data from an impact evaluation. A quarterly knowledge publication from Independent Development Evaluation at the African Development Bank Group