Impact Evaluation AND Human Development
Impact evaluation carefully measuring the capacity of poverty reduction programs to deliver results is central to the mission of the World Bank s Human Development Network. The network mainstreams evaluation into projects across three core areas: health, education, and social protection. But it also works to ensure that the reach of evaluation efforts goes well beyond the World Bank s own projects. Working through strategic partnerships, HDN makes evaluation happen all over the world, wherever it is needed most to fill critical knowledge gaps. As a case in point, HDN recently completed an evaluation of Save the Children s preschool program in Mozambique, the first impact evaluation of such a program in rural Africa (see p. 2). The results show that it is possible to successfully deliver early childhood benefits at low- cost, providing a model that can be scaled-up in Mozambique and used to shape programs to reach millions of the world s poorest children. Through partnerships, HDN has grown The Human Development Network (HDN) is at the to become one of the leading funders of forefront of mobilizing evidence for better policymaking evaluations worldwide. HDN manages the in health, education, and social protection and labor. Strategic Impact Evaluation Fund (SIEF), Across the globe, HDN impact evaluations are generating knowledge that improves the design of programs government. The fund is being used to started with $40 million from the British for world s poorest families. HDN is a global knowledge support impact evaluations in key areas of hub on what works and what doesn t in development, childhood development, health systems, and has trained thousands of professionals from governments around the world in impact evaluation tech- education, and water and sanitation. The niques. Looking forward, HDN shares six broad lessons fund builds on the successful model of the for working smarter in impact evaluation. Spanish Impact Evaluation Fund, which ran from 2007 2012 and supported more than 50 impact evaluations in strategic policymaking areas, in addition to running an extensive training program for researchers and policymakers. Generating Evidence for Policy: Action that Changes Lives A key feature of these evaluation studies is that they actually change policy and results on the ground. This is because HDN doesn t just support impact evaluations, but also is engaged in delivering programs. Working through the World Bank s close partnerships with governments all over the world, the network is in a position to apply evaluation results to program design and implementation. This results in a tight and positive link between evaluation and policy action. As the following three case studies illustrate, HDN evaluations lead to policy impact in a number of ways: 2 Impact Evaluation and Human Development
Small pilot programs that deliver results may be scaled up to positively affect the lives of millions. Existing programs may be improved via design changes informed by evaluation to maximize results for beneficiaries and increase cost-effectiveness. New ideas may be put through their paces, encouraging innovation and saving public resources from being spent on ineffective approaches. Case 1: Scaling up a small, pilot preschool program in Mozambique HDN recently financed and conducted an impact evaluation of a small Save the Children preschool program in Mozambique. HDN chose to support this evaluation because of the critical knowledge gap it fills. While research clearly shows the importance of early child development interventions in raising life chances of moving out of poverty, most interventions known to work are too costly to be affordable on a large scale in the world s poorest countries. For as little as about $2.50 a month per child, children and families in Mozambique reap the rewards of early education both now and in the future. Carolyn Miles, President & CEO, Save the Children 1 While similar small preschool programs are scattered across rural Africa, no robust impact evaluation of these programs had been conducted to determine whether they actually present a viable alternative to traditional, more costly models of early child development programs. Save the Children s preschoolers in Mozambique. Photo credit: Reuters Impact Evaluation and Human Development 3
The evaluation results demonstrated to Save the Children and the Government of Mozambique that small investments in children can deliver tremendous rewards. Children in the program are 24 percent more likely to enroll in school and are better prepared to benefit from that schooling, with more advanced cognitive, fine motor, and socio-emotional skills. The program also has important family spillover effects; siblings are more likely to enroll in school, and parents more likely to become income-earners. After seeing the evaluation results, the government presented an official request to the World Bank for help in developing a national Early Childhood Education project a first for Mozambique. At the same time, the Ministry of Education is expanding the preschool program from 30 to 600 communities. And there will likely be ripple effects far beyond Mozambique, as the program is viewed as model for low-income countries around the world. Case 2: Improving the supply chain for lifesaving medicines in Zambia In Zambia, shortages of anti-malaria medication and other basic lifesaving drugs are frequent, leaving children and adults at greater risk of sickness and death. The Government of Zambia correctly diagnosed the problem as centered not on the availability of these lifesaving drugs in the country overall, but on bottlenecks in the distribution system. Results-based financing and HDN With funding from the Health Results Innovation Trust Fund (HRITF), the World Bank is supporting countries to design, implement and evaluate sustainable results-based financing (RBF) pilot programs that improve maternal and child health outcomes. A key element of this program is to ensure that a rigorous and well designed impact evaluation is embedded in each country s RBF project. What policy change would work best to reduce the prevailing medical supply bottlenecks? To help the Government of Zambia solve this problem, and as part of the Malaria Impact Evaluation Initiative, HDN sponsored an evaluation testing two distinct supply models. Model A strengthened the role of district stores in coordinating between local clinics and the central medical store. Model B, on the other hand, minimized the district role, allowing local clinics to submit orders directly to the central medical store. Results showed only moderate improvement under Model A. But Model B giving local facilities control over ordering medical supplies resulted in a dramatic reduction in the number of days that essential drugs were unavailable. At the end of the evaluation period, malaria treatments for adults were out of stock in just 6 percent of facilities operating under Model B, compared with a previous rate of 48 percent. Researchers estimated that if Model B were implemented nationwide, malaria-related deaths could drop by more than 20 percent, and households would save more than $1.6 million annually in income otherwise lost due to disease or death of wage earners. 4 Impact Evaluation and Human Development
Rural Zambia Health Clinic Photo credit: jhpiego.org Model B was so successful that it has been extended to all districts that took part in the evaluation, and the Government of Zambia is seeking funding to allow a national scale-up. Case 3: Testing an innovative approach to youth unemployment in Uganda Unemployment, especially among youth, is a widespread problem in many poor countries. In addition to individual impoverishment, it can often lead to wider social problems, such as gang violence and crime. Youth unemployment is also a particularly intractable policy problem. Programs to raise employment, often called active labor market programs, are challenging to put in place successfully. Finding what works rests on testing different approaches, something that can be costly and thus difficult to gain support for because of public finance constraints. Unconditional and unsupervised cash transfers may be a more effective and cost-efficient form of large-scale aid than commonly believed. Chris Blattman, Innovations for Poverty Action 2 In Uganda, HDN joined with Innovations for Poverty Action and the Government of Uganda to test whether unconditional, unsupervised cash transfers to unemployed youth could meet the dual objectives of raising youth incomes and reducing conflict. On the surface, the plan seemed risky: common sense would suggest that giving a group of people cash worth several times their annual earnings with little supervision is not a way to encourage good use of money. The evaluation yielded surprisingly positive results, showing that unconditional and largely unsupervised Impact Evaluation and Human Development 5
Uganda Youth Opportunities Program Photo credit: Innovations for Poverty Action cash transfers can work. Fears that money would be mismanaged were unfounded. Overall, young adults who received the unsupervised grants stuck to their stated plans, using the majority of funds on training and business supplies. The economic impacts of the program were significant. Hours working outside the home went up by 25 percent for men and 50 percent for women, with real income gains averaging 35 percent per year. The program had a measurably important social impact, too, with interpersonal male aggression declining by 50 percent. Disseminating Results, and Building Evaluation Capacity Worldwide The HDN impact evaluation effort focuses not just on generating evidence for policy action, but also on making information widely available about what works and what doesn t in development, and on building evaluation capacity worldwide. A Global Knowledge Hub As a global knowledge hub, HDN generates evidence around key knowledge gaps in human development. It directs resources towards evaluations that help build a body of knowledge where evidence is thin, and on deriving systematic results that are found across contexts. For example, HDN has built up a library of conditional cash transfer (CCT) program evaluations that, taken together, advance global knowledge on how to foster investment in human capital and demonstrate the applicability of CCTs across a wide range of settings. In order to derive maximum policy impact from the evaluations it supports, HDN makes sure that policymakers are engaged in all stages of the evaluation process, and that evidence generated is widely disseminated across governments, international donors, and the academic community. 6 Impact Evaluation and Human Development
Building Evaluation Capacity Worldwide Since 2007, HDN has trained more than 2,500 people in impact evaluation techniques in Africa, Asia, Eastern Europe, the Middle East and Latin America. Many have gone on to incorporate impact evaluation results in their work, and three-quarters have managed and/or designed an impact evaluation since attending the training. Six Ways to Work Smarter in Impact Evaluation HDN s experience leading one of the world s largest impact evaluation programs yields the following six broad lessons for working smarter as impact evaluation moves forward. Make the choice of which programs to evaluate more strategically. One of the main factors contributing to HDN s success at deriving policy impact from evaluations is its strategic selection of which programs to evaluate. Greater coordination across the international development community in identifying and carrying out evaluation priorities would yield far more benefits for the world s poor from each dollar spent on evaluation. Combine survey data collection with complementary research methods. Qualitative research, for example, can complement impact evaluation to get at crucial and otherwise unanswerable questions about optimal program design. For example, information about why minority groups are failing to access an otherwise successful program may be gathered from a focus group, or other qualitative research method. Impact Evaluation across the World Bank The Development Impact Evaluation Initiative is a World Bank-wide program to generate knowledge on the effectiveness of development policies. Working across 18 thematic areas, DIME collaborates with 300 agencies in 72 countries to improve the effectiveness of policies and programs and strengthen country capacity for real-time evidence-based policy-making. Don t assume a costly, one-off data collection effort is always necessary. Look for opportunities to reduce costs and time by using existing administrative or other regularly collected data to conduct an evaluation, as opposed to mounting a new survey instrument. Administrative data can also be used to test program design alternatives within a particular program. Use impact evaluation to maximize program efficiency, not just impact. Increasingly, impact evaluation is being used to test the cost-effectiveness of different program modalities, not just impact. This is a worthwhile endeavor, as it allows policymakers not only to ensure outcomes, but also to achieve those outcomes at the lowest cost. Standardize quality control to reduce failed or poorly performing evaluations. Adopting quality standards and peer review protocols helps ensure that evaluation resources are well-spent and yield solid, reliable results. Impact Evaluation and Human Development 7
Further strengthen the feedback loop between impact evaluation results and programs on the ground. In addition to the five points above, factors most likely to maximize the link between evaluation studies and policy change include close engagement with policymakers, timeliness and dissemination of results, and building a culture of using evidence in decision-making. The Human Development Network, part of the World Bank Group, supports and disseminates research evaluating the impact of development projects to help alleviate poverty. The goal is to collect and build empirical evidence that can help governments and development organizations design and implement the most appropriate and effective policies for better educational, health and job opportunities for people in developing countries. For more information about who we are and what we do, go to: http://www.worldbank.org/sief Endnotes 1 (Reuters, February 29, 2012. http://www.reuters.com/article/2012/02/29/idus205799+29-feb-2012+bw20120229) 2 Blattman, Christopher et al. 2011. Employment Generation in Rural Africa: Mid-term Results from an Experimental Evaluation of the Youth Opportunities Program in Northern Uganda For more information please visit: www.worldbank.org/sief January 2013