Module 4 Deciding to Evaluate: Front-End Analysis

Transcription

1 Module 4 Deciding to Evaluate: Front-End Analysis

2 Module Objectives By the end of this module you will be able to: Identify whether an evaluation may be appropriate for a particular program or project. Identify 6 possible evaluation triggers during project implementation. Identify the key users and other important evaluation stakeholders. Assess the readiness or evaluability of a program or project for evaluation. Facilitate the development of a theory of change for the project if one is not in place. Recognize 3 potential benefits and 4 potential costs of a proposed evaluation. 2

3 ADS References for this Module ADS Reference ADS ADS and ADS and ADS ; and ADS ADS ; ADS and Topic or Issue Evaluation decisions are part of design Allocate funds for evaluations during the design phase Stakeholders should be involved in evaluation planning Program theory and development hypotheses are pre-requisites for good evaluation Evaluations are appropriate when performance information indicates unexpected positive or negative results affect either men or women (refer to Gender Equality and Female Empowerment Policy referenced and hyperlinked in ADS ) 3

4 Evaluation Decisions During Design When doing your CDCS: Operating units are encouraged to identify at least one opportunity for an impact evaluation for each DO during CDCS planning. Evaluation for learning: Acknowledge what we don t know OUs can also consider evaluation questions, not only those requiring an IE IE is not for the full DO, but something within it ex/ DO: Increased agricultural productivity IE: What is the best way to encourage farmers to adopt enhanced fertilizer technology? Actual feasibility will be determined during design stage of the project. Allocate 3% of program funds to evaluations! 4

5 Evaluation Decisions During Design During Project Design: Plan for baseline Determine feasibility of IE if planned In a Performance Management Plan (PMP): Include possible evaluation efforts to complement performance monitoring Estimate the costs of collecting, analyzing, and reporting performance data, and plan how these will be financed. When developing a project cost estimate and financial plan, USAID teams must include the costs of evaluation. 5

6 USAID Evaluation Triggers During Implementation 1 A key management decision is required, but there is inadequate information to make it. 2 Performance information indicates an unexpected result (positive or negative) that should be explained, such as unanticipated results affecting either men or women Customer, partner, or other informed feedback suggests that there are implementation problems, unmet needs, or unintended consequences or impacts. Issues of sustainability, cost-effectiveness, or relevance arise. The validity of Results Framework hypotheses or critical assumptions is questioned; for example, due to unanticipated changes in the host country environment. Periodic Portfolio Reviews have identified key questions that need to be answered or that require consensus. ADS Revision

7 Evaluation Needs of Key Potential Users Who uses evaluations? Users are specific individuals or units that will make decisions based on the evaluation. Who are the key potential users? - In USAID - Other Stakeholders What do they want or need to know about the program or project? Can they envision how they would use an evaluation? Is there real demand for an evaluation? 7

8 Evaluation Stakeholder Analysis A technique to identify and assess the importance of an evaluation to key people, groups, or institutions. C-3: Evaluation Stakeholder Analysis Helps determine what roles USAID and key stakeholders should play in an evaluation including in defining evaluation questions. Assists in anticipating the influence stakeholders may have on mobilizing support or resistance to the evaluation use of evaluation findings and implementation of recommendations Evaluation of Mindanao Multi-Donor Trust Fund supported by World Bank, USAID and other donors 8

9 Matrix of Potential Stakeholder Evaluation Roles Evaluation stakeholders Make policy Make operational decisions Make inputs to evaluation (describe) React to findings For interest only Proponent of the evaluation (describe why) Resistant to the evaluation (describe why) USAID mission COR for program USAID/Wash technical bureau USAID/Wash regional bureau Implementing partners Host county line ministry Program clients and beneficiaries Congress / Other Funders Other Partners 9

10 Assess Program or Project Readiness or Evaluability Classic evaluability assessment questions: Does the program or project have clear theory of change and measurable objectives? Are the program or project s objectives (intended results) known and shared by all key stakeholders? Is the program design reasonable and are there reasonable resources to achieve program/project objectives? Has the program design considered and adjusted to the contextual factors? How likely is it that program/project managers or strategic planners will use the evaluation results? 10

11 Theory of Change - What is it? A cause and effect model of the building blocks needed to achieve the long-term goals of a development program. Results Framework DO IR1 IR2 Sub-IR Sub-IR Sub-IR Assumptions - Usually a graphic representation of how USAID and development partners are expected to achieve results and underlying assumptions. Purpose Output 1 Output 2 Output 3 Goal Assumptions 11

12 Theory of Change - USAID Tools USAID s approach links its theory of change at the program level to projects that support them. Development Objective (Highest Result for which Mission Assumes Responsibility ) Program Level Results Framework Intermediate Result 1 Intermediate Result 2 Intermediate Result 3 Intermediate Result 4 Sub-IR 1.1 Sub-IR 1.2 Narrative Summary In d ic a to rs D a ta S ources Assumptions Goal Then Goal Purpose Sub-Purpose If Purpose Then Purpose If Sub-Purposes Then Sub Purposes And Assumptions And Assumptions Outputs I f Outputs Then Outputs And Assumptions Inputs If Inputs And Assumptions Project Logical Framework 12

13 Development Hypotheses Take out a piece of paper and clarify the DEVELOPMENT HYPOTHESIS for the project example below: In Burundi, malaria accounts for almost half of child deaths nationwide; malnutrition is the second leading cause of death. Education of all caregivers to promote optimal infant and child feeding; community-based rehabilitation of malnourished children via the Hearth project. 13

14 Consider Contextual Factors and Resources How will the stage of program or policy implementation affect the evaluation? Is there baseline info? How much time is available to complete the evaluation? What is the nature and extent of resources for the evaluation? What is the political climate for the evaluation? Are there gender issues (evaluation needs or implications) that should be taken into consideration? 14

15 Balance Evaluation Benefits and Costs Potential Benefits Potential Costs Evidence-based decision making leading to sound decisions: Cost of evaluation vs. cost of the program? What to scale-up or Future replicate programming What to improve Burden of the evaluation on What to cut - $50 back mil or lion termin on fate uture intervention(s) beneficiaries, implementers, - Risk host country government Contributions to streams of knowledge about what works, what doesn t and why Political costs to USAID and implementing partners if results are highly controversial Building local capacity Reputational costs if insufficient time and resources are provided to conduct a comprehensive evaluation 15

16 Should we evaluate? A Congressional priority project works with the host country government, the World One year Bank, into and an other agricultural stakeholders development to implement. program, Each monitoring stakeholder and believes anecdotal the objectives data reveal of that the there project are to disparities be a little different. in the distribution The project of outcomes is coming between to a close men and we and want women. to know whether or not our project was successful. An Country education X is phasing program out is finishing all interventions implementation related to and civil we society want to participation know whether due to changes or not our in funding program streams caused and a change country in prioriti knowledge s. One for such students. CS project is coming to a close and we want to know the extent to which it was successful. - Reliable data (school test scores) exist. End of project monitoring data reveals that, despite successful completion of all planned activities, citizens in the northern province did not change their behavior as envisioned in the design phase. A nation-wide project allows each county to select one or more interventions from a pool of options to improve governance via education, capacity building, and/or economic approaches. The project is planned to begin implementation in 6 months and we want to know the potential extent to which our project can cause changes in governance. 16

17 Evaluation Purpose Evaluation Purpose: To conduct an evaluation of the Rural Enterprise Project The Evaluation Purpose should explain why the evaluation is being undertaken: - what prompted it? - what management purposes will it serve? 17

18 Evaluation Purpose Be specific about why the evaluation is needed. Is it being undertaken to help make a specific decision whether to fund a follow-on project? Was it triggered by something that occurred during program or project implementation weak performance? Is it being undertaken to rigorously test the effects of an innovative approach or model? Is it being undertaken to justify your program to Congress or other donors? 18

19 Purpose Examples Accountability We need to show stakeholders how resources were spent. We have some data, but aren t sure if it s accurate, complete, and/or reliable. Performance Management Was our project targeting the right people? What, if anything, needs to change in our implementation during the next phase of our project? Were there unintended consequences? Obstacles to implementation? 19

20 Purpose Examples Continued Understanding Development Hypothesis Contribute to subject matter body of knowledge: is this a valid development hypothesis? Would this theory of change work in other contexts? Does one intervention work better than another at causing certain changes? Empowerment We want to build evaluation capacity for local partners and stakeholders. We want stakeholders to take ownership of successes and gaps to adjust behavior accordingly after the project ends. Publicity Everyone should know what a great job we re doing! 20

21 Purpose Intended Users The primary intended users of the evaluation USAID decision-makers? The project implementation team? Important secondary audiences for the evaluation The government of the assisted country? USAID staff in other countries who might want to design a similar project? 21

22 Purpose Intended Uses Decisions about an on-going project Decisions about a follow-on project Planning future USAID interventions based on a rigorous test of the development hypothesis (consider cost/benefit) Fostering the scaling-up of what works by others Accounting for funds expended by interested stakeholders 22

23 Quick Exercise You have been asked to review and approve this Evaluation Purpose: III. PURPOSE OF THE ASSIGNMENT USAID/E requires a team of three consultants to conduct an end-of-project evaluation of the MSG project. With the MSG project having existed for three years now and with IntraHealth ending this 2009, this evaluation team will collect information about MSG implementation, progress, and challenges, specifically looking at whether the MSG project is achieving its objectives. The evaluation team will assess the viability and collaborative advantage of facility-based MSG in rapidly scaling up access to quality HIV/AIDS services. In addition, the evaluation is expected to help stakeholders understand successful approaches that should be emphasized for future programming. The evaluators will share programmatic, management, and/or financial obstacles and challenges affecting program implementation and recommend any changes in program or management strategies that would increase the efficiency and impact of the program. The recommendations will inform follow-on programs. The evaluation will cover the MSG program performance period of 2005 to Overall, the evaluation report will help the Ethiopian Government, USAID, and other partners to address topics of management, quality of services, and sustainability around MSG, and ultimately to decide if the MSG strategy should be scaled up nationally. How will you respond? 23

24 Case Study 1: USAID Burma SHIELD: Health, Education, Humanitarian Assistance

25 Case Study 2: USAID/Georgia: AgADVANTAGE- Agricultural value chains

26 Case Study 3: USAID s A Ganar Alliance program

27 Exercise Based on your project documents: 1. Develop and clarify your project s theory of change. 2. State the purpose of the evaluation you are planning. 3. Complete a first cut of your evaluation stakeholder analysis. 4. Determine likely impacts of contextual factors on the evaluation. 5. Consider benefits and costs on balance is the project ready for and will it benefit from the evaluation that is being planned? 6. Put your work on a flip chart and be prepared to present your project and ideas about readiness to evaluate. 27

28 Review Questions What factors should trigger you to consider evaluating a given project? What makes a program or project ready for evaluation? What potential costs and benefits should be considered when making an evaluation decision? 28