Module 5: Project Evaluation in

Transcription

1 Module 5: Project Evaluation in IUCN 5-1 Module 5: Project Evaluation in IUCN

2 5-2 Module 5: Project Evaluation in IUCN Table of contents 1. THE ROLE OF PROJECT EVALUATION IN IUCN THE PURPOSE OF EVALUATION IN IUCN Learning and Improvement Accountability WHAT IS PROJECT EVALUATION? TYPES OF PROJECT EVALUATIONS ROLES AND RESPONSIBILITIES OF PROJECT MANAGERS MANAGING PROJECT EVALUATIONS ESTABLISHING THE NEED FOR AN EVALUATION ASSESSING THE ABILITY AND READINESS TO EVALUATE DEFINING HOW BIG OR SMALL - SETTING THE FOCUS AND SCOPE FOR THE EVALUATION PROVIDING RESOURCES FOR THE EVALUATION DEVELOPING TERMS OF REFERENCE FORMULATING EVALUATION QUESTIONS USING AN EVALUATION MATRIX ENGAGING THE EVALUATOR OR EVALUATION TEAM APPROVING THE WORKPLAN IMPLEMENTING AND MONITORING THE EVALUATION COMMUNICATING THE RESULTS TO DIFFERENT AUDIENCES DEVELOPING A FOLLOW-UP PLAN USING EVALUATION RESULTS FOR IMPROVEMENTS EXERCISE DEVELOPING TORS AND WORKPLAN FOR AN EVALUATION WRAP-UP: DAY 5... ERROR! BOOKMARK NOT DEFINED.

3 Module 5: Project Evaluation in IUCN 5-3 Notes:

4 Module 5: Project Evaluation in IUCN 5-4

5 Module 5: Project Evaluation in IUCN 5-5 Learning Objectives: At the end of Module 5, participants will: Be familiar with the concept of project evaluation as practiced in IUCN Understand the responsibilities of project managers for evaluation Understand the key elements of an evaluation process and the standards to which they should be carried out in IUCN Understand the different ways of effectively communicating evaluation results Approximate Duration: 6 hour Overview: In this module: Presentation: Evaluation Exercise 5.1: Developing ToRs and workplan for an evaluation Notes:

6 Module 5: Project Evaluation in IUCN 5-6 Basic Concepts 1. The Role of Project Evaluation in IUCN IUCN undertakes evaluation at a number of different levels project, programme and institutional levels. This course focuses on project level evaluation, touching briefly on the relevance of project evaluation for programme and institutional evaluations. Other IUCN materials and courses are available on programme and organisational evaluation. Since IUCN delivers approximately 80 per cent of its work through projects, it is critical that it has in place a system of feedback on the relevance, effectiveness, efficiency and impact of its project work. For this to happen, project managers will need to both respond to the contractual needs of donors for evaluations, as well as be able to initiate internal evaluations and reviews in order to learn and improve from experience The Purpose of Evaluation in IUCN Project evaluations serve the same purposes as all other types of evaluations in IUCN Learning and Improvement Evaluation is part of the learning environment for IUCN, its partners and members. It involves the creation of an environment that engages staff and their partners in creative ways to learn how to improve work. In this context, evaluations are instruments for making IUCN s policies, programmes, projects, and organisational units more effective through the provision of useful feedback and a commitment to act on that feedback. By doing so, evaluations are a way to understand why IUCN activities succeed or not. As learning tools, evaluations add to our body of knowledge with respect to best practices in evaluation and conservation Accountability Second, evaluations are part of IUCN s overall accountability system. IUCN is answerable to its members, partners and donors for effective and efficient delivery of policies and programmes, and for appropriate use of funds. The evaluation process, including the fulfillment of contractual and internal evaluation requirements enforces performance What is Project Evaluation? Project evaluation is - a systematic, objective assessment of the design, implementation process and results of an on-going or completed project. The central issue is to determine whether the project is contributing to the change that it is designed to make, and to examine those aspects of the project that are contributing to or hampering its success, whether they lie in the design or implementation process. A project evaluation should provide credible, useful information that makes it possible to identify and incorporate lessons drawn from previous experience into the decision-making process. An evaluation addresses the following issues:

7 Module 5: Project Evaluation in IUCN 5-7 Relevance Was the project well conceived given the situation? Is it still relevant to the original problem it was intended to address? To what extent is it contributing to the strategic direction of IUCN s Programme framework (KRAs, KEG) 1, members, and partners? Is it appropriate in the context of its environment? Efficiency - Is the project delivered in a timely and cost-effective manner? Have resources been used cost effectively? Do the results -- quantity and quality -- justify the resources expended? Effectiveness To what extents are planned results achieved. What are the reasons for the state of achievement? What supports and barriers have affected achievement? Impact - To what extent has the project contributed to longer-term outcomes and goals of IUCN? Are there unanticipated positive or negative consequences? Why did they arise? Sustainability Is there an enabling environment that supports ongoing positive impacts? Can the outcomes be sustained beyond the life of the project? Will impacts continue to be realised? External Utility - To what extent is the project replicable? Might its approaches, methods, and/or content have potential value in another situation or project? Project evaluation is an important step in the project cycle. During the process of designing projects, problems are identified and solutions proposed. It is rare that all the information required to make decisions is available at the onset. Many assumptions are made during the planning process. Evaluation therefore serves as a necessary feedback mechanism, providing information to project managers so that they can adjust the design or implementation process Types of Project Evaluations The IUCN Evaluation Policy outlines a broad range of types of evaluation that IUCN undertakes, including meta evaluation, synthesis evaluation, policy evaluation, organisational evaluation, programme thematic evaluation, project evaluation, strategic reviews and self assessments. While it is useful for project managers to know the full range of evaluations in IUCN, it is unlikely that they will be expected to undertake more than two types; usually mid term and post project evaluations as well as self-assessment at key moments during the life of the project. Mid term evaluation This can be an internal review initiated by the project manager, the programme manager, or required externally by the donor. Evaluations that take place during the life of the project are referred to as formative or process evaluations. Their purpose is to provide feedback on the implementation process. They examine the appropriateness of the design given the prevailing field conditions, and the results achieved. They also help to identify obstacles to performance. They are the most common type of evaluation for IUCN projects. End of project evaluation This is often called a summative evaluation. It is done at the end of the project or several years after project completion to assess 1 The Key Result Areas of the IUCN Programme, and the strategy of Knowledge, Empowerment and Governance.

8 Module 5: Project Evaluation in IUCN 5-8 outcomes, impacts, sustainability, external utility, and the lessons learned. This type of evaluation contributes to the collective knowledge of using projects to solve conservation and development problems by providing pointers about the processes that work/fail under given circumstances. Self assessment IUCN includes self assessments as a type of evaluation. Self assessments are internally guided and controlled processes, using an agreed framework of performance, agreed questions and a facilitated process to gather the information and promote reflection, discussion, debate and improvements in both the delivery of our work as well as the way in which we are organised and managed to deliver our work. Evaluations can also be classified as internal and external, depending on who is commissioning and/or taking a lead in undertaking them. Project managers often initiate internal evaluations while an external authority, usually the donor, initiates external evaluations. Whether internal or external, evaluations should be carried out by a team comprising a mix of persons who are both external and internal to the project. The external people are independent and therefore assumed to be objective, while the internal people know the project: both stances are important in an evaluation Roles and Responsibilities of Project Managers Even though your role may vary in different types of evaluations and reviews depending if they are internal or external evaluations, managers should take responsibility for the following and be prepared to: Know and implement donor contractual requirements for evaluation of your project; Identify needs for internal evaluations and self-assessment processes that foster learning, reflection, and improvement; Ensure that resources for evaluation are built in at the target level of 3-5% of the total budget for your project; Seek out the specialised training needed to keep staff skills up-to-date on the use of evaluation techniques and tools; Link project evaluations to the programme results (outputs and outcomes) in both the component programme and the IUCN Intersessional Programme; Build time into workplans for anticipated evaluations; Liase with the M&E global and regional staff to obtain the necessary support to design, manage and deliver high quality evaluations; and Create a culture of learning and reflection by actively using the results of evaluations for project planning improvements. Regional and global M&E staff are prepared to assist you in designing a good evaluation process and tools, however the responsibility for ensuring that evaluation takes place, and is used to improve performance remains with managers.

9 Module 5: Project Evaluation in IUCN Managing Project Evaluations Most evaluations in IUCN will follow a standard process that begins with a planning stage and ends with the actions taken to follow up on the evaluation s recommendations. While your responsibilities at each stage of an evaluation may vary depending on the type of evaluation, you are expected to be involved throughout the entire process. Sometimes you will be responsible for carrying out the particular stage e.g., writing terms of reference--while other times you might simply review the work of others. A detailed guidance on managing evaluations is in the IUCN Handbook on Managing Evaluations in IUCN, available from the IUCN M&E Initiative or your regional M&E staff Establishing the Need for an Evaluation An important question to first ask - Is an evaluation the right tool or is an audit or performance appraisal more appropriate for the issues at hand? If there are issues related to the inappropriate use of funds, or problems related to the behaviour of staff or management of the project, it is advised to first discuss these with senior finance and human resources staff before deciding whether an evaluation, an audit or a staff appraisal is the appropriate process. There may be several reasons for doing a project evaluation. You will need to determine the needs and opportunities for evaluation in your particular project: Donor requirement a mid term or end of project evaluation may be part of your donor contractual obligations. You should check your donor contract and consult with your programme manager to clarify the expectations of your donors, and to ensure that the requirements are still timely and appropriate. Accountability - an evaluation may be necessary for you as a manager to ensure that IUCN remains accountable to its donors, members and partners. It may help a project to recover credibility with key stakeholders if there have been long standing problems that have not been adequately addressed. Innovation - an evaluation of the project may or may not help to determine whether the approach can be applied with confidence elsewhere? Learning and change similarly an evaluation may present a major opportunity for learning and change. To enhance this learning you need to allow time to capture the reasons for success and/or failure. Responding to changed circumstances If there has been a significant change in the external or internal operational environment of your project, an evaluation presents a good opportunity to obtain the necessary feedback to make appropriate adjustments Assessing the Ability and Readiness to Evaluate Even if an evaluation is required, you will need to determine if it is feasible at this time, and if the project is able and ready to undergo a useful evaluation. Does it have resources (people, time, money)? Can relevant information be acquired? What is the probability that evaluation will bring about changes? 2.3. Defining How Big or Small - Setting the Focus and Scope for the Evaluation At this stage, we need to clarify whether the evaluation is external or internal, what level of effort and resources will be allocated, and which stakeholder groups should be involved and

10 Module 5: Project Evaluation in IUCN 5-10 how. Full participation is as important as equitable participation across different groups. If this is not feasible, you should consider engaging representatives of stakeholders at the following critical moments: Deciding whether or not to evaluate. Defining the type of evaluation, its scope, and criteria. Defining the evaluation questions, what are the key issues to explore in the evaluation? Defining evaluation workplan. Evaluation activities must be scheduled and fit into the stakeholders' agendas. Deciding which recommendations to adopt and which to reject. Disseminating and gathering feedback on the results Providing Resources for the Evaluation Evaluations require substantial investments of financial and human resources. IUCN s Evaluation Policy recommends that projects include a budget line for evaluation of 3-5% of the total project budget. These funds are often used for external evaluations. Managers often face the challenge of pooling funds from different sources to meet internal evaluation requirements. Project budget lines earmarked for training, communication and learning can often be used to finance monitoring and evaluation and to collate lessons learned. When these funds are insufficient, the Suggested Outline for A TOR M&E unit should be consulted for advice Developing Terms of Reference Terms of reference (TOR) are the key guide for an evaluation. They clarify the reasons for the evaluation, highlight issues that have become apparent, indicate the general depth and scope required, and spell out any imperatives for the evaluators. They also provide details about methodology, scheduling, cost and the mix and qualifications of the members on the evaluating teams. The project manager is responsible for ensuring that clear and focused TORs guide evaluations. M&E staff will help where necessary. All TORs should contain the following sections. Context for the Evaluation A clear description of the project goals, objectives, expected results; how and when did it began; 1. Context for the evaluation 2. Rationale or purpose for the evaluation 3. Evaluation issues and questions 4. Evaluation stakeholders 5. Methodology 6. Qualifications of evaluators 7. Schedule 8. Outputs and Deliverables 9. Cost 10. Action Plan 11.Appendices - Evaluation Matrix, Evaluation Policy, LFA who it serves; the social, economic, cultural, political and environmental contexts that influence its implementation and/or achievement of results. Rationale or Purpose for the Evaluation the reason for the evaluation; is it a formative or summative evaluation, internal or external. Evaluation Questions the major issues and questions to be explored by the evaluation.

11 Module 5: Project Evaluation in IUCN 5-11 Evaluation Stakeholders - the major stakeholders in the evaluation, their main interests and concerns about the project. Methodology the data collection and analysis methods to be used, the stakeholder groups to be consulted and how, the geographic conditions or other aspects to be considered while designing the evaluation. Qualifications of the Evaluators the specific skills or characteristics required of the evaluator or evaluation team. Schedule - the start and finish dates for the evaluation. Outputs and deliverables the deliverables required of the evaluator or evaluation team and due date. (Workplan, briefings, interim report, draft findings, presentations and the final report). Cost - budget the resources that are available for the evaluation, the project budget estimate for consultant fees, travel, and other categories. This is not a detailed budget, but general allocations for broad budget categories. Appendices support material such as project LFA, IUCN Evaluation Policy Formulating Evaluation Questions Using an Evaluation Matrix Evaluations are designed to answer a series of questions about the project. Good questions are clear, focused, and relevant and make sense to all stakeholders. Five broad issues guide evaluation questions in IUCN: relevance, effectiveness, efficiency, impact and sustainability. Questions can be raised on each of the issues. Several factors guide the choice of questions: the objectives and results of the project, the stage of the project in the project cycle, and who is likely to use the findings. Other considerations include relative importance of each question, and relative ease, resource requirements and data availability for answering the possible questions. The questions chosen should demonstrate a clear link to the issues of concern and therefore the purpose of the evaluation. The following resources should be consulted during the process: Project design document Does it specify objectives, outputs, outcomes and indicators for the intervention? Funding agreements - Determine whether specific evaluation questions were part of the funding agreements Stepping into the new millennium: IUCN s Intersessional Programme - How do the questions link to the key result areas (KRAs) or the KEGO? Other evaluations or studies that have been done - What questions did they ask and what did they learn? Key project stakeholders - What do they think are important questions? Experts in the field - What insights can they bring to bear on formulating the evaluation questions? It is advisable to remain open to a wide variety of possible questions and to resist locking yourself into a question before having consulted adequately. It is also important to be realistic

12 Module 5: Project Evaluation in IUCN 5-12 about the number of questions that can be answered in a single evaluation. Obviously, the evaluation should focus on priority questions. An Evaluation matrix is a way of organizing your evaluation questions around the major issues of concern to the evaluation. It also provides a structure to ensure that multiple data sources are used to answer each question, and a balance of stakeholder views is sought during the process. Exhibit 5.1: Evaluation Matrix Suggested Format Issues Key questions Sub-questions Indicators Relevance Effectiveness Efficiency Impact Sustainability Other evaluation criteria Examples of data sources 2.7. Engaging the Evaluator or Evaluation Team The ideal evaluation team should have a mix of evaluation skills, technical knowledge of the issues the project is addressing, and an understanding of the socio-cultural-political context of the project area. Choosing an evaluator is obviously critical in an evaluation process. Too often IUCN evaluation teams lack evaluation expertise and are dominated by technical experts who may provide interesting technical advice, but fail to adequately address the priority evaluation questions. M&E staff can assist with this step by recommending good evaluators to consider for the team Approving the Workplan A workplan is the document that governs the implementation of the evaluation. It is an important communication and operational tool, and should not be too rigid. It updates the preliminary evaluation matrix provided with the TOR, and specifies the specific evaluation issues, questions, methods of data collection and analysis that will be undertaken at a specified time and within a specified budget. It establishes commitments in terms of roles, deliverables, schedules, and budget. The workplan contains the same sections as the TOR. It is prepared as proposed response to the TOR by the evaluator or the evaluation team Implementing and Monitoring the Evaluation The project manager plays the important role of supervising and monitoring the progress of the evaluation team, and uses the workplan as a tool for guiding this. The manager and the evaluation team should discuss all major deviations from the workplan. Milestones in the process include facilitating the start-up, engaging in regular reporting and feedback, reviewing draft findings and assessing the results of the evaluation.

13 Module 5: Project Evaluation in IUCN Communicating the Results to Different Audiences It is critical to communicate the findings of the evaluation. Phone calls and s are informal means of communicating. They should be complemented by formal briefings and presentations. Different audiences will have different needs and the reporting approach should reflect those needs Developing a Follow-up Plan Evaluations are useful when their findings are used. Unfortunately, follow-up is often a weak part of the evaluation processes. An action plan, outlining how the project will incorporate the recommendations of the evaluation should be included in the final report. This action plan should be prepared together with project stakeholders. A format for the action plan is suggested below. A mechanism to monitor implementation of the action plan should be devised. Exhibit 5.2: Action Plan Conclusion or Recommendation Accepted not accepted and why What needs to be done? Who will do it? When will it be finished? Using Evaluation Results for Improvements Evaluation findings and results can provide a major input into the iterative learning cycle of projects and institutions. For this to happen, the results and findings need to feed into management and implementation, and into major strategic planning sessions. Project evaluation results need to be included in the agendas of staff meetings, discussed, debated and programme managers need to look at the issues and trends arising from a range of project evaluations over time. Looking across a body of project evaluations that have taken place over a ten-year period may help managers to see if there are common factors across IUCN that support or hinder project achievements. Relating the results of project evaluations to programme and organisation level is also important. What are we learning from the hundreds of IUCN projects that we implement across the Secretariat, and across the Union? What can we learn from evaluations of similar projects implemented by others (agencies and organisations)? Project managers should look at the results of project evaluations of other projects in their thematic or geographic area to see if there are common trends and issues, and what others are learning from evaluation.

14 Module 5: Project Evaluation in IUCN 5-14 Time: 3 hours 3. Exercise Developing ToRs and workplan for an evaluation Instructions: Working in a small group Instructions: Five years later - Assume the project we have developed was funded and has been implemented over the last five years. The project is about to undergo a formative evaluation and you are managing the evaluation process. Working in a small group design TORs for the evaluation including an evaluation matrix, as described in this module. 1. Identify evaluation questions for the five evaluation criteria (40 minutes) 2. Outline the methodology (data required, data sources & data collection methods) to be used in answering the evaluation questions (40 minutes) 3. Determine the qualifications, skills and experiences mix of an evaluation team (15 minutes) 4. Draft the evaluation workplan and a budget (30 minutes) 5. Provide the evaluation report outline (15 minutes) 6. We will discuss this in plenary (40 minutes)

15 Module 5: Project Evaluation in IUCN 5-15 Notes:

16 Module 5: Project Evaluation in IUCN 5-16 Test your understanding 1. What are the purposes of project evaluation in IUCN? 2. List, with examples, the common types of evaluation a project manager is likely to be involved with in IUCN 3. What is evaluation criteria? What evaluation criteria are adopted in IUCN? 4. What are the roles and responsibility of the manager in a project evaluation? 5. What are the steps involved in managing a project evaluation? 6. Give an example of an outline for TORs of an evaluation. 7. Why is it important to agree on an evaluation workplan? 8. What are evaluation questions? Why are they important guides to evaluation? 9. In your experience, how well are results of evaluations used to improve management in IUCN? i