United Nations Development Programme GUIDELINES FOR EVALUATORS CONTENTS

Transcription

1 United Nations Development Programme GUIDELINES FOR EVALUATORS CONTENTS

2 PREFACE 1. BASIC DEFINITIONS 2. POLICY FRAMEWORK FOR EVALUATION 3. PURPOSES OF EVALUATION 4. TYPES OF PROJECT EVALUATIONS 5. BASIC PRINCIPLES OF EVALUATION 6. SOURCES OF DATA 7. CONDUCT OF A MID-TERM, TERMINAL OR AN EX-POST EVALUATION 8. CONDUCT OF AN EVALUATION OF AN INSTITUTION-BUILDING PROJECT 9. PARTICULARS OF THE EVALUATION OF A NATIONALLY EXECUTED PROJECT 10. ASSESSMENT OF IMPACT AND SUSTAINABILITY 11. FORMULATION OF RECOMMENDATIONS AND LESSONS 12. ORGANISATION OF THE EVALUATION EXERCISE 13. PRESENTATION OF THE REPORT 14. PROJECT EVALUATION INFORMATION SHEET.

3 4 PREFACE The Guidelines for Evaluators is a package of information for the operational guidance of consultants called upon to perform UNDP project evaluations. They are based on the current rules on evaluation set forth in the Manual for Monitoring and Evaluation issued in April 1987 and in the relevant chapters of the PPM. These guidelines formalize the evaluation process by breaking it down into a sequential set of discrete steps. They also highlight some lessons drawn from previous evaluation exercises and identify some commons mistakes. In addition, this material also provides guidance in the completion of the project evaluation information sheet, which is the basic source of data for the Central Evaluation Office database. The guidelines, which are to be provided to the team of consultants together with the terms of reference (TOR) and the project evaluation information sheet at the outset of the evaluation mission, are to be used as a handy reference throughout the evaluation exercise.

4 5 1. BASIC DEFINITIONS Evaluation is the process that attempts to determine, as systematically and objectively as possible, the relevance, efficiency, effectiveness, impact and sustainability of activities in light of their objectives. Evaluation thus is concerned with achievements rather than with input delivery. Evaluation is also important in the identification and solution of problems. Relevance: the degree to which the purpose of the project remains valid and pertinent. Efficiency: the productivity of the implementation process - how good and how cost effective the process of transforming inputs into outputs was. Effectiveness: a measure of the extent to which a project achieves its objectives through an effective use of its results. Capacity building: a complex, long term phenomenon requiring the development of human resources, the establishment of well-functioning organizations within a suitable work environment and a supportive socio-political environment. Impact: an expression of the change actually produced in human development, environment, institutions, etc. as a result of the project. Sustainability: an assessment of the likelihood that the project results will endure after the active involvement of UNDP has ended. 2. POLICY FRAMEWORK FOR EVALUATION The evaluation is needed to enable the Administrator of UNDP to fulfil his duties of accountability to the Governing Council with regard to all phases and aspects of the implementation of the UNDP programme and to provide the information necessary for the decision-making process to improve current implementation and future planning and delivery of UNDP assistance. The UNDP evaluation is a tripartite exercise involving the recipient Government, any United Nations executing agency involved in the project implementation and UNDP. In the case of government-executed projects, evaluation is a bipartite undertaking involving the government and UNDP. UNDP evaluations policies as well as procedures are designed for the UNDP decentralized system under which operational responsibility for the programme and its evaluation rests with Regional Bureaux and Field Offices in individual countries or groups of countries for regional or interregional programmes/projects. All UNDP-supported projects and programmes are subject to evaluation. 3. PURPOSES OF EVALUATION In addition to supporting the Administrator in the fulfilment of his duties of accountability, evaluation is a management tool (used in decision-making), a learning process (used for improving future projects) and in some instances an instrument of technical cooperation in its own right (used for transferring evaluation know-how to the nationals who participate in the evaluation). 4. TYPES OF PROJECT EVALUATIONS UNDP conducts several types of evaluations that are distinguishable according to the agent

5 6 (who conducts the evaluation); the timing (when the evaluation is conducted); and the scope (the span of the evaluation). Based on the agent of evaluation, two types of evaluations can be distinguished: internal and external. Internal evaluations are evaluations carried out by people directly involved with the formulation, appraisal, approval, implementation or daily management of the project. External evaluations are evaluations carried out by people not directly involved with the formulation, appraisal, approval, implementation or daily management of the project. The present guidelines are primarily for external evaluations but might be used for internal evaluations. Based on the timing of the evaluation, three categories may be distinguished: mid-term, terminal and ex-post. Mid-term evaluations are evaluations carried out during the course of project implementation. Terminal evaluations are evaluations carried out at the end of project implementation. Ex-post evaluations are evaluations carried out two years or more after the end of project implementation. Based on the scope of the evaluation three types may be distinguished: project, sectoral and thematic. Project evaluations, are evaluations of a single project. Sectoral evaluations are evaluations of a group or cluster of projects. They are designed to address specific issues in a common sector or sub-sector based on the review of these projects. Thematic evaluations are evaluations of a group or cluster of on-going or completed projects addressing themes cutting across two or more sectors or sub-sectors. Thematic evaluations may be substantive or process oriented. For sectoral and thematic evaluations, the review of the related projects serves the purpose of developing case studies for addressing programme, process, strategic and policy issues rather than project assessment. In addition to these evaluations, UNDP also carries out joint programme evaluations with interested parties (recipients, donors, United Nations agencies) and promotes participatory project evaluations with the full involvement the beneficiaries. Evaluations in UNDP are based on qualitative and quantitative measurements. They are also "summative" or retrospective for accountability purposes and "formative" or forward looking for decision-making purpose. 5. BASIC PRINCIPLES OF EVALUATION The recipient Government, any United Nations specialized agency involved in project implementation and UNDP select their own independent consultant. The leader of the evaluation mission is the UNDP-designated member. He represents the Administrator of UNDP. He is responsible for the conduct of the evaluation team, for ensuring that the TORs are fully understood by all the team members, and for ensuring that the evaluation report and the project evaluation information sheet are completed in accordance

6 7 with UNDP guidelines. UNDP evaluations take place mainly in the field. The evaluators work closely with the recipient Government (coordinating and technical agencies), project management, representatives of relevant United Nations agencies located in the recipient country, donor representatives (if warranted) and the UNDP field office. Operational responsibility for the conduct of evaluations is shared between UNDP (the field office and the regional bureau), the Government and the specialized agency. The views and expertise of the beneficiaries must be made an integral part of the evaluation. The beneficiaries include both men and women, rural and urban dwellers, etc. The value of evaluation lies in the exto which the information it generates is useful and used. Hence, the evaluation process must be objective, transparent, operational, gender sensitive, forward looking and independent, thereby giving credibility to its findings and recommendations. The evaluation report is the product of an independent team. As such, its content is at the total discretion of the evaluation team, provided it complies with the TOR. Therefore, the evaluation report should not be altered in its substance by the Government, the executing agency or UNDP. Each party, however, has the right to express in writing its views on the report. The evaluation report must be as free as possible of technical jargon in order to ensure accessibility to its wide and diverse audience. Prior to the departure of the evaluators from the country, the main findings and recommendations must be brought to the attention of the parties concerned. 6. SOURCES OF DATA The evaluation team should consult the following sources of information: - National development plans - Sectoral development plans - Manpower development plans - Structural adjustment programmes - NATCAP documents - The UNDP country programme - Any relevant UNDP country reports - World Bank country reports - United Nations agency country sectoral reports - Other documents related to external assistance to the sector or sub-sector - Project document - Project budget sheets - Project performance evaluation reports, (PPER) - Project evaluation reports, if any - National documents - Project files - Other evaluative material that might be available in the regional bureaux, and the Central Evaluation Office including the CEO computerized evaluation database (CEDAB)

7 8 The evaluation team should also use interviews as powerful means to collect data. 7. CONDUCT OF A MID-TERM, A TERMINAL OR AN EX-POST EVALUATION A mid-term evaluation is intended to be a corrective mechanism that diagnoses the problems presently faced by the project and suggests the necessary corrections to ensure the achievement of project objectives. It is not meant to be a substitute for a technical review of the project. A mid-term evaluation usually focuses on: (a) The continuing relevance of the project (approach, objectives, modalities of implementation,etc.) with regard to the prevailing context; (b) The interim results with regard to the indicators of progress or, if not available, with regard to the end-results; (c) The effectiveness of the approach being used to produce these results; (d) The efficiency of project management, including the delivery of inputs in terms of quality, quantity and timeliness; and the monitoring system; (e) The transfer of capacity to the nationals; and (f) The views of the direct beneficiaries on the preliminary outcomes and on the consultative process taking place for the project. Mid-term evaluations identify problems and constraints and propose immediate actions. Midterm evaluations could present, even if too late, a good opportunity to reach a consensus on the criteria for success for the project if these were not clearly stated at the formulation stage. Mid-term evaluations could also be used to clarify the target groups if the beneficiaries were vaguely defined at the formulation stage. Terminal evaluations focus on project results, effectiveness and performance. They should provide an assessment of the likely impact and sustainability of project results. Capacitybuilding efforts should also be assessed. These evaluations should provide a judgement on the impact of the project on sustainable development and on capacity-building in the following areas: poverty alleviation, environment, women in development and transfer of technology. Terminal evaluations are also expected to make recommendations for future action, particularly to consolidate the sustainability, identify the conditions for a successful replication and draw generic lessons. Terminal evaluations should try to identify the groups that are benefitting or losing as a result of the project implementation. Ex-post evaluations are designed to measure the impact of a project and the sustainability of its results. The project focus is no longer relevant since the capacity and the performance of the recipient organization in achieving its mandate become the central concern. Ex-post evaluations require an interactive, gender sensitive dialogue with the end users to assess the real impact of the project. Ex-post evaluations are to be retrospective and forward looking. Expost evaluations looking at a cluster of projects could be a powerful means to consider policy issues. 8. CONDUCT OF AN EVALUATION OF AN INSTITUTION-BUILDING PROJECT

8 9 While such an undertaking does not require the evaluation team to assess the recipient institution or the current government policy in the context of a specific project, it might be useful for the team to have, at the outset of the evaluation, a broader picture of institutional issues in order to narrow their analysis to its appropriate level, the institution-building project, in the course of the evaluation. Institutional issues (a) An institution is an operational arm servicing a policy through a mission. The following need to be considered: - Relevance of the policy; - Relevance of the programme; - Relevance of the institution and alternative choices; - Relevance of its mandate; (b) An institution provides services and/or products. The following aspects of this activity need to be reviewed. - Response to the expectations; - Response to the demand; - Quality and quantity of the outputs; - Cost for the client, the tax payer and the community; - Efficiency; and - Intended and actual beneficiaries. (c) An institution is a live organization in a dynamic socio-economic context. The following need to be addressed. - Capacity to respond to changes; - Capacity to streamline its activities; and - Potential for long term growth. (d) An institution needs financial resources to perform, survive and develop. The following need to be assessed: - Affordability and financial sustainability. (e) An institution is a human organization. Its impact on the following must be considered: - Gender issues; - Environment; and - Human resources development. (f) An institution develops external and internal working relationships. The following points should be considered: - Linkages with other organizations and relationships with customers and stakeholders; - Culture of the organization (top-down approach, autocratic, participatory, centralized, decentralized, bureaucratic, business-like,etc.). From a consideration of these issues, the evaluation team should assess the relevance of the project, its likely impact and the sustainability of its results. 9. PARTICULARS OF THE EVALUATION OF A NATIONALLY EXECUTED PROJECTS Nationally executed projects are characterized by certain features that require attention, namely: - Nature and extent of government support; - Appropriateness of the implementation modality; - Effectiveness of project management, including the review of the management information

9 10 system; - Support from UNDP and from any of the United Nation specialized agencies cooperating with the Government; - Use of the support cost arrangements by the United Nations agencies eligible for this new formula (DESD, FAO, ILO, UNESCO and UNIDO). - Sustainability of the project results and the impact with this modality of execution compared with the traditional modality of execution by United Nations agencies. 10. ASSESSMENT OF IMPACT AND SUSTAINABILITY The impact should be assessed on: (a) The assisted institution; (b) Its staff (direct beneficiaries including gender consideration); (c) Its clients (end-users including specific groups); (d) The community (specific groups); (e) The resources used; and (f) The physical environment. If the objective of the project is partially of fully designed to alleviate poverty, then the evaluation should assess the impact of the project on the planned target group and other groups. Sustainability of the results needs to be reviewed in light of the following considerations: (a) Host government policies - Commitment (b) Socio cultural factors - Acceptability of the project outputs by the beneficiaries - Involvement of the local organizations (participatory process) - Role of women (c) Management and organizational factors - Administration - Human resources management - Logistic and maintenance systems - Information and feedback systems - Managerial leadership - Local participation (d) Financial factors - Funding (e) Environmental and ecological factors (f) Human resources development - Attrition - Retraining - Selection of trainees - Training approach - Quality of training - Training needs assessment - Employment and related manpower development issues - Social dimension (health, welfare and living conditions) (g) Technological factors - Suitability of the technology

10 11 - Cost of the technology introduced - Level of transfer of technology reached with the institution, the trainees and the beneficiaries 11. FORMULATION OF RECOMMENDATIONS AND LESSONS Recommendations The evaluation team should: (a) Ensure logical linkages between findings and recommendations. (b) Identify basic assumptions or key factors (external or internal) from which the recommendations should derive. (c) Try to assess the specific impact of each recommendation. (d) Identify the linkages, if any, between the recommendations in order to establish the chronological order for their implementation. (e) If appropriate, present the recommendations in the form of scenarios. (f) Try to limit the number of recommendations or classify them by order of importance to avoid a selection of the easiest and less effective ones. (g) Estimate the cost of implementing a specific recommendation or a group of recommendations and the expected burden on each stakeholder to be sure that the recommendations can be implemented. (h) Make the recommendation realistic, understandable and operational. A major sectoral level policy revision should not be recommended when a few regulatory changes might suffice. (i) Assess whether the recommendation is specific to the project or has a wider application in the context of the country. (j) Make the recommendations from the perspective of maximizing impact and sustainability of results rather than only from a technical or institutional stand-point. Lessons While findings are project specific, lessons have a bearing beyond the project at hand. Most of the lessons extracted from previous evaluation reports are too specific, trivial or vague. Lessons could be either positive or negative. The following are examples of lessons found in evaluation reports recorded in the CEO database: Weak/bad/incorrect lessons (a) Projects involving rain-based agriculture must start during the dry season. (b) A bad selection of international experts might lead to delays in project implementation. (c) Mid-term evaluations should be more frequently conducted for projects with substantial implementation delays. Good (a) The best entry point for identifying the driving forces of an institution is a review of its management information system to identify the kind of data on which the decision-making process, including the system for allocation of resources, is based. (b) An institution dealing with research development should build a strong information center to disseminate the outcome of its research programme to the benefit of the users.

11 PRESENTATION OF THE REPORT A sample cover page, the outline of the report and the checklist for drafting the report follow. The evaluation report has 7 sections. Section 1, Executive summary, provides a synopsis of the report. It should be clear and concise. Sections 2 to 7 constitute the core of the report. Section 2, Project concept and design, aims to establish the extent to which the problem being solved has been properly identified and the extent to which the solution proposed has been and still is appropriate and has been properly formulated. Section 3, Project implementation, analyses the implementation of the various activities planned in the project document and determines the extent to which they have been carried out successfully or unsuccessfully. Section 4, Project results, establishes the extent to which the project's planned results have been achieved, beginning with the outputs, which are normally measurable, and extending to the immediate objectives and the development objectives which must be logically linked to the outputs. This section also addresses issues related to effectiveness, capacity building, impact and sustainability of results which are paramount. Section 5, Findings, translates the analysis carried out under sections 2, 3 and 4 into specific answers to the questions raised in the TOR and to the three set of questions related to relevance, performance and success as mentioned in page 22. Section 6, Recommendations, provides suggestions for how to proceed depending on alternatives that might be proposed in the TOR, or for solutions that might emerge during the course of the evaluation. Section 7, Lessons learned, provides the opportunity to transcend the immediate context of the project being evaluated and to draw generic lessons that might be applicable in future UNDP assistance. One common mistake for new-comers to evaluation is to be repetitious in the report. For example, in Section 2, which is related to project design, judgements on project implementation might be made, early conclusions might be drawn and preliminary recommendations might be formulated. Each section must be clearly self contained and not overlapping with other sections. The report should present an internal logic for the reader.