INDEPENDENT SYSTEM-WIDE EVALUATION OF OPERATIONAL ACTIVITIES FOR DEVELOPMENT

Transcription

1 JIU/REP/2016/6 INDEPENDENT SYSTEM-WIDE EVALUATION OF OPERATIONAL ACTIVITIES FOR DEVELOPMENT META-EVALUATION AND SYNTHESIS OF UNITED NATIONS DEVELOPMENT ASSISTANCE FRAMEWORK EVALUATIONS, WITH A PARTICULAR FOCUS ON POVERTY ERADICATION Prepared by Gerard Biraud Gopinathan Achamkulangare Joint Inspection Unit Geneva 2016 United Nations

2

3 JIU/REP/2016/6 Original: ENGLISH INDEPENDENT SYSTEM-WIDE EVALUATION OF OPERATIONAL ACTIVITIES FOR DEVELOPMENT META-EVALUATION AND SYNTHESIS OF UNITED NATIONS DEVELOPMENT ASSISTANCE FRAMEWORK EVALUATIONS, WITH A PARTICULAR FOCUS ON POVERTY ERADICATION Prepared by Gerard Biraud Gopinathan Achamkulangare Joint Inspection Unit United Nations

4

5 iii ACKNOWLEDGEMENTS In my capacity as inspector of the Joint Inspection Unit, and as Chair of the Evaluation Management Group, I would like to thank and acknowledge the important group of stakeholders who played a role in supporting the present evaluation. At the outset, I should like to express my deep appreciation and gratitude to former Inspector Gerard Biraud who was the lead author of this pilot evaluation till 31 December 2015 when he completed his second term as Inspector of the Joint Inspection Unit. I would like to thank Member States for their foresight in selecting a topic that is relevant for the 2030 Agenda for Sustainable Development and for providing extrabudgetary resources to pilot the conduct of this independent meta-evaluation and synthesis of the United Nations Development Assistance Framework evaluations. The present evaluation benefited from the substantive and strategic guidance of a key stakeholder reference group that included representatives of Member States, United Nations system entities, Resident Coordinators Offices, global experts and non-governmental organizations. Appreciation goes to the six evaluation offices of United Nations system entities that contributed to the evaluation as members of the Evaluation Management Group to enhance its professional and technical quality. I am grateful to the consultants who participated in the evaluation exercise and in particular to the significant contribution of Dale Thompson. The Joint Inspection Unit, in line with its mandate for independent system-wide evaluation, played a critical leadership and substantive role. Sincere thanks goes to all Unit inspectors who helped to guide the management of the evaluation and who, in the context of the collective wisdom of the Unit, conducted careful reviews of the evaluation findings, conclusions and recommendations. I would like to thank the secretariat of the Joint Inspection Unit and the coordination secretariat of the independent system-wide evaluation mechanism for their extensive support in the day-to-day management and administrative support of the pilot evaluation. Last but not least, I would like to thank the Joint Inspection Unit focal points for their usual extensive support in coordination. Gopinathan Achamkulangare Inspector Joint Inspection Unit of the United Nations System Chair, Evaluation Management Group

6 iv FOREWORD The present meta-evaluation and synthesis exercise of the United Nations Development Assistance Framework intends to examine both the quality of Framework evaluations conducted in the period and to subsequently draw conclusions from those evaluations to provide an assessment of the contribution of the United Nations system to poverty reduction. The evaluation provides a good opportunity to identify and highlight, in a systematic manner, the challenges related to the evaluative process of Framework activities, in order to guide decision-making for the strengthening of the overall value of Framework evaluations, as a mechanism for United Nations system-wide accountability and learning at the country level. The evaluation concludes that there is an alarming lack of commitment from stakeholders in the Framework evaluation process, highlighted by the low level of compliance with the requirement for an evaluation and with the quality standards promoted in related guidance. Furthermore, a low level of participation from national stakeholders has been identified, along with significant issues of coordination and cooperation in the evaluation activities conducted by the United Nations entities at the country level. The recommendations presented in the present review are aimed at addressing the strategic implications of the strengthening of the Framework evaluation system as a vital part in the overall United Nations planning framework. The enhancement of the Framework evaluation mechanism will involve a commitment by senior decision-makers within the United Nations system and those of Member States. An active partnership will be necessary to enhance the quality and relevance of the Framework evaluation mechanism so that it can play its rightful role as a mechanism to promote cooperation, organizational leaning and positive feedback. The present review is a first step in that process of renewal and redirection.

7 v EXECUTIVE SUMMARY UNITED NATIONS DEVELOPMENT ASSISTANCE FRAMEWORK META-EVALUATION AND SYNTHESIS JIU/REP/2016/6 On 21 December 2012, the General Assembly adopted its resolution 67/226 on the quadrennial comprehensive policy review of operational activities for development of the United Nations system. Following that, in paragraph 8 of its resolution 68/229, the Assembly mandated that two pilot independent system-wide evaluations be conducted in 2014, subject to the availability of extrabudgetary resources, one of which on the meta-evaluation and synthesis of United Nations Development Assistance Framework evaluations, with a particular focus on poverty eradication. The meta-evaluation and synthesis was designed to identify and assess the benefits of the United Nations Development Assistance Framework evaluations. The specific purposes of the present review have been to: Conduct an assessment of Framework evaluations in the period to determine the overall quality, credibility and use or evaluations, in order to provide advice on possible improvements and adjustments to existing Framework evaluation guidelines Identify key findings, conclusions and recommendations of Framework evaluations undertaken in the period in order to assess the contribution of the United Nations system in achieving national development goals for planning and adjustments to current strategies Internal administrative and operational considerations, especially the challenges with regard to the mobilization of extrabudgetary resources, resulted in the assignment being carried over to Main findings and conclusions The first significant finding of the review is the very low rate of compliance with the 2010 United Nations Development Group guidelines that mandate an evaluation at the end of each Framework cycle. Between 2010 and 2014, 33 out of 88 programme countries (or 37.5 per cent) with active Framework cycles submitted evaluations as per the Guidelines, issuing 36 Framework evaluations. This low rate of compliance calls into question the degree to which evaluations play an organizational learning role in the Framework process. A lack of adequate financial resources most likely contributed to this low rate. However, it may not have been the sole factor for the low levels of participation: factors such as overlap and duplication with other evaluations being undertaken at the country level, along with a lack of commitment to the Framework evaluation itself, also resulted in the low rate of compliance. Therefore, if the Framework evaluation is to continue to be the feedback and organizational learning component of the overall process, measures will need to be taken to strengthen commitment to the evaluation process. Of equal importance, only 23 of the 36 Framework evaluations were of sufficient quality as to warrant a more in-depth examination of their content, with a view to drawing conclusions with respect to the contribution of the United Nations system in achieving national development goals. This further underscores the challenge of assessing the degree to which Framework evaluations play a feedback and organizational learning role. The low level of participation of Member States in the Framework evaluation process points to another challenge. The philosophy that underpins the 2030 Agenda for Sustainable Development implies that the primary responsibilities of the United Nations system at the country level are to support the development plans of Member States and their strategies to attain the Sustainable Development Goals, utilizing contextually sensitive programming and support. The low rate of participation in the Framework implies that the evaluation process did not engage Member States as active stakeholders. Moreover, the degree of Member State participation in the design and implementation of the Framework itself may not have been optimal. Therefore, it will be important to provide guidance to the United Nations system to ensure it actively encourages the participation of programme countries in the Framework and to encourage programme countries to engage actively in the development and implementation of Framework activities and their monitoring and evaluation. Several methodological and conceptual gaps emerge with regard to the quality of Framework activities as analysed through the Framework evaluations. Notwithstanding the low rate of compliance of evaluations (37.5 per cent), some common threads could be highlighted. First, key building blocks for an evaluation appear to be missing, or of limited importance, in many Framework

8 vi evaluations. The utilization of robust evaluation methodologies in accordance with the norms and standards of the United Nations Evaluation Group has only been partially undertaken. Stakeholder input has largely been limited to country team members. Similarly, data sets have largely been restricted to desk-based document review. Significantly, the recommendations sections of the Framework evaluations are generally deficient in addressing more strategic issues and proposing actionable recommendations, as opposed to ones that cannot be measured or easily implemented. These deficiencies limit the utility of evaluations as a tool for feedback and organizational learning. Second, only 23 of the Framework evaluations, or approximately half, appear to have generated management responses, suggesting that country team management has not invested sufficient effort in assessing the implications of the Framework evaluations. Taken together with the low number of evaluations in the first place, the willingness of country team management to support the important role of evaluations in the entire Framework cycle can be called into question. Measures to fill these gaps will need to be taken. To gauge how the Framework evaluation process has provided a platform for organizational learning, it is necessary to review how the evaluations have assessed the content of the Framework reports themselves with respect to the relevance, effectiveness, efficiency, potential impact and sustainability of United Nations actions. The 2010 United Nations Development Group guidelines refer to the issue of impact to be addressed via Framework evaluations. Although the contribution of Framework activities to poverty reduction is central to their reason for being, only two Framework evaluations established a solid link between such activities and their impact and government poverty alleviation goals and strategies. This limits the value of the Framework evaluation mechanism as a means of strengthening the Framework system. It also illustrates the extent to which the Framework evaluation process is significantly limited in its ability to address primary issues, such as the demonstration of how the Framework process contributes to national poverty alleviation goals. Furthermore, not all of the programming principles of the Framework have been integrated. Environmental sustainability and capacity-development are seldom mentioned. This leads to the conclusion that the principles themselves are probably insufficiently articulated in Framework activities, resulting in a significant programmatic gap. The present review examined a number of internal processes of the United Nations at the country level designed to strengthen the relevance and subsequent implementation of Framework activities. However, the limited evidence garnered from the Framework evaluations indicates less than optimal levels of coordination and cooperation, thus hindering the overall effectiveness of the delivery of United Nations programming at the country level, and inefficiencies in joint programming and inter-agency work under different Framework pillars. Moreover, the four Delivering as one programme countries that prepared evaluation reports experienced the same kind of coordination challenges as other country teams. The present meta-evaluation and synthesis has laid out a sobering picture of the current state of the utility of the Framework evaluation process and, by implication, some aspects of the Framework process itself. There now appears to be a degree of willingness to recognize that the past performance with respect to the Framework evaluations has been inadequate and that corrective measures are necessary. These shortfalls are addressed through a set of strategic recommendations. Caution was exercised to develop a set of actionable recommendations that were sufficiently detailed so as to address the issues at hand, but not excessively prescriptive as to constrain the development of context-specific solutions. The recommendations are designed less to fix specific challenges and shortfalls, than to address the implications of the strengthening of the Framework evaluation system as a vital part in the overall United Nations planning framework. With the emphasis on nationally contextual solutions as a primary focus of the 2030 Agenda for Sustainable Development, the Framework process as a whole acquires greater salience. Recommendation 1 Recommendations The Secretary-General should, in consultation with the Chief Executives Board for Coordination as appropriate, revise, through the United Nations Development Group, programming guidance for the United Nations Development Assistance Framework to emphasize the centrality of the active participation of Member States, especially programme countries, throughout the design and subsequent implementation and evaluation. This revised guidance for country teams and Resident Coordinators Offices should lay out clear guidelines and systematic approaches for interaction with officials of the

9 vii Governments of the respective programme country and sensitization on programming principles related to their national development priorities and plans; and, introduce a more systematic approach to mainstream the programming principles into the Framework process. Recommendation 2 The General Assembly should, through the quadrennial comprehensive policy review process, encourage Member States, especially programme countries, to more fully participate in the United Nations Development Assistance Framework process at the country level by means of early engagement during the development of upcoming Frameworks and participation to the fullest extent throughout their implementation, monitoring and evaluation. The Economic and Social Council should, within the quadrennial comprehensive policy review process, keep under constant review and monitor the measures taken by the United Nations country teams to facilitate the engagement of the programme countries throughout the Framework cycle, including its evaluation. Recommendation 3 The Secretary-General should, in consultation with the Chief Executives Board for Coordination as appropriate, initiate, through the United Nations Development Group, a process for sensitizing and specifically instructing Resident Coordinators on the importance of the United Nations Development Assistance Framework evaluation process as a critical element in the overall United Nations country-level planning processes; and, specifically, for requiring Resident Coordinators to develop an action plan and timetable for conducting Framework or follow-up evaluations on instruments, for example, sustainable development frameworks, and to integrate the conduct of Framework evaluations into the performance management framework of the Resident Coordinators. Recommendation 4 The Secretary-General should, in consultation with the Chief Executives Board for Coordination as appropriate, request other United Nations agencies to coordinate their evaluation activities at the country level so that such activities can be better integrated into the United Nations Development Assistance Framework evaluation process. Recommendation 5 The Secretary-General should, in consultation with the Chief Executives Board for Coordination as appropriate, revisit, through the United Nations Development Group, the 2010 evaluation guidelines and subsequent direction in the light of the findings of the present report, with a view to strengthening their methodological rigour and design and to increasing the rate of compliance; in particular, the modified guidance should: (a) Emphasize the requirement for actionable recommendations with a clear target audience and time frame for implementation that comply with the United Nations Evaluation Group norms and standards; (b) Document the mainstreaming of the Framework programming principles and, in particular, those relating to environmental sustainability; (c) Utilize a robust evaluation design based on multiple data sources, in addition to desk-based (document) reviews and stakeholder interviews.

10 viii CONTENTS ACKNOWLEDGEMENTS... FOREWORD... EXECUTIVE SUMMARY.... ABBREVIATIONS... Page iii iv v ix Chapter Paragraphs I. INTRODUCTION: PURPOSE AND BACKGROUND A. Purpose B. Background II. OBJECTIVES, METHODOLOGIES, SCOPE AND DELIVERABLES A. Objectives B. Overview of methodologies C. Scope D. Limitations III. THE META EVALUATION A. Specific methodologies and limitations of the meta-evaluation B. Coverage C. Key findings from the quality screening of Framework evaluations IV. SYNTHESIS A. Specific methodologies and purpose B. Findings V. CONCLUSIONS VI. RECOMMENDATIONS VII. CONCLUDING REMARKS ANNEXES I. List of Framework evaluation reports by countries concerned II. Overview of actions to be taken on recommendations... 27

11 ix ABBREVIATIONS CEB FAO ILO UNDP UNEP UNFPA UNICEF UNIDO WFP WHO United Nations System Chief Executives Board for Coordination Food and Agriculture Organization of the United Nations International Labour Organization United Nations Development Programme United Nations Environment Programme United Nations Population Fund United Nations Children's Fund United Nations Industrial Development Organization World Food Programme World Health Organization

12 1 I. Introduction: purpose and background A. Purpose 1. On 21 December 2012, the General Assembly adopted its resolution 67/226 on the quadrennial comprehensive policy review of operational activities for development of the United Nations system. Following that, in paragraph 8 of its resolution 68/229, the Assembly mandated that two pilot independent system-wide evaluations be conducted in 2014, subject to the provision and availability of extrabudgetary resources and decided that the themes of those two evaluations should be metaevaluation and synthesis of United Nations Development Assistance Framework evaluations, with a particular focus on poverty eradication and evaluation of the contribution of the United Nations development system to strengthening national capacities for statistical analysis and data collection to support the achievement of the Millennium Development Goals and other internationally agreed development goals. 2. In paragraph 8 of its resolution 68/229, the General Assembly took note of the policy for independent system-wide evaluation of United Nations operational activities for development. According to that policy, 1 the definition of an independent system-wide evaluation is a systematic and impartial assessment of the relevance, coherence, efficiency, effectiveness, impact and sustainability of the combined contributions of United Nations entities towards the achievements of collective development objectives. 3. In April 2015, pending the formal establishment of the Evaluation Management Group and the key stakeholder reference group, the Joint Inspection Unit of the United Nations system (JIU) established an ad hoc advisory group consisting of representatives of Member States and United Nations entities. 4. The Evaluation Management Group was established by the Joint Inspection Unit in close consultation with the United Nations Evaluation Group in June 2015, and comprises seven members, including the Unit as Chair. 5. In the third quarter of 2015, the Evaluation Management Group approved an inception paper for the meta-evaluation and synthesis of the United Nations Development Assistance Framework. The members of the Group established the key stakeholder reference group on the basis of a mapping of key stakeholders and composed of 13 members (4 of whom were also members of the ad hoc advisory group). 6. Internal administrative and operational considerations, especially the challenges with regard to the mobilization of extrabudgetary resources, resulted in the assignment being carried over to B. Background Need to evaluate the United Nations Development Assistance Framework 7. The United Nations Development Assistance Framework refers to a unified approach towards common development goals at the country level. The common country assessment and the Framework were adopted as strategic planning tools for the United Nations system. Guidelines for their preparation were first issued in April 1999 and later revised in 2002, 2004 and 2010 to reflect lessons at the time and to take into account the United Nations Millennium Declaration. 8. Five years after it encouraged the United Nations development system to intensify its collaboration at the country level through the Framework, the General Assembly, as a conclusion of its 2007 Triennial comprehensive policy review, decided in 2012 to assess how the new tool had been 1 Available from

13 2 effective and efficient, in particular from 2010 to While ongoing efforts to improve transparency, accountability and results-based management in the United Nations system were recognized by the General Assembly in its resolution 67/226 on the quadrennial comprehensive policy review of operational activities for development of the United Nations system, there continued to be a need to strengthen independent and impartial system-wide evaluation of operational activities for development. 9. In the policy framework for independent system-wide evaluation of operational activities for development, meta-evaluation was identified as the most rapid and economical of the three approaches embedded in the independent system-wide evaluation policy, elaborated in 2013 in a note by the Secretary-General (A/68/658-E/2014/7), submitted to the Economic and Social Council and taken note of by the General Assembly in its resolution 68/ The present review becomes significant, given that past efforts to assess the quality of Framework evaluations had noted several weaknesses. 2 No systematic evaluation of the Framework has yet been attempted. As the present review was primarily based on desk-based methods, it invariably limits the extent to which conclusions can be drawn regarding the performance of the United Nations system in its contribution to development results at the country level. Need to evaluate Framework evaluations 11. Prior to undertaking any synthesis of the lessons learned from the Framework process, it was necessary to review and assess the technical adequacy of the Framework evaluations. This, therefore, is the purpose of the meta-evaluation part of the report. 12. Structured as a primarily desk-based analysis, the first part of the report (meta-evaluation) examines the technical quality of Framework evaluations and the adequacy of resources dedicated to them. The second part (synthesis) looks back at the learning and performance of the United Nations country teams as assessed by Framework evaluations and seeks to provide forward-looking recommendations, designed to improve the process of strategic planning, programme development and implementation. The report is also designed as an input into the deliberations of the 2016 quadrennial comprehensive policy review of operational activities of the United Nations development system. 2 Past efforts to assess Framework evaluation quality include: R. Longhurst, "Review of the Role and Quality of the United Nations Development Assistance Frameworks" (2006), available from Quality_of_UNDAFs.pdf; and P. Balogun, "The relevance, effectiveness and efficiency of the United Nations Development Assistance Framework" (2012), available from

14 3 II. Objectives, methodologies, scope and deliverables A. Objectives 13. In accordance with General Assembly resolution 68/229, the objectives of the meta-evaluation and synthesis are to assess, in line with the United Nations Evaluation Group norms and standards for evaluation in the United Nations system, the overall quality, credibility, utility and utilization the Framework evaluations; and synthesize the major and key findings, conclusions and recommendations of Framework evaluations published between 2010 and The intended purposes of the meta-evaluation and synthesis are to: Conduct an assessment of Framework evaluations in the period to determine the overall quality, credibility and use of evaluations in order to provide advice on improvements and adjustments to existing Framework evaluation guidelines Identify key findings, conclusions and recommendations of Framework evaluations undertaken in the period in order to assess the contribution of the United Nations system in achieving national development goals to inform the 2016 quadrennial comprehensive policy review for planning and adjustments to current strategies. 15. The target audiences for the meta-evaluation and synthesis are States Member of the United Nations, Governments of programme countries, officials of the United Nations system organizations at the headquarters, regional and country levels, and evaluators who conduct Framework evaluations. B. Overview of methodologies 16. The present review employed a mixed-methods approach so as to better calibrate limited and sometimes incomplete data. Framework evaluations and similar national-level reports were gathered online and by other direct means to establish the scope of the assignment. An electronic survey of Resident Coordinators was used to fill in data gaps and identify qualitative factors influencing the development of Framework evaluations. An evaluation matrix containing key evaluation questions was developed. For each of the components of the meta-evaluation and synthesis, a separate rating tool was developed. The assignment was designed to be a pilot, examining the effectiveness of a deskbased review, with only limited direct interviewing. 17. In order to contextualize the review, it is necessary first to present the evaluation questions for both the meta-evaluation and synthesis components. 18. The evaluation questions guiding the meta-evaluation segment are presented in table 1 below. Substantive area for the metaevaluation Coverage Quality Table 1 Evaluation questions guiding the meta-evaluation Key evaluation questions How many Framework programme countries have produced Framework evaluations? What are the main reasons for not undertaking Framework evaluations? What is the technical quality of Framework evaluations? To what extent is the evaluative evidence from the Framework evaluation credible, useful and timely? 3 3 Credibility is measured through the 42 checkpoints in the report quality screening tool.

15 4 Substantive area for the metaevaluation Key evaluation questions To what extent are the Framework evaluations resourced in terms of funding, human resources, data inputs and time allocated? 4 To what extent are the four inputs adequate? Are current Framework evaluation guidelines and tools adequate? 19. The evaluation questions guiding the synthesis are illustrated in table 2 below. Table 2 Evaluation questions guiding the synthesis Substantive area for the synthesis Poverty eradication Mainstreaming the principles United Nations system performance Key evaluation questions Based on the report findings and conclusions, To what extent has the Framework contributed to and made progress towards the achievement of the national poverty alleviation goals and strategies? 5 What are the challenges or main factors contributing to progress? How well have the five programming principles been mainstreamed into the agreed Framework? What are the challenges or main factors contributing to progress? (if any) 6 How well aligned is the Framework with national development strategy goals and planning cycles? Does the Framework facilitate national ownership and leadership during design and implementation stages? Has the Framework led to a more coherent and coordinated United Nations response to the development needs and priorities of the programme country? 7 Does the Framework facilitate national access to the full spectrum of United Nations system mandates and resources? How has the Framework strengthened partnerships between the existing United Nations system at the country level and other development actors (development partnerships beyond the United Nations system)? 4 The set of questions related to the adequacy of inputs (e.g., data and guidelines), and financial and human resources could not be comprehensively addressed owing to incomplete data. 5 This key evaluation question is only partially addressed in the report owing to limitations in data availability. 6 The five programming principles are: the human rights-based approach, gender equality, environmental sustainability, results-based management, and capacity development. 7 This includes evidence of: (a) the United Nations system reducing gaps and duplication; (b) improving the division of labour among United Nations actors at the country level and enhanced coordination (synergies) with the respective national entities concerned at the strategic and operational levels; (c) a dynamic process during implementation of joint United Nations initiatives (cross sector, advocacy, thematic and/or joint programmes); and (d) the Framework serving as a facilitator of South-South cooperation.

16 5 C. Scope 20. The inception phase of the assignment included the identification of all Framework evaluations from various online information sources within and outside the United Nations system. 21. An initial online request was sent to all Resident Coordinators that any relevant Framework evaluations be made available for the assessment. After considering the number of programme countries with evaluations published between 2010 and 2014 (an initial total of 27 evaluations from 24 programme countries), the number of programme countries without evidence of an evaluation was identified to be 98. Given the low rates of completion in terms of compliance, a follow up online questionnaire was developed with the support of the Evaluation Management Group and sent to all programme countries with an active Framework cycle but without evidence of an Framework evaluation. 22. This follow up questionnaire was sent to 88 Resident Coordinators Offices in July and August 2015, with a remarkable 100 per cent response rate. 23. Nine additional Framework evaluations published between 2010 and 2014 were received and one programme country reported not having a Framework at all. This, in turn, reduced to 88 the total number of programme countries that ought to have produced a Framework evaluation but did not (out of a possible 121 Framework programme countries). 24. Therefore, 36 Framework evaluations were identified. A systematic screening process then took place by using a report quality screening tool. This resulted in 23 Framework evaluations being deemed to meet the criteria from the screening process, which were labelled validated evaluations for the purposes of the analysis required for synthesis. The remaining 13 Framework evaluations that did not meet these criteria were labelled not-validated evaluations. 25. Figure 1 shows the process of how Framework evaluations were identified and screened. Figure 1 Screening process for Framework meta-evaluation and synthesis

17 6 D. Limitations 26. There are major limitations that serve to weaken the usefulness of the present review and need to be taken into consideration when considering the findings, conclusions and especially recommendations. 27. First, the small number of Framework evaluations results considerably limits the scale of the review. Moreover, 15 of the 36 evaluations stem from Latin American or Caribbean countries, clearly a disproportionate geographical representation. By contrast, only 9 stem from Africa. 28. Second, and more importantly, only 23 of the evaluations were deemed to be valid for the purpose of analysing them to draw conclusions about the effectiveness of the Framework process to contribute to the attainment of national development goals the prime object of the synthesis element of the report. This even lower rate, combined with regionally disproportionate reports, hinders the drawing of substantive conclusions. 29. Given the disproportionate nature of the Framework evaluations, it was not possible to develop, as part of the present review, any region-specific findings or conclusions.

18 7 III. The meta evaluation A. Specific methodologies and limitations of the meta-evaluation 30. A report quality screening tool was developed to systematically identify differences in quality within the Framework evaluation reports. 8 The tool was fully operationalized after pre-testing with 10 evaluation reports. It consisted of two parts: a quality screening and a report content identification, which included a total of 42 checkpoints designed to measure different elements categorized under 11 sections. A five-point scale was used on the checkpoints (if applicable), ranging from 1 (very poor) to 5 (excellent). 31. The tool was applied to the 36 identified Framework evaluation reports. Additional evidence linked to quality, including funding, human resources and time allocated was collected to support the analysis of report quality. 32. Background and contextual interviews with key officials of the United Nations Development Group and United Nations Evaluation Group were very few, again reflecting the decision to undertake the assignment to the greatest degree possible as a desk-based and document review pilot process. 33. Likewise, stakeholder involvement in the Framework evaluations appears limited, with the reports tending to include national Governments, United Nations system entities, including the nonresident agencies, civil society organizations and, to a lesser extent, representatives of donor Governments at the country level. A few reports note the participation of those benefitting from the United Nations system contributions, stakeholders from academia, foundations, non-governmental organizations, private sector and United Nations regional offices. B. Coverage 34. In their penultimate year, all Framework activities are required to undergo an evaluation that, in turn, is designed to contribute to the strategic orientation of the subsequent Framework. 9 Between 2010 and 2014, 33 out of 88 programme countries (or 37.5 per cent) with active Framework cycles submitted Framework evaluations as per the 2010 United Nations Development Group guidelines, resulting in the issuance of 36 Framework evaluations (see figure 1 above). 35. Given this low level of coverage, the intended roles of the Framework in determining the contribution of the United Nations system to national development priorities and in identifying emerging priorities for future programming, have not been widely achieved. The capacity to use evidence and information from evaluations to guide decision-making and planning is limited. 36. The evidence drawn from the online questionnaire to United Nations Resident Coordinators Offices indicated the following main reasons for this level of coverage: Lack of adequate financial resources or funding Inadequate data to assess the results of the contributions from the United Nations system (e.g. financial data, monitoring indicator data or assessments) 37. In specific terms, evidence from the Framework evaluations shows the average budget for the evaluation process to be $34, This average cost is consistent with other research undertaken by 8 This tool draws heavily from multiple sources, including the United Nations Evaluation Group quality standards (2005), the United Nations Development Group guidance (2010), and a tested tool produced by the Office of Internal Oversight Services. See the complementary annexes to this report on the JIU website for the list of documents consulted and the report quality screen tool, respectively United Nations Development Group guidelines are available from 10 Budget information gathered from the United Nations Development Programme evaluation resource centre ( was only available for 22 of the 36 assessed reports, or 61 per cent.

19 8 the Joint Inspection Unit. 11 Reactions from the United Nations Development Group, the United Nations Evaluation Group and the Evaluation Management Group stakeholders highlighted that the amount was well below what they considered to be adequate, namely, in the range of $100,000, for conducting a high quality evaluation process in line with the United Nations Evaluation Group norms, standards and ethical guidelines. However, no formal guidance about the costs of such evaluations has ever been established or provided. A request from the Group about the extent to which the Framework budgets include a description of resource mobilization efforts and an actual financial allocation for Framework evaluation costs revealed that none of the Framework documents had any information about planning for or allocating funding for the Framework evaluation. 38. Therefore, while compelling to some degree, the budgetary argument should not be seen as axiomatic. Rather, it is one of a number of important factors that could have influenced low rates of coverage. 39. Equally, respondents to the online questionnaire indicated that lack of data, sometimes caused by resource shortages, but also the result of inadequate country level data gathering by the United Nations agencies, impeded the development of Framework evaluations. 40. Along with (perceived) financial and data gaps, lack of precise direction and coordination has limited the rate of coverage of Framework evaluations. 41. Additional evidence from the questionnaire indicates the inadequate level of participation of the national Government partners and the country teams to conduct Framework evaluations. In some programme countries, it was reported that national partners were frequently burdened with different evaluation teams asking similar questions. Questionnaire responses highlighted that similar observations typically focus on the area of the United Nations system coordination and collaboration, or the lack thereof. 42. From the few United Nations Development Group and United Nations Evaluation Group stakeholders interviewed, it was observed that each individual United Nations agency may conduct its own country programme evaluation independently of the Framework, and that those evaluations are not sufficiently coordinated to contribute to the Framework evaluation. Data gathered from the online questionnaire points to the need to provide direct, explicit guidance to support coordination and systematic harmonization of evaluation efforts. This way, all evaluative inputs can be fed into the next Framework cycle and thus support the Framework evaluations. C. Key findings from the quality screening of Framework evaluations 43. Before examining the quality of Framework evaluations, it is important to review the profiles of the 36 Framework evaluations assessed in the meta-evaluation during the period Specifically, the majority (53 per cent) are in English language, followed by Spanish (33 per cent), French (11 per cent) and Portuguese (3 per cent). Four evaluations are issued from Delivering as one countries in the meta-evaluation sample, including Albania, Cabo Verde, Mozambique and Rwanda. 44. The 11 elements reviewed during the quality assessments were: executive summary; introduction; evaluation methodology; application of evaluation criteria; reporting on results; conclusions; recommendations; format; United Nations coherence analysis; national ownership; and results for results-based management only. 45. While generally satisfactory, several of the factors were lower rated, most notably recommendations, thereby generally weakening the effectiveness of the Framework evaluation process. 46. An overview of the results of the report quality screening tool, presented in figure 2 below, demonstrated the range of the ratings, with notable weaknesses in the: Executive summary 11 See JIU/REP/2014/6, Analysis of the Evaluation Function in The United Nations System, p. 58.

20 9 Evaluation methodology Application of evaluation criteria Recommendations Analysis of the monitoring and evaluation framework, or results for results-based management only. Figure 2 Overview of the results from the report quality screening tool 47. No attempt has been made to develop an average rating across all factors. In the absence of a weighting of individual factors, doing so by a simple equal weight average would have resulted in an artificiality. 48. The most significant shortfalls relate to the relatively low quality of the recommendations. For the Framework evaluation process to constitute a process of learning and strengthening, relevant and high quality recommendations are essential. Recommendations are a core component of evaluations and should be aimed at enhancing the effectiveness, efficiency or sustainability of the project or programme. In terms of content and quality, recommendations should have clear messages, follow from evidenced-based conclusions and should be actionable. 12 However, a large percentage of the observed recommendations were not specific, time-bound or sufficiently actionable. Some 72 per cent of the recommendations sections fell into the fair, poor or very poor categories. The majority did not include specific timeframes for implementation, the steps needed to complete them and specific responsibilities for implementation. 49. Limitations in relation to evaluation methodology and application of evaluation criteria demonstrate overall methodological weakness, thereby detracting from the validity of the Framework evaluation process. 12 See United Nations Evaluation Group Quality Checklist for Evaluation Reports (2010) (sect. 7, p. 6), available from mainsite/about_iom/eva_techref/uneg_eval_report.pdf.

21 The weaknesses demonstrated emphasize that the methodologies used for Framework evaluation processes are not necessarily robust, thereby detracting from the quality and scope of the analysis. Almost all of the reports have methodology sections that rely nearly exclusively on two main methodologies: document analysis and stakeholder interviews. Using responses to questionnaires to support findings is rare and only a few reports make use of monitoring and evaluation data that is systematically collected by United Nations agencies and/or national Governments. Therefore, data sets have been insufficiently calibrated. 51. The sources of evidence used in most evaluations rely extensively on a narrow range of stakeholder perceptions and are unlikely to offer an accurate account of the United Nations system performance, especially since the stakeholder participation was limited to the Government, United Nations system and civil society organizations. There is also a lack of baseline information, the absence of benchmarks and, in some cases, the absence of financial information. Given these major shortcomings, most of the methodology sections rank as fair (13) or poor or very poor (13 combined); with only 10 reports categorized as either good or excellent. 52. In terms of the quality and use of evaluation criteria, while many reports utilized the criteria of relevance, effectiveness, efficiency and sustainability as part of the overall analytical framework, the evaluations reveal a notable absence of impact assessment. 13 The United Nations Evaluation Group Quality Checklist for Evaluation Reports specifies that findings should respond directly to the evaluation criteria and questions detailed in the scope and objectives section of the report should be based on evidence derived from data collection and analysis methods described in the methodology section of the report. In the period of the present review, in 8 out of 36 evaluations, or approximately one quarter, evaluation criteria were either not included or given as not applicable. In most cases, however, this analytical framework was not required in their terms of reference. The reports that did not apply the evaluation criteria were published in 2011 and 2012, which was before the full roll-out of the 2010 United Nations Development Group guidelines. 53. Only a handful of reports addressed the mandated evaluation criterion of impact even in the most tangential fashion. Moreover, only 2 out of 36 evaluation reports included a results chain, which facilitates the identification of anticipated results through different steps (e.g., input, output, outcomes and impact). 14 Furthermore, nearly all of the reports ranked fair or poor demonstrated analytical gaps and a lack of robust supporting evidence in the way their conclusions were justified. 54. These shortfalls in methodology point to an overall limitation in the Framework evaluation process, namely, lack of sufficient evaluation quality as part of the report development process. While the present review was unable to determine the direct consequences of these shortfalls, combined with the shortfalls related to recommendations, these gaps call into question the worth of the Framework evaluations as they are currently undertaken. 55. The overall rating of the other assessed factors, while satisfactory, demonstrates a degree of inherent weakness that detracts from the overall effectiveness of Framework evaluations. 56. Figure 2 demonstrates the extent to which the other assessed factors (for example, format and ownership ) averaged in the upper ranges of the fair category (3-4). Limitations were found in all these factors, with the ratings being degraded because a number of the reports simply did not address them. For example, the rating for quality of results reporting is weakened by the fact that five of the reports had either poor or very poor reporting on results. 57. Nevertheless, the extent to which the present review has demonstrated only moderately satisfactory aggregated ratings points to the need for the United Nations system to provide both improved direction for strengthening the Framework evaluation process and enhanced leadership for addressing issues of low coverage. 13 According to the 2010 United Nations Development Group guidelines, the results expected from the Framework evaluation are considered judgement about the relevance, efficiency, effectiveness, impact and sustainability of Framework results (page 17). 14 United Nations Evaluation Group, standard 4.12.

22 The overall use of Framework evaluations as instruments to promote organizational learning appears to be mixed, raising questions about their value. 59. Results regarding the utilization of Framework evaluations have been based on the online questionnaire, the few selected interviews and the availability of management responses. The 2010 United Nations Development Group guidelines call for a written and agreed management response by the country teams and the Government of the programme country following the conduct of an evaluation. While half of the evaluations have a management response, the remaining either do not have a management response posted, or there is no information available. Some additional management responses may exist, but may not be available via electronic means. Nevertheless, this gap points to a challenge with respect to how United Nations country teams appear to conceptualize the Framework process and view its overall worth. This finding can be inferred to imply a broadbased tendency to view the process as one of compliance, whereby the evaluation is performed as a matter of process, as opposed to one of organizational learning, whereby it becomes a catalyst for renewal and redirection.

23 12 IV. Synthesis A. Specific methodologies and purpose 60. The synthesis is formulated from 36 Framework evaluations that have been screened for quality, using the report quality screening tool. Out of 36 Framework evaluation reports, 23 reports, or 64 per cent, meet the qualifying criteria, namely, the Framework evaluations which had an overall assessment of fair, good, or excellent and are labelled validated evaluations. 61. The report quality screening tool was then used to generate an overall view of identified substantive areas in poverty eradication and the five programming principles, based on the key findings, conclusions and recommendations of all 23 validated Framework evaluations. It is based on a three-point scale: 1 = mostly negative results reported; 2 = mixed results reported; and 3 = mostly positive results reported. The five programming principles constitute a starting point, and for all of the stages of the Framework formulation, including results planning, implementation, monitoring and evaluation. According to the United Nations Development Group, all Framework evaluations should address these key cross-cutting issues The purpose of the synthesis was to review the past experiences of country teams with respect to the contribution of the United Nations system to the achievement of national goals, by: Understanding the extent to which the United Nations system achieves concrete results in assisting the programme countries to eradicate poverty (effectiveness); Learning about the alignment of the Framework to national needs and priorities (relevance), national ownership (sustainability) and how coherence and coordination among United Nations organizations at the country level are progressing (efficiency), including any challenges or opportunities; Non-resident agency engagement; Partnerships with actors beyond the United Nations. 63. The synthesis is an assessment of the Framework evaluation reports that have been deemed to be of sufficient quality for more detailed analysis. The synthesis, therefore, is not an assessment of the quality of the Frameworks themselves. B. Findings 64. The findings below are organized on the basis of the analytical framework (see table 2 above) for the synthesis, namely, on poverty eradication, mainstreaming of programming principles and United Nations system performance. Poverty eradication, alignment and leadership 65. Only two of the 23 validated Framework evaluations established a factually based link between Framework activities and their impact and government poverty alleviation goals and strategies, thereby limiting the value of the Framework evaluation mechanism as a means of strengthening the Framework system. 66. This finding underscores the extent to which the Framework evaluation process is limited not only in scope, but also in its ability to address primary issues, such as how the Framework process contributes to national poverty alleviation goals. 15 See and (Part%20II).pdf.