UNEG Study on the Evaluability of the UN Development Assistance Framework

Transcription

1 UNEG Report UNEG Study on the Evaluability of the UN Development Assistance Framework This report was prepared by Paul Balogun, consultant, and was published in December This report assesses the evaluability of recent UNDAFs in terms of: a) clarity of intent of the subject to be evaluated (relevance and design of the expected outcome statements and results matrices); b) the existence of sufficient measurable indicators (collection of reliable data for analysis); c) quality of joint monitoring systems; and d) external factors (positive or negative) that have influenced the process and the realization of expected outcomes. UNEG/RPT/UNDAF(2006)1

2 Abbreviations and Acronyms AWP CCA CP MDGs M&E PRSp RM TCPR UNCT UNDAF UNDG UNDGO UNEG Annual Work Plans Common Country Assessment Country Programme UN Millennium Development Goals Monitoring and Evaluation Poverty Reduction Strategy paper Results Matrix UNGA s Triennial Comprehensive Policy Review UN Country Team UN Development Assistance Framework UN Development Group UN Development Group Office United Nations Evaluation Group 2 UNEG Study on the Evaluability of the UN Development Assistance Framework

3 Summary 1. The UN Development Assistance Framework (UNDAF), introduced in 1997, aims to support identification and implementation of a collective, coherent and integrated UN system response to national priorities and needs. In 2004, the Guidelines issued by the UN Development Group on the development and use of UNDAFs were revised. As part of this revision, guidance on monitoring and evaluation was included for the first time. 2. This report assesses the evaluability of recent UNDAFs in terms of: a) clarity of intent of the subject to be evaluated (relevance and design of the expected outcome statements and results matrices); b) the existence of sufficient measurable indicators (collection of reliable data for analysis); c) quality of joint monitoring systems; and d) external factors (positive or negative) that have influenced the process and the realization of expected outcomes. 3. The conclusions and recommendations outlined are expected to provide inputs into the update of the UNDAF M&E guidance currently being prepared by UNDG led Working Group on Programming Policy. They are also meant to yield benefits for the on-going process of conducting joint country-level evaluations and to provide inputs to refine the scope of the next round of UNDAF evaluations that will be commissioned by the relevant UNCTs. 4. The assessment is based upon a textual analysis of the 35 UNDAF documents completed in 2004 and 2005, where it can be expected that the 2004 Guidelines would have affected the approach adopted both to programme design and monitoring and evaluation and hence their evaluability. This analysis was then strengthened with evidence collected through telephone interviews with 6 selected UNCTs and from the questionnaire completed in mid 2006 by 65 of the possible 80 UNCTs on the effectiveness of the guidelines. The evidence and findings were then triangulated against that found in other reviews of UNDAF performance. Major findings and conclusions Overall findings and conclusions 5. The present Guidelines establish a rigorous set of normative standards for monitoring and evaluation, although possibly due to constraints on the length of document allowed they do not provide detailed guidance on how to implement these normative standards. This finding also applies to the draft revision of the Guidelines circulated in November 2006 to a lesser extent. 6. The overall conclusion is that if fully implemented against these normative standards, the M&E approach proposed would significantly enhance the evaluability of the UNDAFs. This conclusion particularly applies to the definition and relevance of outcomes identified and the existence of sufficient measurable indicators. Norms for joint monitoring systems and especially for assessing risks and assumptions are comparatively less developed, although the draft revision of the Guidelines circulated in November 2006 does strengthen the treatment of key assumptions and risks. UNEG Study on the Evaluability of the UN Development Assistance Framework 3

4 7. However, it is important to understand that the normative standards implied by the Guidelines are extremely challenging to implement. In general, all of the UNDAF documents reviewed show that UNCTs (UN Country Teams) have attempted to follow the M&E approach suggested in the Guidelines, but the evidence and findings suggest that no UNDAF will fully meet the normative standards suggested in the Guidelines. Evaluation of the UNDAFs against the scope suggested in the Guidelines would require significant additional investment in addressing issues not adequately dealt with to date during the design and initial implementation of the UNDAFs. This conclusion is strengthened by the finding of MacKenzie (2006) that most UNCTs are not managing to implement their UNDAF monitoring plans, due to competing demands upon their time. However, it is also concluded that significant opportunities remain to further enhance the evaluability of the UNDAFs. Clarity on the evaluation s purpose? 8. The UNDAF documents, Guidelines and UNCTs interviewed identify no clear and specific accountability demand that the evaluation should meet. Within the Guidelines, the implication is that the UNDAF evaluation should inform the development of the subsequent UNDAF through identification of lessons learned. However, the evidence suggests that UNCTs have given insufficient thought about when the evaluation would need to be scheduled to meet this purpose. The danger is that by the time they do consider this issue properly, it will be too late to implement an effective evaluation. 9. Evaluability is also adversely affected by a lack of clarity on what it is a priority to evaluate. The UNDAF is supposed to be the common strategic framework for the operational activities of the UN system at the country level and provide a collective, coherent and integrated UN system response to national priorities and needs, yet the Results Matrices and indicators do not adequately reflect this. The infeasibility of measuring impact 10. The evidence rather suggests that most UNCTs want an evaluation to identify the impact of the UN. But review of both the documentation and discussion with UNCTs strongly suggests that most in the UNCTs have failed to appreciate that this would probably be impossible in the 4 th year of a 5 year programming cycle. Quite simply, not enough time would have elapsed for the contribution to be detectible in any shift in national level indicators. If this is the major area of interest for UNCTs, then enhancing both evaluability and the utility of monitoring systems needs to start from the question of what it is technically feasible to monitor and then to evaluate during programme implementation and then ensure that M&E systems focus the limited resources at the appropriate level. Sufficient measurable indicators 11. In general, UNCTs appear to have had great difficulty identifying relevant indicators that actually track performance and that are assigned at the correct level in the logical hierarchy. For example, at Country Programme Outcome level, few of the indicators actually identify specific results which an agency could be expected to realise within the time frame of the UNDAF and be held accountable against. The review also found that there is not the evidence required across the 32 UNDAFs reviewed for an evaluator to easily understand the logic between UN interventions, and associated indicators, and expected impact upon the national goals as shown in the Results Frameworks. 4 UNEG Study on the Evaluability of the UN Development Assistance Framework

5 12. The M&E matrices also either have not included any UNDAF Outcome indicators or have pitched them at too high a level in the results hierarchy. This means that there is a lack of indicators on the added value from the UNDAF process itself in delivering a more collective, coherent and integrated UN system response. Quality of joint monitoring systems 13. Neither the CCA-UNDAF Guidelines nor the UNDAF documents discuss the issue of either the capacity or resource requirements required to implement an M&E plan which will deliver against the normative standards implied in the Guidelines. Nor is there evidence that UNCTs have adequately considered how the demands for monitoring and evaluation of the UNDAF should affect the monitoring and evaluation activities of the individual agencies. This is a major over-sight, since until these two issues are considered any M&E plan is de facto no more than a statement of intent. 14. Discussion with six UNCTs also suggests that the M&E plans have not driven monitoring of the UNDAFs by the UN agencies and supports the findings of Mackenzie (2006), as stated below: Current practice is to re-organize theme groups (TGs) around the agreed UNDAF outcomes. These UNDAF outcome groups are meant to use the M&E framework as an operational tool and report on a regular basis to the UNCT about progress. This seems to be happening only rarely. It appears that the pressure for agency reporting to HQs, who are responsible for reporting to executive boards and to other governance bodies takes precedence almost immediately. This is understandable. But programme staff in the field have to fulfil their agency monitoring and reporting tasks, as well as participate meaningfully in theme groups, do joint monitoring and report on overall progress for achievement of the UNDAF outcomes. There is duplication, particularly because each agency maintains its own separate monitoring framework in the CPAP or other project instrument. Overall, it seems very difficult to do both well. Experience from the field suggests that the result is: Outcome groups that meet rarely and almost never meet with all members; Responsibilities that are usually not put into performance appraisal instruments; Group members who are not rewarded for their UNDAF monitoring and reporting efforts; Outcome groups that quickly become paper entities; and most importantly No regular or coherent reporting to the UNCTs about overall progress towards UNDAF outcomes. External factors 15. External factors (risks and assumptions) impact in two ways on the evaluability of the UNDAF: First, those risks and assumptions directly tied to the efficient and effective delivery of interventions by the UN agencies. UNEG Study on the Evaluability of the UN Development Assistance Framework 5

6 Second, those that relate to what it is assumed will be delivered by others, if the UNDAF Outcome is to be delivered. This is required if an attempt is going to be made to evaluate the UN s contribution. 16. Review of the UNDAF documents showed that eight out of 33 UNDAFs include no reference to identified risks and assumptions, but more importantly, in no case, does any UNDAF discuss how risks and assumptions will be monitored. This significantly diminishes the evaluability of the UNDAF s, since if an evaluation cannot directly assess the impact of UN interventions on national goals, it can assess whether or not the assumptions on why the interventions should have such impacts are still valid. 17. The second set of assumptions relate to what others are doing. The UNDAFs do enhance evaluability in that they normally identify who the other key partners in achievement of the UNDAF outcomes are supposed to be. However, since the monitoring systems are not designed to track whether or not partners actually do what is expected, evaluability is still adversely affected. Recommendations 18. The below recommendations focus upon enhancing the evaluability of the UNDAFs and therefore only discuss monitoring in this context. It is also difficult to identify specific recommendations, since their relevance will depend upon whether or not the conclusions and recommendations of the High Level Panel on System Wide Coherence are adopted and implemented. Recommendation 1: The relevant sections of the Guidelines need a further revision. 19. It is recommended that any future revision of the evaluation and monitoring related aspects of the Guidelines needs to be based upon conclusions to the following four questions: Who needs what evidence, when and for what purpose? This would require finding out why UNCTs don t adequately address these issues when developing their UNDAF monitoring and evaluation plans. Is it methodologically possible to collect the needed evidence when it is needed and at what cost? The key recommendation in this report is that both monitoring and evaluation activities need to focus more on whether or not assumptions remain true, since timing means that evaluations or reviews will be formative in nature. Are the resources and capacity likely to be in place to derive the required evidence? Any plan that does not address these two issues during development is not a plan but a statement of intent and highly unlikely to be implemented, as is clearly seen with the present UNDAF M&E plans. What is the minimum level of monitoring and review activity that should be expected of an UNCT? 6 UNEG Study on the Evaluability of the UN Development Assistance Framework

7 Recommendation 2: Drop the requirement for UNCTs to commission UNDAF evaluations. 20. Most UNCTs have neither the expertise nor the resources to commission a good quality evaluation (against the UNEG norms and standards). It is therefore recommended that UNCTs instead carry out less resource intensive reviews of performance against the UNDAF at the beginning of penultimate year as part of the process for developing a new UNDAF. Recommendation 3: Establish what the likely demand for evaluative evidence actually is. 21. Assuming that UNCTs, in future, carry out a review of UNDAF implementation in the penultimate year, this study has not been able to identify any other current demand for evaluation of the UNDAF. How the demand for UNDAF evaluations for accountability purposes within the UN will develop is unknown and will depend upon whether the conclusions and recommendations of the High Level Panel on System Wide Coherence are adopted and implemented. 22. However, as a first step, and to improve demand for both monitoring and evaluation evidence, any future revision of the Guidelines should high-light the need for UNCTs to embed monitoring and evaluation of the UNDAF within the systems that the partner governments are creating, as proposed in the Paris Declaration, to monitor and evaluate donor and government performance against aid-effectiveness commitments. Recommendation 4: Task UN evaluation functions to carry out UNDAF evaluations. 23. Given capacity constraints within both partner governments and UNCTs and the methodological challenges involved, UNDAF evaluations should be managed by UN evaluation functions. If such evaluations are to be commissioned, the central evaluation function should consider: The use of evaluability studies at the start of the UNDAF cycle in targeted countries as an approach to ensuring that sufficient evidence is collected and address the problems UNCTs have in implementing M&E plans. That no comparable evaluations have been carried out to date but that EvalNet s pending evaluation of harmonisation and alignment may provide concrete lessons on evaluation approaches and methodologies that would be relevant to any future UNDAF evaluation. Recommendation 5: UNEG should review M&E guidance for the individual agencies and identify whether they can increase the availability of evidence that could underpin future UNDAF evaluations. 24. In practice, data and evidence for evaluation of an UNDAF is more likely to come from evaluations carried out by the individual UN agencies and by other stakeholders. Therefore, UNEG UNEG Study on the Evaluability of the UN Development Assistance Framework 7

8 should consider whether amending evaluation guidance issued by the individual UN agencies might not be the easiest way to enhance UNDAF evaluability. Recommendation 6: Establish the principle that monitoring from the level of the CP Outcome downwards should be the responsibility of the individual implementing agencies and of national goals should lie with the partner government. Recommendation 7: Focus the UNDAF monitoring framework and the M&E plan on what the added value of a coherent and coordinated UN country programme would be to delivery of national goals. 25. Assuming that the UNDAF actually does lead the programming processes within the individual UN agencies, the principle that monitoring from the level of the CP Outcome downwards should be the responsibility of the individual implementing agencies needs to be established more clearly. This would help ensure that scarce resources within the UNCT are not wasted on developing duplicate UNDAF monitoring systems at these levels and start to address the lack of attention to developing joint monitoring systems within the current Guidelines. 26. The focus of monitoring under the UNDAF process should then be to monitor the degree to which the overall UN programme is likely to add value: Through a more coherent and coordinated approach within UN country programme to delivery of national goals; and Through its influencing and political brokerage role in delivering the UNDAF Outcomes more effectively. 8 UNEG Study on the Evaluability of the UN Development Assistance Framework

9 1. Background and objective of the study 27. In response to the Secretary-General s 1997 call for the United Nations to articulate a coherent vision and strategy for a unified approach towards common development goals at country level, the Common Country Assessment (CCA) and the United Nations Development Assistance Framework (UNDAF) were adopted as strategic planning tools for the UN system. 28. The UNDAF is the common strategic framework for the operational activities of the UN system at the country level. It aims to provide a collective, coherent and integrated UN system response to national priorities and needs, including PRSps and equivalent national strategies, within the framework of the MDGs and the commitments, goals and targets of the Millennium Declaration, summits and instruments of the UN system. The UNDAF emerges from the analysis of the Common Country Assessment an instrument used to analyze the national development situation and identify key development issues and provides a basis for the preparation of UN system country programmes Initially (up to 2004), monitoring and evaluation (M&E) were not addressed in the guidelines issued by the UN Development Group on the development and use of UNDAFs. However, in 2002, an M&E task force was created that produced M&E recommendations in 2004, which were then incorporated into the guidelines issued. These revised guidelines should therefore have affected the 16 UNDAF documents finalised in 2004 and the 19 finalised in This study is expected to provide inputs into the updating of the UNDAF M&E guidance, currently being prepared by UNDG led Working Group on Programming Policy. It is also meant to yield benefits for the on-going process of conducting joint country-level evaluations and to provide inputs to refine the scope of the next round of UNDAF evaluations that will be commissioned by the relevant UNCTs. 2. Scope and methodology 2.1 Scope 31. The study focuses on the 35 UNDAF documents (listed at Annex 1) completed in 2004 and 2005, where it can be expected that the revised Guidelines would have affected the approach adopted both to programme design and monitoring and evaluation and hence their evaluability. 2.2 Methodology 32. The ToRs for this study indicate that the study should assess the evaluability of UNDAF in terms of: a) clarity of intent of the subject to be evaluated (relevance and design of the expected outcome statements and results matrices); b) the existence of sufficient measurable indicators (collection of reliable 1 U.N, Common Country Assessment and United Nations Development Assistance Framework Guidelines for UN Country Teams preparing a CCA and UNDAF, undated, pp.11, 17. UNEG Study on the Evaluability of the UN Development Assistance Framework 9

10 data for analysis); c) quality of joint monitoring systems; and d) external factors (positive or negative) that have influenced the process and the realization of expected outcomes. 33. Assessing evaluability requires making a judgement based on the evidence and therefore needs to be based on the use of commonly agreed normative standards wherever possible and the transparent application of said standards. Therefore, under the four broad questions identified in the ToRs, the UNEG Norms and Standards were used to develop a set of more detailed questions 2. Selection of relevant UNEG Standards was also influenced by review of the: Approach of UNDP when assessing the evaluability of the Assessment of Development Results for Serbia and Montenegro 3. The paper outlining Issues & Recommendations for the revision of the 2004 CCA-UNDAF Guidelines by A. MacKenzie (Sept. 2006). The Review of the Role and Quality of the United Nations Development Assistance Frameworks (UNDAFs) by R. Longhurst (May 2006). Approach to assessing evaluability used by the IADB when assessing the evaluability of its country strategy documents 4 ; 34. Selection of these questions also revealed that it would not be possible to interpret the evidence based on the purely technical aspects of the results framework, without making assumptions on what the purpose of the evaluation would be. Therefore a fifth group of questions was included that focused on the purpose of the evaluation. A listing of the questions used is at Annex The questions developed were then used to assess whether: i. The Guidelines for UN Country Teams (2004) contain the elements required to allow the evaluation of the UNDAFs. ii. For the 35 UNDAF documents finalised in 2004 and 2005, to what degree these documents include the elements that would allow the future evaluation of these UNDAFs. iii. These documents actually followed the relevant suggestions in the Guidelines for UN Country Teams. 2 The UNEG Norms only briefly touch upon evaluability, whilst evaluability is not mentioned in the UNEG Standards at all. However, many of the Standards directly deal with the issue of evaluability and therefore can be grouped under the four broad questions identified in the ToRs. 3 Assessment of Development Results Serbia and Montenegro. Inception Report, August Report on the Evaluability of Bank Country Strategies, Office of Evaluation and Oversight, Inter-American Development Bank, September UNEG Study on the Evaluability of the UN Development Assistance Framework

11 36. The above assessment was based upon a textual analysis of documentation. However, this approach has three key limitations. First, a lack of documentary evidence does not mean that something has not been done in reality. Second, the degree to which something can be evaluated depends upon the capabilities of those commissioning and implementing the evaluation and the level of resourcing that one is willing to commit to the evaluation. Third, the CCA and UNDAF documents are not the only sources of information and evidence that affect the evaluability of the UNDAF. Therefore, evidence on these three key issues was collected through telephone interviews with 6 selected UNCTs 5 and from the questionnaire completed in mid 2006 by 65 of the possible 80 UNCTs on the effectiveness of the guidelines. The evidence and findings were then triangulated against that found in other reviews of UNDAF performance. 3. Findings 3.1 Do the Guidelines for UN Country Teams (2004) contain the elements required to allow the evaluation of the UNDAFs? 6 Finding 1: Do the Guidelines cover the purpose, scope and management of the evaluation? 37. The Guidelines are not clear on what the purpose of the evaluation is purpose is defined here as being for either accountability or lesson learning or a mixture of the two. The suggested timing at the beginning of the penultimate year implies that the evaluation is intended to inform development of next UNDAF, which would suggest that its primary purpose is to learn lessons. However, Section 4.3 of the Guidelines, which covers the suggested scope of the evaluation, makes little reference to drawing lessons for the development of the next UNDAF. Neither does the guidance on the development of the CCAs make any substantive reference to using lessons learned from evaluations of past performance as a criterion for selection of future areas of cooperation. 38. The suggested scope of the evaluation is clearly set out in the Guidelines (Section 4.3), as shown below: 7 5 Bangladesh, Serbia, Swaziland, Guyana, Armenia, and China. 6 Evidence supporting thee below findings is summarised at Annex 3. 7 The scope of the UNDAF evaluation defined in the November 2006 draft revision of the Guidelines is as follows: (i) Did the UNDAF make the best use of the UN s comparative advantages in the country?; (ii) Did the UNDAF generate a coherent UNCT response to national priorities?; and (iii) Did the UNDAF help achieve the selected priorities in the national development framework? This represents a narrowing of the scope to focus on areas (a), (b) and (f) of the evaluation scope identified within the current Guidelines. UNEG Study on the Evaluability of the UN Development Assistance Framework 11

12 a. Impact To what extent has progress in attaining the UNDAF Outcomes impacted national development? How has the progress in attaining the UNDAF outcomes benefited the poorest and most vulnerable people? b. Relevance - Did UNDAF outcomes strategically position the UN within the development community, especially in pursuit of national MDGs? Are the outcomes still valid for the next UNDAF? c. Sustainability Are positive changes in the development situation sustainable? To what degree have strategies and programmes under the UNDAF been institutionalized? Have complementarities, collaboration and/or synergies fostered by the UNDAF contributed to sustainability? How have national capacities at the levels of government, local community, NGOs and civil society been enhanced? d. Effectiveness vis-à-vis UNDAF and CP outcomes What progress has been made towards the CP and UNDAF outcomes and CP outputs? e. Efficiency Were results achieved at reasonably low or lowest cost? How did strategies result in a more efficient, simplified and harmonised UN? f. Effectiveness of the UNDAF as a coordination framework - Has the UNDAF contributed to more complementary and collaborative programming by agencies? Did the UNDAF make programming by agencies more strategic and synergistic? Has value been added by these synergies? Has effectiveness been enhanced? Finding 2: What relevant guidance is given on constructing a plausible logic model connecting UN support with national development goals? 39. The Results and Monitoring Matrices should effectively summarise: What the UN agencies intend to do; Expected results at UNDAF Outcome, Country Programme Outcome and Country Programme Output level; How, in broad terms these link to particular national goals and outcomes; Who the agencies will be working with (both inter-un agency and other partners) and the relative contributions being made by the partners; and What the UN agencies consider the main risks and assumptions to be. 40. They should therefore provide an evaluator with a baseline of the situation at the start of an UNDAF s implementation and the logic assumed to underpin proposed interventions. 41. For the logic linking UN interventions with national goals, the CCA guidance clearly high-lights that the better the analysis of cause and effect is, the more accurate will be the resulting development 12 UNEG Study on the Evaluability of the UN Development Assistance Framework

13 assistance frameworks in identifying steps necessary for the achievement of national and international goals. For instance the guidance points out that a statement about the inadequacy of certain sector policies is not sufficient to guide development interventions; at the minimum the missing or unsupportive elements should be indicated. In fact, a significant part of the guidance on CCAs is on approaches to analyzing the situation and identifying the immediate and root causes. Hence, the assumption should be that the detailed cause and effect logic required by an evaluation of the UNDAF outcomes should be found in the CCA, rather than in the UNDAF, given that it would be difficult to include this level of detail in an UNDAF that should not exceed 15 pages in length. 42. The Guidelines also state that the UNDAF emerges from the analyses of the CCA and is the next step in the preparation of United Nations system country programmes and projects of cooperation 8. Within the Guidelines, the first identified step in development of the UNDAF is selection among, and agreement with Government on, areas of cooperation that will be taken forward by the UN. It is recommended that this process be carried out in a Prioritisation Workshop. However, since a record of the discussion within the Prioritisation Workshop is not appended to the UNDAF, the danger is that an insufficiently detailed rationale for selection of the areas of cooperation will be included in the UNDAF. In this case, insufficient is defined as meaning not enough evidence to allow robust examination of the logic underpinning selection of areas of cooperation. 43. It is during the Prioritisation Workshop that the UNCT are supposed to also identify between three and five UNDAF Outcomes. 9 However, whilst Guidelines clearly define what an UNDAF Outcome is, they do not outline how the UNCT is supposed to derive the UNDAF Outcomes from the areas of cooperation. The danger is that not spelling out this difference clearly would mean that the two are seen as being the same thing, when in fact an UNDAF Outcome should be a much more specific and concrete statement of what results the UN will commit to deliver. 44. Finally, those responsible for assessing the quality of UNDAF s should assess the robustness of the logic presented, when answering Question 7 in the Quality Review Template, which specifically asks whether the UNDAF convincingly presents a strategic & results driven framework for UN agency programming to address poverty challenges prioritized in the CCA. Finding 3: What does the results matrix seek to measure? 45. The guidelines state that linkages between national goals or targets (as related to specific MDGs and/or other international commitments, goals and targets) and UNDAF and Country Programme Outcomes are elaborated within the Results Matrix, together with resource requirements. For each outcome, the Matrix illustrates how the UNDAF guides the design of agency-supported Country 8 Part 3.1, Definition of an UNDAF 9 An UNDAF Outcome is defined as the specific results which the United Nations system expects to realize within the time frame of the UNDAF as its contribution towards the achievement of the national development priorities and goals in each area of cooperation. The United Nations system is collectively accountable for these outcomes, working in collaboration with the Government and other development partners, and must be able to demonstrate progress towards their achievement. UNEG Study on the Evaluability of the UN Development Assistance Framework 13

14 Programmes and projects and parallel or joint programming (see the Guidance Note on Joint Programming 19 December 2003). Conversely the Matrix also shows how the major outcomes of agency-supported Country Programmes and projects lead to the achievement of the shared UNDAF outcomes that exceed the sum of individually planned efforts. 46. Before, examining the UNDAF guidelines further, it is necessary to first be clear on what the expected added value of having UNDAF results and monitoring frameworks might be for an evaluator. Looking at the counterfactual scenario, it can be assumed that: Monitoring and analysis at the national goal level would occur anyway and is unlikely to be affected by the presence or absence of UNDAF results and monitoring frameworks and there is little reason to believe that the existence of the UNDAF leads to additional UN support in this area over and above that which would be provided by the individual UN agencies anyway. Monitoring and evaluation by the individual UN agencies would also occur anyway and given the lack of resourcing would be the major source of information on UN agency performance even in the presence of an UNDAF results and monitoring framework. Monitoring and evaluation of joint or collaborative programming between two or more agencies might be affected by the presence, or not, of the UNDAF results and monitoring frameworks, but the Guidelines do not discuss this issue. Prioritisation, at either the level of UNDAF Outcomes or below would be an exogenous factor to the results and monitoring frameworks. This is because prioritisation is dealt with before the results and monitoring frameworks are designed and the frameworks contain no information on what the alternatives open to the UNCT were. 47. For an evaluator, the added value from the matrix would logically be in the degree to which it can show how the major outcomes of agency-supported Country Programmes and projects lead to the achievement of the shared UNDAF outcomes that exceed the sum of individually planned efforts. This is effectively the raison d etre for the UNDAF process. However, within the Guidelines, there is no guidance upon how UNCTs might define UNDAF Outcomes that reflect this additionality, nor is the issue of additionality flagged in the sections dealing with analysis during the CCA process or in the criteria suggested for prioritising and selecting between possible areas of cooperation. The risk therefore is that the results and monitoring matrices may fail to incorporate additionality and therefore this issue is not monitored. Finding 4: What type of indicators does the Guidance suggest be used? 48. The Guidelines give a clear set of normative standards on what indicators should be included, as stated below 10 : 10 It should be noted that those responsible for monitoring the quality of the UNDAF document are not asked to assess the degree to which the indicators included in the UNDAF monitoring matrix meet these standards. 14 UNEG Study on the Evaluability of the UN Development Assistance Framework

15 The framework lists, for each UNDAF outcome and related CP outcomes, one or more quantitative and/or qualitative indicator(s) for monitoring progress, including baseline data and sources of verification as well as risks and assumptions. CP outputs are also listed and indicators and baselines added when available. Indicators, including baselines, should be disaggregated by gender and any additional characteristics that may be relevant to disparities between population groups in the country. 49. The Guidelines suggest that the indicators be formulated, together with the relevant national institutions, in such a way that progress towards their achievement can be objectively verified, by direct measurement or other means. The Guidelines however provide no examples of what indicators that could be used at either UNDAF or Country Programme Output level might look like or how they might be developed. Finding 5: What Guidance is given on the use or collection of other evaluative evidence? 50. The Guidelines briefly note the need to identify data gaps, but this is not then reflected in the issues covered in the Monitoring & Evaluation Programme Cycle Calendar, so increasing the danger that UNCT s will not identify gaps in time to ensure that measures are put in place to fill the gaps. 51. The Guidelines do not explicitly discuss the role of evaluation data from evaluations carried out by individual agencies in evaluation of the UNDAF. Nor do the Guidelines flag that such evaluations are likely to be the main sources of evidence upon which the evaluation of the UNDAF would be based. Finding 6: Treatment of risks and assumptions 52. The Guidelines do suggest that risks and assumptions be identified in the monitoring matrix. However, the Guidelines do not suggest that programmes monitor these on an on-going basis or identify risk mitigation strategies. 3.2 Do recent UNDAF documents include the elements that would allow the future evaluation of these UNDAFs? Finding 7: Definition of purpose and scope in UNDAF documents 53. The Guidelines were not explicit on what the purpose of the evaluation would be, but were clear on suggesting what its scope might be (see para. 13). Table 1 below presents an analysis from 33 of the 35 UNDAF documents agreed in 2004 and 2005 of what evaluation purpose was specified and the scope of the evaluation Only 33 of the 35 UNDAF documents completed in 2004 and 2005 were included, since the reviewer s Spanish wasn t good enough to credibly review the two documents that were drafted in Spanish Peru and Guatemala. UNEG Study on the Evaluability of the UN Development Assistance Framework 15

16 Table 1: Number of UNDAF documents (n=33) specifying the evaluation purpose and scope of the evaluation. Scope of evaluation Evaluation purpose Lesson learning Accountability Both Not Specified Total Impact on national goals Relevance of UNDAP outcomes Sustainability Effectiveness of implementation Efficiency of implementation Effectiveness of the UNDAF as a coordination framework Not specified The table reveals four major findings: None of the UNCTs appear to see the function of the evaluation as being to meet accountability needs. This finding was confirmed in discussions with UNCTs, who did not identify any current specific and operational accountability function for the evaluation. Two UNCTs included no scheduled UNDAF evaluation within their UNDAF documents. Half of the UNDAF documents identified evaluating the impact of the UNDAF Outcomes on the national goals as a role of the evaluation. Approximately a third of the UNDAF documents identified evaluating the effectiveness of the UNDAF coordination framework as a role of the evaluation. 55. No UNDAF document identified in anything but the broadest terms who would be responsible for the planning and implementation of the evaluation. Nor were the milestones for the planning and implementation of the UNDAF evaluation included in the Monitoring & Evaluation Programme Cycle Calendar included in finalised UNDAFs, as had been suggested in the Guidelines. Finding 8: Do UNDAF documents include an evidence based rationale for selection of the UNDAF outcomes? 56. All UNDAF documents state that the UNDAF Outcomes were defined through Prioritisation Workshops and most state that the criteria identified in the Guidelines were used during the Prioritisation process. However, none of the documents provide details of what evidence was used and how these criteria were applied. The seven UNCTs contacted all stated that they had documented the proceedings of their Prioritisation Workshop and retained these in their records, which implies that the basic information required for evaluating the relevance of the Outcomes selected is available, even if not in the UNDAF document itself. 16 UNEG Study on the Evaluability of the UN Development Assistance Framework

17 Finding 9: Is there sufficient evidence/information for an evaluator to develop or understand the logic between UN interventions and impact upon the national goals? 57. The main finding is that there is not the evidence required across the 32 UNDAFs reviewed for an evaluator to easily understand the logic between UN interventions and impact upon the national goals as shown in the Results Frameworks. Issues include that: In only 24 of the 32 UNDAFs were the problems impeding achievement of the national goals clearly identified. In the other eight cases the documents started from government goals/targets but did not identify the problems with achieving those targets/goals. However, in theory this issue can be dealt with by reference to both the CCA and PRSp documentation by an evaluator. In only 11 of the 32 cases was there any discussion of what the causes of the problem to be addressed were. In no case, was any evidence presented that would justify the assertions made of what the causes were, although review of a sample of CCAs suggests that it is possible in some cases to find detailed discussions of the causes, and sometimes supporting evidence. In only two cases, was there any discussion of why, and how, it was expected that the identified UN interventions would affect either the problem or cause identified. This makes development or review of the logic chain more challenging for the evaluator, because this information is not included in other documents either. In the case of the CCA, there is no requirement to discuss this issue. Programme or project documents, on the other hand, may look at the logic linking achievement of a particular CP Outcome with a national goal, but don t discuss the logic of why the sum of all of the CP Outcomes under one UNDAF Outcome will affect achievement of the national goal. Finding 10: The utility for evaluation of UNDAF and CP Outcomes 58. As shown in Table 2, all UNDAF documents reviewed, barring one that included no results framework in the version publicly available, identify UNDAF and CP Outcomes. Thirty of the 32 frameworks also include CP Outputs. Table 2: Number of UNDAF documents (n=32) including UNDAF Outcomes, CP Outcomes and CP Outputs by year of preparation 2004 UNDAF documents 2005 UNDAF documents UNDAF outcomes CP outcomes CP outputs No results framework However, the usefulness of the outcomes described for evaluation purposes is highly variable. UNEG Study on the Evaluability of the UN Development Assistance Framework 17

18 60. At the UNDAF Outcome level, UNCTs appear to find it is difficult to define Outcome statements that outline the specific results which the United Nations system expects to realize within the time frame of the UNDAF as its contribution towards the achievement of the national development priorities and goals in each area of cooperation. Nor do UNDAF Outcome statements identify the proposed additionality that is supposed to be derived from the interaction of results under the supporting CP Outcomes. Discussion with the seven UNCTs suggests that this may reflect the difficulty for UNCTs in defining precise UNDAF Outcome statements that are inclusive enough to cover the support across all UN agencies. This in turn reflects a tension under-lying the UNDAF approach. Namely, how do UNCTs balance the need to be inclusive and use the framework as an inclusive big tent against using the UNDAF framework and process as a way of focusing and prioritising the combined efforts of the agencies. Finally, the utility of UNDAF Outcomes to an evaluator is severely compromised because nearly half of the UNDAFs include no indicators at this level, and, of the remainder, only 4 UNDAF s include indicators that aren t national government targets. 61. CP outcomes are intended to describe the intended results to which a specific agency-supported Country Programme, rather than the UN as a whole, contributes. CP Outcome statements are more specific and useful for evaluation purposes than those at UNDAF level. Finding 11: The utility of UNDAF and CP Outcome indicators for tracking performance and delivery 62. Overall, monitoring matrices include examples of both quantitative and qualitative indicators. Matrices also either include baselines, where available, or note in the narrative that baselines will be established in the first year of implementation. 63. But as already discussed, in only 4 of the 33 UNDAFs, were UNDAF Outcome level indicators included that could be said to attempt to track performance by the UN 12. In other cases where indicators were included at this level, indicators were usually actually indicators of movement at the level of national goals, and therefore too high level to be useful as indicators at UNDAF Outcome level. Discussion with the seven UNCTs confirms the finding from MacKenzie (2006) that definition of indicators was difficult for UNCTs. 64. Whilst all UNDAF s identified CP level outcome indicators, the major findings are that: Few of the indicators actually identified specific results which an agency could be expected to realise within the time frame of the UNDAF. In the 32 monitoring matrices reviewed, in only four cases did the CP outcome indicators consistently attempt to identify agency results, whilst in a further 11 cases this principle had been applied to some extent. In the remaining 17 cases, the judgement is that the indicators did not identify results at the required level. 12 The criterion used here was that the indicators attempted to measure a change in institutional capacity. 18 UNEG Study on the Evaluability of the UN Development Assistance Framework

19 In only two cases, was there clear evidence that the matrix contained indicators that would allow some assessment of progress towards achievement of the CP Outcome whilst instances of indicators containing dates by when they should be achieved were rare. No indicators were included that monitored either whether strategies were resulting in a more efficient, simplified and harmonised UN or the effectiveness of the UNDAF as a coordination framework. Finding 12: Do the UNDAFs identify what the expected contributions of other partners will be? 65. The UNDAF documents do identify who the UNCT expect the major partners to be, although this principle is inconsistently applied. For instance, some documents identify who the other major donors would be, whilst others make no mention of who these donors might be. Again, some UNDAF documents identify what they expect each partner s role to be, in broad terms, whilst most merely identify who the partners will be. None of the UNDAF documents identify the relative levels of inputs by each partner or identify which non-unct partners are key for delivery of UN Outcomes. Finding 13: What evidence is there of planning for the evaluation? 66. Beyond stating when the evaluation will occur, there is no evidence in the UNDAF documents that UNCT s have considered the requirements for evaluation of the UNDAF when setting up the M&E systems. This was also confirmed in discussion with the seven UNCTs. Finding 14: Treatment of risks and assumptions (external factors) 67. According to the Guidelines, risks and assumptions should be dealt with in two places in the UNDAF document. First, a description of major risks and assumptions which may affect the achievement of UNDAF outcomes should be included in the narrative. Second, risks and assumptions for each UNDAF and country programme/project outcome should be included in the Monitoring and Evaluation Framework. Review of the UNDAF documents shows that eight out of 33 UNDAFs include no reference to identified risks and assumptions. In no case, does any UNDAF discuss how risks and assumptions will be monitored. 3.3 Do recent UNDAF documents follow the relevant suggestions in the Guidelines for UN Country Teams? 68. Table 3 below summarises the degree to which UNCTs have attempted to incorporate the relevant sections of the Guidelines within their UNDAF documents. Issues of the quality of what has been included are not addressed in the table, since it is covered in the section 3.2 above. UNEG Study on the Evaluability of the UN Development Assistance Framework 19

20 Table 2: Analysis of implementation of M&E guidance by year of UNDAF preparation Included in UNDAF document? 2004 UNDAF 2005 UNDAF Overall Yes No Yes No Yes No 1. Frameworks 1.1. Results Matrix included Result matrix includes all suggested topics M&E Framework M&E Framework follows suggested structure M&E Programme cycle calendar Inclusion of rationale for selection of UNDAF outcomes Outcomes 2.1 UNDAF outcomes CP outcomes Indicators 3.1 UNDAF outcome CP outcome Baselines Disaggregated by gender and vulnerable group M&E Management 4.1 Monitoring management identified Building national monitoring capacity Evaluation Management identified Evaluation planning and implementation milestones Scheduled for penultimate year Finding 15: Do the UNDAF documents show evidence of being guided by the CCA-UNDAF Guidance? 69. In general, all of the UNDAF documents follow the structure suggested in the Guidelines and there appears to be little difference between those completed in 2004 and in With specific reference to monitoring and evaluation, the UNDAF documents also follow the Guidelines in the following areas: Structure and broad content of the Results and M&E Frameworks; 20 UNEG Study on the Evaluability of the UN Development Assistance Framework

21 Inclusion of UNDAF Outcome and CP Outcome statements; Inclusion of CP Outcome indicators; Inclusion of a rationale for choice of UNDAF outcomes; Definition of who will be responsible for monitoring during implementation of the UNDAF; and Building of national capacity. 70. UNDAF documents diverge from the Guidelines in the following areas: Inclusion of an M&E Programme Cycle Calendar; Inclusion of UNDAF Outcome indicators; Inclusion of baselines for indicators; Definitions of the evaluation planning and implementation milestones; and The timing of the evaluation. 3.4 Capacity and resourcing for evaluation of UNDAFs Finding 16: The lack of evidence that capacity and resourcing for evaluation is considered during design of the UNDAF 71. Neither the CCA-UNDAF Guidelines nor the UNDAF documents discuss the issue of either the capacity or resource requirements required to implement an M&E plan which will deliver against the normative standards implied in the Guidelines. This is a major over-sight, since until these two issues are considered any M&E plan is de facto no more than a statement of intent. Discussion with the seven UNCTs suggests that the M&E plans have not driven monitoring of the UNDAFs by the UN agencies and support the findings of Mackenzie (2006), as stated below: Current practice is to re-organize theme groups (TGs) around the agreed UNDAF outcomes. These UNDAF outcome groups are meant to use the M&E framework as an operational tool and report on a regular basis to the UNCT about progress 13. This seems to be happening only rarely. It appears that the pressure for agency reporting to HQs, who are responsible for reporting to executive boards and to other governance bodies takes precedence almost immediately. This is understandable. But programme staff in the field have to fulfil their agency monitoring and reporting tasks, as well as participate 13 Please note that this is current practice the guidelines do not describe or suggest this. However, the following are often referred to: GlobalNet Digest discussion on Theme groups, September 2, 2003 to May 24, 2004, and Basic principles for CCA/UNDAF theme groups, UNDG #5834, UNEG Study on the Evaluability of the UN Development Assistance Framework 21