UNICEF Evaluation Management Response 1

Transcription

1 UNICEF Evaluation Management Response 1 Review Title: DAC/UNEG Peer Review of the Evaluation Function of UNICEF Region: Global Office: New York headquarters Evaluation Year: 2017 Person-in-charge for follow-up to management response: Executive Director/Deputy Executive Director, Management Overall response to the Peer Review: We welcome the Peer Review as an opportunity to take stock and consider the steps to improve the evaluation function of UNICEF across the organization. The 2016 Multilateral Organization Performance Assessment Network (MOPAN) institutional assessment of UNICEF, the 2016 Evaluation of UNICEF Development Effectiveness, this Peer Review, and the Evaluation Office internal self-assessment preceding it all noted that the UNICEF evaluation function needs to fully achieve the purposes of accountability and learning in a large, decentralized and complex organization. The evaluation work is being implemented in coordination with other United Nations entities and partnerships with the private sector and civil society. The Peer Review notes that: the quality of evaluation has improved over time; there is significant improvement in compliance with management responses; the Evaluation Office plays an active role in the international evaluation community and in strengthening national evaluation capacity development; and UNICEF is committed to achieving the target of allocating a minimum of 1 per cent of total programme expenditure to evaluation. Despite these improvements, the overall assessment of the Peer Review against the three core principles of independence, credibility and utility rated the UNICEF evaluation function either short of satisfactory or close to satisfactory. Remedies for shortcomings identified will include a new evaluation policy that clearly defines the accountability framework for evaluation, a human-resource strategy to strengthen evaluation capacity at all levels of the organization, and further professionalization of the evaluation function. All the above can improve the selection, timeliness, quality and use of evaluations at corporate, regional and country-office levels. The key instruments to systematically address issues of independence of the evaluation function are a revamped evaluation policy (and relevant executive directives) and the strengthening of independent oversight of the evaluation function. There is also the need to improve performance (delivery and quality) of the evaluations at both corporate and decentralized levels. Planned use of the Peer Review: To strengthen the UNICEF evaluation function to assure independence, credibility and utility in line with the United Nations Evaluation Group (UNEG) norms and standards. 1 This Management Response to the Peer Review of the UNICEF Evaluation Function has been updated since the Second Regular Session of the Executive Board in September The update is based on a request by several members of the Executive Board during the Second Regular Session to extend the period of consultation for the revised Evaluation Policy. Consequently, the timeline for all activities related to the revised Evaluation Policy has been extended to June 2018 when the revised Policy will be presented to the Executive Board.

2 Peer Review of the Evaluation Function: Recommendations Management response Person/office responsible Timeline Evaluation recommendation 1: An independent and decentralized UNICEF Evaluation Function A new policy for the UNICEF evaluation function should be developed, that takes full account of the decentralized structure of the organization and integrates all the requirements for the independence, credibility and professionalization of the function, in line with the 2016 UNEG Norms and Standards, and with the other recommendations and actions proposed by the Peer Review. Management response: Agree. UNICEF will prepare a new evaluation policy that will take into account the UNEG norms and standards, and will address the requirements of the UNICEF decentralized structure. The process of developing the new policy will include: (a) an analysis of the implementation of the current policy; (b) recommendations from the external Peer Review, the Evaluation Office internal self-assessment and other recent assessments, including the 2016 MOPAN assessment of UNICEF, the new UNICEF Strategic Plan, and EvalSDGs; and (c) consultations with the UNICEF Executive Board and relevant UNICEF staff at all levels. In revising its Evaluation Policy, UNICEF will consider a number of related issues that have been highlighted by external reviews and internal assessments and that have a bearing on the recommendations of the Peer Review: (a) How best to strengthen the evaluation function, noting the need to customize the UNEG norms and standards to the needs and capacities of a decentralized organization, for both accountability and learning; (b) The need for the evaluation function to be forward-looking to address the priorities of the UNICEF Strategic Plan, , the imperatives of the quadrennial comprehensive policy review of operational activities for development of the United Nations system, including for system-wide evaluation, and the report of the Secretary-General on repositioning the United Nations development system to deliver on the 2030 Agenda for Sustainable Development; (c) The need to pursue evaluation methods and systems that focus on learning and accountability for realizing children s rights and that contribute to organizational innovation, adaptive management, understanding of complexity and assessment of how partnerships and alliances can be catalytic and transformative in addressing the great challenges of our time; (d) In response to the increasing operational focus of UNICEF in fragile and humanitarian settings, the need for the evaluation function to meet the emerging requirements specific to evaluation of humanitarian action, as well as to support the coherence between humanitarian and development programming within the evaluation policy; (e) The need to strengthen learning from timely evaluations in general and take into consideration issues that are more strongly cross-cutting (gender, equity, human rights, etc.); (f) Outlining the UNICEF contribution to national evaluation capacity-building in all country contexts; 2/12

3 (g) Making provision to meet the target of at least 1 per cent of programme expenditure on evaluation; (h) Enhancing the synergies and coordination between audit, evaluation, data, research and knowledge management for more coherent learning and evidence-generation in a knowledge-based organization. The new policy and associated directives and plans will address the specific complementary measures of the recommendation in a comprehensive manner. The policy will be submitted to the UNICEF Executive Board for approval at its first regular session of Specific complementary measures for the implementation of recommendation 1 include: 1.1 Establish dual reporting line for Regional Evaluation Advisers: a direct administrative reporting line to the Regional Director, who would also ensure financial resources for the position; and a technical reporting line to the Evaluation Director in headquarters. Agree. This will be reflected further in the new evaluation policy in the context of the accountability of Regional Directors and the Evaluation Director. Please also refer to Recommendation 4.3 Office of the Executive Director (OED) June Develop adequate impartiality provisions and safeguards for the behavioural independence of staff with responsibility in planning, commissioning and managing evaluations. 1.3 Make publicly available on UNICEF s external web site, all Management Responses to evaluations. 1.4 Revamp the Global Evaluation Committee as a platform for substantive discussion between the EO and UNICEF Management on, among others: evaluation topics and planning also on strategic management issues; emerging and compliance issues; sharing of key evaluation findings of corporate relevance. Agree. Impartiality provisions to protect behavioural independence will be included in the new evaluation policy and specific guidance developed. Agree. UNICEF already makes all its evaluation reports publicly available, and a system for making the management response also public will be instituted. Agree. UNICEF will revise the Global Evaluation Committee terms of reference along the lines recommended. OED June 2018 OED Immediate OED December /12

4 1.5 Development of a Theory of Change for the UNICEF evaluation function. Agree. A draft Theory of Change exists but will need to be included in the revised policy. Evaluation Office (EO) Director in consultation with OED June Development of a Policy implementation strategy, in the form of an Executive Directive, for the future evaluation policy, to guide its operationalization. Agree. This accords with past practice. The executive directive will explain how the policy will be implemented. OED September 2018 Evaluation recommendation 2: The internal governance of the evaluation function UNICEF should assign to the EO, full responsibility for the internal governance of the evaluation function, and adequate resources for fulfilling this role. Management response: Agree. The Evaluation Office will be strengthened to enhance its role in setting standards for utility, coverage and quality assurance. The Office will retain overall technical oversight of the function with Regional Evaluation Advisors technically reporting to the Evaluation Director. Specific complementary measures for the implementation of this recommendation 2 include: The EO should take the lead, with inputs from the Regional Evaluation Advisers and in consultation with other relevant stakeholders in the organization, on the following actions among others: 2.1 Revision of the Key Performance Indicators (KPIs) on evaluation coverage at country level and on the budget allocated to the evaluation function to simultaneously achieve: adequate coverage of UNICEF s work; more accountability for the function itself; and more flexibility to adjust to the very diverse contexts and circumstances of UNICEF country and regional offices. Agree. Taking the UNEG norms and standards and what is appropriate in the diverse contexts and circumstances of UNICEF country and regional offices, UNICEF will address the issues identified under this specific complementary measure. EO Director March Ensuring that the Global Evaluation Plan includes strategic corporate-level evaluations, as well as evaluation syntheses or meta-evaluations that draw Agree. The Global Evaluation Plan (GEP) already includes corporate-level evaluations and, where required, these are coordinated with regional and country EO Director to draft new Global Evaluation Plan in March /12

5 on country and/or regional level evaluations identified in consultation with the Regional Evaluation Advisers. 2.3 Country level Costed Evaluation Plans (CEPs) should be discussed with both Regional Evaluation Advisers and the EO; these CEPs should include to the extent possible evaluations that will feed into regional, multi-country and global evaluations. 2.4 Regional Costed Evaluation Plans should be discussed with the EO and include as appropriate, evaluations that will feed into corporate-level evaluations. 2.5 Development of a corporate strategy for National Evaluation Capacity Development (NECD), taking into account the evolution of the debate on NECD within UNEG. costed evaluation plans. This model, which is essential to ensuring adequate oversight for organization-wide strategic evaluations, will be retained. Evaluation syntheses/meta-evaluations will take into consideration evaluations undertaken by regional and country offices. Agree. The Evaluation Office will work with regional offices to provide increased oversight and advice. Agree. The Evaluation Office will work with regional offices to strengthen support and guidance on costed evaluation plans and monitor their quality through the Global Evaluation Reports Oversight System. Agree. UNICEF will work with UNEG, the United Nations Development Programme and the United Nations Entity for Gender Equality and the Empowerment of Women (UN-Women) to coordinate the organization s contribution to NECD and outline a UNICEF-specific strategy for NECD. consultation with GEC and Regional Evaluation advisers EO Director March 2018 EO Director March 2018 EO Director March 2018 Evaluation recommendation 3: Financial resources for the evaluation function UNICEF should establish new modalities for the funding of the evaluation function at all levels. Management response: Agree. 5/12

6 UNICEF will achieve the target of allocating a minimum of 1 per cent of programme expenditures for evaluations in the period of the Strategic Plan, , by investing regular (core) resources (RR) and other resources (OR) at all levels of the organization. In this context, provisions have been made for a 33 per cent increase in the Integrated Budget of UNICEF, , (from RR) for evaluation. The proposed increase will help to strengthen the capacity of regional offices and headquarters to produce and disseminate independent, credible and useful evaluations. It is also expected that partners will increase OR to leverage the projected growth in the allocation of RR, especially towards high-quality evaluation products and processes at the country level. Specific complementary measures for the implementation of Recommendation 3 include: 3.1 The allocation of 1% of the financial resources of the organization spent on evaluation should be a target at the regional level, to enable a flexible and more efficient use of resources. 3.2 All EO and Regional evaluation staff positions should be funded through Regular Resources, or Programme Resources transferred to the EO and to the Regional Directors, managed under their respective direct responsibility. 3.3 A sustainable pool funding mechanism should be developed, to leverage resources from headquartersbased programme divisions and from country-offices, to fund evaluation specialist positions that cannot be funded through Regular Resources, and for conducting evaluations at the regional and/or multi-country level. Disagree. It is not advisable to have mechanical regional and country targets in the distribution of financial resources; instead, they will be based on criteria and efficient use of evaluation resources for coverage. The goal is value-for-money for learning and accountability and taking into consideration the uniqueness of particular country situations for example, those responding to humanitarian crises. Agree. However, UNICEF proposes a phased approach. The Integrated Budget, , has provided for six new positions on RR, including three P-5 posts at the regional level. OR will be needed to fill funding gaps during this period. Agree. This has been proposed in the recently submitted Office Management Plan (OMP), OED January 2020 OED June /12

7 3.4 The multi-country evaluation specialist model tested in Cambodia, Malaysia and Myanmar, should be replicated across groups of countries where this may prove appropriate and useful to overcome scarcity of resources for evaluation at country-level. Agree. UNICEF agrees that the multicountry evaluation specialist model has proven to be successful, and where appropriate the model would be replicated. UNICEF will continue to consider other options, including long- term agreements. EO Director Ongoing Evaluation recommendation 4: Human resources for the evaluation function UNICEF should develop a Strategic Human Resource plan for the evaluation function. The Plan should also contain a section that defines the provisions for the selection and appointment of the Director of the Evaluation Function. Management response: Agree. Further professionalization of the UNICEF evaluation function requires a strategy for human resource development that includes capacity-building for evaluation staff at all levels. This strategy will draw on the recently completed UNEG Evaluation Competencies Framework. The Strategic Human Resource plan for the evaluation function should include the following provisions: 4.1 Establishment of the position of an EO deputy director, at D-1 level. 4.2 Ensuring that the Evaluation Director and Deputy Director together provide credible evaluation experience and competence as required by UNEG Norms and Standards and by the UNEG Competencies Framework for UNEG Heads. 4.3 Establishment of the position of Regional Evaluation Adviser at the P-5 Agree. The establishment of an EO deputy director at D-1 level will be phased-in based on available resources as part of the midterm review of the UNICEF Strategic Plan, Agree. UNICEF agrees that leadership of the Evaluation Office requires senior staff with strong skills in management and leadership, as well as experience in evaluation matters. The new evaluation policy will address related provisions in UNEG norms and standards and related guidance. Agree. Two regional offices (out of seven) currently have dedicated P-5 Evaluation Advisors. Three new positions of P-5 Regional Advisor have been approved for OED January 2020 OED January 2020 OED January /12

8 level in each and all UNICEF Regional Offices. 4.4 Through gradual re-allocation of resources over time, establishment of teams of full-time evaluation advisers and specialists in each region, at regional, multi-country and national level where justified, by consolidating the resources currently used for monitoring and evaluation specialists at country level. 4.5 Development of specific provisions for the rotation of evaluation staff in UNICEF that allows staff to pursue a career in evaluation whilst maintaining their behavioural independence. the period. The remaining regional functions will be reviewed as part of the midterm review of the UNICEF Strategic Plan, Partially Agree. UNICEF recognizes the advantages of building and maintaining a cadre of full-time evaluation specialists within the organization. However, the funding cannot be reallocated away from programme needs. Thus, additional resources will be sought. Agree. UNICEF will ensure that clear provisions for behavioural independence are provided in the new evaluation policy in line with standard practice. OED January 2020 OED June 2018 The Strategic Human Resource plan for the evaluation function should include the following provisions for the selection and appointment of the Director of the UNICEF Evaluation Function: 4.6 The title of the position should become UNICEF Evaluation Director. 4.7 The Director of the Evaluation Function should be selected and appointed in agreement between the Executive Director and the Executive Board. Agree. This will be reflected in the new evaluation policy. Agree. While the management of the evaluation function will remain with the Office of the Executive Director, the Executive Director will consult the Audit Advisory Committee and UNICEF Executive Board on the appointment of the Evaluation Director. This will be reflected into the new evaluation policy. OED June 2018 OED June /12

9 4.8 External evaluation expertise should be part of the selection panel for the Director, e.g. at the level of UNEG heads. Agree. This requirement will be incorporated into the new evaluation policy. OED June The Evaluation Director should report directly to the UNICEF Executive Director, on all matters. Agree. No change is envisaged, as the Evaluation Director already reports directly to Executive Director. OED Ongoing 4.10 The terms of reference for the Director of UNICEF Evaluation should include the systematic presentation of all global evaluation reports to the Executive Board, in addition to the Global Evaluation Plan and the Annual reports on the evaluation function as already the case. Agree. The job description of the Evaluation Director already includes the presentation of global evaluation reports to the UNICEF Executive Board, in addition to the global evaluation plan and the annual report. Recognizing that there are many evaluations conducted every year for a large organization such as UNICEF, the Executive Board will have to decide (based on the corporate plan) how to set the priorities and agree on which reports need to be presented so as to overcome self-selection bias. The UNICEF Executive Board Ongoing Evaluation recommendation 5: Quality of evaluations in UNICEF All evaluations planned and commissioned by UNICEF, whether by EO, programme divisions, regional and country offices should aim at achieving the same standards of independence, credibility and utility, and align with the guidance and procedures established by the EO. The EO should be given the responsibility to strengthen the quality of the evaluation process in UNICEF, with an active oversight role on the Decentralized Evaluation Function (DEF) in this respect. Management response Agree. Technical guidance will be developed and quality assurance systems reviewed for the entire evaluation function of the organization. Specific complementary measures for the implementation of Recommendation 5 include: 5.1 Establishment of an external evaluation advisory committee, similar to Agree. UNICEF recognizes the value of an external advisory committee and will establish the same. Special attention will be made to ensure that top evaluation EO Director January /12

10 the one established by the Independent Evaluation Office in UNDP. 5.2 Development of standard guidance for quality assurance of the evaluation process. 5.3 Establishment of a real-time Quality Assurance mechanism in support of evaluations at country level in all regional offices, under the responsibility and management of the Regional Evaluation Adviser and the oversight of the EO. 5.4 Development and/or adaptation of evaluation guidelines and manuals to fit the UNICEF evaluation process, from inception to completion. 5.5 Development and/or adaptation of evaluation guidelines and manuals to improve the integration of human rights and gender equality perspectives in evaluations. 5.6 Revision of the criteria that differentiate evaluations from other types of assessments and reviews, and consider 10/12 experts in areas of concern to UNICEF will become members of this committee. Agree. UNICEF already has guidance and standards for quality assurance, and they have been proven useful at the regional offices as well. UNICEF will continue to expand and improve the guidance and standards. The guidance and qualityassurance mechanisms will be reviewed, and if necessary, adapted. Agree. Some UNICEF regional offices have long-term agreements with independent evaluation firms for support in the area of real-time quality assurance. Moving forward, the UNICEF Evaluation Office will expand the coverage of realtime Quality Assurance mechanisms to ensure that all offices have access. Agree. UNICEF already has a range of guidelines and manuals for the evaluation process. These will be reviewed, taking into account those that exist within UNEG, and where necessary, be adapted and expanded accordingly. Agree. UNEG has guidance for a range of topics, including human rights and gender. This guidance adequately covers the needs related to UNICEF evaluations. All guidance will be reviewed, and if necessary, adapted. Agree. The Taxonomy for Defining and Classifying UNICEF Research, Evaluation EO Director December 2017

11 a more consistent application of the taxonomy in the titles of evaluation reports. 5.7 Revision, in consultation with Senior Management and the Regional Evaluation Advisers of the quality standards for evaluation recommendations, to make them more useful to UNICEF managers while maintaining the possibility for strategic, corporate and cross-cutting issues to be captured. 5.8 Revision of the timeframes for the implementation and closure of recommendations that address strategic and corporate-wide issues. and Studies will be updated, and reflected in the evaluation policy. Agree. The 2017 UNEG quality standards for evaluation recommendations will be adapted for use by managers in UNICEF. Agree. An integrated system of tracking evaluations and their management response, implementation and closure of recommendations is currently being developed and will be implemented after its testing. The testing is planned to take place by the end of This will provide an opportunity for discussion on the appropriate length of time for closure of actions around recommendations. Evaluation recommendation 6: Management of the EO The Director of UNICEF Evaluation should revise the internal management processes of the EO. Management response: Agree. The 2017 OMP submission, which covers the period 2018 to 2021, has taken this recommendation into account and made changes required for the efficient management process of the Evaluation Office, which includes additional staffing for the Evaluation Office. Specific complementary measures for the implementation of Recommendation 6 include: 6.1 Improving the efficiency of the corporate-level evaluation processes and Agree. The annual report by the Director of Evaluation on the evaluation function in UNICEF includes an indicator for the EO Director Ongoing 11/12

12 the timely delivery of all evaluation products by the EO. 6.2 Rotation of the EO evaluation advisers and specialists for the management and conduct of evaluations, so as to enable diversity of perspectives and experience in evaluations of the same thematic areas and programmes. timely completion of the evaluation products. The Director will pay special attention to this indicator and take corrective action as needed. Agree. The OMP foresees dissolving strict sector-bound work areas for Evaluation Office evaluation advisors and specialists. EO Director January Inclusion of highly reputed specialists in the subject matter of the evaluation, into the evaluation teams responsible for carrying out evaluations commissioned by EO and the DEF. Agree. UNICEF currently maintains the practice of including subject-matter specialists in its evaluation work and will reinforce this at all levels. EO Director Ongoing 6.4 Ensuring that all criteria and KPIs identified in the Evaluation Policy are adequately monitored and reported upon. Agree. The Evaluation Director will set up a system to monitor and report on the criteria and KPIs in the updated guidelines and manuals. 12/12