UNICEF Evaluation Office Terms of Reference: External Assessment of UNICEF s Global Evaluation Reports Oversight System (GEROS) The Evaluation Office is planning to assess the Global Evaluation Reports Oversight System (GEROS) of the organization. The assessment is expected to be done by external consultants under the supervision of the Evaluation Office. The TOR outlines the background of GEROS, its objectives, the rationale for the assessment, the scope and assessment questions. It also outlines the methodological principles, the management arrangements as well as the workplan and key deliverables. 1. Background and Rationale UNICEF s Revised Evaluation Policy was approved in 2013. In line with the Evaluation Policy, the Evaluation Office (EO) maintains oversight of the effective implementation of the policy at all levels in the organization. Given the decentralized nature of the Evaluation Function in UNICEF, EO works collaboratively with the Regional Offices (ROs) and Country Offices (COs) in a wide range of initiatives aimed at strengthening the Evaluation Function in UNICEF. Within the efforts towards enhancing the evaluation function at UNICEF, a Global Evaluation Report Oversight System (GEROS) was launched in 2010. GEROS aims to contribute to the quality of evaluations managed or commissioned by UNICEF at the global, regional and country levels by providing assessment of the quality of evaluation reports and feeding results back to management. GEROS is an organization-wide system. The quality assessment of final evaluation reports is managed by the Evaluation Office. This is complemented by customised qualityassurance mechanisms designed and implemented by COs and ROs. GEROS has four main objectives: To provide senior managers with a clear and independent assessment of the quality and usefulness of evaluation reports To strengthen internal evaluation capacity, through practical feedback on how to improve future evaluations To contribute to corporate knowledge management and organizational learning, making available good-quality evaluations To report to management and the Executive Board on the quality of evaluation reports Through the GEROS system, all UNICEF final evaluation reports submitted to the Global Evaluation and Research Database (EDB) each year are rated against the UNEG/UNICEF Evaluation Report Standards. GEROS also provides feedback on the quality of each 1
evaluation report as well as a meta-analysis leading to recommendations aimed at informing decisions on ways to improve the overall quality of future evaluations. GEROS data is also used for corporate reporting through the Annual Report on the Evaluation Function prepared by the Evaluation Office. In recent years, this has used a set of Key Performance Indicators of which several draw on GEROS results. According to data obtained through the GEROS system, the quality and use of evaluation reports has significantly improved over the years since the system was put in place. Between 2011 and 2013, the number of good quality reports rated either outstanding/best practice or highly satisfactory increased from 40% to 69%. After 5 years of implementation, the Evaluation Office intends to carry out an external assessment of GEROS. Although the system seems to be working well and the overall results are gratifying, there is a need for an in-depth analysis of GEROS overall approach and tools in light of the experience gained so far, some observed shortcomings and the new challenges the evaluation function currently facing. The external assessment of GEROS is timely for UNICEF as the Long Term Agreement (LTA) with the Institution commissioned to implement GEROS is coming to an end and a new bidding process for a new LTA is expected. The assessment is then meant to inform the implementation of a new cycle of GEROS that will run from 2016 to 2018. 2. Purpose and objectives: The purpose of the assessment is to generate evidence to inform action to further strengthen the overall evaluation report quality assurance and feedback system at UNICEF. The assessment will be forward looking, aimed at contributing to the quality, consistency, credibility and use of the evaluation report quality assurance system at the global, regional and country levels. The assessment pursues the following objectives: 2
i) Determine the extent to which GEROS has achieved its expected objectives (see GEROS objectives listed above); ii) iii) iv) Determine the relevance and conceptual clarity of the GEROS approach and how far it remains relevant to the changing context in UNICEF, the UN development system and more widely; Review the adequacy and utility of the GEROS methodology, tools, mechanisms and related guidance; Examine stakeholder understanding and perceptions of the system and the capacities of key stakeholders to adequately fulfil the requirements of the system; v) Identify constraining and enabling factors for effective operation of the system as well as opportunities and risks; vi) vii) 3. Scope To the extent possible, undertake a rapid comparison with quality assurance mechanisms in other agencies for the purpose of benchmarking as well as identification of good practices and alternative options; Identify good practices and lessons for wider application and propose evidencebased recommendations on the way forward. The assessment will focus on the relevance, effectiveness and efficiency of the GEROS system, taking account of the changing operational context in UNICEF, the UN development system and in partner governments, donors, implementing agencies and other organizations. It will include an analysis of the extent to which the GEROS has contributed to effective implementation of the revised evaluation policy (2013). It will also provide an in-depth analysis of the GEROS approach, methodology and feedback mechanisms; and the extent to which feedback has been used to enhance the overall quality, credibility and use of evaluations at different levels of the organization. The assessment will consider the following elements of the GEROS system: (a) the Evaluation and Research Database and its role in managing evaluation information and knowledge; (b) the evaluation report quality assessment mechanism, including associated procedures, tools and guidance; (c) feedback to management; and (d) use of GEROS information and outputs by other stakeholders. It will not consider the management response tracking system. The assessment will consider the range of key internal and external stakeholders involved in managing or using the GEROS system or outputs at each level of the organization. Five cycles of GEROS implementation will be assessed i.e. 2010 to 2014. While the main focus will be UNICEF and the GEROS system, attention will also be given to rapid comparison with similar systems in other agencies. 3
4. Assessment criteria and questions Relevance: To what extent are the GEROS conceptual approach and methodology aligned with the QCPR, UNEG Norms and Standards and related guidance documents; and with UNICEF s Evaluation Policy and quality standards? To what extent is the GEROS system responsive to corporate organizational priorities e.g. renewed focus on equity? To what extent is the GEROS system responsive to the needs and priorities of its main internal and external stakeholders at the global, regional and country levels? Are the associated procedures, tools and guidance sufficiently clear to support users? How does it need to be adapted to address the needs of external stakeholders (e.g. UN country teams undertaking UNDAF evaluation; and national partners, including government institutions, in particular where UNICEF assists country-led evaluations)? Is there a clear and coherent theory of change linking inputs to intended outcomes? Have there been any significant conflicts of interests? Effectiveness: Is there evidence on how far GEROS has fulfilled its specific objectives and how far it has contributed to a more effective implementation of UNICEF s Evaluation Policy? To what extent has GEROS contributed to the quality, consistency and coherence of evaluation reports? To what extent and in which ways has the GEROS contributed to boost the consistency, credibility and accountability of the evaluation function? Has it helped, directly or indirectly, to build internal evaluation capacity? Are there significant numbers of evaluation reports falling outside the scope of GEROS? Is there a significant number of evaluation reports not being submitted to the Evaluation and Research Database (e.g. some health evaluation reports?) To what extent have the GEROS feedback and the meta-evaluation reports contributed to better knowledge management and organizational learning, to increased use of evaluation and to better informed decision-making? To what extent and with which results has the GEROS triggered or supported divisional or regional quality assurance and knowledge management systems? Are there any unexpected results? Has the system exerted a deterrent effect on teams considering whether or not to undertake evaluation activities? Efficiency: To what extent is the GEROS coherent/compatible with other system strengthening initiatives undertaken by the Evaluation Office, the Regional Offices and the Country 4
Offices? Is the GEROS approach and guidance consistent with the UNICEF Management Response tracking system? Is there complementarity between the GEROS and the regional quality assurance systems? Are the transaction and coordination costs reasonable? Are there duplications or redundancies? To what extent does the investment in GEROS justify the results achieved so far? Could comparable results be achieved with reduced effort (for example, by rating only a sample of the evaluation reports completed each year)? Cross-cutting and emerging issues To what extent has the GEROS contributed to mainstreaming gender equality, human rights, ethics and equity considerations in evaluations? To what extent is the GEROS methodology supported by clear, appropriate, sufficient and up to date guidance for implementation? How widely is this guidance known and used? Have dissemination efforts been sufficient? Can GEROS systems, guidance and procedures deal appropriately with different kinds of evaluations including joint evaluations, UNDAF evaluations, country-led evaluations, multi-country evaluations, and country case studies undertaken as components of wider thematic evaluations? Is the GEROS treatment of impact evaluation reports appropriate (including reports of evaluations using randomized control trials) To what extent the GEROS is enabling or inhibiting innovations in evaluation? To what extent is the Evaluation and Research Database (ERDB) fulfilling its role as a tool or platform for effective sharing of evaluation evidence within UNICEF and more widely? 5. Methodology The evaluation will be forward looking, and guided by the following methodological principles: Consultation: the methodology should allow the participation of the various stakeholders at different stages of the assessment. Triangulation: data/information is expected to be analyzed taking into account a variety of perspectives as well as of sources of information. Mixed-methods approach: a combination of qualitative and quantitative methods of data collection and analysis will be applied. The assessment is expected to use the following frameworks and approaches: theory of change stakeholder analysis, detailing needs, interests, capacities and incentives 5
analysis of the treatment of equity and gender equality validation of the quality ratings for a small sample of reports to the extent possible, an estimation of costs and benefits ( value for money ) The assessment will include the following: a rapid desk review leading to an inception report individual interviews with Sr. management and M&E focal points at HQs, regional and CO levels brief documentation of good practices light-touch reviews of regional QA systems in UNICEF light-touch benchmarking or comparison with similar systems in other UN entities preparation of an inception report and a draft final report leading to a final report. A full methodological proposal is expected in the Inception report to be prepared by the consultant/s. 6. Management arrangements The assessment will be managed by the Sr. Evaluation Specialist Systemic Strengthening who will be supported by the Knowledge Management Specialist. A reference group will be established to act as a sounding board to the assessment exercise. The Reference group is expected to provide inputs to the inception and draft reports. The reference group will be made of representatives of key stakeholders in UNICEF namely, the Office of Research (OoR), the Division of Research and Policy (notably the Data and Analytics section (D&A), the Field Results Group (FRG), Regional M&E chiefs and nominees of selected Country Office Representatives. 7. Work plan, deliverables & timeline Inception phase: Desk review; Tasks Expected Products/Deliverables Timeline Interviews with Sr. Management and EO staff. Meetings with Assessment Manager An inception report containing inter-alia, the following : Unfolding of all sections of Evaluation TOR Theory of change - ToC Assessment framework Full methodological proposal Workplan 3 weeks 6
Data collection and analysis Reporting Presentation of preliminary findings to EO and to Reference Group Two updates to the Assessment manager Draft Assessment Report including annexes Final Assessment Report including annexes. Power point with summary of key findings and recommendations 4 weeks 3 weeks 8. Estimated duration of the Assessment Exercise The assessment is expected to last 10 weeks, over a period of September and November 2015. The final deliverable should be submitted no later than 30 November 2015. 9. Official Travel Involved The consultants are expected to visit UNICEF Headquarters twice i.e. during the inception phase; and towards the end of the analysis period to present the preliminary findings, conclusions and recommendations. 10. Experience and qualifications required for the assignment: Two consultants (one team leader and one team member) are required for this assignment with the following qualifications and experience: Team Leader Advanced post-graduate degree, at least Master s Degree, in international development, social sciences or related fields; 15 years of recognized international experience in leading evaluations or evaluative enquiries; Proven experience as a team leader. Experience in leading similar assessment is a must. Familiarity with human rights-based approaches and with gender equality. Knowledge of the UN System is highly desirable. Excellent communication and facilitation skills and ability to manage diversity of views Strong analytical and writing skills Ability to perform consultations Excellent written and oral English. Working knowledge of Spanish and French is desirable. 7
Team Member: Advanced University degree i.e. Master s in international development, social sciences or related disciplines. At least 7 years of experience in evaluation notably in quality assurance systems. Good knowledge and understanding of gender and equity issues. Strong analytical and writing skills Ability to perform consultations Excellent written and oral English. Working knowledge of Spanish and French is an asset. Application Process UNICEF s Evaluation Office (EO) is seeking to contract two qualified evaluation professionals (one team leader and one team member) to conduct an assessment of the Global Evaluation Reports Oversight System (GEROS) of the organization. Interested professional evaluators should send an application including the following: a. Updated CV/Resume, and completed Personal History Profile (P11 http://www.unicef.org/about/employ/files/p11.doc ), b. A sample report of a similar exercise/subject or an evaluation report, with a clear indication of the applicant s contribution in the report. c. Availability and fee. The application should be transmitted via email by 17 th August, 2015, at the very latest, using the following email: evalofficeapplications@unicef.org Candidates should also indicate in the email subject the consultancy they are applying for: Email Subject: Application for Assessment of GEROS Team Leader Or Application for Assessment of GEROS Team Member 8
ANNEX 1 Expression of Interest Assessment of GEROS 1. Name and contact details 2. Evaluation Experience Summarize your experience highlighting how it is relevant for the assignment. Include information on previous similar assessments or evaluations done (2 or three paragraphs or no longer than one page) 3. Proposed Evaluation approach Including an overview of approach to answer the evaluation questions; and to conduct the evaluation process including methodology (one page) 4. Proposed timeline 5. Proposed fees 9