Fighting Hunger Worldwide In brief: WFP s Evaluation Function Do the best you can until you know better. Then when you know better, do better Maya Angelou June 2016
II
Introduction WFP s new evaluation function is framed by three foundational pillars: The Evaluation Policy (2016-2021), approved by the Executive Board in November 2015, sets the vision, strategic direction and model for WFP s evaluation function to embed evaluation as an integral part of all our work and thereby, help strengthen WFP s contribution to ending global hunger and achieve the Agenda 2030 Sustainable Development Goals. The Evaluation Charter, issued by the Executive Director in May 2016, confirms the mandate and governance of the evaluation function, and establishes the necessary staff authorities, roles and institutional arrangements to operationalise the policy. This Corporate Evaluation Strategy 2016-2021, endorsed by the Executive Management Group in April 2016. It sets out a phased implementation plan, comprising all the elements and activities required for building the model of a combined centralized and demand-led decentralized evaluation function which meets UN evaluation norms and standards, and achieves the Policy s vision. The evaluation function supports WFP s accountability, learning and continued strengthening, which comes not only from confirming and amplifying what we are good at, but also from asking challenging questions, welcoming external perspectives, and acting on lessons learned. WFP s Evaluation Function Foundational Documents Policy: sets vision & strategic direction for WFP s new evaluation function Evaluation Policy Evaluation Strategy Evaluation Charter Strategy: describes all the elements/workstreams necessary for phased implementation Charter: sets new mandate, governance, authorities & institutional arrangements 1
WFP s Evaluation Policy Evaluation is: the systematic and impartial, periodic assessment of the performance of WFP s activities, operations, strategies and policies. It provides evidence on achievement of intended and unintended results, causal contributions and performance (accountability); and, helps to understand the reasons and factors affecting performance and results for continuous improvement (learning). Recognizing the contribution evaluation makes to the evidence base on WFP s impact on the people it serves, the WFP Executive Board approved a new WFP Evaluation Policy in November 2015. The 2016-2021 Evaluation Policy aims to strengthen WFP s contribution to ending global hunger by embedding evaluation into the heart of its culture of accountability and learning, ensuring that evaluation is planned for, and evaluation findings are comprehensively incorporated into, all WFP s policies and programs. The policy sets the vision and purpose of evaluation in WFP s contemporary internal and external contexts. Its phased implementation will shift evaluation being mostly the business of the Office of Evaluation (OEV) to its being an integral part of all WFP s work. As illustrated in the Theory of Change (Figure 1) the Policy will be achieved through adopting a phased approach to attain the following outcomes: 1) Independent, credible and useful evaluations embedded into the policy and programme cycle, with all evaluations managed in accordance with the United Nation Evaluation Group s (UNEG) Norms & Standards, and WFP s Evaluation Quality Assurance System (EQAS). 3) Capacities for evaluation enhanced across WFP, with management arrangements that meet the United Nations Evaluation Group norms and standards. 4) Best practices in evaluation are developed and modelled through partnerships with other international humanitarian and development evaluation actors relevant to WFP s work. The policy reaffirms WFP s commitment to international evaluation principles, norms and standards, that form the foundations for WFP s overall evaluation function - which combines centralized and demand-led decentralized evaluation. The application of the foundational evaluation principles of independence, credibility and utility (Figure 2) ensure evaluation quality, and enhance organisational accountability and learning by enabling confidence in the independence and credibility of evaluation findings, recommendations and lessons for continual improvement of WFP s performance and results. To underpin WFP s commitment to safeguarding the independence and impartiality of all evaluations, the policy identifies specific provisions for impartiality; and clarifies the roles and accountabilities of the main stakeholders in the evaluation function: Executive Board; Executive Director; Director of Evaluation; Directors of HQ Divisions, Regional and Country Directors. As appropriate, evaluations in WFP consider application of the UN Charter humanitarian and related principles on: gender, protection and accountability to affected populations, ethics, principles for interventions in fragile situations and the Paris Declaration principles for aid effectiveness. 2) Appropriate application of evaluation coverage norms to WFP s policies, strategies and programmes, either by the Office of Evaluation (centralized evaluations) or by other Headquarters divisions, Regional Bureaux and Country Offices (decentralized evaluations). 2
Figure 1: Evaluation Policy Theory of Change asc Vision WFP s contribution to ending global hunger is strengthened by evaluative thinking, behaviour and systems embedded in its culture of accountability and learning Purpose Assumptions External stakeholders demand for evaluation Adequate internal demand for evaluation Organizational leadership, ownership and support Evaluation results are consistently and comprehensively incorporated into WFP s policies, strategies and programmes 1. Independent, credible and useful centralized and decentralized evaluations 4. Active evaluation partnerships in international arena Outcome s 2. Appropriate centralized and decentralized evaluation coverage 3. Adequate evaluation management capacity across WFP Assumptions Sustainable and predictable financing Skilled human resources available Optimal use of evaluations Quality monitoring data available Drivers of change at centralized and decentralized levels Normative framework Quality assessment Quality assurance International engagement Knowledge Management Communication Planning Partnerships Coverage norms Reporting Capacity development Resourcing Figure 2: Evaluation Principles Evaluation Principles Impartiality INDEPENDENCE CREDIBILITY Transparency UTILITY Intentionality Timeliness Accessibility QUALITY Accountability and learning 3
The evaluation function comprises the normative framework and the set of accountabilities applicable at centralized and decentralized levels to meet the policy objectives. It includes the following: Planning and selection: Evaluation is integrated into WFP's policy and programme management cycle and stakeholder requirements. Quality: adherence to WFP Evaluation Quality Assurance System based on UNEG Norms and Standards; and all completed evaluations are independently quality assessed. Use, communication and follow-up: stimulate learning by actively communicating evaluation results to all stakeholders and applying them in policy, strategy and programme design. All evaluations and management responses are publicly available. Partnerships: aligned with the Agenda 2030 call for stronger evaluation partnerships worldwide, the Policy commits to interagency collaboration and strengthening of national evaluation capacities. An effective evaluation function requires secure, predictable and adequate financial and human resources. Through the evaluation policy, WFP is committed to progressively allocating 0.8% of its total contribution income to address the needs of its entire evaluation function over the life the policy; to sustainable financing solutions for decentralized evaluations; and, to the establishment of evaluation advisor posts in Regional Bureaux by 2017. There are two categories of evaluations in WFP: those commissioned and managed by the Office of Evaluation - Centralized Evaluations; and those commissioned and managed by the Country Offices, Regional Bureaux and HQ-based Divisions - Decentralized Evaluations. Figure 3 explains the alignment of the types of various evaluations conducted in WFP to Centralized and Decentralized evaluation categories. All evaluations are conducted by independent consultants and made publicly available (www.wfp.org/evaluation) Figure 3: Evaluation Types CENTRALIZED EVALUATIONS (OEV managed) DECENTRALIZED EVALUATIONS (non-oev managed) Strategic Policy Country Portfolio Corporate Emergency Response Impact Operation Joint Activities Pilots Themes Transfer modalities Any other area of action 4
The policy envisages a phased approach (Annex 2) for progressive application of the agreed minimum coverage norms (Table 1). Table 1: Minimum Evaluation Coverage Norms Centralized evaluation Decentralized evaluation Strategic evaluations providing balanced coverage of WFP s core planning instruments, including Strategic Plan elements and related strategies Evaluation of policies 4 6 years after implementation starts 3 Evaluation of at least 50% of each country office s portfolio of activities 1 within a 3-year period 2 Recommended: before scale-up of pilots, innovations, and prototypes; for high-risk 4 interventions; and before third repeat of an intervention of similar type and scope Country portfolio evaluations: every 5 years for the 10 largest country offices (2 per year) every 10 12 years for all other country offices (7 per year) Evaluation of all corporate emergency responses, sometimes jointly with IASC Centrally managed operation evaluations providing balanced coverage 5 All country programmes 1 PSA, Project funds and other sources including multilateral and trust funds, SRAC allocation (Contingency Evaluation Fund) managed by the Evaluation Function Steering Group 1 In terms of USD value of resourced requirements and implemented through operations or trust funds 2 In countries with only one development project or country programme, evaluations can be every five years. 3 WFP/EB.A/2011/5-B 4 WFP/EB.A/2015/5-B 5 The current temporary series of centrally managed operation evaluations is expected to wind down as the decentralized evaluation function develops. Operation evaluations can also be decentralized 5
WFP s Evaluation Charter The Evaluation Charter, issued by the Executive Director in May 2016, enshrines the mandate and governance of WFP s evaluation function; establishes the staff authorities required by the Policy and identifies 14 institutional arrangements to operationalise the policy, summarized in Table 2. Table 2: Institutional Arrangements for WFP s Evaluation Function Adaptation of Existing Arrangements EB Annual Consultation on Evaluation (ACE): In addition to review of OEV s work plan and approval of OEV s budget as part of WFP s Management Plan, the focus of the ACE will be broadened to consider progress on implementation of the Evaluation Policy and effectiveness of WFP s entire evaluation function. Management Response and Follow Up system: To meet accountabilities for decentralized evaluation, the current system which covers only Centralized Evaluation will be expanded to include all WFP evaluations, thus ensuring that management responses to all evaluation recommendations are publicly available, appropriate follow up actions are taken and reported annually to the Executive Board. Strategic Programme Review Process (SPRP): To ensure that evaluation evidence is incorporated into programmes, policies and strategies at design stage, adjustments will be made to Country Strategic Plan, Project concept note and document templates. Staff Performance Management and Directors Assurance on Internal Control Frameworks: To embed evaluation responsibilities for impartiality, coverage and use, requirements will be integrated into relevant staff performance and competency enhancement system (PACEs) and work plans across WFP, and Directors Assurance Statements. Post-hoc Quality Assessment: To contribute to the credibility and continuous improvement in WFP s evaluations, the current arrangement will be expanded to cover all completed evaluations, including DEs. Evaluation Repository and Knowledge Management System: OEV s current system will be enhanced to cover DE s and to provide better access to evaluation results for all WFP stakeholders for accountability and learning purposes. New Arrangements Evaluation Function Steering Group (EFSG): Chaired by the Chief of Staff, OEV s secretariat support, and cross-functional membership at Director level, the EFSG will support the ED s role in safeguarding the Policy s provisions; fostering and embedding the evaluation culture into decision-making and practice across WFP. It will review policy implementation progress, provide strategic guidance, resolve issues and take decisions on some resourcing aspects of the Policy. Regional Evaluation Committee (REC): Mirroring the EFSG at Regional level, the REC will support the Regional Director s new accountabilities and build awareness, demand, use and planning for evaluation across Regions and CO s. Regional Evaluation Officers will provide the secretariat for the REC. Sustainable Financing Mechanism: Will be established in 2016 and stewarded by the EFSG, in recognition of the criticality of adequate resources for achieving the policy s aims, coverage norms for both centralized and decentralized evaluations, and expanded responsibilities set by the Evaluation Policy at all organizational levels. DE Committee (temporary): To ensure due process in evaluation management, and minimize bias, a DE committee will be convened for each DE commissioned. Hotline: To facilitate resolution of staff or evaluator concerns, the Director of Evaluation will manage a confidential hotline to safeguard evaluation impartiality and independence, with follow up consistent with general WFP policies and procedures. Evaluation Community of Practice: To help build WFP s evaluation culture and capacity, OEV s current informal network will be replaced by WFP communities platform for evaluation to share knowledge, experience and learning as Policy implementation proceeds. 6
WFP s Corporate Evaluation Strategy Building on the Policy s Theory of Change and normative framework, and the Evaluation Charter s institutional arrangements, the Corporate Evaluation Strategy 2016-2021 sets a phased plan for Policy implementation. The evaluation strategy comprises all the elements and activities required for WFP s model of a combined centralized and demand-led decentralized evaluation function that meet UN evaluation norms and standards, and achieves the Policy s vision. It describes various Workstreams to achieve each Policy Outcome, and others that cut across all Outcomes, together with their corresponding expected results, activities and internal partnerships timelines and phasing. Figure 4 provides a summary overview of the Strategy and Annex 1 summarizes the workstreams, key activities and partners to achieve it. In view of the importance of adequate human and financial resources for policy implementation, the Strategy also proposes funding sources 6 for all the workstreams, supporting the needs of the overall evaluation function and meeting of the policy s evaluation resourcing target of 0.8% of contributed WFP income. Figure 4: Corporate Evaluation Strategy Overview Workstream (WS) A: Normative Framework Outcome 1: Independent, credible & useful centralized & decentralized evaluations Outcome 2: Appropriate centralized & decentralized evaluation coverage Outcome 3: Adequate evaluation management capacity across WFP Outcome 4: Active evaluation partnerships in international arena WS 1.1: Impartiality provisions WS 2.1: Coverage norms WS 3.1: WFP capacity development WS 4.1: Partnerships WS 1.2: Quality assurance WS 2.2: Planning WS 3.2: Institutional arrangements WS 4.2: National & regional capacity development WS 1.3: Quality support WS 2.3: Funding WS 3.3: Evaluator expertise WS 1.4: Post-hoc quality assessment WS 3.4: Staffing WS 1.5: Use of evaluations WS B: Reporting WS C: Communication & Knowledge Management 6 PSA, Project funds and other sources including multilateral and trust funds, SRAC allocation (Contingency Evaluation Fund) managed by the Evaluation Function Steering Group 7
Looking Forward Implementation of the workstreams, activities and institutional arrangements elaborated in WFP s Evaluation Charter and Strategy, will enable WFP to meet its evaluation policy vision and purpose through: Planning independent and impartial evaluation into its policies, strategies and programmes from the outset, to generate the evidence and knowledge WFP needs to achieve its goals in an increasingly complex world; Commissioning a greater number of independent and credible evaluations to meet all stakeholders needs - at the right time, and with the right partners to maximise feedback and use of evaluation results; Broadening WFP s culture of accountability and learning, by building from evaluations managed only by the Office of Evaluation, to generating and sharing evaluation lessons across HQ, Regional Bureaus and Country Offices; Developing WFP s evaluation skills and capacity to better engage in the country-led Zero Hunger and evaluation partnerships expected to increase under Agenda 2030, helping to meet the expectations of the people WFP serves, worldwide. 8
Annex 1: Summary of Expected results, Activities and Key Partners by Policy Outcome Outcome 1: Independent, credible and useful centralized and decentralized evaluation WS 1.1: Impartiality provisions (By their nature impartiality is supported by several provisions and workstreams, reflected throughout the Strategy) Expected result: systems and arrangements for ensuring independence and impartiality are applied to all WFP evaluations. Provide guidance on operationalization of impartiality provisions Establish an evaluation committee for each DE Review evaluation service provider contracts Provide guidance evaluation managers to assess and manage conflict of interest Set up Hotline Key Partners: COs, RBs and HQ Divisions other than OEV (e.g. OSC, HRM). WS 1.2: Quality assurance Expected result: quality assurance systems for CE (EQAS) and DE (DEQAS) meet international norms and standards and are well-aligned with WFP s international commitments; the quality assurance system is systematically applied on all evaluations. Review and update EQAS Pilot and finalize DEQAS in 3 languages Key Partners: COs, RBs and HQ Divisions other than OEV (e.g. GEN, OSE), UNEG WS 1.3: Quality support Expected result: Aligned with the DEQAS Guidance package, each DE process is supported by independent, real-time technical advice, to strengthen their quality, credibility and usefulness. Establish OEV Help-desk REOs provide direct support Establish & apply outsourced quality support advisory service Key Partners: RBs WS 1.4: Post-Hoc Quality Assessment Expected result: quality of all WFP completed evaluations is independently assessed, thus contributing to CE and DE transparency, credibility and utility. Establish and apply independent Post-Hoc Quality Assessment Key Partners: HQ Divisions other than OEV (GEN), UNEG. WS 1.5: Use of evaluations Expected result: learning from high quality (timely, relevant, credible and publicly available WFP evaluations informs design and implementation of all new strategies, policies and programmes. Upgrade evaluation internet and intranet Ensure that project and Country Strateic Plans draw on & plan for evaluation evidence Update Programme Guidance Manual Review and upgrade system for Management Responses Review draft corporate strategies and policies Key Partners: HQ Divisions other than OEV (e.g. OED, RMT, PGM, OSZ, OSN, OSE, RMP). 9
3 Outcome 2: Appropriate centralized and decentralized evaluation coverage WS 2.1: Application of the coverage norms Expected result: phasing process for meeting the minimum corporate evaluation coverage norms over the life of the Policy is explained and clarified. Set up framework to meet the coverage norms for CE Develop coverage norms calculation method for DE Key Partners: RBs WS: 2.2 Planning Expected result: annual and longer term evaluation plans are produced, integrated more closely into WFP s policy and programme management cycle, and implemented incrementally to meet the coverage norms. Prepare OEV work plans Review s-prp and Country Strategic Plan templates for planning evaluation Advise on DE plans and budgets Management Information System for complementarity between CE and DE evaluation plans Key Partners: RBs and HQ Divisions other than OEV (e.g. OED, OSZ) WS 2.3: Funding Expected result: adequate funding for both CE and DE. Set up and run a sustainable financing mechanism (e.g. Contingency Evaluation Fund) Embed DE budgeting in Country Strategic Plans and all other project documents Budget OEV's work plan in line with CE coverage norms Key Partners: HQ Divisions other than OEV (e.g. RM, OSZ). Outcome 3: Adequate evaluation management capacity across WFP WS 3.1: WFP capacity development Expected result: WFP has the capacity to manage DEs, apply evidence-based programme design, and policy engagement as envisaged under Agenda 2030. Develop and roll out evaluation module for Monitoring and Evaluation Learning Programme Develop and manage comprehensive evaluation learning programme including embedding evaluation into other corporate learning initiatives RBs strengthen capacities of CO to manage DE Set up and maintain an evaluation Community of Practice Organize annual Global Evaluation Meetings and participate in regional meetings Key Partners: COs, RBs and HQ Divisions other than OEV (e.g. RMP, HRM, OSZ). WS 3.2: Institutional arrangements Expected result: Charter s evaluation institutional arrangements are operational, supporting WFP s evaluation function and culture of evaluation. Establish Evaluation Function Steering Group Establish Regional Evaluation Committees Key Partners: : RBs and HQ Divisions other than OEV (e.g. OED). WS 3.3: Evaluator expertise Expected result: access to evaluation service providers is improved. Expand access to existing Long Term Agreements (LTA) to DEs Tendering to expand pool of pre-qualified Long Term Agreements (LTA) Update, expand and maintain consultant database Key Partners: RBs and HQ Divisions other than OEV (e.g. OSC). 10 WS 3.4: Staffing Expected result: augmentation in number of WFP staff with requisite evaluation skills and experience. Review staffing requirements in OEV Establish posts and recruit six Regional Evaluation Officers Include appropriate outputs and Key Performance Indicators for evaluation in PACE templates Internal control assurance statements Key Partners: RBs and HQ Divisions other than OEV (e.g. HRM, RMF)
Outcome 4: Active evaluation partnerships in international arena WS 4.1: Partnerships Expected result: WFP s evaluation practice is shared with and benefits from others experience so that WFP s evaluation evidence systematically contributes to wider accountability and learning, especially in the international humanitarian arena. Engage in UNEG on: Agenda 2030; Humanitarian evaluation issues; normative framework and system wide evaluation work Engage with evaluation offices of Rome-Based Agencies Engage with Inter-Agency Standing Committee (IASC) on Inter-Agency Humanitarian Evaluation (IAHE) Engage with thematic global, regional and national evaluation networks Develop guidance for engagement in joint evaluations at global, regional and national levels Key Partners: COs, RBs and HQ Divisions other than OEV (e.g. PGP), UNEG, UN System, RBAs, OCHA, IASC) WS 4.2: National and regional capacity development Expected result: RBs and COs are engaged in partnerships at national and regional levels that meet WFP s commitments to UNGA on strengthening national evaluation capacity in support of Agenda 2030. Develop guidance for regional and national evaluation capacity development Increase participation in regional/national evaluation associations Integrate national evaluation capacity development in Country Strategic Plans Key Partners: RBs and HQ Divisions other than OEV (e.g. PGP, OSZ). Cross-cutting workstreams WS A: Normative Framework Expected result: WFPs entire evaluation function is aligned with UNEG norms and standards, and other internationally-agreed principles relevant for evaluation. Monitor trends in international standards and principles, and update normative framework Key Partners: relevant external stakeholders, including the UNEG-DAC Peer Review mechanism. WS B: Reporting Expected result: implementation of the Policy, Charter and Strategy is evidenced, reported on and adjustments recommended, to WFP s Evaluation Function Steering Group.and Executive Board Develop MIS, KPIs and apply them Provide Secretariat for EFSG Annual Evaluation Report Key Partners: COs, RBs and HQ Divisions other than OEV (e.g. RMP, RMBP) WS C: Communication and Knowledge Management Expected result: all evaluations are accessible, widely-known and used for learning and accountability. Publish and disseminate CEs and DE s Stimulate knowledge and learning from evaluations, accountability to affected populations, including through the Evaluation Community of Practice Innovate in new products and communication of evaluation results Embed evaluation in WFP corporate KM systems Key Partners: COs, RBs and HQ Divisions other than OEV (e.g. PGM, RMT) 11
Annex 2: Phased approach timeline 2015 2016 2017 2018 2019 2020 2021 Evaluation Policy Approved 6 Regional Evaluation Advisers in place Next Peer Review Next Evaluation Policy Finalizing, testing & roll out DEQAS Development & roll out Strategy Progressive application of the coverage norms Capacity Development 12
13
. Printed: June 2016 Front Cover, Back cover, inside photos: WFP/Diego Fernandez For more information on overall evaluation issues contact: wfp.evaluation@wfp.org For Decentralized Evaluation please send all queries to:wfp.decentralizedevaluation@wfp.org. World Food Programme Office of Evaluation www.wfp.org/evaluation