Decentralized Evaluation (DE) Mini guide for WFP Management (CD/DCD) at Country level

Transcription

1 Decentralized Evaluation (DE) Mini guide for WFP Management (CD/DCD) at Country level

2 Photocredits: WFP/Diego Fernandez

3 Introduction Purpose: This Mini Guide is intended to support and enable WFP Management at Country level (CD/ DCD) to plan, commission and undertake evaluations that are useful, impartial and credible in supporting WFP s work in line with WFP s new evaluation function described in: i) Evaluation Policy ( ); ii) Evaluation Charter, issued by the Executive Director as an ED Circular, which establishes the mandate and governance framework for the evaluation function; and iii) WFP s first Corporate Evaluation Strategy Content: This Guide focuses on the relevant authorities and accountabilities in addition to the key decision points 1 and approval processes required for each evaluation phase summarized below: Key phases of a Decentralized Evaluation management process: This guide includes annexes to support understanding and operationalization of impartiality provisions: 1. Options available for appointing the evaluation manager; 2. Quality Support Advisory Mechanisms (QS); 3. Minimum Decentralized Evaluation Coverage Norms; 4. Purpose and role of an Evaluation Reference Group and Evaluation Committee. Key resources: Evaluation Policy ( ) Corporate Evaluation Strategy ( ) Evaluation Charter (Circular OED2016/007) Decentralized Evaluation Process Guide 1 Bolded in each evaluation phase for easy reference

4 Phase 1: Plan Include plans for evaluation in the design of interventions, consistent with the Evaluation Policy s coverage norms (see annex 3) such that evaluation is factored into: office work plans, PACEs and reporting systems Budget adequately for the management (staff time) and conduct of independent decentralized evaluations Promote joint evaluations whenever feasible and relevant Ensure the evaluability of WFP s undertakings establishing baseline information, performance indicators and targets for expected results Decide the type of evaluation needed and its scope Approve the evaluation indicative budget and its timing Ensure that CO staff are aware of and understand the provisions for impartiality and independence Appoints an evaluation manager (see annex 1) Set up an evaluation committee and a reference group (see annex 4) Main outputs: Evaluation plan embedded in CO work plan, project documents and budgets Type of evaluation agreed about Initial timeframe and outline budget set up Evaluation manager identified Evaluation committee (EC) set up

5 Phase 2: Prepare Establish and chair the evaluation committee and the evaluation reference group for the entire duration of the evaluation process Comply with the Evaluation s Policy s provisions and safeguards for impartiality for the entire duration of the evaluation process Provide feedback on ToR Approve the final ToR as the chair of the evaluation committee (CD/DCD) Approve the evaluation team selection and budget as chair of the evaluation committee (CD/DCD) Ensure use of competitive and performance-based procedures for recruitment Ensure that all decentralized evaluations are conducted by qualified independent evaluators who sign the Code of Conduct for Evaluators in the UN system Main outputs: Draft Terms of Reference (ToR) Feedback from outsourced quality support advisory service (DE QS) on the ToR (see annex 2) Completed comments matrix Final approved ToR

6 Phase 3: Inception Provide feedback on draft Inception Report Approve the Inception Report as the chair of the evaluation committee (CD/DCD) Main outputs: Draft Inception report (IR) Feedback from outsourced quality support advisory service (DE QS) on the IR Completed comments matrix Final approved IR

7 Phase 4: Collect and Analyse Data Provide access to required information for evaluation conduct Main outputs: End of Phase debriefing presentation, which summarizes the evaluation team s preliminary findings and acts as a validation tool

8 Phase 5: Report Provide feedback on draft Evaluation Report Ensure that reporting is unbiased and transparent Approve the Evaluation Report as the chair of the evaluation committee (CD/DCD) Main outputs: Draft Evaluation Report (ER) Feedback from outsourced quality support advisory service (DE QS) on the ER Matrix of comments completed Final approved ER Phase 6: Disseminate and Follow-up Lead the Management Response preparation and ensures that it is publicly available as the chair of the evaluation committee (CD/DCD) Undertake and report on follow up actions, and use evidence from decentralized evaluations in the revision and preparation of new policies, programmes, strategies and other interventions Ensure that decentralized evaluation report is publicly available Main outputs: Posting of the ER on the internet A management response to the evaluation recommendations posted on the internet Wide availability of the ER

9

10 Annex 1: Options available for appointing the evaluation manager Key principles to take into consideration when identifying the evaluation manager: Attribution of the evaluation management to a staff is the responsibility of the CO management (CD/DCD) The CD/DCD should ensure that provisions for impartiality are operationalized to the extent possible in view of their Country Office size and composition and that the staff envisaged to manage the evaluation has the capacity to do so in particular a basic understanding of evaluation terminology and an ability to engage meaningfully with colleagues in the CO and outside who might be more senior. Option 1: M&E officer The preferred option is to have the head or a member of the M&E team taking the role of evaluation management. The advantage is that s/he knows about the programme/activities under evaluation while not having managed or implemented them directly (it is recognized that the M&E team is highly likely to have been involved in monitoring the activities under evaluation, except in cases where country offices have different monitoring arrangements for different options). This option should be feasible in very large and large COs and some medium COs who have an M&E officer post. If option 1 is not feasible because the CO does not have an M&E post, or all M&E staff have been directly involved in the management and implementation of the subject being evaluated, or the M&E staff do not have the requisite capacity, then the CD/DCD should consider option 2. Option 2: The CO VAM Officer or a programme officer If and only if the M&E officer cannot be the evaluation manager the alternative could be to appoint the VAM officer or a programme officer or a compliance officer who has not directly managed or implemented the programme/activities to be evaluated.

11 If option 2 is not feasible because the CO does not have a VAM or compliance officer or all programme officers have been directly involved in the management of the subject being evaluated, then the CD/DCD should consider option 3. Option 3: M&E officer from another CO or external consultant If nobody in the CO can ensure the role of evaluation manager then the CD/DCD could explore the possibility of having an evaluation manager from another CO on TDY. For a staff member to be eligible to manage an evaluation taking place in another CO, s/he must have completed one decentralized evaluation management as per the process guide and/or have followed the capacity development initiative set up by OEV. This option might not be available to COs before The TDY could be undertaken partly remotely with presence in the CO at peak times of the evaluation process. Alternatively the CD/DCD might opt to hire an external consultant to manage the evaluation process. This can be accomplished remotely, with limited or no presence in the country. Given the nature of the evaluation process, this would not be a full time contract, but rather on a when actually employed basis, with number of days established based on required level of effort at each stage. If option 3 is not feasible because there are no available staff for TDY or the CO is not in a position to meet the costs of the TDY/external consultant, then the CD/ DCD should consider option 4. Option 4: The Regional Evaluation officer (REO) This option should be of last recourse when none of the three others are possible and only available for very small or small COs. Indeed it would be unsustainable for the regional evaluation advisor to manage all decentralized evaluations in his/her region especially those who have many country offices.

12 Annex 2: Quality Support Advisory Mechanism (QS) This outsourced DE QS mechanism is mandatory for all decentralized evaluations. It contributes to: Useful and impartial evaluations which adhere to UN principles and norms and standards; Enhancing capacity across WFP to meet stakeholders' requirements for accountability and learning and in staff capacities to commission, manage and use high-quality decentralized evaluations; Improving the quality of evaluation products through timely feedback; Building evaluation capacity of Evaluation Managers as well as increasing the application of sound approaches and methods within WFP decentralized evaluations; Improving programme quality, organisational learning and knowledge management. This service provides real-time feedback and recommendations for improvements from an evaluation perspective to Evaluation Managers on the following draft decentralized evaluation deliverables: Draft Terms of Reference (ToR) Draft Inception Report (IR) Draft Evaluation Report (ER) Evaluation Managers are responsible for submitting the draft ToR, IR and ER for review. It is expected that feedback and advice received on draft IR and ER will be shared by the Evaluation Manager with the Evaluation Team Leader. The service is available in English, Spanish and French and specifies a review and feedback to the Evaluation Manager as follows: on ToR within 3 working days of receipt; on Inception Report and Evaluation Report within 5 working days.

13 Process to be followed by Evaluation Managers (EMs) to access the service 1. Check that the draft ToR, IR or ER includes all the elements of the corresponding DEQAS template If not, revise the draft ToR or return the draft IR or ER to the Team leader for revision; If yes, go to Send with draft document to be reviewed and submit service request to and cc and your RMEA/REO The service request form is available at the following address: 3. IODPARC to acknowledge receipt and confirm return date for the feedback 4. IODPARC to return completed feedback form in 3 working days for draft ToR; 5 working days for draft IR and ER. 5. For the ToR, the EM makes appropriate changes based on feedback received. For the IR and ER, the EM shares feedback with evaluation team leader to make the appropriate changes based on feedback received 6. Fill in customers satisfaction form on quality and utility of service after feedback received on draft ToR, IR and ER, EM and send by to deqs@iodparc.com and cc wfp.decentralizedevaluation@wfp.org

14 Annex 3: Minimum Required Decentralized Evaluation Coverage Norms Phase 1: To ensure that an evaluation is completed within 3 years in all COs by 2018, regardless of the scope and size, so that every CO builds experience and evaluation capacity. This implies that a CO with no evaluation (either CE or DE) since 2014 undertakes a DE in 2017; and similarly a CO with no evaluation (either CE or DE) since 2015 undertakes a DE in 2018, etc. Phase 2: Evaluation of at least 50% of each country office s portfolio of activities in terms of USD value of resourced (i.e. funded) requirements and implemented through operations or trust funds, by In countries with only one country programme, evaluations can be every five years. All country programmes (either through centralized or decentralized evaluation). Recommended to evaluate: Before scale-up of pilots, innovations, and prototypes; High-risk interventions, where level of risk is established in line with the WFP enterprise risk management policy; Before third repeat of an intervention of similar type (EMOP, PRRO, CP, DEV) and scope.

15 Annex 4: Purpose and role of an Evaluation Reference Group and Evaluation Committee 1. The overall purpose of the evaluation committee (EC) is to ensure a credible, transparent, impartial and quality evaluation process in accordance with WFP Evaluation Policy It will achieve this by supporting the evaluation manager through the process, reviewing evaluation deliverables (ToR, inception report and evaluation report) and submitting them for approval by the CD/DCD who will be the chair of the committee. The CD sets up and chairs an evaluation committee for each evaluation commissioned by his office, in order to: nominate and support the evaluation manager; make decisions on the evaluation budget, funds allocation, selection of the evaluation team; approve ToR, inception and evaluation report. The chair role may be delegated to DCD, but it is important to note that that the chair of the EC CANNOT be delegated to the Deputy Country Director if s/he is the Head of Programme. The EC membership is a subset of the ERG membership (see TN Evaluation Reference Group). While the ERG has only an advisory role, the EC Chair is making the decision on key aspects of the evaluation. 2. The overall purpose of the Evaluation Reference Group (ERG) is to support a credible, transparent, impartial and quality evaluation process in accordance with WFP Evaluation Policy ERG members review and comment on evaluation TOR and deliverables. The ERG members act as experts in an advisory capacity, without management responsibilities. Responsibility for approval of evaluation products rests with the Country Director/Deputy Country Director as Chair of the Evaluation Committee. The ERG always includes external stakeholders. The Chair of the ERG is Country Director or the Deputy Country Director (by delegation). The chair of the ERG can also be delegated to the Evaluation Manager if considered in a senior enough position to engage with external stakeholders. It is important to note that the chair of the ERG CANNOT be delegated to the Head of Programme unless s/he is the Deputy Country Director. Or to the Programme Officer directly in charge of the activities evaluated.

16 For more information on Decentralized Evaluation visit or contact us at: Printed: June 2016 Photo credits: WFP/Diego Fernandez