Indicative content of evaluation final reports Draft, Jérémie Toubkiss, Evaluation Office, NYHQ (updated 4 February 2016)

Transcription

1 Indicative content of evaluation final reports Draft, Jérémie Toubkiss, Evaluation Office, NYHQ (updated 4 February 2016) 0. Opening pages - The first pages provides the key information on the programme evaluation 0.1 Name and timeframe of the evaluated programme 0.2 Name of the country (and regions) 0.3 Type of evaluation (mid-term, final, etc.) 0.4 Name of the commissioning agency 0.5 Authors of the report (institution, names of individuals) 0.6 Date of the evaluation report 0.7 Status of the report (draft, final ) 0.8 Table of content 0.9 List of acronyms 0.10 List of tables and figures 0.11 Executive summary presenting only the following information that are intended to decision makers and senior management (max 5 pages, can stand-alone): 0.12 Brief description of the evaluated programme 0.13 Evaluation purpose, objectives, criteria, scope 0.14 Methodology (list of main data source, collection methods, triangulation approach, equity lens, ethical aspects, and main limitations) to 3 key findings and the overall conclusion on the programme performance for each evaluation criteria to 3 key lessons learnt to 4 priority recommendations (urgent, strategic or easy-to-implement) by stakeholder 1. Context and objectives of the evaluation - The report includes sufficient and relevant contextual information, and explains relevance and usefulness of the evaluation Context of the evaluation and description of the evaluated programme The ToR adequately describes the particular policy, social, economic, governance and programmatic environment of the country/region/area and of the programme/project/intervention to be evaluated, in a way that illuminates/contextualize the findings Purpose and specific objectives of the programme/project/intervention to be evaluated, and its theory of change or logical framework, which should be well articulated (can be revised/updated by the evaluator) Main assumptions/hypotheses and risks associated with the theory of change or logframe (based on the programme documents or on the evaluator's analysis) Timeframe of the programme/project/intervention: starting and end dates (or implementation status); milestones in the project history; timing of the evaluation within the project management cycle Geographical boudanies: number and name of regions and districts; rural/small towns/peri-urban/urban areas Key stakeholders and their respective roles (UNICEF, government counterparts, implementing partners, service providers, beneficiaries) Financial aspects (amount and origin of funds, future financing prospects...) Existing knowledge and evidence on the programme/project/intervention (previous, on-going and planned evaluations & studies, incl. findings from previous evaluation or evaluations of similar interventions), knowledge gaps and added-value of this evaluation Aspects of complexity or characteristics that can potentially make the evaluation work difficult, e.g. absence of logframe, of baseline, of M&E system, changes in program objectives, strategy or implementation over time, multiplicity of actors, multiplicity of program components, complexity of the policy, institutional, cultural or security context etc Purpose and objectives of the evaluation Purpose of the evaluation (transparency & accountability; learning for decision making; advocacy etc.) Specific objectives of the evaluation specified and coherent with purpose Who has commissioned / initiatied the evaluation (UNICEF, donor, etc.) Primary and secondary audiences for the evaluation Why evaluation is being done at this time and how the evaluation will be useful (intended use, incl. what decisions might likely be influenced by the findings of this evaluation) 1

2 2. Methodology - The report adequately defines the criteria that have been utilized to guide the evaluation, the evaluation questions, the scope of the evaluation, the methodology used for data collection and analysis, and the overall organisation of the evaluation work Evaluation criteria Evaluation criteria used such as: relevance, efficiency, effectiveness (incl. cost-effectiveness or value for money), impact, and/or sustainability - and additional evaluation criteria if any (equity, gender, disability, vulnerable / deprived segment of the population, human/children rights-based approach, results based management, robustness of the monitoring-evaluation-accountabilitylearning system, innovation, humanitarian principles, quality of process, exportability / potential for scaling up, added-value, coherence Why some standard evaluation criteria may not have been used Definition of each evaluation criteria used Description and justification of any change made to what was foreseen in the ToR 2.2. Evaluation questions Final evaluation questions and by who and how they have been chosen and formulated (incl. any change made to the ones proposed in the initial ToR, for example knowledge gaps identified during the initial document review and recommendations drawn from previous evaluations & studies) Sub-questions or hypothesis to be tested, if any Description and justification of any change made to what was foreseen in the ToR 2.3. Scope of the evaluation Components of the programme/project/intervention that have been evaluated (and what was excluded or impossible to evaluate) Geographical coverage of the evaluation (with map comparing intervention areas with areas visited) Time boundaries of the evaluation (or phase in the project) Activities funded by specific donors (if relevant) Stakeholders and beneficiaries included / excluded Description and justification of any change made to what was foreseen in the ToR 2.4. Evaluation design, and methodology for data collection and analysis Conceptual framework and evaluation design (non-experimental, quasi-experimental, experimental) Overall methodology: quantitative, qualitative, participatory, self-evaluation where the evaluation team will play a facilitation role only, utilization-focused, theory-based, etc Data to be collected for each evaluation question Data source: M&E data generated by the evaluated programme; documents reviewed (number and list in annex) and their credibility, stakeholders consulted (program staff; implementing partners such as gov, NGOs, and consultants; contractors such as construction companies and engineering firms; direct and indirect beneficiaries (specify any focus on sex, age, religion, ethnic group, professional occupation, poverty level, vulnerability status...); external partners or resource persons (other NGOs, donors, universities and researchers...) etc Triangulation plan: how key information have been triangulated Methods used to collect each data from each data source: semi-structured interviews, focus-group discussions, problem tree/analysis, community mapping, document review, field observation, surveys, measures/tests, case studies / monography, etc Description and justification of any change made to what was foreseen in the ToR Description of data collection instruments used (actual tools in annex) Sample size and sampling method (for key informants, implementing partners, beneficiaries, intervention sites) Training of enumerators' and pilot testing of survey tools (if applicable) Supervision of & support provided to enumerators' Other methods used for quality assurance of collected data Data analysis methods: how data have be coded, displayed, processed, aggregated, synthetized, compared use of quantitative and qualitative data analysis methods; basis for jugment (subcriteria, performance indicators, benchmarks, standards, comparisons etc.); how Unicef's contribution to results has been assessed e.g. causal contribution analysis, rival hypothesis etc To what extend the evaluation has been participatory (involving stakeholders including beneficiary at every stage of the evaluation, from design to formulation of recommendations) How aspects of complexity have been addressed 2.5. Equity aspects How equity and gender have been taken into account 2

3 2.6. Ethical aspects Identification of ethical issues throughout the evaluation project as well as the measures and methods adopted to mitigate against these issues: methods or practices to ensure the avoidance or minimization of harm and stress to participants; security matters and protection protocols utilized - both for enumerators and people interviewed; protection of privacy and safety of all participants; obtention of informed consent / verbal assent from participants; confidentiality & anonymity of data collected; absence of benefit or compensation offered to interviewees; training of enumerators in these issues and on enumeration/communication skills; declaration of any conflict of interests; if needed: ethical review and registration on clinicaltrial.gov. Reference to UNEG/UNICEF Ethical guidelines for evaluations, Evaluators' code of conduct, and Procedure for ethical standards. URL link to these documents are included Organisation of the evaluation work List of main phases (inception phase, data collection, data analysis and report drafting, stakeholders workshop on preliminary findings and formulation of recommendations, etc.) and timeline Description of each phase: objective and activities undertaken, duration, profile and role of people involved, logistics, support received from Unicef and other stakeholders, challenges faced and measures taken Quality assurance process used for each deliverable (internally within the evaluation team, externally with the evaluation manager, steering committee, technical reference group or during the final stakeholders workshop) and reference to the UNICEF standards for evaluation reports 2.8. Methodological and organizational limitations Methodological limitations: Issues of sampling and sample size; over-reliance on self-report; lack of use of relevant documents or M&E data; identification of potential bias (e.g. from respondents); lack of triangulation of key findings etc Organizational limitations: lack of access to relevant documents, M&E data, key informants; logistical issues, impossibility to make relevant observations in the field; etc How these limitations have been managed and to what extend they weaken the credibility of findings 3. Findings for evaluation criterion #1 - It is suggested here to dedicate one section per evaluation criterion 3.1. Methodological approach Detailed definition of the evaluation criterion and of potential sub-criteria Description of the evaluations questions and sub-questions related to the evaluated criterion, and hypotheses related to this criterion; definition of key concepts Results and analysis Factors explaining the findings (causal analysis) or formulation and assessment of hypotheses Good or bad practice that can be documented, useful for and shared with the various categories of readers (examples, case studies, quotes from the interviews etc.) 3.3. Conclusion on (criterion #1) For each question and programme component: clear, short, objective, evidence-supported and nuanced response Overall assessment of the performance of the programme with reference to the evaluated criterion 4. Findings for evaluation criterion # Methodological approach Detailed definition of the evaluation criterion and of potential sub-criteria Description of the evaluations questions and sub-questions related to the evaluated criterion, and hypotheses related to this criterion; definition of key concepts

4 4.2. Results and analysis Factors explaining the findings (causal analysis) or formulation and assessment of hypotheses Good or bad practice that can be documented, useful for and shared with the various categories of readers (examples, case studies, quotes from the interviews etc.) 4.3. Conclusion on (criterion #2) For each question and programme component: clear, short, objective, evidence-supported and nuanced response Overall assessment of the performance of the programme with reference to the evaluated criterion 5. Findings for evaluation criterion # Methodological approach Detailed definition of the evaluation criterion and of potential sub-criteria Description of the evaluations questions and sub-questions related to the evaluated criterion, and hypotheses related to this criterion; definition of key concepts Results and analysis Factors explaining the findings (causal analysis) or formulation and assessment of hypotheses Good or bad practice that can be documented, useful for and shared with the various categories of readers (examples, case studies, quotes from the interviews etc.) 5.3. Conclusion on (criterion #3) For each question and programme component: clear, short, objective, evidence-supported and nuanced response Overall assessment of the performance of the programme with reference to the evaluated criterion 6. Findings for evaluation criterion # Methodological approach Detailed definition of the evaluation criterion and of potential sub-criteria Description of the evaluations questions and sub-questions related to the evaluated criterion, and hypotheses related to this criterion; definition of key concepts Results and analysis Factors explaining the findings (causal analysis) or formulation and assessment of hypotheses Good or bad practice that can be documented, useful for and shared with the various categories of readers (examples, case studies, quotes from the interviews etc.) 4

5 6.3. Conclusion on (criterion #4) For each question and programme component: clear, short, objective, evidence-supported and nuanced response Overall assessment of the performance of the programme with reference to the evaluated criterion 7. Findings for evaluation criterion # Methodological approach Detailed definition of the evaluation criterion and of potential sub-criteria Description of the evaluations questions and sub-questions related to the evaluated criterion, and hypotheses related to this criterion; definition of key concepts Results and analysis Factors explaining the findings (causal analysis) or formulation and assessment of hypotheses Good or bad practice that can be documented, useful for and shared with the various categories of readers (examples, case studies, quotes from the interviews etc.) 7.3. Conclusion on (criterion #5) For each question and programme component: clear, short, objective, evidence-supported and nuanced response Overall assessment of the performance of the programme with reference to the evaluated criterion 8. Findings for evaluation criterion # Methodological approach Detailed definition of the evaluation criterion and of potential sub-criteria Description of the evaluations questions and sub-questions related to the evaluated criterion, and hypotheses related to this criterion; definition of key concepts For each question and programme component: Presentation of the type of data collected, data source and data collection method used to answer each question (bias mitigation, triangulation) 8.2. Results and analysis Factors explaining the findings (causal analysis) or formulation and assessment of hypotheses Good or bad practice that can be documented, useful for and shared with the various categories of readers (examples, case studies, quotes from the interviews etc.) 8.3. Conclusion on (criterion #6) For each question and programme component: clear, short, objective, evidence-supported and nuanced response Overall assessment of the performance of the programme with reference to the evaluated criterion 9. Conclusions - The report objectively summarizes the key findings by evaluation criterion, programme component, and relevant unit of disaggregation 9.1 Summary of conclusions by evaluation criterion (hightlighing both strengths and weaknesses), and implications for the programme 9.2 Summary of conclusions by programme component (hightlighing both strengths and weaknesses, and specifying which activities or stakeholders contributed the most/least to the results?) and implications for the programme 9.3 Summary of conclusions by sub-category of population, geographical area, etc. and implications for the programme 5

6 10. Recommendations - The report formulates relevant, feasible, users-oriented and prioritized actions logically following the findings and conclusions 10.1 Priority recommendations: Urgent or quick-wins actions (listed by stakeholders/target group, and by programme component) 10.2 More longer term recommendations, likely to be slower to plan and implement (listed by stakeholders/target group, and by programme component) 11. Lessons learnt - Contribution of the evaluation to general knowledge outside of the particular scope of this evaluated programme; good practices worth replicating in other areas, countries or sectors; bad practices to avoid and also to communicate to a wider audience as part of the sector learning 11.1 For the sector stakeholders (programmatic aspects) 11.2 For evaluators of similar programmes (methodological aspects) 11.3 Others 12. Annexes - Provides all data and information needed to better all aspects of the evaluation but that are too long to be included in the body of the report 12.1 Full evaluation matrix linking eval criteria to questions, to hypothesis, to data to be collected, to data source, to data collection methods and sampling (and possibly to criteria or standards that will be used by the evaluation team to make a judgement on whether or not UNICEF as performed well in each particular evaluation question) 12.2 Detailed agenda of the evaluation work 12.3 List of documents read and used 12.4 List of key informants interviewed (if they accept to be identified in the report) 12.5 List of sites visited 12.6 Data collection tools 12.7 Enumerators interview/survey guide (if applicable) 12.8 Full table/set of cleaned data 12.9 Synthesis of comments received on the draft report and how they have been taken into account in the final report (or why not) ToR of the evaluation Short biography / professional background of the evaluators Others 6