Terms of reference GENERAL INFORMATION Title: International Consultant SC-DRR II Final Evaluation and Concept Note Writing Project Name : Safer Communities through Disaster Risk Reduction (SC-DRR) II Reports to: Evaluation Manager/Monitoring and Evaluation Analyst PMEU Duty Station: Jakarta Expected Places of Travel (if applicable): If travel/s required, will be upon approval from evaluation manager Duration of Assignment: End Oct 2015 End Dec 2015 (30 working days within 2 months) REQUIRED DOCUMENT FROM HIRING UNIT v TERMS OF REFERENCE 7 CONFIRMATION OF CATEGORY OF LOCAL CONSULTANT, please select : (1) Junior Consultant (2) Support Consultant (3) Support Specialist (4) Senior Specialist (5) Expert/ Advisor CATEGORY OF INTERNATIONAL CONSULTANT, please select : () Junior Specialist (7) Specialist (8) Senior Specialist v APPROVED e-requisition REQUIRED DOCUMENTATION FROM CONSULTANT v CV v Copy of education certificate v Completed financial proposal v Completed technical proposal ( if applicable ) Need for presence of IC consultant in office: partial (explain), meetings at the PMU SC-DRR office if required intermittent (explain) full time/office based (needs justification from the Requesting Unit) Provision of Support Services: Office space: Yes No Equipment (laptop etc): Yes No Secretarial Services Yes No If yes has been checked, indicate here who will be responsible for providing the support services: N/A I. BACKGROUND Safer Communities through Disaster Risk Reduction in Development (SC-DRR) II scales up initiatives delivered by Phase I of the Project (2007-mid 2012). Its main goal is to make disaster risk reduction a normal part of the development process, one that is embedded in core functions of the central and local government and its public and private partners. The project aims to achieve two main outputs: (a) Policy guidance developed to support the integration of DRR in development planning and specific sectors; (b) Technical capacities of DM actors strengthened to plan, implement and monitor DRR.
The National Agency for Disaster Management (Badan Nasional Penanggulangan Bencana BNPB) is the Implementing Partner of this project with engagement of the National Development Planning Agency (Badan Perencanaan Pembangunan Nasional/Bappenas) and Ministry of Home Affairs (Kementerian Dalam Negeri/Kemendagri) as members of project board. Strategic partnerships is maintained and strengthened with other relevant GOI ministries and agencies especially Ministry of Environment and Forestry (Kementerian Lingkungan Hidup dan Kehutanan/KLHK)and Ministry of Agrarian and Spatial Planning (Kementerian Agraria dan Tata Ruang/KemenATR), as well as with public and private partners at international, national and local levels. SC-DRR II has produced strategic policy recommendations and guidelines since its establishment in early 2013. Unlike its predecessor, this project has limited scope and resources, and has focused on developing policy guidelines and improving technical capacities for DRR planning and monitoring. Regardless its smaller size and resources, the project has been able to contribute to the formulation of critical plans and policy inputs for DRR mainstreaming. SC-DRR II was able to assist Bappenas in retaining DM in the national development agenda and as part of one of the countries seven priority agendas in the new RPJMN 2015-2019. It assisted MoHA/Kemendagri to develop a guideline to mainstream DRR within local development planning and budgeting, as well as drafting minimum service standard for DM. SC-DRR Phase II worked with the National Coordinating Board for Spatial Planning (Badan Koordinasi Penataan Ruang Nasional/BKPRN) and KemenATR to undertake a background study and develop a DRR-sensitive spatial planning guideline. Last, SC-DRR II helped BNPB to develop several chapters of DM Plan and worked with KLHK to formulate a national convergence framework on CCA-DRR. This evaluation will focus on assessing the implementation of the SC-DRR II project from the start of the project in 2013 to the end of the project in 2015. The evaluation will also cover the subsidiary projects under the umbrella of SC-DRR II such as: Enhancing Policy Planning for Disaster Risk Reduction (ED-DRR), Integrated Climate-induced Disaster Risk Management (IC-DRM) and Urban Climate Risks Management (UCLIM). The contributions of these initiatives and the overall achievement of the two aforementioned outputs are to be evaluated. The objective of assignment is two-fold i) conducting a final evaluation of SC-DRR II which substantiates good practices in DRR and lessons learned from the project implementation and ii) developing a concept note for a follow-up project. A report shall be formulated by the consultant in which includes a set of recommendations for immediate response by the project board. The concept note incorporates recommendations and feedback from project board and key ministries such as Ministry of Environment and Forestry and Ministry of Agrarian and Spatial Planning. II. SCOPE OF WORK, ACTIVITIES, AND DELIVERABLES Scope of Work The consultant will develop the intended deliverables under direct supervision of the Monitoring and Evaluation Analyst of UNDP s Planning, Monitoring and Evaluation Unit (PMEU). The evaluation shall substantiate good practices in DRR and lessons learned from project implementation in substances and processes. It will review programme successes and identify approaches that have not worked and/or require future development. In reviewing these approaches, it will advise on mechanisms for making DRR in development more effective. An evaluation report will be formulated by the consultant and will include a set of recommendations for integration into any future UNDP DRR initiatives. Based on the collection and analysis of data and information from relevant sources during the evaluation exercise, a concept note for follow-up interventions will be developed. The detailed purposes of the evaluation are as follows: 1. To assess to what extent the project has created an enabling environment which has helped shape government performance in the execution of service delivery in relation to DRR; 2. To identify good practices and lessons learned in the area of DRR; 3. To identify good practices and lessons learned in the area of project implementation modality; 4. To gain insights into the level of client satisfaction with the project. The clients include local government beneficiaries; national government partners and donors; 5. To review other DRR programmes currently operating in Indonesia and identify compatibility, overlaps and future links that will strengthen DRR across Indonesia.. To review the effectiveness of the gender mainstreaming strategy, as well as the effectiveness and efficiency of its partnership strategy and approach to capacity development; and
7. To provide recommendations for sustaining the benefits of the project and strategic issues and initiatives for a potential next project; 8. To develop a concept note for a future DRR project based on evaluation recommendations A sample of the beneficiaries of the project at national, provincial and/or district levels (government and nongovernment) will be consulted in the evaluation. The evaluation will apply standard evaluation criteria of relevance, appropriateness, efficiency, effectiveness, sustainability and impact. (for details: see pages 50-51 Handbook on Planning, Monitoring and Evaluating for Development Results); http://www.undp.org/evaluation/handbook. Relevance: Evaluate the extent to which intended outputs of the SC-DRR Phase II project are consistent with national and local policies and priorities and the needs of intended beneficiaries. Also evaluate the extent to which SC-DRR II was able to respond to changing and emerging development priorities and needs in a responsive manner. Appropriateness: Evaluate the cultural acceptance and feasibility of SC-DRR II. While relevance examines s the importance of the initiatives relative to the needs and priorities, appropriateness examines whether the initiative as it is operationalized is acceptable and feasible within the local context. Effectiveness: Evaluate the extent to which the intended results of SC-DRR II have been achieved. This includes an assessment of cause and effect- that is attributing observed changes to project activities and outputs. Assessing effectiveness involves three basic steps: 1) Measuring changes in the observed output, 2) Attributing observed changes or progress toward changes to the initiative, and 3) determining SC-DRR II contribution toward observed changes. Efficiency: Evaluate how economically resources or inputs (such as funds, expertise and time) were converted to results. An initiative is efficient when it uses resources appropriately and economically to produces the desired outputs. Sustainability: Evaluate the extent to which benefits of the SC-DRR II continue after external development assistance has come to an end. This includes evaluating the extent to which relevant social, economic, political, institutional, and other conditions are present and, based on that assessment making projection about the national capacity to maintain, manage and ensure the development results in future; Impact: Evaluate changes in human development and people s well being that are brought about by development initiatives, directly or indirectly, intended or unintended. Evaluate the extent to which gender mainstreaming is applied in project implementation to engage and empower vulnerable groups especially women. 4. EVALUATION QUESTIONS The evaluator will develop a list of questions and detailed methodology that will help generate information needed. Below is a sample of questions for reference for the evaluator: Were stated outputs or outcomes achieved? What progress toward the outcomes has been made? What factors have contributed to achieving or not achieving intended outputs? To what extent have the outputs contributed to the outcomes? Has the UNDP partnership strategy been appropriated and effective? What factors contributed to effectiveness or ineffectiveness? What could be improved or scaled-up in future interventions? Evaluation questions will be consulted with the project board who has commissioned the evaluation. The evaluator should detail evaluation methodology in an interim report. Details should include sources of information, sampling methodology, a work plan and a clear time frame. Questions should also be geared towards finding data and information to design a concept note for follow-up interventions in the area of DRR in Indonesia. The results from evaluation is synthesized into a concept note focusing on areas of intervention commensurate to UNDP strengths and comparative advantages, capacity gaps in disaster risk reduction and interest of donors. Partnership with project board and key ministries (e.g. Ministry of Environment and
Forestry and Ministry of Agrarian and Spatial Planning) should also be sufficiently analysed to provide an insight on enhancing future partnership including opportunities for Indonesian Government to share the cost with UNDP through Government Cost Sharing (GCS) arrangement. Sufficient consultations with key ministries will be sought to ensure alignment with development needs and priorities of Indonesian Government. 5. METHODOLOGY The evaluator will design a detailed step-by-step plan of work that specifies the methods the evaluation will use to collect the information needed to address the evaluation criteria, and answer the evaluation purposes, analyse the data, interpret the findings and report the results. The overall approach and methodology should yield the most reliable and valid answer to the evaluation questions and criteria within the limit resources (for more detail see pages 172-177 of Handbook on Planning, Monitoring and Evaluating for Development Results). http://www.undp.org/evaluation/handbook. Data Collection Methods Primary data: The evaluator may use questionnaires to collect primary data from beneficiaries, stakeholders, key informants, and expert panel. The data can also be collected through direct observation, interviews, focus groups, and case studies. SC-DRR II project will provide key information and documentation generated through monitoring during project implementation cycle. The information includes: Project document of SC- DRR II (this include: Result Resources Framework with detail indicators, baseline and target), Quarterly Monitoring Reports, Internal Project Assurance Reports (IPAR), Mid Term review of EP-DRR, Minutes of Board Meetings, Project Fact Sheet, Donor Reports, etc. Secondary data: Secondary data will be collected by the consultant from other sources that have direct relevance for the evaluation purposes. This includes amongst others: National Planning Documents (RPJMN) 2015-2019; Strategic Plan (Renstra) BNPB, Laws pertaining to DRR and DM; Policy recommendations and guidelines developed by the project, DM and DRR Action Plans 2015-2019 (whenever available), Monitoring and Evaluation reports of relevant projects / programme; Sampling: The sample must be selected on the basis of a rationale or purpose that is directly related to the evaluation purposes and is intended to ensure accuracy in the interpretation of findings and usefulness of evaluation results. Number of sample and sampling procedures for beneficiaries will be developed by the evaluator. Data analysis: The evaluator will develop the procedures used to analyse the data collected to answer the evaluation questions and criteria. It should details the various steps and stages of analysis that will be carried out, including the steps to confirm the accuracy of data and results. Findings: should be presented as statement of fact that are based on analysis of the data. They should be structured around the evaluations questions and criteria. Conclusions: Should be comprehensive and balanced, and highlight the strengths, weakness of the SC-DRR II project Recommendations: The report should provide practical, feasible recommendations that can be transpired into a project concept note for follow-up project. Lessons Learned: The report should include discussion on lessons learned for the evaluation that is newly gained from the particular circumstances. Concept Note: The document should incorporate the results and recommendations of final evaluation and input from project board and key ministries as mentioned above. Approach and Consultations The aforementioned objectives should be underpinned by an approach which is consistent with that outlined in the UNDP Handbook on Planning, Monitoring and Evaluation. This approach should also encompass the following:
A desk review of relevant project and governmental documentation including project documents, work plans, progress reports, Quarterly Monitoring Reports including Internal Project Assurance Report, national policies/laws, guidelines, plans and so on; Meeting and interviews with a broad range of DRR stakeholders including development partners such as Australian Government, NZAID, USAID, JICA, GFDRR, etc. A mixed evaluative approach including in-depth, participatory interviews with clients, partners and stakeholders at national level; An assessment of the needs and priorities of the GoI in the area of disaster risk reduction and the potential interest of development partners for collaboration and government interest on cost sharing Consultations with project and programme staff, relevant DRR stakeholders and; Field visits to project sites and activities for review of project results whenever necessary. Present the results of evaluation and concept note on follow-up project to project board SC-DRR II Reporting Requirements The Evaluation Consultant is expected to submit the following documents: 1. Inception Report 2. Draft evaluation report 3. Draft concept note 4. Final Evaluation Report 5. Final Concept Note. The Consultant is fully responsible for the content of the reports and concept note, which should be reviewed by the project board. The findings and conclusions may not necessarily reflect the views of the Government or UNDP. Logistical Arrangements The consultant will work closely with relevant UNDP Programme and Project team members, in particular monitoring and evaluation staff. Expected Outputs and deliverables Review/approval time required to review/approve the outputs prior to authorizing payments. The consultant will submit one electronic copy in MS Word format with following deliverables. Deliverables/ Outputs Estimated number of working days Completion deadline Review and Approvals Required (Indicate designation of person who will review output and confirm acceptance) Inception report, including evaluation questionnaire (15% payment) Draft Evaluation Report, draft Concept Note (35% payment) Final Evaluation Report, Final Concept Note (50% payment) 5 Day 5 PMEU, Evaluation Manager 15 Day 20 PMEU, Evaluation Manager 10 Day 30 PMEU, Evaluation Manager III. WORKING ARRANGEMENTS
Institutional Arrangement This evaluation would be conducted by the Planning Monitoring and Evaluation unit UNDP Indonesia. Table 1: Key roles and responsibilities in the evaluation process Person or Organization Roles and Responsibilities SC-DRR II Project Board as Determine which output will be evaluated and when commissioner of the Provide clear advice to the evaluation manager at the onset on evaluation how the findings will be used Respond to the evaluation by preparing a management response and use of findings as appropriate Take responsibility for learning across evaluation on various content areas and about evaluations Safeguard the independence of the exercise Allocate adequate funding and human resources Quality Assurance (DCD-P and Head of PMEU) Evaluation Manager: M&E Analyst (PMEU) Reference Group: Representative of the stakeholders: Staff from Bappenas; Staff from BNPB and staff from Kemendagri Evaluation Team : One international Review documents as required and provide advice on the quality of the evaluation and option for improvement Lead the development of the evaluation TOR Manage the selection and recruitment of the external evaluator Manage the contractual arrangements, the budget, and the personnel involved in the evaluation Provide executive and coordination support to the reference group Provide the evaluator with administrative support and required data Liaise and respond to the commissioners Connect the evaluation team with the wider programme unit, senior management and key evaluations stakeholders, and ensure a fully inclusive and transparent approach to the evaluation Review the inception report and the draft evaluation report; ensure the final draft meet quality standard Participate in drafting and review of draft TOR Assist in collecting required data Oversee progress and conduct of the evaluation Review the draft evaluation report and ensure final draft meets quality standard Fulfil the contractual arrangements in line with the UNEG norms and standards and ethical guidelines; this includes developing an evaluation matrix as part of the inception report, drafting reports, briefing the commissioner and stakeholders on the progress and key findings and recommendations as needed.
Figure 1: Proposed management structure for SC-DRR II project evaluation Commissioner (SC-DRR II) Project Board Quality Assurance (Head of PMEU) Evaluation Manager (M&E Analyst, PMEU) Reference Group (Staff from Bappenas and BNPB, Kemendagri and DFAT Australia) Evaluation Team (consultant) Duration of the Work Activities Time Frame Briefing of evaluator Day 1 Desk Review Day 1 to Day 5 Finalizing the evaluation design and methods and preparing the Day detailed inception report In-country evaluation mission (visit to the field, interviews, Day 7 to day 17 questionnaire) Preparing the draft reports Day 18 to day 25 Stakeholder meeting and review of the draft report (for quality Day 2 assurance) Incorporating comments and finalizing the evaluation reports Day 27 to day 30 Duty Station Jakarta. Travel Plan: None No Destination Frequency Duration/days None IV. REQUIREMENTS FOR EXPERIENCE AND QUALIFICATIONS Minimum of a Master s degree in management, social, political, economic sciences, public policy, regional planning and other related majors. Minimum 10 years experiences working in development in developing countries, preferably related to disaster risk reduction or recovery programmes Experience in evaluation, field research and report writing Familiar with monitoring and evaluation techniques including in-depth interview; focus group discussion and participatory information collection techniques;
First-hand knowledge of the Indonesian development context is required Fluent in spoken and written English. Proficiency in Bahasa Indonesia is an asset Strong writing and analytical skills Knowledge of Indonesian disaster risk management related policies, regulations and plans Substantive experience in drafting concept note is preferable Excellent communication and interpersonal skills, team oriented work style, interest and experience of working in multi-cultural environment. Ability to work independently with minimum supervision Willing to work at tight deadlines and under pressure V. EVELUATION METHOD AND CRITERIA Individual consultants will be evaluated based on the following methodology: Cumulative analysis When using this weighted scoring method, the award of the contract should be made to the individual consultant whose offer has been evaluated and determined as: a) responsive/compliant/acceptable, and b) Having received the highest score out of a pre-determined set of weighted technical and financial criteria specific to the solicitation. * Technical Criteria weight; [70%] * Financial Criteria weight; [30%] Only candidates obtaining a minimum of 70 point would be considered for the Financial Evaluation Criteria Weight Maximum Point Technical Criteria A: qualification requirements as per TOR: 1. Relevance of Education/ Degree 2. Years of Relevant Experience 3. Regional Experience 4. Adequacy of Competencies for the Assignment 5. Others / Special Skills, Language, etc. 30% Criteria B: Brief Description of Approach to Assignment 70% 70 Criteria C: Further Assessment by Interview (if any) n/a