Evaluation of Environmental and Social Impact Management at ADA Terms of Reference

Transcription

1 Evaluation of Environmental and Social Impact Management at ADA Terms of Reference 27 June Background Environmental and Social Impact Management (EGSIM) The Austrian Development Cooperation (ADC) has been taking measures to manage environmental and social impact for a considerable time. Processes for environmental and gender assessments of projects were introduced in the 1990s and have been continuedly revised. Also, government policies or specific policies drafted by the Austrian Development Agency (ADA) 1 on related issues (Gender, Human Rights, Environment, Poverty Reduction, Good Governance) were put in place. It was in July 2015 when the Environmental and Social Impact Management was formally launched and a Manual was approved and published. The manual aimed to assist applicants and ADA staff in developing interventions that go in line with ADA s environmental and social standards where environmental and/or social risks which may stem from the intervention are identified and addressed accordingly. The Manual attempts to establish a systematic approach to managing environmental and social impact, with special attention to the project appraisal process. It describes: - overarching policies and principles relating to poverty reduction, gender equality, environment, human rights, good governance and peacebuilding - project level standards, including an exclusion list - assessment procedures (standard and simplified procedure) for project and programme appraisal In retrospect, ADA management acknowledged that there were several unresolved issues at that time, including 2 : - the question of how to deal with strategic partnerships or third-party funded projects 3 that ADA implements directly - the question of how to deal with emergency relief and humanitarian aid that usually has short planning periods - the distinction between quality assurance and risk management - the monitoring of recommended mitigation action stemming from the ESIM process 1 The Austrian Development Agency (ADA) was founded in 2004 as the operational unit of Austrian Development Cooperation. 2 Environmental, Gender & Social Impact Management System Introduction, Presentation at ADA Programme Manager and Advisors Meeting, 19. April Mainly EU funded projects delegated to the ADA for implementation - 1 -

2 Green Climate Fund accreditation process and ESIM s successor In 2015, ADA started a process of accreditation with the Green Climate Fund (GCF). GCF is a funding organization that invests in adaptation and mitigation activities in developing countries implemented by its accredited partner organisations. Apart from fiduciary standards, GCF looks at the ability to manage environmental and social risks as well as adherence to its Gender Policy. During the accreditation process, the following issues relating to ADA s ESIM were identified by GCF 4 : - International Finance Cooperation (IFC) environmental and social performance standards are insufficiently addressed - unclear differentiation of risks assessment and risk management in terms of procedural steps and allocation of responsibilities within the institution - lacking guidance on the Who and How - insufficient transparency towards beneficiaries and stakeholders (other than project partners) - lack of a dedicated channel for grievances by beneficiaries and stakeholders (e.g. an address or hotline for complaints) Environmental, Gender and Social Impact Management (EGSIM) In response to this criticism ESIM has been revised into an updated version, called Environmental, Gender and Social Impact Management (EGSIM) 5. It was introduced to ADA staff during a workshop mid-april 2018 and is being rolled out gradually. ADA staff has been advised to start applying the EGSIM for those projects that are still in planning stage from May 2018 onwards. For projects in process for approval, where the change in the procedure would lead to additional work, it was recommended to use the old system 6. Management of E(G)SIM The unit responsible among several other functions - for designing and overseeing the implementation of E(G)SIM is the ADA Unit on Themes and Quality (T&Q), a unit within the Department for Programmes and Projects International. A key aspect of managing E(G)SIM is the risk assessment during project appraisal. It is currently conducted by: - two Advisors for Environment & Natural Resources (environmental appraisal), and - three Advisors for Governance, Rural Development and Peace and Security (social impact appraisal), - an Advisor for Gender, supported by a JPO, whereby the Advisor for Governance functions as alternate. Furthermore T&Q unit has the function to monitor the implementation of the recommendations made in the appraisal process. 4 Ibid. 5 The EGSIM Manual can be accessed online: nd_social_impact_management/egsim_manual_juni2018.pdf 6 from the Head of of Themes and Quality Unit to Programme Managers and Advisors, 8. May

3 ESIM versus EGSIM ESIM EGSIM Effective since Jul 2015 June 2018 Underlying documents Main purposes: Includes: Covers: New aspects (compared to before) Source: ADA Evaluation Unit. The evaluation Manual Checklists for project appraisal Description of procedure Risk mitigation Enhancement of development results Capacity development overarching policies and project-level standards short section on delivery system all interventions in developing countries funded by ADA xxx Social Impact standards Differentiates between standard and simplified procedure Exclusion List Manual Staff Guidance Note Checklists for project appraisal Description of procedure Risk mitigation Enhancement of development results Capacity development overarching policies, the principles and standards processes for its application all interventions in developing countries funded or implemented by ADA, with a few exemptions Additional guidance for ADA staff is provided in the EGSIM Staff Guidance Note Different risk categories (two-phased approach) Disclosure policy and grievance mechanism One checklist incorporating all topics ADA s Evaluation Unit 7 (EVAL) is commissioning a strategic evaluation of the Environment and Social Impact Management at ADA. The evaluation is to be conducted by a team of external consultants between August 2018 and January These ToR are the main planning document that describes the purpose, scope and main evaluation questions. They have been drafted through a consultative process with inputs from the main stakeholders. The further refinement of the approach and methodology, as well as a detailed work plan will happen in the inception phase (after contracting of the evaluation team). 2. Evaluation purpose E(G)SIM is a relatively new management system and involves changes in ADA s project and programme appraisal and management. At the time of the publication of the manual it was envisaged that a review would take place in 2017 to provide for early learning and potential adjustments. The ongoing accreditation process with the GCF, which had already induced an adaptation from ESIM to EGSIM was given as another reason to include it in ADA s strategic evaluation plan A subgroup of ADA s Executive Unit of Evaluation and Statistics

4 The primary purpose of this evaluation is to learn from the recent experience of ADA in introducing and implementing different aspects of E(G)SIM. The evaluation aims to provide evidence and lessons as an input into the ongoing process of implementing the follow up system, EGSIM. It will provide recommendations that likely consider a variety of scenarios (in terms of resource needs and prioritization) for moving ahead with EGSIM. It will analyze the introduction and application of ESIM and its quality and reflect on how EGSIM can be optimally used to help ADA achieve its objectives. In that regard the evaluation wants to contribute to the further development of EGSIM by increasing its relevance, efficiency and effectiveness. ADA management is committed to use the evaluation findings and recommendations for this purpose. The primary users of the evaluation results are - ADA senior management and strategic decision makers, and the actual users of E(G)SIM, consisting of: - the E(G)SIM appraisal team (T&Q) - implementers of the results of E(G)SIM appraisals (project owners) - those responsible for following up and monitoring the implementation of the results of the E(G)SIM appraisals (project owners, ADA programme managers, coordination offices) 3. Evaluation scope and key questions 3.1. Scope The timeline to be covered is from mid-2015 (introduction of ESIM) until mid-2018 (introduction of EGSIM). The evaluation will analyse the underlying management framework of ESIM (on paper) as well as its actual implementation process (in practice). EGSIM will be assessed ex-ante, in terms of its design as a follow up system of ESIM (on paper). The evaluation will focus on the processes and procedures at project and programme as well as organizational level. It will explore envisaged outcomes at the organizational level and review results of a sample of projects and programmes. (overall ADA). The evaluation will focus on the processes at ADA but also assess implications for its partner organizations. The evaluation will explore the main drivers for moving towards better environmental and social impact management as well as the constraints that have been experienced or have been perceived. It will also assess achievements so far in implementing ESIM in ADA s specific context. This evaluation will focus on the assessment of the criteria relevance, efficiency and effectiveness

5 3.2. Key questions The key evaluation questions to be covered by the evaluation are: Design and fit for ADA (relevance) covers both ESIM and EGSIM, differences between the two will be noted but not actively explored - Is E(G)SIM a relevant tool to increase ADA s contribution to the international development agenda, particularly the SDGs? - What have been the main drivers and strategic considerations to introduce ESIM and later EGSIM? Are they in line with the strategic priorities of ADA? - Have both management tools been adequate and useful for fulfilling their purpose? - To what extent is E(G)SIM in line with ADA s project and programme management and current working conditions and structures? Management (efficiency, effectiveness) focuses on the experience of implementing ESIM - How well was ESIM implemented at organizational (design of processes and quality standards, change management, etc.) and project/programme-level? - To what extent did ESIM support project owners in the coherent management of environmental and social impact? - How well did ADA support the implementation? What parts of the support were effective or not so effective? - Have resources dedicated to managing ESIM been appropriate? Have there been bottlenecks or/and opportunity costs because of ESIM? - How do the benefits of introducing ESIM relate to the cost? Moving from ESIM to EGSIM (relevance, efficiency, effectiveness) focuses on the organizational development and change process of moving from one system to the other - What have been factors supporting or hindering the implementation of ESIM? - To what extent does the updated EGSIM address shortcomings of ESIM s conceptualisation and implementation? Results (effectiveness) covers both ESIM and EGSIM, differences between the two will be noted but not actively explored Organizational level: - What does ADA intend to achieve with E(G)SIM and are those objectives clear to the main stakeholders? - What are the actual likely outcomes of introducing E(G)SIM? - 5 -

6 Project and programme level: - What have been the effects (intended and unintended) on the project and programme performance? - What are the perceived advantages and disadvantages of using E(G)SIM? 4. Evaluation approach and methods 4.1. Evaluation approach The evaluation of E(G)SIM will be formative, in that it will examine the implementation process of ESIM and aims to provide inputs into the further development and use of EGSIM. Thus, it has elements of an ex-ante evaluation of EGSIM, as the prospects of successful implementation of EGSIM will be analysed before its implementation. The evaluation will have a non-experimental design and will rely on qualitative data. Accordingly, methods and sources will be chosen to assure data and methods triangulation 9, thus enabling multiple perspectives on every evaluation question. As no Theory of Change of E(G)SIM exists to date, the evaluation will collect information on the possible causal pathways, including assumptions. The aim is to construct a Theory of Change that will guide the inquiry and will be reviewed and adapted through an iterative process with evaluation stakeholders and informants. It will be participatory and will aim to involve relevant stakeholders throughout the evaluation process. This will be done to assure the relevance and subsequently the use of the evaluation results 10. As a means to this end, an evaluation reference group will be created and regular feedback loops with ADA management established. The focus will be on internal stakeholders, especially the users of E(G)SIM, which have been specified under evaluation purpose. The evaluation follows a utilization-focused approach in that there should be clarity from the beginning about: 11 - what the evaluation will contribute to - who will be the main users of the evaluation results and - how it will be disseminated and used Considering the evaluation will likely identify issues affecting E(G)SIM performance that lie outside the actual scope of influence of E(G)SIM implementers, it is important that ownership from ADA senior management is created. Also, the recommendations should be clearly addressed to the different stakeholders and management response carefully overseen and followed up. 9 See for example Flick, U. (2004): Triangulation: Eine Einführung. Springer. 10 Patton, M.Q. (2008). Utilization-focused evaluation, 4th edition. Thousand Oaks, CA: Sage. 11 Ibid

7 The evaluation team will build up its work on the relevant OECD DAC standards for evaluations of the OECD DAC and those of the Austrian Development Cooperation, where applicable. All work needs to be documented by using transparent criteria Evaluation methods The evaluation is desk-based and no field visits are foreseen. Face-to-face interaction with evaluation stakeholders will happen at the kick-off meeting and at the meeting for the presentation of the draft report. Evaluation methods will likely include: A. Portfolio review (preparatory work by EVAL) It is planned to conduct a review of the appraisal process of projects and programmes since 2015 when ESIM was applied through ADA s grant management system (FMS). The FMS allows the evaluation to track information on the assessment and following appraisal process of environment, gender and social impact. A portfolio level analysis should be done at the beginning of the evaluation. Data to be analyzed can include, for example: - Distribution of different assessments - Distribution of different assessment processes (external, internal, etc) - Types of projects (funding modality, funded by ADA or implemented by ADA under third-party funding, etc) and their appraisal history - Other descriptive analysis B. Synthesis of evaluative information Evaluations that cover aspects of ESIM should be reviewed for evidence that is relevant for answering the key evaluation questions. A preliminary list is below: - Recent DAC Peer Reviews (2015 and 2009) - Evaluation of the Environment policy of the Austrian Development Cooperation and its implementation by the main ODA actors between (2016) - Evaluation of the Austrian Development Cooperation (ADC) Gender Policy between (2012) - Impact assessment of environment and development Western Balkans - Evaluations of bigger projects - Other evaluated evidence C. Desk review Relevant documents to be reviewed will include - ESIM Manual and underlying policies - Documents provided to the GCF during accreditation process - EGSIM Manual and Staff Guidance Note (Drafts) - 7 -

8 - Documents about the process of drafting ESIM and EGSIM - Internal reviews, reflection about ESIM - Documents, including evaluations about other environmental and social impact management systems, especially of other donor organizations, including evaluations D. Interviews and focus group discussions Key informants will be interviewed through semi-structured interviews or, where useful, through focus group discussions. They will include: - ADA senior management - ADA management responsible for ESIM implementation - ADA project and programme managers - Partner organizations - Other comparable organizations with experience in introducing environmental and social standards E. In-depth process analysis To better understand how ESIM has worked in practice, a few projects and/or programmes (5-8 in total) will be selected for which the process of applying ESIM and results will be assessed. Here, projects which received substantive feedback regarding environmental, gender and social impact standards should be considered. This will be done through document review and selected interviews, covering both ADA as well as partner organizations F. Cost/benefit analysis To assess how the costs (staff time, money) of implementing E(G)SIM relate to the achieved and likely achieved results (outputs, outcomes), the evaluation will conduct a separate analysis. The exact methodology will be defined in the inception phase Evaluation challenges ESIM or EGSIM?: It might be difficult in relation to the data collection to make a clear distinction between ESIM and the follow up system EGSIM. The differences between the two management systems should be established right at the beginning of the evaluation. In the inception report, define very clearly when such a differentiation is useful and when ESIM, EGSIM or both (E(G)SIM) will be assessed. ESIM is relatively new: ESIM has only been introduced mid-2015 and is already in the process of being adapted. Therefore, from an organizational perspective this is a relatively short assessment period

9 The evaluation will focus on the project and programme level and the process of implementing ESIM. Causalities between ESIM and achieved outcomes will only be looked at in the case studies and not at overall ADA level. The perceived added value of E(G)SIM: It is expected that the evaluation will find a lack of consensus on the value and relevance of environmental and social impact management for ADA. The evaluation will interact with as many different stakeholders as possible. When there are differences in opinion this will be analyzed and noted rather than attempting to make an overall conclusion. 5. Envisaged process and timeline The evaluation is scheduled to take place between August 2018 and January The evaluation will be commissioned by ADA Evaluation Unit which will be responsible for managing the evaluation and providing quality assurance. ADA s Themes and Quality Unit which oversees the implementation of E(G)SIM will support ADA Evaluation Unit in facilitating data collection for the evaluation team. An Evaluation Reference Group consisting of key stakeholders (T&Q, Unit Countries and Regions, Executive Units Risk-, Third Party Funding & Knowledge Management and Evaluation, project managers, etc.) will provide inputs at important stages of the evaluation (ToR, inception report, preliminary findings, draft report). The Evaluation Team will be responsible for the conduct of the evaluation and for the timely submission of deliverables. PREPARATORY PHASE Timeframe Scope of Work May 2018 Establishment of Evaluation Reference Group June 2018 ToR June 2018 Recruitment of evaluation team (incl. expert panel) Mid Aug 2018 Portfolio Analysis (through EVAL) Aug 2018 INCEPTION PHASE (estimated total of 15 working days) Document review and portfolio analysis Aug 2018 Kick Off Meeting in Vienna Early Sep 2018 (after summer break) Inception Report Sep 2018 INQUIRY PHASE (estimated total of 35 working days) Evaluation synthesis Sep- Oct 2018 Data Collection Sep Nov 2018 Data Analysis & Synthesis Nov 2018 Presentation of preliminary findings in Vienna Nov 2018 REPORTING PHASE (estimated total of 15 working days) Draft Report Submission Dec 2018 Presentation of Draft Report (solid final draft report) at Dec 2018 ADA, Vienna travel to be budgeted by evaluation team Final Report Submission/Approval Jan

10 6. Deliverables and payment The evaluation team is expected to deliver the following products: draft and final inception report, including a preliminary ToC, analytical framework as well as data collection instruments presentation of preliminary findings (for virtual meeting) draft and final evaluation report All reports must be written in English. The final report must contain an executive summary and a list of recommendations made by the evaluation team. It should not comprise more than 40 pages (excluding annexes). An advance payment of one-third of the total budget will be transferred upon acceptance of the inception report by ADA. A second payment will be released upon reception of the draft report. The third and final payment will be settled by ADA after the submission and approval through EVAL of the final report. 7. Evaluation team The evaluation should be conducted by a team of two consultants. The team leader will take the lead in designing the methodology of the evaluation and writing the evaluation report. The team should combine the following qualifications: Experience/knowledge: Extended experience in conducting strategic and thematic evaluations in the international development cooperation arena Extended experience in organizational development and change management Extended experience with environmental, gender, and social impact management systems Familiarity with ADC is a major asset Languages: Fluency in both oral and written communication in English is a requirement Fluency in German is a strong asset Other skills: Strong analytical skills and ability to work with and as a part of teams Excellent report-writing and editing skills Excellent data presentation and visualization skills

11 8. Contracting and selection process For the hiring of the evaluation team, ADA will pursue a direct contracting process. Several experts/companies will be invited to submit a proposal. Furthermore, a call for expression of interest (including the Terms of Reference) will be published in a variety of platforms and networks. Interested candidates/firms should provide a technical offer (maximum 10 pages) and the estimated costs. The technical offer should include the following: Brief description of the understanding of the assignment Presentation of a proposed approach, particularly regarding the cost/benefit analysis Presentation of a detailed work plan including time schedule and division of tasks Detailed CVs and references of similar assignments of the team leader, the other evaluator and the consulting firm (if applicable) as annexes The cost proposal (financial offer) should follow the following breakdown (in EUR): Personnel costs: name of experts, estimated number of working days, fee rate per working day, total amount Costs for kick off meeting and meeting to present draft report in Vienna Other costs (for communications, etc) VAT (if applicable) The received proposals will be assessed by a commission. The technical and financial proposals will be weighed 70:30 respectively. Criteria that will be used to assess the proposals include: Expertise, relevant experience and suitability of evaluation team Relevant experience and capacity of consulting firm (if applicable) Quality and coherence of the proposal, including methodology, timeline and work plan Methodological rigor and individuality of approach that matches this evaluation s information needs Coherence of financial and technical proposal; efficient use of resources. Please note that personal data included in the proposals (like CVs) will be stored and used by ADA internally during the review of the Proposals. Proposals should be submitted until 30 July 2018 to Ms. Sophie Zimm, Evaluation Advisor at the Austrian Development Agency: sophie.zimm@ada.gv.at Please use Evaluation of ESIM Proposal as subject header