Asia: Earthquake and Tsunami Operation Evaluation Framework

Transcription

1 Asia: Earthquake and Tsunami Operation Evaluation Framework International Federation of Red Cross and Red Crescent Societies Monitoring and Evaluation Department Introduction Following the Asia Earthquake and Tsunami, the International Federation of Red Cross and Red Crescent Societies launched the largest-ever emergency, relief and recovery operation in its history. To date, combined resources of its members for the response total about CHF 2 billion. An operation of this size and involving funds generously donated mostly by individuals across the world demands high degree of accountability to both for whom the resources have been raised as well as the people from whom the resources have been raised. Red Cross and Red Crescent Movement, being the foremost among the humanitarian community in the world, has a vital leadership role to play in terms of how efficiency, effectiveness, impact and accountability of humanitarian operations are demonstrated by humanitarian actors. This document, by providing a framework for management of Asia Earthquake and Tsunami operation evaluations, forms a crucial step in ensuring that role is fulfilled. Guiding Principles The evaluations will be conducted in accordance to the following guiding principles: 1) Independence: The evaluation process should be carried out independently of program management and decision-making processes. This independence serves to safeguard the credibility of evaluations and prevent any potential conflict of interest. While all possible efforts should be put to reach the decisions through consultations, discussions and consensus building, the evaluations carried out by the Monitoring and Evaluation Department should have the independence from the line management of the program, activity or entity being evaluated at all stages of the process, including resource allocation for evaluations, formulation of terms of reference, staffing of evaluation teams, and clearance of reports. 2) Transparency: The findings of evaluations should be made public. Decisions regarding dissemination of findings should be taken at the beginning of an evaluation. Presentations of evaluation findings should be accompanied by a clear description of the object of the evaluation, its purpose, its intended audience, the questions asked, the methods used and their limitations, and the arguments and criteria which led to these findings. 3) Pluralism: The evaluations should consider in a balanced manner all the legitimate points of view expressed about the evaluated activity. This recognition of multiple viewpoints will be achieved, wherever possible, by involving beneficiaries and other various stakeholders in the evaluation process, or by other appropriate means. 4) Inclusiveness: The evaluations should reflect the fact that the Federation s work needs to be regarded as single set of programs rather than as programs of individual National Societies. In addition, PNSs whose funding is being used but which are not present need to have some feedback on the use of their funds; the learning opportunities for the Federation and the National Societies are increased by sharing monitoring and evaluation activities; it is wasteful of resources and unhelpful to beneficiary communities to multiply evaluation exercises beyond what is necessary for accountability and learning purposes. However it also is recognised that individual National Societies may want or need to carry out their own evaluations. In these cases they are encouraged to consult with the Monitoring and Evaluation Department regarding the best ways to integrate these exercises within the Evaluation Framework, and

2 to share the process and the products as much as possible with other Societies and with the International Federation. Evaluation Approach In addition, evaluations and related analytical functions will make a significant contribution to the achievement goals and objectives of the operation through being: focused on giving voice to beneficiaries. While the constraints of a fully participatory approach are recognised, the evaluations will attempt to understand the humanitarian response from a beneficiary perspective as much as possible. The ultimate criteria for judging the operations should be those which concern the victims and affected community, and these include their survival, livelihood, dignity, self-respect and rebuilding their homes in the midst of the destruction. Efficiency, effectiveness and impact will be measured or judged primarily in terms of these concerns of the affected community. appropriate and relevant to the needs of the International Federation of the Red Cross and Red Crescent and key stakeholders; balanced between learning and accountability; implemented in a collaborative and non-confrontational manner consistent with winning acceptance to the findings and recommendations by all interested parties, the host country National Society, the International Federation and the donors; reflective of the special position of the most vulnerable - marginalised groups, gender, children, and people with disabilities; coherent and cooperative with, and complementary to, the implementation of evaluation exercises of other actors and outside agencies; quality driven - implemented objectively, rigorously, and benchmarked to highest professional standards. Objectives The objectives of the evaluations conducted under this framework are to: 1) provide an independent and objective basis to review performance of all Asia Earthquake and Tsunami operations, their processes and systems; 2) continuously improve operation by providing lessons from experience; 3) ensure active accountability in the achievement of the objectives and utilisation of resources; 4) demonstrate accountability to our beneficiaries, public, partners and donors in the use of resources entrusted to us. Scope All objectives, sectors, activities, systems and processes related to the Asia Earthquake and Tsunami operation are subject to being covered by this evaluation framework. Specific evaluations to be performed will potentially encompass, but not be limited to, the following areas: 1) Adequacy of the planning and goal setting: the extent to which goals were adequate, appropriate, and clearly defined. 2) Speed of response: documenting the evolution of the response and commenting on decisions being made; examining whether the operations could be implemented faster without undue risk. 3) Identification of strengths and weaknesses in the International Federation operations and resource utilisation. 4) Relevance of the operations, including the: degree to which the operations are focused on needs of most vulnerable in the affected countries, and on topics and problems assigned priority by the National Societies; relevance of the operations to individual country s demands and needs, including the extent to which they are supplementary/complementary to the support provided by other sources;

3 extent to which the operations represents the most appropriate response from International Federation and take advantage of RC/RC comparative advantages and strategic priorities. 5) Opportunities for improving the operations, especially in terms of management processes and effectiveness, coordination and relationships, and financial management, as per the following definitions: Management processes and effectiveness: assessment of the operations management from Geneva and field, including decision making process; coordination and relationships: the nature and extent of functional relationships, cooperation and exchange of information within the Secretariat, among the Movement components, and with external actors; financial management processes: cost effectiveness of the operations, efficiency of financial processes (transfer, disbursement etc). 6) The responsiveness and efficiency of the disaster operation, including the: extent to which Federation is able to draw on the areas of particular competence of its members; cost-efficiency in producing the outputs; balance of staff and non-staff resources. 7) The quality of disaster operation (benchmarked to a) Performance Indicators 1, and b) other major active humanitarian organisations), including the: participation of the beneficiaries and volunteers in programme planning and implementation at local level; satisfaction of beneficiary needs in terms of factors such as appropriateness, relevance and implementation; comprehensiveness - in particular, the extent to which concerns including children, gender, people with disabilities, and vulnerability and poverty implications have been adequately taken into account. 8) Effectiveness and impact, including the extent to which: health, basic welfare, and water and sanitation systems in affected communities are rebuilt; assistance leads to sustainable capacity and livelihood's of the affected population; affected communities receive appropriate support, complementary to government efforts, to reconstruct housing and social infrastructure in a manner which is consistent, in keeping with local standards and expectations, conducive to community cohesion, and resilient to possible future disasters. livelihoods are re-established or improved to enable the affected population to become economically self-sufficient. operations result in enhanced institutional capacity of National Society, in terms of: a) capability to implement relevant programming under the Tsunami operation, and reach or exceed the standards of institutional effectiveness embodied in the Well-Functioning National Society concept; b) ability to prepare for and respond to disasters in a timely, efficient, and coordinated manner clearly linked to international response mechanisms (e.g. Early warning systems); and c) ability to mobilize communities at risk to cope with future disasters. 9) Coverage: The adequacy of planning to meet the needs in various geographical areas and sectors. 10) Compliance of all partners with related agreements, policies, and standards such as Sphere. Aspects related to adequacy of the agreements and relevance of the policy and standards should be commented upon. 11) Validity of assumptions made about inputs and outcome of the operation. More specifically, were important assumptions been left out or ignored? Did the operations have any unintended side effects, social or economic? 12) Red Cross/Red Crescent value-added in terms of principles and values, focus on most vulnerable, utilisation of local volunteers, etc. Periodicity A comprehensive round of real time evaluations, covering all activities, sectors and countries, will be conducted every four months. As the operation develops, the frequency of the evaluation rounds may 1 See Annex 1

4 decrease. Each evaluation round should consist of up to four independent evaluation exercises, either country-based or sectoral; one evaluation exercise per round should concentrate on the Secretariat management of the operations. Evaluation exercises should be, whenever possible, done simultaneously across the countries, so as to ensure that the results are comparable and allow for timely management decisions/actions. The turnaround period from start of the evaluation to the submission of final reports, including feedback on evaluation findings and conclusions, should not exceed 35 days. In addition, smaller reviews of the particular aspects of the operation may be launched at any time, upon request by management or initiated by the Monitoring and Evaluation Department in consultation with the management. Toward the end of the Asia Earthquake and Tsunami operation, a final evaluation of the response of the International Federation will be conducted. Evaluation Teams Each evaluation exercise will be conducted by an external independent consultant. In addition, sector departments, donors and National Societies in the operational countries will be given an opportunity to nominate team members; however, the team leader will always be an independent consultant. The decisions regarding the selection consultants and the composition of evaluation teams will be made by the Monitoring and Evaluation Department, through consultation with key stakeholders. The consultancy positions for every evaluation will be openly advertised. In addition, the Monitoring and Evaluation Department will utilize its Consultant Register, containing the information on potential expertise available across different sectors. The consultants will be pre-selected on the basis of their relevant expertise and according to the requirements outlined in the Terms of Reference. The pre-selected consultants will be asked to submit a bid, stating their availability and fee expectations, and may be invited to prepare a proposal, giving details of their suggested approach (in less than 500 words). Consultants may participate in more than one round of the real time evaluation. The evaluations will be managed independently by the Monitoring and Evaluation Department. Reporting, dissemination of results, and follow-up Management of the International Federation, as well as the stakeholders of the evaluation, will be given an opportunity to comment on the reports prior to their being finalized. The evaluation teams are, however, fully responsible for their independent reports, which may not necessarily reflect the views of the International Federation. The procedure for finalizing and distributing the reports follows steps outlined in the Operational Framework for Evaluations. On the basis of the evaluation reports, management will prepare a memorandum ( management memo ), commenting on all the recommendations of the evaluation reports. Based on the management memo, the completion memorandum will be prepared by the Evaluation Department, summarising all the recommendations made and including the proposed management response to each one. Focal points for each action will be identified, and, whenever possible, deadlines set. The completion memorandum will be signed off as approved by the relevant director. Follow-up on the implementation of the agreed responses will be submitted to the Monitoring and Evaluation Department and reviewed every four months. In addition, each round of real time evaluations will include listing all the recommendations suggested by the previous evaluations, and an assessment of the progress made in relations to each one. The Monitoring and Evaluation Department will coordinate the circulation, responses, transmission and preparation of the evaluation report, management memo, and completion memorandum for all evaluation exercises to all stakeholders. Final evaluation reports will be made public through the website of the International Federation.

5 Annex 1: Performance Indicators To the extent possible, all evaluations should reflect on the performance with respect to the following indicators: Performance Indicators 1. Fundamental principles and humanitarian values/advocacy RC/RC distinguished from other organizations by its values and principles RC/RC known to stand on the side of vulnerable people 2. Disaster preparedness did RC/RC have in place: trained local volunteers community based preparedness mechanism 3. Disaster response did RC/RC response result in: meeting basic needs of affected people (food/water/shelter) beneficiaries satisfied with service received no food riots or long lines for relief material in any of the affected areas 4. Health no disease outbreak other organisations know with clarity: what we do, and how we operate in an emergency situation known for substantial work in health and care 5. National Society meets performance indicators in the areas listed above known as credible and effective partner of choice of: governments international organizations civil society private sector 6. Working together effectively acting as one organization determined to maximize the full potential of our global reach and resources in order to achieve greatest impact for our beneficiaries and greatest influence on other actors all participating and host NSs feel equally valued and respected contributors Red Cross Red Crescent seen as the 'Heartbeat of Humanity' and is a welcoming space for all people who wants to help or be helped, regardless of nationality, race, religion, political affiliation or any such consideration Where we stand