Internal Monitoring and Evaluation Guidelines

Transcription

1 Internal Monitoring and Evaluation Guidelines DISS-M&E-GL-1.0

2 CONTENTS INTRODUCTION AND METHODOLOGY WHEN TO CONDUCT M&E? WHAT TO MEASURE? M&E METHODOLOGY HOW TO TRACK AND REPORT M&E ACTIVITIES?... SECTION ONE : USER MONITORING SECTION TWO: REFERENCE MONITORING... 6 SECTION THREE : USAGE MONITORING AND EVALUATION... 8 SECTION FOUR : ENGAGEMENT MONITORING... 9 TEMPLATES AND TOOLS DISS-M&E-1.1.: M&E TEMPLATE IMPACT INTERNAL MONITORING AND EVALUATION GUIDELINES - JUNE

3 INTRODUCTION AND METHODOLOGY Monitoring and Evaluation (M&E) is a key component of all IMPACT programms, aiming to evaluate their impact and assess whether results are achieved. This is key both for internal capitalisation and lessons learned, feeding into the improvement of ongoing and future programs, and for transparency towards external stakeholders, such as humanitarian partners and donors. 1.1 WHEN TO CONDUCT M&E? M&E activities assess the entire programme cycle as shown in Figure 1 below (refer to activities in yellow), including engagement with external actors, project implementation, outputs/products, dissemination/utilisation and the impact of the project. Figure 1: IMPACT Programme Cycle Management Engagement with external actors Resources mobilization/ resources Results Programme Design / PD Project inception Project implementation Outputs/ products Outcomes/ utilization Impact Country strategy Implementation Monitoring Evaluation Regional strategy Drafting of dissemination plan Approval of dissemination plan Implementation of dissemination plan Evaluation of dissemination impact Dissemination An M&E framework should be developed for all research and projects. Research M&E frameworks should be incorporated in Research Terms of Reference, while Projects M&E frameworks should be incorporated in proposals. Specific, Measurable, Attainable, Relevant and Time-bound (SMART) indicators, aiming to measure and evaluate programme activity impact, should be developed for proposals and Research TORs based on IMPACT programme objectives and measured during and after programme implementation. IMPACT INTERNAL MONITORING AND EVALUATION GUIDELINES - JUNE

4 1.2 WHAT TO MEASURE? The following specific objectives of IMPACT programmes will be assessed through the M&E framework: 1. Humanitarian stakeholders are accessing IMPACT products 2. Humanitarian stakeholders are using IMPACT products 3. Humanitarian stakeholders are using IMPACT products > to inform decision-making 4. Humanitarian stakeholders take ownership of IMPACT outputs 5. IMPACT activities contribute to better humanitarian programme implementation 6. IMPACT activities are contributing to better coordination of the humanitarian response The humanitarian coordination mechanism that IMPACT operates within can be divided into three levels, strategic, coordination and operational (see Figure 2 below). Sampling when implementing the IMPACT M&E framework is conducted according to these three strata, with accompanying strata specific methodologies and tools (please refer to DISS-M&E-1.1.). Figure 2: Humanitarian Coordination Strata STRATEGIC HCT, HC, UN Rep, NGO CD, Senior donor representatives, Government representatives Assessed by: IMPACT Global Coordination Team during country deployments Assessed by: Country Offices COORDINATION Cluster/Sector coordinators, Cluster/ Sector members, Government coordination counterparts Assessed by: OPERATIONAL Country Offices Program Managers, field staff, local authorities IMPACT INTERNAL MONITORING AND EVALUATION GUIDELINES - JUNE

5 1.3 M&E METHODOLOGY Different methodologies can be used to monitor and evaluate IMPACT programmes throughout the programme management cycle: 1. User monitoring 2. Reference monitoring 3. Usage monitoring and evaluation 4. Engagement monitoring The choice of the methodology will depend upon what we want to measure and the stage of the project/research implementation. As mentioned above, an M&E framework should be developed for all projects/research and included in proposals and Research TORs respectively. 1.4 HOW TO TRACK AND REPORT M&E ACTIVITIES? M&E activities should be tracked and report to country and HQ focal points using IMPACT s M&E tool (DISS-M&E-1.1). This tools includes an indicator list and data collection tools for each M&E activity. One M&E file should be completed for each project or research (as agreed with IMPACT HQ at project/programme inception phase) and shared with IMPACT HQ on a monthly basis throughout the implementation, to be reviewed by the Global Communications and Dissemination department. M&E data should also be stored in country level project files and, when relevant, shared with ACTED project development departments to ensure that data collected is referenced as appropriate in interim/final donor reports. IMPACT INTERNAL MONITORING AND EVALUATION GUIDELINES - JUNE

6 SECTION ONE : USER MONITORING 1 Humanitarian coordination strata sampled: Strategic, Coordination, Operational. STRATEGIC OBJECTIVE 1: Humanitarian stakeholders are accessing IMPACT products User monitoring will be conducted to identify the number of users accessing IMPACT products, such as reports and maps. This allows IMPACT to track the effectiveness of dissemination and to identify whether adjustments are needed to dissemination activities, where findings indicate that the recorded number of users is lower than the expected number of users. User monitoring data collection will be conducted using a quantitative methodology, simply counting the number of instances of access using Google Analytics (by IMPACT HQ); requests to Reliefweb (IMPACT HQ) and other relevant platforms (Country teams); Mailchimp (IMPACT HQ); and Map request and printing logs (Country teams). Tool: TO BE PREPARED SECTION TWO : REFERENCE MONITORING 2 Humanitarian coordination strata sampled: Strategic and Coordination. STRATEGIC OBJECTIVE 2: Humanitarian stakeholders are using IMPACT products STRATEGIC OBJECTIVE 3: Humanitarian stakeholders are using IMPACT products > to inform decision-making STRATEGIC OBJECTIVE 6: IMPACT activities are contributing to better coordination of the humanitarian response Reference monitoring will be conducted to measure the number of documents where IMPACT outputs have been used as secondary data and identify occasions where IMPACT outputs have been shown to inform decision-making. The reference monitoring is closely linked with REACH product objectives, which are expected to inform a) key milestones of the Humanitarian Programme Cycle (HPC) and b) agency-specific strategies, analysis and aid delivery planning. Reference monitoring will be conducted using quantitative data collection: Reference check of HPC key milestone documents (refer to Figure 3), such as Humanitarian Needs Overview, Strategic Response Plans, Flash appeals, cluster/sector strategies and other key milestones. Such milestones should be identified by the country team during programme design/ programme inception phases, and reflected in the dissemination plan (refer to DISS-EXT-1.1.). IMPACT INTERNAL MONITORING AND EVALUATION GUIDELINES - JUNE

7 Figure 3: Humanitarian Programme Cycle Management Reference check of single agency documents should also be conducted by country/regional/global teams (for country/regional/global documents respectively) on a rolling basis, to track the number of references to IMPACT products and to assess the influence of such products on agencyspecific decision-making. Reference monitoring data collection will be conducted using a quantitative methodology, simply counting the number of instances as follows: 1. Count frequency of references to IMPACT products in the HPC and single agency documents. 2. Identify and describe instances where IMPACT products (primarily REACH) informed and influenced the decision making process of the HPC e.g: i. Priority areas for aid delivery were identified based on the REACH assessment; ii. Number of WASH units to be constructed in the camp were calculated based on REACH WASH assessment. Tool: Reference log tool in DISS-M&E-1.1. IMPACT INTERNAL MONITORING AND EVALUATION GUIDELINES - JUNE

8 SECTION THREE : USAGE MONITORING AND EVALUATION 3 Humanitarian coordination strata sampled: Strategic, Coordination, Operational. STRATEGIC OBJECTIVE 4: IMPACT activities contributes to better humanitarian programme implementation. Usage monitoring and evaluation will be conducted to assess how IMPACT outputs are used and how IMPACT programme activities are perceived. This M&E component is thus central to identify best practices and lessons learned at country/regional/global level. Usage M&E data collection will be conducted using different tools tailored to the three humanitarian coordination strata (see Figure 2 above): 1. Strategic level: IMPACT HQ or regional (as identified by HQ) staff on field deployment should conduct individual interviews with key stakeholders (ex. UN Rep, Humanitarian Coordinator, NGO Country Directors, senior donor representatives, etc.) during meetings with key partners. During such meetings, the IMPACT representative should inquire on the perception of IMPACT country-programmes, which can be related to their relevance and contribution to the coordination of the humanitarian response, the interaction of IMPACT staff with the various stakeholders, their capacity to build ownership and influence aid practices, the utility of IMPACT programme outputs (ex. Reports, maps) for their specific agency, etc. Recommendations on how to improve/ re-adjust the strategic direction of IMPACT programmes should also be gathered during such meetings, and fed back to country teams for review. Tool: Strategic_level_interview tool in DISS-M&E Coordination level: Country teams should gather feedback before (baseline) and after (endline) programme implementation at two levels a) Mandating body (such as Cluster/sector coordinators, bilateral donors, etc.) and b) Cluster/sector members. Feedback should enable comparison of the initial expectations of the mandating body and its members on the IMPACT programmes against perception of final product. It should also evaluate their engagement in IMPACT programmes (e.g. development of ToRs, questionnaires, provision of resources) and evaluate how they used the information provided in IMPACT products for planning/delivery of aid. Similar to the strategic level, the feedback should also include any suggestions/recommendations for improving programme implementation and impact of IMPACT outputs. This qualitative data collection using a semi-structured tool will be conducted as follows: a. Mandating body (cluster/sector coordinators, donors, etc.): Structured feedback from coordinators via or individual face-to-face interviews (with minutes shared with participant). b. Mandating body or cluster/sector members: Individual or group discussions (with minutes from the discussion shared with all participants). Tool: Coordination_level_interview tool in DISS-M&E-1.1 IMPACT INTERNAL MONITORING AND EVALUATION GUIDELINES - JUNE

9 3. Operational level: Country teams should gather feedback from individuals working directly on humanitarian programme implementation (e.g. programme managers, field staff, local authorities, etc.), using structured questionnaires as follows: a) After implementation: a short online questionnaire (max 10 questions) should be administered following implementation of key activities (e.g. food security assessment) to gather feedback on programme implementation (engagement with stakeholders; perceived technical capacities of IMPACT teams) and outputs (usefulness for informing programme design/implementation; quality of outputs), as specified in DIS-EXT-GL-1.0. b) During implementation [at humanitarian hubs only]: a short printed feedback form should be administered at humanitarian hubs for visitors to the hub to provide feedback on the relevance of the service delivered, the timeliness, the quality of the information, the process for requesting information, and to also provide suggestions on how to improve the functionality and relevance of the hub. Tool: TO BE PREPARED SECTION FOUR : ENGAGEMENT MONITORING 4 Humanitarian coordination strata sampled: Coordination and Operational. STRATEGIC OBJECTIVE 5: Humanitarian stakeholders take ownership of IMPACT outputs STRATEGIC OBJECTIVE 6: IMPACT activities are contributing to better coordination of the humanitarian response A key element for the success of IMPACT programmes, especially REACH, is the engagement with external stakeholders throughout the programme management cycle. End-users are more likely to use information provided by sources they believe to be credible. Sources are more likely to be considered credible if the end-user harbours a sense of ownership and the source has buy-in from key stakeholders. By conducting pro-active consultations and encouraging collaboration with key humanitarian stakeholders during the design, planning and implementation of programme activities, IMPACT programmes can increase their level of credibility and consequently the level appropriation and effective use of programme outputs. Within the IMPACT PCM, the process of developing information products is thus as crucial as the quality of the final product. Engagement monitoring could relate to the provision of resources (staff, cars, etc.) by other agencies, their engagement in the development of the research TORs, indicators, etc. Engagement monitoring will be conducted by country teams by completing a simple quantitative tool measuring frequency in types of engagement, throughout the programme cycle. Tool: Engagement_log in DISS-M&E-1.1. IMPACT INTERNAL MONITORING AND EVALUATION GUIDELINES - JUNE

10

11

12 CONTACT US International Environment House n.2 Chemin de Balexert Châtelaine (Geneva) Switzerland Landline : geneva@impact-initiatives.org