Evaluation Capacity Building Strategy October 2017
Table of Contents Table of Contents... 2 Abbreviations... 3 1 Introduction... 4 2 Background... 4 2.1 Defining Evaluation Capacity Building (ECB)... 4 2.2 Previous ECB approaches in Timor-Leste... 5 2.3 Proposed roles and responsibilities for M&E House... 5 3. Evaluation Capacity Building Strategy... 6 3.1 M&E House goal and end of program outcomes... 6 3.2 ECB strategy Theory of Change... 6 3.3 Stakeholders and stakeholder engagement... 8 4. Evaluation Capacity Building Plan... 9 4.1 ECB needs assessment... 9 4.2 Delivery of the ECB plan... 10 4.3 Next Steps... 10 4.4 Budget and resourcing... 11 5. Monitoring, Evaluation and Learning... 11 5.1 Monitoring and Evaluation... 11 5.2 Risk Management... 12 Annex 1: List of training needs identified to date... 13 2
Abbreviations AIP Aid Investment Plan APPR Aid Program Performance Report AQC Aid Quality Check CoP Community of Practice DFAT Department of Foreign Affairs and Trade ECB Evaluation Capacity Building EOPO End of Program Outcomes ES Engagement Strategy GfD Governance for Development IP Implementing Partner KEQ Key Evaluation Questions M&E Monitoring and Evaluation MEL Monitoring Evaluation and Learning MELP Monitoring Evaluation and Learning Plan MIS Management Information System PAF Performance Assessment Framework PHD Partnership for Human Development PNDS Programa Nasional Dezenvolvimentu Suku /National Development Program for Suco R4D Roads for Development ToC Theory of Change TOMAK To os Ba Moris Diak /Farming for Prosperity Program in Timor-Leste WDPTL Workforce Development Program of Timor-Leste 3
1 Introduction Monitoring & Evaluation House Buka Hatene (M&E House) is a DFAT funded program designed to strengthen the delivery of Australia s aid program in Timor-Leste. M&E House will fortify monitoring evaluation and learning (MEL) systems, enhance motivation, and build skills to generate and use credible information; key to measuring and improving the performance of Australia s aid program in Timor-Leste. The M&E House Evaluation Capacity Building strategy (ECB strategy) is set within the M&E House Theory of Change and contributes to M&E House s program goal which is to ensure that Australia s aid program in Timor- Leste is high performing and continually improving. The ECB strategy outlines the scope and approach to strengthening capacity across DFAT in Timor-Leste and their Implementing Partners (IPs) and is aligned with M&E House end of program outcomes (refer to Figure 1). This document represents the first step in operationalising the ECB strategy which intends to develop the capacity of DFAT and IPs around design, monitoring, evaluation, learning and reporting, and to learn from, and adapt to, the complex environment in which they operate. It is grounded in an approach that will: promote, motivate and enable individual and institutional learning by focusing on organisational systems and programs that support performance measurement and adaptive management; provide skills development to targeted stakeholders delivered through a mix of training, mentoring and technical advice; and motivate key decision makers to demand evidence-based performance information to inform and influence operational and strategic decisions. M&E House will implement the ECB strategy over the life of the investment. It will be regularly monitored, evaluated, and adapted according to feedback and evidence. 2 Background 2.1 Defining Evaluation Capacity Building (ECB) ECB is a globally defined field which has broadened over the past decade. Early approaches focussed predominantly on training people to improve their ability to conduct effective evaluations. Recently, there is increased awareness that the supply of people with evaluation skills is not the only issue. Organisational drivers and demand for evidence is critical to create an environment where monitoring and evaluation information is being used to assess and improve program performance. Labin et al (2012) defined ECB as an intentional process to increase individual motivation, knowledge, and skills and to enhance a group and/or organisation s ability to conduct and use monitoring and evaluation. 1 M&E House has extended this definition to focus on pathways of monitoring, evaluation, learning, communicating (reporting), and using evidence to improve program performance. In line with current thinking around ECB, we include both the need to build skills (supply) but also to foster an environment where evaluation and evidence is valued, demanded, and used by senior staff (demand). 1 Susan Labin et al. (2012) A Research Synthesis of the Evaluation Capacity Building Literature; Volume: 33; Issue: 3, page(s): 307-338. September 1, 2012 https://doi.org/10.1177/1098214011434608 4
Our working definition is therefore: ECB is an intentional process to increase an individual s motivation, knowledge and skills, and the organisational ability of DFAT and partners to conduct or use monitoring and evaluation for improved development effectiveness. 2.2 Previous ECB approaches in Timor-Leste In 2012, the DFAT Timor-Leste country program developed an evaluation capacity building program, informed by a similar program in Indonesia. 2 In Timor-Leste the approach focussed on DFAT staff, and involved workplace training and mentoring covering program design, basic program management, and assessment of the quality of implementing partner/contractor monitoring and evaluation products. The M&E House design document developed by DFAT in 2016 reported that the ECB program in Timor-Leste achieved significant gains in terms of improving DFAT staff confidence and skills in performance management, and confidence to articulate M&E requirements to contractors, assess the quality of those products and provide constructive feedback. Also, that the program led to improvements in the quality of key documents such as investment designs and monitoring and evaluation plans. However, these key improvements did not translate into the generation of more credible information about investment performance, much less the use of that information to improve investment or program performance. 3 The document also reported that DFAT senior managers felt that they were not able to obtain evidence when they needed; for example, to make strategic decisions. During its inception period in early 2017, M&E House undertook the M&E Stocktake of DFAT and IP s internal M&E systems. The stocktake found that while the majority of MEL Plans (MELP) were well developed, the quality diminished along the MEL pathway, in relation to implementing the MELPs, communicating information, and using information for learning. 4 Additionally, it was identified that IPs were often dependent on M&E advisers, often with varying approaches, and in some cases, varying quality, and that turnover in national DFAT staff meant that some current staff had limited training or mentoring. Drawing from DFAT s assessment in the design document, and evidence and lessons learned from the M&E Stocktake, M&E House will provide a more consistent, cohesive, and targeted approach to ECB. 2.3 Proposed roles and responsibilities for M&E House M&E House will implement the ECB program. Given our permanent presence in Timor-Leste and in-depth knowledge of the aid portfolio and investments, we have tailored ECB to the specific needs of stakeholders. Our team is supported with short-term inputs from Clear Horizon which focus on high-level strategic capacity needs, and provide lessons learned and training materials from various DFAT and global aid programs. DFAT and IPs have committed to participating in ECB initiatives and providing feedback to M&E House. 2 In 2010, DFAT s Indonesia Country Program launched an evaluation capacity building program to support DFAT staff to strengthen the performance management of investments. Following this, similar programs were developed in Timor-Leste and Vanuatu. 3 DFAT (2016) M&E House Buka Hatene (Timor-Leste) Design Document Annex 1. 4 M&E House (June 2017) M&E Stocktake of DFAT s Bilateral Investments in Timor-Leste. 5
3. Evaluation Capacity Building Strategy 3.1 M&E House goal and end of program outcomes The M&E House overall goal is to ensure that Australia s program is high performing and continually improving. M&E House has been designed to strengthen systems and skills to generate and use credible information; key to managing the performance of the Australian Aid program. We contribute to this goal through two end-of-program outcomes (EOPOs): 1. DFAT is equipped with evidence and capacity to continually improve decision-making and tell a clear performance story; and 2. DFAT s IPs generate and use evidence to learn, adapt, and produce user-focused reports. The ECB strategy Theory of Change will contribute to the achievement of these EOPOs as outlined below. 3.2 ECB strategy Theory of Change Within the M&E House Theory of Change (ToC), the ECB strategy has a single intended outcome: DFAT and IPs have the capacity to effectively design, monitor, evaluate, adapt and report on the Australian aid program. The ECB strategy will contribute to three key practice changes across DFAT and IPs, identified through M&E House ToC: DFAT and IPs use effective processes to report country-level performance information; DFAT and IPs use good quality MEL systems for effective performance management; and DFAT and IPs reflect on evidence, learn, and engage in adaptive management. The ECB strategy as part of M&E House ToC is presented in Figure 1. M&E House has identified five intermediate outcomes (IOs) that provide the pathway to achieving the ECB strategy outcome. These IOs provide the basis for monitoring progress towards the ECB strategy outcome (refer to Figure 1 and Table 2). In order to achieve the intended outcome of the ECB strategy, M&E House will strengthen the performance management capacity of DFAT and IPs through: improving skills and knowledge through training, workshops and targeted capacity building interventions; generating awareness and motivation by creating an interest in improved MEL through social media, newsletters etc.; providing technical support, reference materials, mentoring and coaching to those designing or implementing MEL; and sharing lessons and experiences through communities of practice and seminars. 6
M & E H OUSE EVALUATION C A P A C I T Y BUILDING S T RATEGY Figure 1: M&E House Theory of Change and ECB Strategy 7
3.3 Stakeholders and stakeholder engagement M&E House submitted an Engagement Strategy (ES) to DFAT on 30 th June 2017 outlining how M&E House will effectively engage with our key stakeholders; DFAT and IPs. Key to the strategy is building trust and relationships, with a shared understanding of how M&E House and key stakeholders work together for improved M&E; we anticipate that ECB activities will themselves encourage greater engagement in MEL and with M&E House. M&E House will develop Care Plans for each IP which include ECB needs. The ECB strategy will engage at key levels within DFAT and their IPs, and to a lesser extent provide support for the broader network of MEL stakeholders in Timor-Leste. Each group will have different capacity needs, and as such require different levels of resourcing (refer to Table 1). Table 1: ECB stakeholders and resourcing requirements STAKEHOLDER GROUP ROLES LEVEL OF RESOURCING DFAT Implementing Partners MEL Community of Practice DFAT program/activity managers responsible for managing designs, evaluations, monitoring and evaluation, progress reporting, back-to-office reports and other quality reporting processes such as the AQC and the APPR Leadership level DFAT senior executives responsible for approving and guiding change to program delivery (including program-wide) based on evidence-based performance information Implementing partner (IP) - M&E and project managers responsible for monitoring and evaluation, writing progress reports, contribute to conducting or managing evaluations, and feeding into other reporting processes Oversight/leadership level IP team leaders/managers who as decision makers need to trust M&E information and be motivated to engage in adaptive management Awareness level TL community of M&E practitioners which includes DFAT, IPs and a range of M&E practitioners from non-dfat and non-ip staff High structured approach to knowledge required, mentoring High need strategies for effective targeted engagement given limited availability High maintain a focus on key staff, identify targeted M&E needs High time-poor thus need strategies for effective targeted engagement Low/medium build an interest in good quality MEL through sharing information and resources
4. Evaluation Capacity Building Plan 4.1 ECB needs assessment M&E House will undertake an ECB needs assessment in November-December 2017 to identify capacity building requirements at the individual and organisational level in areas of design, monitoring, evaluation, and learning for DFAT and IPs. Through both the M&E Stocktake and informal meetings, key stakeholders have identified their areas of interest (refer to Annex 1). However, a comprehensive needs assessment will provide specific information on current capacity and needs, and ensure that ECB activities are targeted to priority areas. This information will be used to develop ECB activities and guide their implementation to ensure that our inputs are targeted and fit-for-purpose and context. Table 2 provides a summary of potential ECB activities and planned frequency, based on consultations to date. These will be further refined after the needs assessment. Table 2: Potential ECB activities POTENTIAL ACTIVITIES High quality MEL products across DFAT and IPs Facilitate IPs to review and/or develop their ToC and MELF/MELP o High quality MEL products and systems provide a foundation for organizational MEL capacity o Aim to increase staff understanding of change pathways, practical M&E approaches, indicators, risk assessment etc. Facilitate IPs and DFAT to develop and operationalize their MELPs/performance management systems o Strengthen skills and organisational systems to ensure effective data collection tools, data analysis methods, reporting, and presentation of evidence o Advise and strengthen processes for preparing for and utilizing learning dialogues etc. Timely, effective and trusted advice, help desk, mentoring and coaching available for DFAT and IPs Help desk support service, where people can ask questions around DME. Targeted technical assistance based on jointly-identified needs including strategic-level advice to DFAT and IPs on any specific M&E needs Build trusting relationships, a culture of participation, learning, and open communication; ensure that M&E advice, mentoring, and coaching are high-level and credible FREQUENCY As requested As requested Deliver M&E training to DFAT and IP staff Conduct M&E training series based on capacity needs assessment M&E House will monitor changes in DFAT and IP staff M&E capacity periodically to measure the application of skills and knowledge, and to identify further training needs Strengthen awareness of and motivation for quality MEL Provide strategic support to senior executives in DFAT and IPs to strengthen organizational performance management systems Convene MEL community of practices including seminars, discussion groups, sharing of resources, and access to relevant MEL information Produce a quarterly newsletter and website to facilitate discussion around quality MEL Quarterly Annually As required Bi-monthly Quarterly 9
4.2 Delivery of the ECB plan The ECB program will be delivered by M&E House. It is anticipated that the specialist training/masterclass series and the development of systems to provide data will predominately be facilitated by expert evaluators and trainers from Clear Horizon. Mentoring, advice, coaching, and timely support will be provided by in-country M&E House specialists. The roles and responsibilities for key stakeholders are shown in Table 3. Table 3: Evaluation Capacity Building - Roles and responsibilities WHO KEY ECB TASKS M&E House team leader Manage the overall ECB strategy and plan Support team members to implement their respective elements of the ECB plan Ensure effective monitoring of ECB activities and the overall strategy Manage ECB progress reporting M&E House team members and Clear Horizon Other team members and short-term associates/advisers from Clear Horizon Coordinate and implement the ECB strategy and plan Undertake an ECB needs assessment Support external trainers, including CH consultants in ECB implementation Implement monitoring of key ECB strategies Contribute to the impact assessment on the ECB program Collect data on ECB activities, identify needs, issues, and evidence of progress towards relevant skills and knowledge Report progress on ECB implementation Provide high-level ECB training and mentoring for DFAT and IP s senior staff Provide strategic level support to organizational performance assessment processes for both DFAT and IPs DFAT and IPs Ensure key staff are available to participate in the ECB program Provide feedback on ECB performance as requested Participate in satisfaction surveys to assess ECB success as requested 4.3 Next Steps On approval of this ECB strategy by DFAT, M&E House will undertake the steps outlined in Table 4. Table 4: ECB next steps ACTIVITY PROPOSED DATE ECB strategy reviewed and approved by DFAT October 2017 Undertake an Evaluation Capacity Needs Assessment November -December 2017 Further develop plans for ECB approaches and activities based on the findings of the Evaluation Capacity Needs Assessment Implement capacity building activities January- March 2018 Commenced and ongoing 10
4.4 Budget and resourcing The budget for ECB program activities will be confirmed each financial year; Table 1 provides an indicative budget. Funding will primarily be for short-term consultants and local or international costs associated with regular ECB program activities (hire of venues etc.). Table 1: Indicative budget ECB 2017-2020 ACTIVITY 2017/18 2018/2019 2019/20 Short-term consultants $ 25,400 $ 21,000 $ 21,000 ECB program: training, seminars, materials, CoP $ 16,300 $ 18,000 $ 18,000 Total $40,700 $39,000 $39,000 5. Monitoring, Evaluation and Learning 5.1 Monitoring and Evaluation M&E House will monitor and evaluate the ECB strategy as part of the M&E House M&E Plan. In developing the ECB strategy, M&E House has identified five intermediate outcomes for the ECB strategy (refer to Figure 1). Potential indicators for each of the ECB intermediate outcomes are provided in Table 2 below. These will be further developed and incorporated into the M&E House MELP. Table 2: ECB intermediate outcomes and potential indicators INTERMEDIATE OUTCOMES POTENTIAL INDICATORS DATA COLLECTION TOOLS DFAT and IPs have skills and knowledge in MEL Applying M&E knowledge and skills to improve deliverables Stakeholder/participant surveys Training evaluation DFAT and IPs have experience in developing and implementing a quality MEL M&E system compliance with DFAT M&E standards Instances of design and adaptation by implementing partners to apply learning Client feedback session M&E Stocktake (baseline) Stakeholder survey Evidence and data on program performance is available Robust data collected, analysed and used as evidence for achievements reported Stakeholder survey Interviews Case study Review of MIS DFAT and IPs have effective program design and MELFs to guide practices Instances of design and adaptation to apply new learning Stakeholder survey Interviews Supportive networking in place to promote learning Level of satisfaction with community of practice Client feedback session Stakeholder survey 11
The ECB strategy, and progress towards achieving ECB outcomes, will be assessed as part of DFAT s review of M&E House. M&E House will also annually assess progress towards the intended outcome. Progress markers will be developed to monitor changes in behaviours and practices by DFAT and IP staff aimed at contributing to the achievement of the ECB outcome (refer to Table 3). Table 3: List of potential ECB progress markers STAGE OF THE ECB PROGRAM PROGRESS MARKERS FOR DFAT AND IPS End of year 1 Jan 2018 Learning needs assessment completed Regular COP events are well attended with positive feedback Mid-term (year 2) December 2019 Clear improvements in DFAT and IPs capacity in design and MEL Improvement in effective monitoring, analysis and sharing evidence Improved learning and reflection across the investments End-of program December 2020 High quality investment reporting that is able to inform strategic investment and portfolio level decision making Implementing partners are contributing to a country-wide MEL and MIS systems 5.2 Risk Management M&E House will undertake regular assessments to monitor risks. A detailed risk plan and associated mitigation is provided in the M&E House M&E Plan and M&E House work plan. 12
Annex 1: List of training needs identified to date Suggested training from DFAT and M&E Stocktake Investment designs Developing M&E frameworks, methods and tools Infographics and data visualisation Aid management cycle Progress reporting and DAC criteria Reporting against performance assessment framework indicators Evidence and causal inferences for AQC and PPA preparations Information management systems linked to M&E Impact evaluations; theory and practice Back-to-office reports Gender mainstreamed into M&E Social inclusion mainstreamed into M&E Most significant change stories Significant instance of policy and institutional influence M&E capacity building for Planning, Monitoring and Evaluation Unit (UPMA) staff and line ministries Results Measurement (RM) in specific sectors, particularly women s economic development How to integrate cross-cutting issues, particularly gender equality, into M&E process and analysis How to implement M&E for iterative programs Qualitative approaches of M&E Participatory M&E and strength-based M&E in the activity level Advanced M&E training Suggested training topics from the MEL community of practice Series of M&E training, for e.g. M&E methods and tools Qualitative and quantitative data analysis. Approaches and tools to develop case studies and success stories Qualitative and quantitative indicators Master classes on undertaking rigorous baselines Approaches to measure changes in organizational capacity, policy changes etc. Tools to measure progress and sustainability of development initiatives Techniques and approaches to develop an evidence-base for policy making 13
Contact Information CONTACT INFORMATION M&E House Suite 2A, Level 2, Seaview Offices Avenida de Portugal Dili Timor-Leste Disclaimer This report has been prepared by M&E House for the Australian Government Department of Foreign Affairs and Trade (DFAT). M&E House is implemented by GHD Pty Ltd., in a practical association with Clear Horizon Pty Ltd. GHD disclaims responsibility to any person other than DFAT arising in connection with this report. GHD also excludes implied warranties and conditions, to the extent legally permissible. The services undertaken by M&E House in connection with preparing this report were limited to those specifically detailed in the report and are subject to the scope limitations set out in the report. The opinions, conclusions and any recommendations in this report are based on conditions encountered and information reviewed at the time of preparation of the report. M&E House has no responsibility or obligation to update this report to account for events or changes occurring subsequent to the date that the report was prepared. Document Status VERSION NO. DESCRIPTION OF VERSION AUTHOR REVIEWER APPROVED BY DATE 1 Draft 1 Alberto Correia Keryn Clark Louise Maher 01/10/17 2 Version for Submission Keryn Clark Louise Maher Byron Pakula Louise Maher Allan Pankhurst 09/10/17