TERMS OF REFERENCE. End Line Evaluation of the H4+ Joint Programme Canada and Sweden (Sida)

Evaluation Office TERMS OF REFERENCE End Line Evaluation of the H4+ Joint Programme Canada and Sweden (Sida) 2011-2016 New York June 2015 End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 1

Contents List of Acronyms... 4 1. Introduction and Background of the H4+... 5 2. Rationale, Purpose and Specific Objectives of the Evaluation... 5 2.1 Rationale and Purpose of the evaluation... 5 2.2 Specific Objectives of the Evaluation... 6 3. Evaluation Object and Scope... 7 3.1 The strategic context of the H4+ JPCS... 7 3.1.1 A brief overview of the H4+ partnership... 7 3.1.2 The Secretary General s Global Strategy for Women s and Children s Health... 8 3.2 The Canada and Sida H4+ Joint Programme (H4+ JPCS)... 10 3.2.1 The evolution of the H4+ JPCS... 10 3.2.2 The structure of the H4+ JPCS... 11 3.2.3 Work of the H4+ Global Technical Team (financed by H4+ JPCS)... 12 3.2.4 Country-level Work of the H4+ JPCS... 12 3.2.5 Stakeholders of the H4+ JPCS... 13 3.3 Scope of the Evaluation... 14 3.3.1 Temporal Scope of the Evaluation... 14 3.3.2 Geographical Scope of the Evaluation... 14 3.3.3 Thematic Scope of the Evaluation... 15 4. Evaluation criteria and indicative areas of investigation... 15 4.1 Evaluation Criteria... 15 4.2 Indicative Areas of Investigation... 17 5. Evaluation Approach and Methodology... 20 5.1 Required Elements of the Evaluation Methodology... 20 5.1.1 Logical Reconstruction of the H4+ Intervention Logic and Theory of Change... 20 5.1.2 Finalization of the Evaluation Questions and Assumptions... 22 5.1.3 Well-designed Country Case Studies... 22 5.1.4 A Wide Range of Data Collection Tools (Quantitative and Qualitative)... 24 End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 2

5.1.5 A well-structured evaluation matrix to ensure the validity of evaluation findings... 25 6. Evaluation Process... 26 6.1 Preparatory Phase... 27 6.2 Inception Phase... 28 6.3 Data Collection Phase... 28 6.3.1 Desk Study... 29 6.3.2 Field Study... 29 6.3.3 Online Survey... 30 6.4 Reporting Phase... 30 6.5 Management Response... 31 7. Roles and Responsibilities... 31 7.1 The Consultant (Company)... 31 7.2 UNFPA... 32 7.3 Evaluation Management Group (EMG)... 33 7.4 Joint Evaluation Reference Group (ERG)... 34 8. Quality Assurance... 34 9. Indicative Deliverables and Time Schedule... 35 10. Composition of the Evaluation Team... 38 11. Specification of Tender, Cost of the Evaluation and Payment Modalities... 39 Selected Bibliography... 42 Annex 1: Structure of Evaluation Reports and Country Case Studies Notes... 44 Annex 2: Suggested Structure of Evaluation Matrix... 48 Annex 3: List of Other Specifications of the Deliverables... 49 Annex 4: Joint Results Framework of H4+ Joint Programme (Canada, Sweden / Sida) (H4+ JPCS)... 51 Annex 5: Elements of a preliminary conceptual framework for a Logic Model / Theory of Change of the H4+ JPCS... 56 Annex 6: List of Countdown Countries and H4+ Countries... 60 Annex 7: UNEG Ethical Guidelines and Code of Conduct... 62 Annex 8: Country Profiles of H4+ JPCS Programme Countries... 63 Annex 9. Quality Assurance of the Evaluation Report... 83 End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 3

List of Acronyms DAC DFATD DRC EMG EmONC EQA ERG EWEC HMIS ierg IHP+ M&E MDG MNCH / RH MNH OECD PMNCH PMTCT RMNCH ToR UNEG UNFPA UNICEF WB WHO Development Assistance Committee Department of Foreign Affairs, Trade and Development (Canada) Democratic Republic of Congo Evaluation Management Group Emergency Obstetric and Newborn Care Evaluation Quality Assessment Evaluation Reference Group Every Woman Every Child Hospital Management Information System Independent Expert Review Group International Health Partnership Monitoring and Evaluation Millennium Development Goals Maternal, Newborn and Child Health / Reproductive Health Maternal and Newborn Health Organization for Economic Cooperation and Development Partnership for Maternal and Newborn Health Prevention of Mother-To-Child Transmission Reproductive, Maternal, Newborn and Child Health Terms of Reference United Nations Evaluation Group United Nations Population Fund United Nations Children Fund World Bank World Health Organization End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 4

1. Introduction and Background of the H4+ In 2008, UNFPA, UNICEF, WHO and the World Bank launched the H4 partnership as a joint effort for capitalizing on the core competencies of the four agencies along the continuum of care for maternal, newborn and child health. 1 In 2010, United Nations Secretary-General Ban Ki-moon launched the Global Strategy for Women s and Children s Health to accelerate progress to meet MDGs 4 and 5. This initiated the Every Woman Every Child (EWEC) movement to put the Global Strategy into action. The H4 partnership was expanded to also include UN Women and UNAIDS, thus transforming it into the H4+. At the same time, the focus of H4+ was broadened to include reproductive health and child health, to help countries put into action their commitments under the Global Strategy for the integrated package of reproductive, maternal, newborn and child health (RMNCH) services. H4+ became the technical arm of the EWEC Strategy and assumed the role of supporting the 75 high burden countries where more than 85% of all maternal and child deaths occur, including the 49 lowest-income countries. Canada (2011) and Sweden (2012) provided grants to the H4+ partners specifically to support their collaboration in 10 specific countries. Owing in part to the different start dates, the two collaborations were initially based on separate project documents. In 2013, at the request of both Sweden and Canada, the H4+ partners developed a joint results framework, intended as a basis for closely coordinated implementation under one joint H4+ joint programme. 2. Rationale, Purpose and Specific Objectives of the Evaluation 2.1 Rationale and Purpose of the evaluation With the expiration of the Millennium Development Goals as of 2015, the strategic context of the H4+ is undergoing fundamental change. Many countries did not meet the MDG targets and much remains to be done beyond 2015, particularly in low-income countries in sub- Saharan Africa and South Asia, and in countries affected by conflict. The eventual shape of the new development agenda, including for maternal and child health will be defined by member states at the United Nations General Assembly in September 2015. This means that the global framework will already have changed by the time this evaluation is carried out. At the same time, the H4+ joint programme of Canada and Sweden / Sida (henceforth H4+JPCS ) that has provided funding for the H4+ partners 2 is coming to a close with a body of accumulated knowledge. 2 Four agencies under the Canada Collaboration (UNFPA, UNICEF, WHO and the World Bank) and six agencies under the Sweden / Sida Collaboration (the original four agencies plus UNAIDS and UN Women) End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 5

The purpose of the evaluation is to: Support learning among key stakeholders from the experience of implementing the H4+ JPCS at global, regional, national and subnational levels, with a view to informing similar initiatives to facilitate the delivery of the comprehensive package of services and support in the field of RMNCH in the future and support the H4+ partners review of the partnership mandate in a post-2015 context; Support accountability of the H4+ partners for the results they have achieved under the H4+ JPCS. The intended users for the evaluation will be in particular: The Global Technical Team of the H4+ initiative of all six agencies, who have coordinate and overseen the H4+ JPCS; Members of the H4+ country teams in the 10 countries of the H4+ JPCS, and in the 38 countries that have had established functioning H4+ coordination mechanisms over the duration of the programme; Counterparts of the H4+ JPCS in programme countries and other high-burden countries, including national health entities and other agencies that form part of national health systems; Representatives of Canada and Sweden / Sida as the two donors that have funded the H4+ JPCS; as well as other donors who have been supporting the H4+ initiative outside of this joint programme; 3 Interested parties in the six agencies that have partnered under the H4+ JPCS, including management and technical staff; Global health stakeholders, including stakeholders involved in the management and monitoring of the Global Strategy for Women s, Children s and Adolescents Health. 2.2 Specific Objectives of the Evaluation The specific objectives are to: Assess the relevance of the objectives and the approach of the H4+ JPCS at global, regional, national and subnational levels, including its role and positioning within the context of other partnerships and platforms. 4 Assess the effectiveness and efficiency of the implementation of the H4+ Joint Programme (Canada, Sweden) (H4+ JPCS) at global, regional, national and subnational levels; also, but not exclusively, with regard to: Achievements of the programme regarding the strengthening of national health systems at policy and programme level in the ten programme countries. 3 This includes in particular France (Muskoka grant) and Johnson & Johnson. 4 Such as the Partnership for Maternal and Newborn Health (PMNCH), the International Health Partnership (IHP+), the RMNCH Steering Committee, the Innovation Working Group, the independent Expert Review Group. End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 6

Improvements in the delivery of a comprehensive package of reproductive, maternal, newborn and child health services to the population in intervention areas in the ten programme countries. Assess the sustainability of the results achieved by the H4+ JPCS at global, regional, national and sub-national levels; Assess the added value of the H4+ JPCS approach and actions for the development of tools and guidelines for maternal and children s health programming, awareness raising products, and technical guidance on RMNCH. Assess the extent to which issues of gender equality, social inclusion and equity have been taken into consideration. Identify lessons and good practices from the implementation of the H4+-JPCS, and opportunities to improve both the cooperation between the six agencies and their support aimed at the improved delivery of the comprehensive package of services and support in RMNCH, in a set of concrete and actionable recommendations. 3. Evaluation Object and Scope The H4+ joint programme, financed by Canada and Sweden sits within a set of policies and strategic frameworks, namely: The H4+ partnership, whose consolidation and actions the H4+ JPCS was meant to support, The Secretary General s Global Strategy for Women s and Children s Health and the associated global initiative Every Woman, Every Child (EWEC) that came to serve as the strategic framework for the H4+ partnership. 3.1 The strategic context of the H4+ JPCS 3.1.1 A brief overview of the H4+ partnership The overall mandate of the H4+ partnership was defined in 2008 in a joint statement of the four originally participating agencies (UNFPA, UNICEF, WHO, World Bank). The objective of the partnership was to harmonize approaches by UN agencies to improve maternal and newborn health (MNH) at country level and jointly raise the necessary resources. The H4 effort was meant to help participating agencies to harmonize and unify their actions to strengthen the capacity of national health systems for designing and implementing catalytic and strategic interventions to improve access to RMNCH information and integrated services, using an equitable, rights-based and participatory approach. The resulting continuum of care for RMNCH consists of the integrated service delivery for mothers and children from pre-pregnancy to delivery, the immediate postnatal period, and childhood, and provided by families and communities, through outpatient services, clinics and other End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 7

health facilities. 5 The statement also envisaged the division of labour to each of the four agencies, according to core competencies, along the continuum of care for RMNCH. When UN Women and UNAIDS joined the four original H4 members, the partnership was retitled as H4+ whose mandate was defined as leveraging the collective strengths and distinct advantages and capacities of each of six agencies in the UN system to address RMNCH in the countries with high burden of maternal and child mortality and morbidity. 6 The H4+ partners acknowledge that countries require distinct types of support for RMNCH, and have pledged to ensure that all approaches need to be locally defined, and implemented in a manner that advances the national health plan, and strengthens national structures. Country-driven approaches are meant to allow the H4+ partnership to address differing obstacles to achievement of MDGs 4 and 5, including health system failures, to underlying problems of gender inequality, and economic and social exclusion, to varying degrees. The partnership is based on the notion that a collaborative approach will allow the six participating agencies to operate across parallel sectors, and to advance progress in health, finance, legal, and social sectors simultaneously. Key Objectives of H4+ partnership are to: 1) Mobilize political commitment and support, and maximize synergies between UN agencies, governments, and other global and national partners for women s and children s health; 2) Provide joint technical support for scale up of national integrated RMNCH policies and plans, with a focus on universal rights to access affordable, accessible, easily available, and quality RMNCH services; 3) Strengthen national health systems and further national health plans for RMNCH; 4) Promote evidence-based interventions to address the root causes of poor RMNCH, such as the social, economic and gender inequalities; 5) Strengthen mutual accountability and national capacity to monitor RMNCH interventions through sustainable improvements of country s Health Management Information Systems. 7 3.1.2 The Secretary General s Global Strategy for Women s and Children s Health Launched in 2010, two years after the formal creation of the original H4 partnership, the Global Strategy for Women s and Children s Health of the Secretary General of the United Nations set out the several key areas for action by the global community, to help promote progress towards MDG 4 (a two-thirds reduction in under-five mortality) and MDG 5 (a 5 PMNCH Fact Sheet: RMNCH Continuum of care; http://www.who.int/pmnch/about/continuum_of_care/en/ 6 The H4+ Partnership Joint support to improve women s and children s health, September 2014 at http://www.unfpa.org/sites/default/files/pub-pdf/h4%2b%20synthesis%20report%202014_finallowres.pdf 7 Ibidem. End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 8

three-quarters reduction in maternal mortality and universal access to reproductive health). These include: Support for country-led health plans, facilitated by increased, predictable and sustainable investments; Integrated delivery of health services and life-saving interventions so women and their children can access prevention, treatment and care when and where they need it; Stronger health systems, with sufficient skilled health workers at their core; Innovative approaches to financing, product development and the efficient delivery of health services; Improved monitoring and evaluation to ensure the accountability of all actors for results. The Global Strategy also assigned responsibility to the agencies of the United Nations system for supporting this global effort with specific contributions. These called for the UN to: Define norms, regulations and guidelines to underpin efforts to improve women s and children s health, and encourage their adoption; Help countries develop and align their national health plans; Work together and with others to strengthen technical assistance and programmatic support, helping countries scale up their interventions and strengthen their health systems, including health-care workers and community level care; Encourage links between sectors and integration with other international efforts (such as those on education and gender equality), including harmonized reporting; Support systems that track progress and identify funding gaps; Generate and synthesize research-derived evidence, and provide a platform for sharing best practices, evidence on cost-effective interventions and research findings. 8 Subsequent to the publication of this strategy, the H4+ partnership became the technical arm of the Global Strategy, with the aim of contributing leadership in the areas of reproductive, maternal, newborn and child health (RMNCH). 9 In support of the Global Strategy, the Secretary General Ban Ki-Moon also launched the global initiative Every Woman, Every Child (EWEC) during the United Nations Millennium Development Goals Summit in September 2010. Designed to become a global movement, Every Woman Every Child is meant to help mobilize and intensify international and national action by governments, multilaterals, the private sector, and civil society to put the Global Strategy for Women s and Children s Health into action. As of December 2014, the movement had over 400 commitments, from over 300 partners; including national governments of UN 8 Global Strategy for Women s and Children s Health of the United Nations Secretary General Ban Ki-Moon (2010) 9 http://www.unfpa.org/h4plus End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 9

member states, businesses, philanthropists, research and academic organizations, Civil Society and UN agencies and other multilateral organizations. 10 As technical arm of the Secretary General s Global Strategy, the H4+ partnership is also associated with EWEC. H4+ partners have made specific commitments to EWEC, pledging to mobilize political support for the Global Strategy in the 49 Lowest Income Countries;" increasing the speed of the downward trend in maternal and child mortality by strengthening country and regional technical capacity to implement commitments; advocating for equity-focused approaches that include universal access to an integrated essential package of health services for women and children; and to addressing the root causes of ill-health, in particular gender inequality. 11 3.2 The Canada and Sida H4+ Joint Programme (H4+ JPCS) 3.2.1 The evolution of the H4+ JPCS In order to help strengthen the technical and convening roles of the H4+ partnership at the global and country levels, the Governments of Canada and Sweden mobilized a combined grant of US$ 102 million, respectively in 2011 and 2012, to allocate specific funds for the H4+ Joint Programme to accelerate progress towards achieving MDGs 4 and 5 in 10 African countries. In 2011, H4+ partners started to use a CA$ 50 million grant from Canada (henceforth referred to as the Canada Collaboration ) to work under the umbrella of the H4+ partnership at global and regional level, and in five countries: Burkina Faso, the Democratic Republic of the Congo, Sierra Leone, Zambia and Zimbabwe. 12 In 2012, the H4+ partners received additional grant financing from Sweden / Sida in the amount of US$ 52 million (henceforth referred to the Sweden / Sida Collaboration). In 2013, this money was used to carry out additional joint actions at global and regional level, and also to expand their work at country level, by carrying out joint actions in 5 additional countries: Cameroon, Côte d Ivoire, Ethiopia, Guinea Bissau, Liberia, and provision of increased support to Zimbabwe. 13 All 10 countries rank among the lowest in Human Development. All have a maternal mortality rate of more than 300 deaths per 100,000 live births, and high infant mortality 10 For a complete list of commitments, see http://www.everywomaneverychild.org/commitments/allcommitments?filter_sectors=8. 11 See more at http://www.everywomaneverychild.org/commitments/all-commitments/h4-who-unfpaunicef-unaids-un-women-and-the-world-bank. 12 As of 2014, 76 per cent of the total budget for the Canada collaboration had been designated for countrylevel activities. The average fund utilization was 67 per cent for country level activities. 13 In the first 18-month period of the collaboration, from June 2013 to December 2014, 81 per cent (US$26.8M) of the 2013 2014 budget (US$32.9M), was dedicated to country-level activities. The expenditure rates for this period averaged at 52 per cent. Only Zimbabwe is recipient of grant funding under both collaborations. End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 10

rates. 14 They are also countries in which the H4+ agencies are well-established and where the grant activities can be aligned with existing processes and complement existing funds and programmes. Owing in part to their different start dates, the two collaborations (i.e., the Canada Collaboration and the Sweden / Sida Collaboration) were initially based on two separate project documents, and also used separate work plans. However, in 2012, at the request of both Sweden / Sida and Canada, the H4+ partners developed a joint results framework, intended as a basis for joint implementation of the two collaborations under the H4+ joint programme Canada and Sweden / SIDA (H4+ JPCS). The joint results framework for the H4+ JPCS is shown in Annex 4. Note: The joint results framework of the H4+ JPCS in Annex 4 of these ToR also makes reference to actions financed by the French Government under the Muskoka Collaboration. However, while the actions of the Muskoka Collaboration are listed in the joint result framework, they are not part of the H4+ JPCS, and are therefore also not part of the scope of this end-line evaluation. The support from Canada is scheduled to end in March 2016, representing an implementation period of 5 years and the support of Sweden is scheduled to end in June 2016, representing an implementation period of 3 years. 3.2.2 The structure of the H4+ JPCS The overall H4+ JPCS is managed by UNFPA on behalf of all six H4+ partners. Work at global and regional levels under the H4+ JPCS is carried out by an H4+ Global / Regional Team (also referred to as the Global Technical Team ). This team consists of technical staff from all six H4+ partners. Country level work under the H4+ JPCS is managed by a set of ten H4+ country teams (see Figure 1). Figure 1: The structure of the H4+ JPCS 14 See Country profiles in Annex 8 End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 11

In each of the ten H4+ joint programme countries, one of the H4+ agencies has been appointed as a coordinating agency with a representative acting as the H4+ focal point or country coordinator, overseeing and coordinating implementation at the country level (see list of country lead agencies in Error! Reference source not found.). Table 1: H4+ coordinating agencies and programme countries H4+ coordinating agency UNFPA UNICEF WHO Countries Côte d Ivoire, DRC, Guinea Bissau, Sierra Leone and Zimbabwe Cameroon and Zambia Burkina Faso, Ethiopia and Liberia In the 10 programme countries of the H4+ JPCS, leaders from government, NGOs, civil society, professional bodies, communities and the private sector have stepped forward to join in their efforts to protect the right to health for all women and children. The H4+ teams at global, regional and country levels support the development and implementation of the national SRH plan in close collaboration with the relevant Ministries of Health and key stakeholders (see Error! Reference source not found.), using when necessary a country / sub-national assessment, to analyze the situation and needs. The design of the H4+ JPCS follows an expanded health system building-block approach, which includes leadership and governance, financing, technologies and commodities, human resources, health information, service delivery, along with community ownership and communications for demand generation and advocacy. Human rights, including SRH rights and gender equality are cross-cutting themes integrated in all H4+ interventions at global and country levels. Innovative solutions are developed and tested on an on-going basis to address locally specific RMNCH issues. 3.2.3 Work of the H4+ Global Technical Team (financed by H4+ JPCS) Global and regional level work financed by the H4+ JPCS aims at enhancing the efforts within countries for knowledge management by generating and sharing technical guidelines; building technical capacity at regional and national levels, facilitating global, regional and South to South collaborations in key areas; and advocating for greater global political and financial commitments to the reproductive, maternal, newborn and child health sector. 3.2.4 Country-level Work of the H4+ JPCS Much of the work financed by the H4+ JPCS occurs at country level, where the H4+ agencies set out to coordinate their support to jointly contribute to the strengthening national health systems, including through the development, costing and financing of the components of national health plans that relate to RMNCH. End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 12

At policy level, the H4+ JPCS-financed efforts are meant to build on knowledge of national context, and experience with successful interventions in other countries. The H4+ partnership and the H4+ JPCS that supports this partnership were designed to: (i) mobilize political, technical and financial support for RMNCH; (ii) provide joint technical support to develop quality national RMNCH plans; (iii) promote universal access to integrated essential health services and evidence-based, high impact and cost-effective interventions in RMNCH; and (iv) to work closely with governments to align to national priorities the wide array of complementary national, bi-lateral and multi-lateral RMNCH initiatives underway within the country. H4+ JPCS also was meant to support the development of strategic and policy documents, and efforts to remove financial barriers to accessing emergency obstetric and newborn care (EmONC) services. At programme level, the H4+ JPCS aimed at building national and sub-national technical and managerial capacities to address maternal, newborn, and child health issues; and supporting the implementation of national RMNCH plans in identified districts, focusing on strengthening the quality of RMNCH services and on enhancing community engagement to increase demand for and use of those services. Activities at sub-national level aimed to feed into relevant policy activities for scaling up interventions at the national level to strengthen health systems. 3.2.5 Stakeholders of the H4+ JPCS Table 2 provides a preliminary overview of key stakeholders of the H4+ JPCS, meant as a basis for a more comprehensive stakeholder analysis during the inception phase of the evaluation. Table 2: Stakeholders of the H4+ JPCS Types of Stakeholders Direct (primary) beneficiaries Secondary beneficiaries Key Stakeholders (not exhaustive) Health system strengthening, including capacity development support Ministries of Health, other relevant Ministries Local health authorities Training institutes Health care providers and programme managers. Communities and civil society Local service delivery (pilots) Patients of supported health service providers/institutions (women, children) Health system strengthening, including capacity development support Agents acting on behalf of primary organizational beneficiaries (e.g. local health authorities acting on behalf of supported Ministries of Health) Local service delivery (pilots) End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 13

Executing agencies / implementing organizations Families of patients of supported health service providers. Lead agencies for specific H4+ initiative (global-level, country level offices of H4+ agencies) H4+ global technical team (consisting of global and regional-level staff members of individual H4+ agencies) H4+ country teams (consisting of staff members from country offices of H4+ agencies) Project staff for local pilot projects (e.g.,: innovations) Donors Sweden/Sida; Canada (donors of evaluated H4+ JPCS) France (donor of Muskoka Collaboration) Johnson & Johnson (additional donor supporting the H4+ partnership) Other donors supporting RMNCH (country specific) Other cooperation Members of Partnership for Maternal, Newborn and Child Health (PMNCH) partners United Nations Secretary General Office Country-specific cooperation partners 3.3 Scope of the Evaluation 3.3.1 Temporal Scope of the Evaluation In order for this evaluation to cover as much of the implementation period as possible, the period covered by this evaluation will extend from March 2011 until March of 2016, 15 covering the entire implementation period of the H4+ joint programme (Canada, Sweden / SIDA) (H4+ JPCS). This means that the evaluation will cover the actions financed by Canada and Sweden before the two collaborations were integrated into the H4+ JPCS, as well as the period after the integration into the joint programme. 3.3.2 Geographical Scope of the Evaluation The End Line Evaluation should cover all ten countries that are part of the H4+ JPCS. Four of these countries will be covered by field-based case studies: DRC, Liberia, Zambia and Zimbabwe. The remaining six countries will be covered by desk case studies: Burkina Faso, Cameroon, Côte d'ivoire, Ethiopia, Guinea Bissau and Sierra Leone. See below section 5.2.3 (and related Annex 8) on the allotment of H4+JPCS countries to either field or desk-based case studies. Global knowledge products and tools produced by the H4+ global technical team were aimed at all 75 countdown countries identified in the Global Strategy for Women s and Children s 15 March of 2016 is the projected end of the inception phase of this evaluation; and the start of data collection. End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 14

Health. The H4+ JPCS has also been used to provide technical support to the larger group of 75 countdown countries, 16 for example through dissemination of global public goods produced by the H4+ global technical team. Therefore, the evaluation should cover activities financed by the H4+ JPCS at global level by the H4+ global technical team and regional level (i.e., those carried out by regional offices of H4+ agencies) and should be able to provide an assessment of the contribution of the H4+ effort in countries beyond the ten programme countries of the H4+ JPCS. 3.3.3 Thematic Scope of the Evaluation Thematically, the evaluation covers first and foremost the actions carried out under the outcome areas of the H4+ JPCS. However, as explained in Section 2.2, the H4+ JPCS was designed to help strengthen the H4+ partnership itself, with its mandate of establishing a long-term division of labor between the six H4+ agencies in accordance with their core competencies along the RMNCH continuum of care, and the RMNCH integrated package. The H4+ partnership in turn was meant to contribute to the delivery of the Global Strategy for Women s and Children s Health. Evaluators will not be required to provide an assessment of these broader initiatives, however, the evaluation will need to examine the contributions that the H4+ JPCS has made to the objectives of these initiatives, and the extent to which the H4+ JPCS has been coherent with those of the Global Strategy; Every Woman Every Child and the H4+ partnership at large. The evaluation should also examine direct and indirect (snow ball) effects of the H4+ JPCS, including the relationship between country level,projects, programmes and actions of H4+ partners/agencies financed from other sources, in the respective countries. 4. Evaluation criteria and indicative areas of investigation This section sets out the main evaluation criteria that should be used to define the evaluation. It subsequently introduces indicative areas of investigation that evaluators should use as a starting point to develop the specific evaluation questions. 4.1 Evaluation Criteria The evaluation will be informed by OECD DAC criteria, as well as other relevant criteria. Attention will be given throughout to issues of gender equality, equity and social inclusion. 16 A list of countdown countries and H4+ countries is included in Annex 6. End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 15

Table 3: Evaluation criteria and their significance for the evaluation of the H4+ JPCS Relevance Response to in-country needs in RMNCH (as defined by government strategy papers, key non-governmental sector stakeholders) and alignment with national plans, Response to in-country needs with regard to health systems strengthening (as defined by government priorities; service providers, third party stakeholders of national health sector) Catalytic and strategic nature of H4+ JPCS. Responsiveness The ability of the H4+ JPCS to respond to: Interventions responding to national health systems priorities Changes and/or additional requests from national counterparts, and Shifts caused by external factors in an evolving country context. Effectiveness H4+ JPCS achievements of health systems strengthening outcomes in joint programme countries and beyond; Achievement of RMNCH-related health outcomes; Level of focus on the needs of the most vulnerable groups and marginalized populations (e.g., rural poor women, families in geographically inaccessible areas, adolescents/early pregnancies, pregnant women living with HIV, women/adolescents/children living with disabilities, indigenous people); Achievement of Global Strategy, and Every Woman, Every Child operationalization outcomes Efficiency How economically are resources/inputs (funds, expertise, time, etc.) converted to outputs; H4+ JPCS effects on creation of synergies among H4+ partners, in view of a better use of resources at global level and country level; Avoidance of duplication of efforts between H4+ JPCS actions and actions financed by different sources Comparative H4+ JPCS transaction costs Sustainability Attention to human resources in the health sector, including capacity development of national institutions, such as pre-service training schools of midwives. Likelihood that H4+ JPCS results/benefits will continue after funding is ended. Committed financial and human resources to maintain results; in particular, Ownership of the H4+ concept and initiative among H4+ partners, i.e., the institutionalization of the effort in organizational planning and budgeting processes; Ownership and commitment among country level partners to health system strengthening efforts, ensuring that they are taken-up by the national health system; and the overall End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 16

national governance system in H4+ JPCS programme countries External environment conducive to the maintenance of results Coordination Strengths and weaknesses of mechanisms in place to ensure an appropriate approach to joint programming, management, planning, monitoring and evaluation under H4+ JPCS; between H4+ partners and other development partners; and between the Sweden / Sida Collaboration and the Canada Collaboration (as constituent parts of the H4+ JPCS), as well as with the Muskoka Collaboration (funded by France; not part of the H4+ JPCS); Efforts of H4+ country teams to facilitate sustainability. Added Value Added value of the actions of the H4+ JPCS and the overall H4+ partnership (e.g. before/after, with/without cases) and in contrast to business as usual with regard to cooperation and coordination among the six UN agencies. 4.2 Indicative Areas of Investigation The evaluation criteria have been translated into indicative areas for investigation, presented in Table 4. These will be used as a starting point for developing the specific set of evaluation questions. The indicative areas for investigation are intended to give a more precise form to the evaluation criteria and to articulate the key areas of interest that have emerged from consultation with stakeholders, thereby optimizing utility of the evaluation. Table 4: Indicative areas of investigation for the End-Line Evaluation Indicative Areas of Investigation (DAC Criterion / Criteria covered) (1) The extent to which the H4+ JPCS, through country level activities in combination with regional and global technical support services, tools and knowledge products, was able to contribute to the strengthening of national health systems in joint programme countries. (Effectiveness, Sustainability) Additional Information / Explanations The approach of the H4+ JPCS assumed that successful experiments in health service delivery at local level could inform efforts to strengthen the health system at large; i.e., at national level. Assessment of this issue should therefore examine the linkages between H4+ JPCS-financed activities of H4+ partners in selected districts or regions in programme countries and possible resulting changes in national health systems; such as improved RMNCH-related policies and institutional structures, improved practices for strategic planning and health system management, the adoption of new tools or programme components at the national level, and other possible reforms and changes in the national health sector. End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 17

Indicative Areas of Investigation (DAC Criterion / Criteria covered) (2) Changes in integrated health service provision along the continuum of care, covering both intended and unintended outcomes of country-level support financed by the H4+ JPCS, and their longterm viability. (Effectiveness, Sustainability) (3) The extent to which the original designs of the H4+ JPCS has responded to the needs and priorities of the main stakeholders in national health systems in countdown countries of the Global Strategy and in particular of the ten H4+ JPCS programme countries, in relation to health system strengthening and the identification and scale up of innovative approaches for integrated service delivery. (Relevance) (4) The extent to which the H4+ JPCS has helped to identify innovative approaches, and has successfully promoted their scale up at national level and their replication across countries. (Effectiveness, Sustainability) Additional Information / Explanations In each of the joint programme countries, interventions financed by the H4+ JPCS in selected districts of the country were intended to help demonstrate the benefits of a coordinated, integrated approach, and also were meant to yield accelerated improvements of access to integrated RMNCH health services and improved health outcomes in these districts. Under this issue, the evaluators are expected to assess the success of these interventions / projects, giving attention to issues of gender equality, equity and social inclusion. Under this issue, the evaluators should investigate to what extent the objectives of the H4+ JPCS have been in line with the priorities and needs regarding health systems strengthening and (integrated) health service delivery in programme countries. Evaluators should compare and contrast programme priorities (as expressed in the original programme design) a) with the needs identified in relevant government policies and plans, and the corresponding governmental priorities in programme countries; b) with health-related needs identified in relevant third party analyses of the health situation in programme countries. Under this issue, evaluators should provide a differentiated assessment of the success of the programme to identify new and promising practices, and to prepare for their replication and scale-up in H4+ JPCS programme countries. Issues involved here are: The strategies and methods chosen to identify practices for scale-up, The robustness of the evidence-base that was used to identify these practices as promising, The approach for engaging with partners at national level to identify End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 18

Indicative Areas of Investigation (DAC Criterion / Criteria covered) (5) The extent to which the division of labor among the six agencies and the coordination of work amongst themselves and with other development partners successfully leveraged the collective strengths and distinct advantages and capacities of each of the six H4+ partners, in the short and in the long-term. (Coordination, Efficiency, Sustainability) (6) The added value of H4+ JPCS for the operationalization and implementation of the Secretary General s Global Strategy for Women s and Children s Health, and the global movement Every Woman, Every Child that was launched by the Secretary General to put into action his Global Strategy. (Relevance, Value Added) Additional Information / Explanations what aspects of pilot interventions were to be scaled up; which populations were to become the intended target groups of these new, scaled-up interventions; which agencies in the Government would lead the scale-up effort, and how strategies were devised to ensure that the scaled-up interventions could be successfully taken-over and managed by the national health system at large. This last issue is connected to the issue of strengthening national health systems to deliver improved health services in order to improve RMNCH-related health outcomes. This assessment should cover the division of labor during the implementation of the H4+ JPCS, and should also examine the extent to which the joint programme promoted the development of organizational foundations for a more long-term cooperation between the H4+ agencies that is appropriately institutionalized in the strategic frameworks, planning processes, and structures of the individual agencies. Under this issue, evaluators should examine the advantages and strengths of the H4+ initiative 17 for putting into practice guiding principles of the Global Strategy for Women s and Children s Health (such as country leadership of national health systems strengthening, coordinated support, delivery of an integrated package ) at country and local level. This requires assessment of increased coverage, especially in terms of improvements in gender equality, equity and social inclusion. This should also include an assessment of the role, positioning and added value of the H4+ JPCS and the H4+ partnership at large within the context of other horizontal initiatives; such as the UN 17 Including its global technical team, and the collaborative efforts at country level. End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 19

Indicative Areas of Investigation (DAC Criterion / Criteria covered) Additional Information / Explanations Commission on Information and Accountability (CoIA) and the independent Expert Review Group (ierg). 5. Evaluation Approach and Methodology The evaluation will utilize a theory based approach, which means that the evaluation methodology will be based on the careful analysis of the intended outcomes, outputs, activities, and the contextual factors that may have had an effect on implementation of the H4+ JPCS and their potential to achieve the desired outcomes. Where outcome-level data is lacking, evaluators will base their assessment on the analysis and interpretation of the logical consistency of the chain of effects: linking programme activities and outputs with changes in higher level outcome areas, based on observations and data collected along the chain. This analysis should serve as the basis of a judgment by the evaluators on how well the H4+ JPCS has contributed to the achievement of the intended results foreseen in the programme documents and other relevant results frameworks. However, data on outputs should not be used as a surrogate to judge the achievement of programme outcomes. The evaluation team will develop the evaluation methodology in line with the evaluation approach, and design corresponding tools to collect data and information as a foundation for valid, evidence-based answers to the evaluation questions and an overall assessment of the joint programme. The methodological design will include: an analytical framework; a strategy for collecting and analyzing data; specifically designed tools; an evaluation matrix; and a detailed work plan. 5.1 Required Elements of the Evaluation Methodology The methodology for this evaluation should include the following elements: 5.1.1 Logical Reconstruction of the H4+ Intervention Logic and Theory of Change Although the implementation of the H4+ JPCS has implicitly been guided by a theory of change, this theory has not been explicitly stated in programme documents up to this point. The development of an explicit theory of change, and the reconstruction of its intervention logic will therefore play a central role in the design of the evaluation, in the analysis of the data collected throughout its course, in the reporting of findings, and in the development of conclusions and of relevant and practical recommendations. The two bilaterally funded interventions were launched initially on the basis of two similar but separate programme documents which were later brought together within a joint results End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 20

framework. This evolution will need to be reflected in the steps of the analysis and in the reconstruction of the intervention logic which will likely result in several logic models: first, one for each bilaterally-funded programme as initially defined; and, second, a model to reflect the joint intervention logic of the H4+ JPCS. An analysis of the various linkages between the funding partners and implementing agencies will be key. The analysis and the resulting logic models will need to take into account the respective goals and objectives and other key characteristics of the H4+ JPCS; including, among other things: (a) the strategic context of the H4+ JPCS, including the Secretary General s Global Strategy for Women s and Children s Health and the main principles for support contained therein; (b) the multi-level character of the H4+ JPCS; (c) the role of improved coordination among key stakeholders emphasized by the H4+ JPCS; and (d) the distinction between services and support rendered to national health systems by the H4+ JPCS on the one hand, and improved functioning and service delivery by these health systems to final beneficiaries; (e) the effect of the context of the H4+ JPCS and the individual actions financed by the programme on its performance and the results achieved. In other words, the logic models will need to address the interventions at global, regional, national and subnational levels. In order to adequately reflect this last characteristic of the H4+ JPCS, i.e. the layered nature of improving health service delivery to final beneficiaries, the evaluators should adopt a logic nomenclature to distinguish the different levels of hierarchy of effects of the H4+ JPCS. The terminology proposed below can be considered as a starting point in that regard: Level 1: H4+ JPCS programme activities (at global / regional, country and local (subnational) level) Level 2: Direct Outputs (the products, capital goods, tools, assets and services that result directly from / are produced directly by the H4+ JPCS and that are put at the disposal of its direct beneficiaries, at global / regional, country level and local level) Level 3: Induced Outputs (the products, capital goods, tools, assets and services that are produced with the help of the direct outputs of the H4+ JPCS by the direct beneficiaries of the H4+ JPCS (e.g., actors in the countries health systems) and that are put at the disposal of the health systems wider circle of stakeholders (e.g., individuals and organizations acting as service providers, administrators, etc.). Level 4: Outcomes (Changes in the behavior by health system stakeholders resulting from the use or application of induced outputs provided by the (strengthened) health system). Level 5: Impacts (Changes in the health status of women and children resulting from the strengthening of health systems). Elements of a preliminary conceptual framework for a logical model for the H4+ JPCS are presented in Annex 5 of these ToR. End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 21

5.1.2 Finalization of the Evaluation Questions and Assumptions The finalization of the evaluation questions that will guide the evaluation should clearly reflect the evaluation criteria and indicative areas of investigation listed in these ToR. They should also draw on the findings from the logical reconstruction of the intervention logic of the H4+ JPCS. The evaluation questions should be complemented by sets of assumptions that capture key aspects of the intervention logic associated with the scope of the question; and this will enable evaluators to gauge if logically required preconditions for a contribution of the H4+ JPCS to strengthened national health systems, improved health outcomes and other objectives of the joint programme are fulfilled. The data collection for each of the assumption will be guided by clearly formulated quantitative and qualitative indicators. 5.1.3 Well-designed Country Case Studies The evaluation will include both in-country (field) and desk based case studies. The case studies will contribute to the overall evaluation with in-depth data and information, opinions, and analysis. All ten countries under the joint H4+ collaboration will be featured as a case study: four will undergo an in-country, field-based review (including the pilot/exploratory mission), while the remaining six will be subject to a desk review. Case studies will aim to maximize the breadth and depth of insights into the evaluation questions and provide a comprehensive and nuanced picture of the actions of the H4+ JPCS and their effects. Case studies will, therefore, be illustrative (rather than statistically representative), exemplifying the range of contexts addressed and interventions taken forward under the H4+ initiative. Field-based case study reports will be made available on line (see Annex 1.b). Countries were apportioned (to either a field or desk based case study) via a consultative process, based on the collective judgment of key stakeholders with familiarity of the H4+ initiative, including the Evaluation Reference Group / Management Reference Group and the H4+ Secretariat. To provide country specific contextual information, country profile documents were developed and were referred to in the process of case study allocation. The country profile documents provide a useful snapshot of key contextual factors shaping the programme of work of the H4+ JPCS Key data - including RMNCH related data, social and demographic information, data on government effectiveness, and H4+ expenditure - are included in the profiles, together with additional context specific information. The country profiles can be found in Annex 8. Table 2 below presents the results of the case study selection process. End Line Evaluation of the H4+Joint Programme Canada and Sweden (Sida) 2011-2016 22