Key findings from the evaluation of the 2017 Harvard Ministerial Leadership Forum for Education and Health Ministers

Transcription

1 Key findings from the evaluation of the 2017 Harvard Ministerial Leadership Forum for Education and Health Ministers Key messages from the evaluation 1. Overall, the content of the 2017 Forum for Education and Health Ministers was relevant to the needs of ministers. The Forum was well aligned with ministers expectations and the programme covered issues that were relevant to their ministerial roles in Education and Health. 2. The Forum was a well-structured and facilitated event, however, the course could provide a more conducive learning environment for ministers by allowing more time in the programme for interaction and exchange and possibly reducing the intensity and scope of the programme to allow ministers more time to reflect and process information. 3. Based on ministers self-assessment before and after the Forum, there is clear evidence of improvements in their knowledge and confidence. The Forum resulted in improvements in ministers knowledge and confidence in each of the main themes of: leadership, finance and delivery. 4. The 2017 Forum had a stronger focus on the ministerial legacy statements. This was an important shift from previous Forums and for those involved in the delivery of the Forum, a positive development. Overall, the quality of ministers final legacy statements was good and across the cohort, the ambition of the legacy statements was the highest scoring criteria. The measurability of the legacy statements has the most room for improvement. The Harvard Ministerial Leadership Programme The Harvard Ministerial Leadership Programme (HMLP) aims to stimulate transformational leadership capacities and political acumen among education, health and finance ministers from across South East Asia, Latin America and Africa. The HMLP started in 2012 with a focus on health and finance ministers. It is now being expanded to include education ministers. HMLP has three main components: the Ministerial Leadership Forums which are four-day immersions designed to expose ministers to new tools, build their leadership capacities and support them to develop an ambitious legacy statement; follow up workshops in a subset of participating countries for a selection of senior officials from education, health and finance to support them in planning and implementing the ministers legacy; and ongoing process monitoring that involves ministers and their teams submitting monthly progress updates to HMLP which form the basis for ad hoc problem solving and technical assistance. The 2017 Ministerial Leadership Forum for Education and Health Ministers took place at Harvard University, June. It was the first time education ministers were included in the programme. A total of 16 ministers (nine education and seven health) from 13 countries participated. 1

2 The evaluation of the Forum This briefing presents the finding from the evaluation of the 2017 Ministerial Forum. It is part of a wider evaluation of the overall HMLP that is running alongside implementation over the next five years. The evaluation was informed by four evaluation questions: Was the content of the Forum relevant to ministers needs? Did the Forum provide a conducive learning environment for ministers? To what extent do ministers have improved competencies? To what extent are ministers legacy goals ambitious, clear and measurable? These questions provide the structure for the findings that are discussed below. The evaluation methodology involved: a Pre and post Forum survey of all 16 ministers; exit interviews with ten ministers; interviews with the Forum facilitation team and programme management; and an assessment of Ministers legacy statements against three quality criteria: clarity, ambition and measurability. Was the content of the Forum relevant to ministers needs? Overall, the content of the 2017 Forum was relevant to the needs of ministers. The evidence points to the Forum being well aligned with ministers expectations and the programme covering issues that were relevant to their ministerial roles in Education and Health. The Forum provided ministers with the space and structure to articulate the legacy they wanted to achieve. 15 out of the 16 ministers agreed that the Forum had helped them define a transformative legacy for their ministerial tenure. For some, the Forum provided an opportunity to sharpen an existing set of goals. For others it was an opportunity to validate their thinking with colleagues and peers. The Forum has provided me with an opportunity to further sharpen my goals. When I said that I wanted by % of the schools being quality what did that mean in actuality. I have clarified that I want to emphasize teachers showing up and teaching in the schools, monitored and held accountable. This will lead to improved learning outcomes for boys and girls, but particularly for the girls. In the discussions, I really reflected on this is what I would like to do. The Forum provided ministers with the opportunity to reflect on and understand what transformational leadership means. Seven ministers indicated that the focus on what leadership is and how to achieve it were the most useful insights from the Forum. Likewise, 15 out of 16 ministers agreed that the Forum offered them a unique opportunity to develop their leadership competencies and fulfil their ministerial responsibilities. The importance of focusing on implementation and how to manage delivery were key insights for ministers. This came out in both the interviews and the survey. Five ministers noted how the attention given to delivery helped shift their perspective on how to affect change and their role in this. The structured thinking to define your legacy was important. It dawned on you that you must define clear goals for the implementation period. It dawned upon you that having a vision is not enough. The Forum helps to see the importance of having a vision, of stakeholder mapping and making sure you create an effective delivery system, with a strong M&E system. It invites you to operate in a structured manner... A particular topic covered in the Forum which a small number of ministers had concerns with was Public- Private Partnerships (PPPs). Four ministers specifically mentioned the module on PPPs as disappointing and having not met their expectations. Some ministers also indicated that they felt the focus on education could have been stronger. In the four cases where this was raised, the common point was that the ministers felt they would have benefitted more if they had spent more time in smaller groups having a focused discussion on education specific issues Did the Forum provide a conducive learning environment for ministers? The Forum was a well-structured and facilitated event that has led to improvements in ministers knowledge and confidence across a range of competencies. However, the evidence points to a number of areas where the structure of the course could be improved to provide a more conducive learning environment. The structure of the Forum, for a number of ministers, didn t allow for sufficient time for interaction and exchange. This was raised in five of the exit interviews and in the survey responses. The common message was that there needs to be more time dedicated to fostering quality and meaningful dialogue between ministers and with the facilitators and less time for power point presentations that are more about a one way flow of information. 2

3 Figure 1 Average changes in Minister knowledge and confidence across the main themes of the Forum: leadership, delivery and finance (n=16) source: baseline and endline survey of ministers [The Forum] could do a lot more in terms of becoming more interactive. There are people in the room with real life experiences of how to implement public sector programs. Give them more opportunities. The facilitators have knowledge, but that knowledge should come as a result of the issues described by the ministers sitting there. Less talking to them but facilitating a conversation among them. The intensity and scope of the programme meant some issues were treated superficially and there was limited time for ministers to reflect and process information. Seven ministers commented that they thought the programme was too intensive. They appreciated the need to get through lots of content, and that having a longer course was not an option, but the intensity of the course, for some, meant they had limited time to process, reflect and assimilate what they were hearing. To what extent did the Forum strengthen ministers competencies? Based on ministers self-assessment before and after the Forum, there is clear evidence of improvements in their knowledge and confidence The Forum resulted in improvements in ministers knowledge and confidence in each of the main themes of: leadership, finance and delivery. Figure 1 presents the overall changes in Ministers self-reported knowledge and confidence across the three themes. It shows that on average Ministers moved one point along the scoring scale (e.g. from feeling somewhat equipped to well equipped ). At the level of specific technical issues, again there were improvements across the board. The survey asked respondents at baseline and endline how well equipped they felt to undertake a specific task. The areas where the Forum seems to have had the biggest impact on Ministers knowledge and confidence were: Making operational decisions to increase health and education delivery system performance; Identifying and promoting programmatic opportunities for collaboration between education and health; Implementing performance based budgeting; and Developing close collaboration with the Finance Minister to support your transformational (legacy) goals. The three areas where there was the least improvement were: Identifying specific opportunities in education and health that contribute to the enabling conditions for sustainable economic development in your country; Identifying fiscal space in your budget; and developing public / private partnerships. To what extent did Ministers articulate ambitious and clear legacy statements? The 2017 Forum had a stronger focus on the ministerial legacy statements. This was an important shift from previous Forums. While the legacy statements have always been a focus, there was a more deliberate attempt in this Forum to clearly define and record them. For those involved in the delivery of the Forum this was seen as a positive development. While we did not see Ministers first articulation of their legacy statements on the first day, multiple stakeholders indicated that the final statements were a significant improvement on these. 3

4 Our assessment of the quality of the legacy statements looked at three dimensions: Clarity: The legacy statement includes a clear articulation of the specific changes that are being sought. Ambition: The extent of change detailed in the legacy statement represents a major step change based on the current situation. Measurability: The changes detailed in the legacy statement are measurable; quantifiable targets have been set; and baseline values have been included. Based on our quality assessment we found that: Overall, the quality of ministers final legacy statements was good and across the cohort, the ambition of the legacy statements was the highest scoring criteria. All ministers set far reaching objectives that would have transformational effects on their country. The clarity of the legacy statements was more mixed. The more mixed quality came from ministers not articulating clearly what they wanted to achieve and/or including a wide range of issues in their statement. The clearest legacy statements were those that had a laser sharp focus. The measurability of the legacy statements was the weakest dimension of quality. We identified six statement where there was room for improvement with regards how measurable they were. The three most common areas for improvement were that issues were not sufficiently unpacked to be measured, specific targets not being set and baseline values not being included. 4 We want the resources invested in international development to deliver the best possible results for the poor. Through our innovative consultancy services in monitoring and evaluation we provide the insight and ideas to ensure that they do. Itad and the tri-colour triangles icon are a registered trademark of ITAD Limited. The views expressed in this briefing are those of the evaluators. 12 English Business Park English Close, Hove BN3 7ET, United Kingdom T +44 (0) E mail@itad.com W itad.com

5 EVALUATION : EVALUATION OF THE 2017 HARVARD MINISTERIAL LEADERSHIP FORUM FOR EDUCATION AND HEALTH MINISTERS Submitted by Itad In association with: Big Win Philanthropy and the Harvard Ministerial Leadership Programme Itad 2017

6 Evaluation Report Evaluation of the 2017 Harvard Ministerial Leadership Forum for Education and Health Ministers Table of contents 1. Introduction Harvard Ministerial Leadership Programme Ministerial Leadership Forum for Education and Health Ministers, Background to the wider evaluation Structure of the report 4 2. Methodology 4 3. Findings Was the content of the Forum relevant to ministers needs? Did the Forum provide a conducive learning environment for Ministers? To what extent did the Forum strengthen ministers competencies? To what extent did Ministers articulate ambitious and clear legacy statements? 11 Annex 1 - Dashboard for the 2017 Ministerial Leadership Forum for Education and Health Ministers Error! Bookmark not defined. Annex 2 Harvard Ministerial Leadership Programme s impact pathway 15 Annex 3 Baseline survey for Ministerial Forum 17 Annex 4 Endline survey for Ministerial Forum 19 Annex 5 Exit interview guide for ministers 22 Itad 2017

7 1. Introduction This is the evaluation report from the 2017 Ministerial Leadership Forum for Education and Health Ministers run by the Harvard Ministerial Leadership Programme (HMLP). It provides insights into the relevance and effectiveness of the Forum based on interviews with ministers and the Harvard programme team, a pre and post Forum survey, and a review of ministers legacy statements Harvard Ministerial Leadership Programme The HMLP aims to stimulate transformational leadership capacities and political acumen among education, health and finance ministers from across South East Asia, Latin America and Africa. The HMLP started in 2012 with a focus on health and finance ministers. It is now being expanded to include education ministers. The HMLP has three main components: 1. Ministerial Leadership Forums, which are four-day immersions designed to expose ministers to new tools, build their leadership capacities and support them to develop an ambitious transformative vision In-country follow-up workshops in a subset of participating countries for a selection of senior officials from education, health and finance to support them in planning and implementing the ministers vision Process monitoring that involves ministers and their teams submitting monthly progress updates to HMLP which form the basis for ad hoc problem solving and technical assistance Ministerial Leadership Forum for Education and Health Ministers, 2017 Ministerial Leadership Forums have three main objectives: To support ministers to define their legacy vision and goals; Enhance ministers leadership effectiveness, and Improve ministers competencies in health and education system management, budgeting and resource utilisation, policy implementation and performance management. 3 The 2017 Ministerial Leadership Forum took place at Harvard University, June. It was the first time education ministers were included in the programme. A total of 16 ministers (nine education and seven health) from 13 countries participated. Ministers were selected based on a number of criteria, including targeting education ministers in countries where health and finance ministers are already part of the follow-up component of the HMLP; key countries that have not yet been involved in the programme; and countries where new ministers have recently been appointed. 4 1 Two separate forums are held annually: one for education and health ministers in June, and one for finance ministers in April. The focus of this evaluation report is only on the Education and Health Ministers Forum. 2 Follow-up support is offered to 4-6 ministers each year based on: an assessment of the transformative leadership capacity of the education, health and finance ministers; regional/geographic significance; the in-country health situation; and the governance environment (HMLP Interim Narrative Report, January 1 April 30, 2017.) 3 Taken and adapted from the Harvard Ministerial Leadership Programme Model. 4 HMLP Interim Narrative Report, January 1 April 30, Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 3

8 The programme for the Forum was split over four days with a combination of presentations, group discussion and case study work. At specific points in the programme, parallel sessions were run for education and health ministers to allow for more sector focused discussion. Workbooks were used by ministers to record the progress over the four days. 5 Particular focus was given in this year s Forum to ministers working on their legacy statements. These were recorded in the ministers workbooks Background to the wider evaluation This evaluation of the 2017 Forum is part of a wider evaluation of the overall HMLP that is running alongside implementation over the next five years. The purpose of evaluation is to work alongside HMLP to better understand how the Ministerial Leadership Programme is performing, what is working well and where there are challenges. This will help drive improvements and learning in HMLP, but also help Big Win Philanthropy (BWP) to understand the impact of its investment. As such, the evaluation has both a learning and accountability function. The overall evaluation is structured around testing the HMLP s impact pathway (See annex 1.) 1.4. Structure of the report The report is structured into four sections. Following this introduction, the report moves on to present the methodology (section 2). The core of the report is the findings section (section 3). In this section we take a closer look at whether the content of the Forum was relevant to ministers needs; whether the Forum provided a conducive learning environment for ministers; the extent to which the Forum strengthened ministers competencies; and to what extent ministers were able to articulate ambitious and clear legacy statements. Annex 1 details the impact pathway for the HMLP and Annex 2-4 includes the various data collection tools used in the evaluation. 2. Methodology The evaluation was informed by a series of evaluation questions that focus on the Forum s relevance and effectiveness (See table 1). To answer these we brought together data from a range of data sources, using three main data collection tools: Pre and post Forum survey of all ministers. This was administered at the beginning and end of the Forum to measure changes in ministers self-reported knowledge and competencies (See Annex 2 and 3). Exit interviews with ten ministers. These were conducted on the last two days of the Forum. They were used to explore in more detail ministers experiences of the Forum. The interviews were between minutes and structured around five open-ended questions (see Annex 4.) All interviews were recorded and a summary transcript prepared. Interviews with the Forum facilitation team and programme management. We conducted interviews with a member of the Forum facilitation team and programme management to gather their perspectives on what worked well, what worked less well and lessons learnt from the 2017 Forum. Review of Minister's legacy statements. This was done through observing ministers presenting their visions on the last day of the Forum and taking photos of the final written statements in Ministers workbooks. The statements were then assessed based on three dimensions of quality: clarity, ambition and measurability. 6 See Table 1 for a definition of what we mean by each of these terms. 5 Workbooks have always been used, but their use is going to be given greater prominence in this forthcoming Forum (HMLP Interim Narrative Report). 6 This quality framework is amended from the one originally proposed in the inception report. This was done to better align our assessment of quality with that of the HMLP which defined 6 criteria for assessing the quality of the legacy statements in the Ministers workbook. These included: ambitious, transformative, achievable, urgent, addresses national need, promotes national development. Our amended quality framework focuses on those issues which we felt we could make an informed judgement on. Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 4

9 Table 1 Dimensions of quality used to assess ministers' legacy statements Dimension of quality Clarity Ambition Measurability Definition The legacy statement includes a clear articulation of the specific changes that are being sought. The extent of change detailed in the legacy statement represents a major step change based on the current situation. The changes detailed in the legacy statement are measurable; quantifiable targets have been set; and baseline values have been included. We used a three point scale to structure our assessment against each of these dimensions of quality. High quality meets all aspects of the quality dimension Good quality meets some aspects of the quality dimension Room for improvement meets no aspects of the quality dimension. Table 2 provides a summary of how the different data sources have been brought together to inform our analysis against each evaluation question. Table 2 Evaluation matrix for managing data collection and analysis of the Ministerial Forums Evaluation questions Data collection tools / source Data analysis 1a. Was the content of the Forum relevant to Ministers needs? Pre/post survey with ministers Exit interviews with ministers 1b. Did the Forum provide a conducive learning environment for ministers? 1c. To what extent do ministers have improved knowledge and confidence as a result of the Forum? 1d. To what extent are ministers legacy goals ambitious, clear and measurable? Pre and post survey with ministers Exit interviews with ministers Pre/post survey with ministers We will analyse pre and post survey data to identify areas where there has been the most and least improvement in ministers capacities. As well as analysing data on individual competencies, we will also group the data around the three core themes of the Forum (leadership, finance, delivery) and provide aggregated scores. Exit interviews with ministers We will assess ministers visions and goals Review of workbook against a quality framework using data from Observation of Ministers interviews, observation and the review of presentation of their legacy the workbooks. Final judgements will be vision presented using rating scale with Interviews with HMLP team and accompanying narrative. BWP representative 3. Findings In this section we explore whether (3.1) the content of the Forum was relevant to ministers needs; (3.2) whether the Forum provided a conducive learning environment for the ministers; (3.3) to what extent the Forum strengthened Ministers competencies; and, (3.4) to what extent did ministers articulate ambitious and clear legacy statements. Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 5

10 3.1. Was the content of the Forum relevant to ministers needs? Overall, the content of the 2017 Forum was largely relevant to the needs ministers. The evidence points to the Forum being well aligned with ministers expectations and the programme covering issues that were relevant to their ministerial roles in Education and Health. The three areas of the programme that were commonly identified by the ministers as the most useful were: The space and structure the Forum provided to support them in articulating a legacy; The opportunity provided to reflect on what transformational leadership means; The opportunity to reflect on how to put this into practice; and, the attention given to implementation and delivery. The Forum provided ministers with the space and structure to articulate the legacy they wanted to achieve. This was raised in both the interviews with ministers and the survey. 15 out of the 16 ministers agreed that the Forum had helped them define a transformative legacy for their ministerial tenure. 7 For some, the Forum provided an opportunity to sharpen an existing set of goals. 8 For others, it was an opportunity to validate their thinking with colleagues and peers. 10 The Forum was for some ministers their first introduction to thinking about a legacy. In these cases, ministers commented that the value of the programme was that provided them with a structured framework for thinking about defining their transformative vision. 11 The Forum provided ministers with the opportunity to reflect on and understand what transformational leadership means. Seven ministers indicated that the focus on what leadership is and how to achieve it were The Forum has provided me with an opportunity to further sharpen my goals. When I said that I wanted by % of the schools being quality what did that mean in actuality. I have clarified that I want to emphasize teachers showing up and teaching in the schools, monitored and held accountable. This will lead to improved learning outcomes for boys and girls, but particularly for the girls. In the discussions, I really reflected on this is what I would like to do. 9 the most useful insights from the Forum. 12 Likewise, 15 out of 16 ministers agreed that the Forum offered them a unique opportunity to develop their leadership competencies and fulfil their ministerial responsibilities. 13 The importance of focusing on implementation and how to manage delivery were key insights for ministers. This came out in both the interviews and the survey. 14 Five ministers noted how the attention given to delivery helped shift their perspective on how to affect change and their role in this. For some, it was a case of realising that their tenure as a minister is likely to be short and that if they want to have a legacy they need to move from ideas to action quickly. 15 For others, the Forum helped unpack the specific steps they needed to take in driving change. A particularly important insight for some ministers was that it is not enough to simply have a vision and 7 13 Ministers strongly agreed and 2 agreed with the statement: the Forum helped me define a transformative purpose (legacy) for my tenure as minister (Endline survey) 8 Interview with Minister 1 9 Interview with Minister 1 10 Interview with Minister 2 11 Interview with Minister Endline survey responses Ministers strongly agreed and 2 agreed with the statement: the Forum provided a unique opportunity for me to develop my leadership competencies and improve my effectiveness as a Minister (Endline survey) 14 Three ministers highlighted the focus on implementation and how to manage delivery as the most useful insights from the Forum. 15 Interview with minister 9 Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 6

11 communicate this, but that you need to build an implementation team, monitor results and build coalitions etc. 16 A particular topic covered in the Forum which a small number of ministers had concerns with was Public-Private Partnerships (PPPs). Four ministers specifically mentioned the module on PPPs as disappointing and having not met their expectations. One noted that while this session was of real interest to him, and aligns with his efforts in his country to engage the private sector in society, the session was too general and lacked concrete examples. 19 Three other ministers expressed similar sentiments. The Forum survey also supports this finding. It shows that ministers reported seeing the least improvement in knowledge and skills in developing PPPs to increase investment in education and health (see section 3.3.) Some ministers indicated that they felt the focus on education could have been stronger. Given this is the first time the Forum has been run with education ministers, we think it is important to highlight this issue. In the four cases where this was raised, the common point was The structured thinking to define your legacy was important. It dawned on you that you must define clear goals for the implementation period. It dawned upon you that having a vision is not enough. The Forum helps to see the importance of having a vision, of stakeholder mapping and making sure you create an effective delivery system, with a strong M&E system. It invites you to operate in a structured manner. I can confidently say that the Forum has built my leadership competencies. I am going away with a strong resolve to create this team to drive the change in the health systems As soon as I get back, I will call my senior team together, drum home again to them the priorities. Each of those priorities must have a timeline, we must have a timeline to achieve miniresults that point to the overall goal. And there must be an individual at each of the stations that is responsible, and that person will be responsible to who. It is important to reach out both within and outside the ministry, because it is important to build coalitions around our major priorities. This is one of the things I never did. I used to communicate the priorities and thought that magically we will get there. 18 that the ministers felt they would have benefitted more if they had spent more time in smaller groups having a focused discussion on education specific issues. As one minister commented, parallel session could have been longer, with more opportunities for exchange. 20 A similar point was raised in the survey responses. 21 A separate but related point raised by another minister was that they felt the education side of the programme could have been strengthened, in particular through ensuring it has similar intellectual weight and rigour as the Health sector parts of the programme Interviews with minister 10, 9, 7, 2 17 Interview with minister Interview with minister 8 19 Interview with minister 4 20 Interview with minister 8 21 Endline survey response 22 Interview with minister 1 Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 7

12 3.2. Did the Forum provide a conducive learning environment for Ministers? The Forum was a well-structured and facilitated event that has led to ministers having improved learning and knowledge across a range of competencies. However, the evidence points to a number of areas where the structure of the course could be improved to provide a more conducive learning environment. The key issues raised by ministers were that: The structure of the Forum, for some, didn t allow for sufficient time for interaction and meaningful peerexchange. The intensity of the programme structure meant some ministers didn t have the time to reflect and process information. The structure of the Forum, for a number of ministers, didn t allow for sufficient time for interaction and exchange. This was raised in five of the exit interviews and in the survey responses. The common message was that there needs to be more time dedicated to fostering quality and meaningful dialogue between Ministers and with the facilitators and less time for power point presentations that are just about a one way flow of information. Ministers wanted more time to share their experiences with peers and explore the specific challenges that they face. 25 As one minister remarked, the course structure should be better grounded in the principles of adult learning and take a more participant focused approach that draws on the experience of the Minister much more. 26 Interestingly, this same point was also raised by one of the course facilitators. 27 Not everyone shared this perspective, however. In fact, a majority of ministers indicated in their survey responses that they thought the Forum gave them sufficient time for discussion and exchanges with other Ministers, while six were either undecided or disagreed. [The Forum] could do a lot more in terms of becoming more interactive. There are people in the room with real life experiences of how to implement public sector programs. Give them more opportunities. The facilitators have knowledge, but that knowledge should come as a result of the issues described by the ministers sitting there. Less talking to them but facilitating a conversation among them. 23 There are things I wanted to say, questions I wanted to ask. I kept looking at the agenda to see where I might be able to do this, but it was too packed. The parallel sessions offered that opportunity but more time would have been needed to go into more depth on issues. 24 While it is clear that the Forum was overall successful in supporting learning among the ministers (see section 3.3), the issue raised here is about how the course could be structured and a learning environment created, that could maximise this learning even further. The intensity and scope of the programme meant some issues were treated superficially and there was limited time to reflect and process information. Seven ministers commented that they thought the programme was too intensive. They appreciated the need to get through lots of content, and that having a 23 Interview with minister 1 24 Interview with minister 4 25 Interview with minister 3, 4, 5 26 Interview with minister 3 27 Interview with forum facilitator 1 Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 8

13 longer course was not an option, but the intensity of the course, for some, meant they had limited time to process, reflect and assimilate what they were learning. 28 Some felt the long days produced diminishing returns in terms of the information that people were able to process and retain. 29 A suggestion made by three ministers was to have fewer issues covered in the Forum. 30 As one commented, you could reduce the number of themes that are covered on a superficial level, focusing on a few themes but going into more depth To what extent did the Forum strengthen ministers competencies? Based on ministers self-assessment before and after the Forum, there is clear evidence of improvements in their knowledge and confidence: At the aggregate level, there were improvements in ministers knowledge and confidence across the three main themes of the Forum: leadership, finance and delivery. At the level of specific technical competencies, again, across all ministers, the data reveals improvements in knowledge and confidence. The Forum resulted in improvements in ministers knowledge and confidence in each of the main themes of leadership, finance and delivery. Figure 1 presents the overall changes in ministers self-reported knowledge and confidence across the three themes. It shows that on average ministers moved one point along the scoring scale (e.g. from feeling somewhat equipped to well equipped ). Interestingly, the relative positioning of the baseline and endline values are consistent across the three themes. At baseline ministers felt least equipped in finance, followed by delivery and leadership; at endline, while there had been improvements, this order remained the same. Figure 1 Average changes in minister knowledge and confidence across the main themes of the Forum: leadership, delivery and finance (n=16) source: baseline and endline survey of ministers 28 Interview with minister 2, 3, 6, 7, 9 29 Interview with minister 6 30 Interview with minister 5, 6, 7 31 Interview with minister 4 Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 9

14 At the level of specific technical issues, again there were improvements across the board. Figure 2 presents the overall changes in ministers self-reported knowledge and confidence across a range of specific technical areas. The survey asked respondents at baseline and endline how well equipped they felt to undertake a specific task. The areas where the Forum seems to have had the biggest impact on ministers knowledge and confidence were: 32 Making operational decisions to increase health and education delivery system performance Identify and promote programmatic opportunities for collaboration between education and health; Implement performance based budgeting; Develop close collaboration with the Finance Minister to support your transformational (legacy) goals. The three areas where there was the least improvement were: Identifying specific opportunities in education and health that contribute to the enabling conditions for sustainable economic development in your country; 33 Identifying fiscal space in your budget; Developing public / private partnerships Across each of these technical tasks, on average ministers moved up 1.19 points along the scoring scale 33 For this technical tasks, on average ministers moved 0.75 points along the scoring scale 34 For this technical task, and the one above, on average ministers moved 0.88 points along the scoring scale Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 10

15 Figure 2 Average self-assessment scores across all ministers at baseline and endline* How well quipped do ministers feel to *(n=16) Source: baseline and endline survey of Ministers 3.4. To what extent did ministers articulate ambitious and clear legacy statements? The 2017 Forum had a stronger focus on the ministerial legacy statements. This was an important shift from previous Forums. While the legacy statements have always been a focus, there was a more deliberate attempt in this Forum to clearly define and record them. For those involved in the delivery of the Forum, this was seen as a positive development. 35 While we did not see the ministers first articulation of their legacy statements on 35 Interview with forum facilitator 1 Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 11

16 the first day, multiple stakeholders indicated that the final statements were a significant improvement on these. 36 Based on our quality assessment we found that: The quality of the statements was generally good The ambition 37 of statements was clear The clarity 38 of the statements was variable The measurability 39 of the statement was the weakest dimension of quality. Overall quality of the final legacy statements was good. Table 3 below details our assessment of the statements against three dimensions of quality: the clarity of the statement, the ambition of the statement and the measurability of the statement. Across the cohort, the ambition of the legacy statements was the highest scoring criteria. All ministers set far reaching objectives that would have transformational effects on their country. The following extracts from the statements are illustrative of this: To achieve universal health care coverage in four years. To achieve universal education at the primary level in four years. To eliminate malaria infections. The clarity of the legacy statements was more mixed. The more mixed quality came from ministers not articulating clearly what they wanted to achieve and/or including a wide range of issues in their statement. For example, it is not clear from the legacy statement of one minister s statement (To have 75% of primary school students develop the competencies expected from them, by 2021, against the present 48%, through: School management; Continuous teacher training; and school feeding program) what competencies students need to develop. Likewise, while parts of a minister s statement are clear (reducing the mortality rate of children younger than five years), other parts (such as taking care of chronic illnesses such as cancer and cardiovascular illnesses by repositioning or realigning primary health care ) are not. The clearest legacy statements were those that had a laser sharp focus. For example, one minister states that he will increase basic education completion from five years (now) to nine year basic education by Similarly, another states by 2021, all public primary schools will have one trained, certified, and paid on time teacher per classroom and children in all public primary schools will be assessed through EGRA and EMA to measure literacy and numeracy. 36 Interview with forum facilitator 1; Interview with Harvard programme 37 Our criteria for judging ambition is: The extent of change detailed in the legacy statement represents a major step change based on the current situation. 38 Our criteria for judging clarity is: The legacy statement includes a clear articulation of the specific changes that are being sought. 39 Our criteria for judging measurability is: The changes detailed in the legacy statement are measurable; quantifiable targets have been set; and baseline values have been included. Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 12

17 Table 4 Quality assessment of ministers' legacy statements Name of Minister Clear The legacy statement includes a clear articulation of the specific changes that are being sought. Quality assessment of legacy statement Ambitious The extent of change detailed in the legacy statement represents a major step change based on the current situation Measurable The changes detailed in the legacy statement are measurable; quantifiable targets have been set; and baseline values have been included. Minister 1 Minister 2 Minister 3 Minister 4 Minister 5 Minister 6 Minister 7 Minister 8 Minister 9 Minister 10 Minister 11 Minister 12 Minister 13 Minister 14 Minister 15 Minister 16 Key: High quality meets all aspects of the quality dimension Good quality meets some aspects of the quality dimension Room for improvement meets no aspects of the quality dimension Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 13

18 The measurability of the legacy statements was the weakest dimension of quality. As detailed in table 3 we identified six statement where there was room for improvement with regards how measurable they were. The three most common areas for improvement were: 1) issues were not sufficiently unpacked to be measured (e.g. Attaining universal education at the primary level by implementing the compulsory education policy by 2022); 2) specific targets were not set (e.g. Transform and expand technical education and vocational training in two five years); 3) baseline values were not provided (e.g. Achieve a foundation building block of pupil early grade reading and math s target of at least 90% through reform of curricula within two years.) Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 14

19 Annex 1 Harvard Ministerial Leadership Programme s impact pathway At the heart of the evaluation design is an impact pathway for the HMLP. The impact pathway unpacks the underlying logic of the HMLP and articulates the causal pathway from what the HMLP does, to the changes it hopes to achieve. This provides an overall framework and structure for the evaluation. The HMLP impact pathway details results at a number of levels 40 : 1. Activities/outputs relate to what the HMLP provides ministers and their senior officials through the Ministerial Forums and in-country follow-up support. 2. Short-term outcomes detail the changes HMLP would expect to see in ministers and senior officials both at the forum and in-country follow-up and in the two three months afterwards. These changes include: improved understanding of specific tools, a changed attitude towards leadership and the formulation of a transformative vision. 3. Intermediary outcomes are changes that HMLP would expect to see in ministers and senior officials, after the Forum and in-country follow-up. For example, changes in ministers leadership behaviour and specific steps to move forward ministers legacy visions. Based on feedback we received during the inception phase, we have defined three levels of intermediary outcomes. These reflect the specific changes that we would expect to see in-country as ministers take forward their visions. As illustrated in Figure 2, we have put nominal timeframes against the levels; these outline the likely timeframes during which we would expect to see the different outcomes emerging. Assessing the extent to which (and how) these intermediary outcomes are being achieved will be the primary focus of the evaluation, particularly the monitoring of the implementation of ministers visions and, if we move forward with them, the country case studies. 4. Long-term outcomes are changes that the HMLP would hope to see, but again, only after some time. 5. Institutional impact - These relate to more system-wide changes in how government functions, such as improvements in public service delivery systems and standards in health and education, and greater coordination of human development policy across government. HMLP is only one of a number of initiatives that will be driving these changes. While it is important to clearly articulate these, we do not suggest investing resources in trying to test this link in the impact pathway. 6. Societal impacts are the ultimate changes HMLP hopes to support: improved health and learning outcomes and economic growth. Similar to the institutional impacts, we do not suggest testing this link in the impact pathway. Embedded in the draft HMLP Impact Pathway are two sub theories. The first, illustrated by the black arrows, outlines the causal logic when a minister only attends the Forum. The second, illustrated by the red arrows, details the causal logic when ministerial attendance at the Forum is combined with in-country follow-up support. 40 Underpinning the HMLP is a well-known model for evaluating professional development, programmes, called the Kirkpatrick model. This suggests that professional development should be assessed at four levels: 1. Participant s reaction to the training/support; 2. Participant s learning from the training/support; 3. Participant s use of the learning; 4. The impact of the application of the learning on the performance of the organisation. Itad has used this model in a number of capacity development evaluations and found it a useful framework for thinking about the different levels of change that are of interest when assessing professional development programmes. Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 15

20 Figure 3 Draft impact pathway for the Harvard Ministerial Leadership Programme Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 16

21 Annex 2 Baseline survey for Ministerial Forum Harvard Ministerial Leadership Forum June 2017 COMPETENCY PROFILE (Baseline) This exercise is part of an on-going independent evaluation of the value of this Forum to enhancing your role and impact during your term as Minister. The purpose of your feedback is to understand how well you feel equipped to deliver against the core competencies the ministerial forum is seeking to develop. Below is a list of competencies related to your role as Minister. Please rate do you feel equipped, using a five-point scale: from 1 = not at all equipped, to 5 = fully equipped. Tick [] the chosen rating block. Please note: Itad has requested your name so we can compare your baseline and endline responses. Only Itad has access to your feedback and you will not be named in the evaluation report. The findings will be used to inform planning for future Ministerial Forums. Do you feel prepared to: 1. Not at all equipped 2. Not well equipped 3. Somewhat equipped 4. Well equipped 5. Fully equipped 1 Use your authority and leadership capabilities to make a transformative impact during your tenure as Minister. 2 Identify specific priorities in education and health that contribute to the enabling conditions for sustainable economic development in your country 3 Develop close collaboration with the Finance Minister to support your transformational (legacy) goals. 4 Achieve increased budget effectiveness and efficiency in resource utilization to support your transformational (legacy) goals as Minister. 5 Implement performance-based budgeting to align budget allocations with goals and results. 6 Identify fiscal space in your budget through budget reallocation, new revenues and/or cost savings to support your transformational (legacy) goals. 7 Identify and promote programmatic opportunities for collaboration between education and health in support of critical human development priorities. 8 Make operational interventions to increase health/education delivery system performance to support improved standards of service delivery and better outcomes in education/health. Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 17

22 9 Develop public-private partnerships to increase investment and enhance performance in human development priorities. 10 Organize within your Ministry and collaborate with Cabinet colleagues, other senior government officials and constituencies to achieve your transformational (legacy) goals. Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 18

23 Annex 3 Endline survey for Ministerial Forum Harvard Ministerial Leadership Forum June 2017 COMPETENCY PROFILE (Endline) This exercise is part of an on-going independent evaluation of the value of this Forum to enhancing your role and impact during your term as Minister. The purpose of your feedback is to assess how your competencies have developed during the Forum. On page 1, there are 10 competencies related to your role as Minister. Please rate do you feel equipped, using a five-point scale: from 1 = not at all equipped, to 5 = fully equipped. Tick [] the chosen rating. On page 2, there are 5 questions related to the content and structure of the leadership forum. Please rate the extent to which you agree with the statement: from 1 = strongly disagree, to 5 = strongly agree. Tick [] the chosen rating. Also on page 2, please share two aspects of the Forum that were most and two that were least useful. Please note: Itad has requested your name so we can compare your baseline and endline responses. Only Itad has access to your feedback and you will not be named in the evaluation report. The findings will be used to inform planning for future Ministerial Forums. Now that the Forum has ended, how well equipped do you feel to: 1. Not at all equipped 2. Not well equipped 3. Somewhat equipped 4. Well equipped 5. Fully equipped 1 Use your authority and leadership capabilities to make a transformative impact during your tenure as Minister. 2 Identify specific priorities in education and health that contribute to the enabling conditions for sustainable economic development in your country 3 Develop close collaboration with the Finance Minister to support your transformational (legacy) goals. 4 Achieve increased budget effectiveness and efficiency in resource utilization to support your transformational (legacy) goals as Minister. 5 Implement performance-based budgeting to align budget allocations with goals and results. 6 Identify fiscal space in your budget through budget reallocation, new revenues and/or cost savings to support your transformational (legacy) goals. 7 Identify and promote programmatic opportunities for collaboration between education and health in support of critical human development priorities. 8 Make operational interventions to increase health/education delivery system performance to support improved standards of service Itad in association with Big Win Philanthropy and Harvard Ministerial Leadership Programme Page 19