Multilateral Organisations Performance Assessment Network ANNUAL REPORT May 2013 by the MOPAN Secretariat

Transcription

1 Multilateral Organisations Performance Assessment Network ANNUAL REPORT 2012 May 2013 by the MOPAN Secretariat

2 Foreword The MOPAN Annual Report 2012 outlines the development of the Multilateral Organisations Performance Assessment Network (MOPAN) and its methodology over the last ten years. It looks specifically at the conclusions of the 2012 assessments and reflects on the main lessons and potential developments for The MOPAN reports would not be possible without the support of its multilateral partners, which for 2012, included: African Development Bank (AfDB) Global Alliance for Vaccines and Immunization (GAVI) Joint United Nations Programme on HIV/AIDS (UNAIDS) United Nations Development Programme (UNDP) United Nations Children s Fund (UNICEF) World Bank (IBRD/IDA) We are also grateful to the partner country Governments, civil society partners, the staff from MOPAN members as well as the consultants, on whom we depend to undertake the assessments. The Network made substantial progress in The assessments of the multilateral organisations were concluded at the end of the year with preparations underway for the subsequent country dialogue in early 2013 in the partner countries. An expanded assessment framework was piloted with four of the multilateral organisations which moved beyond measuring organisational effectiveness to examine the results achieved by the multilaterals. It looks specifically at progress being made towards the organisations stated objectives as well as the relevance of their programming. Furthermore, to provide strengthened technical, procurement and administrative capacity to MOPAN successfully concluded negotiations on an arrangement whereby the MOPAN Secretariat will be housed for a three year trial period within the Development Cooperation Directorate of the Organisation for Economic Cooperation and Development (OECD). MOPAN is not being integrated into the OECD but will remain a network that conducts and publishes independent assessments of multilateral organisations which are approved by MOPAN members only. While retaining ownership of its programme, budget and multilateral assessments, MOPAN will continue to work with the OECD in areas of common mutual interest. The MOPAN Organisational Effectiveness Assessments of 2012 of the relevant multilateral organisations together with the respective Management Responses are available for download at:

3 CONTENTS MOPAN AND MULTILATERAL EFFECTIVENESS... 1 MOPAN S METHODOLOGICAL APPROACH 10 YEARS OF CONTINUOUS REFINEMENT... 3 THE MOPAN ASSESSMENT IN African Development Bank (AfDB)... 8 Global Alliance for Vaccines and Immunization (GAVI) Joint United Nations Programme on HIV/AIDS (UNAIDS) United Nations Children s Fund (UNICEF) United Nations Development Programme (UNDP) World Bank (IBRD/IDA) LEARNING FROM THE ASSESSMENT PROCESS IN Achievements Challenges OTHER PROCESSES AND MAJOR DEVELOPMENTS IN ANNEXES A. Roles and responsibilities of members within the network in 2012 B. Expenditure 2012 C. Previous and Current Assessments D. Overview of Bilateral Assessments by MOPAN members

4 MOPAN and Multilateral Effectiveness The proportion of Official Development Assistance (ODA) from OECD Development Assistance Committee (DAC) countries channelled through multilateral organisations increased from 37% in 2007 to 40% (USD 51 billion) in In the current context of fiscal austerity, multilateral ODA has avoided a decrease in volume despite a 2.7% overall drop in ODA in 2011 (Organisation for Economic Cooperation and Development, 2011 and 2012). In an increasingly globalised world an effective multilateral system is more relevant than ever, particularly for the millions of people who do not have access to the basic services envisaged in the Millennium Development Goals (MDGs). Demonstrating the effectiveness of the multilateral system is also an urgent priority for parliaments and taxpayers in MOPAN countries, many of whom are affected by the current financial crisis. The Multilateral Organisation Performance Assessment Network (MOPAN), established in 2002, is a network of 17 donor countries 1 which share a common interest in assessing the effectiveness of the major multilateral organisations they fund. As such MOPAN responds to increasing international demand to better understand how public funds are used for international aid purposes. Every year, MOPAN assesses a select number of multilateral organisations in several developing countries, gathering, over time, a mix of qualitative and quantitative data on each organisation s effectiveness. In 2012, six multilateral organisations were assessed by MOPAN; the African Development Bank (AfDB), the Global Alliance for Vaccines and Immunization (GAVI); the Joint United Nations Programme on HIV/AIDS (UNAIDS); the United Nations Development Programme (UNDP); the United Nations Children s Fund (UNICEF) and the World Bank (IBRD/IDA). The 2012 assessment was the first year in which a Global Fund was assessed. Consultations with staff of the organisations under review were introduced as an additional data source also saw the piloting, in four of the six organisations (AfDB; UNDP; UNICEF and the World Bank) assessed, of an expanded methodological framework which provides an assessment of organisational effectiveness as well as the results achieved by multilateral organisations. The assessment of results focuses on the degree to which progress is being made towards the organisation s stated objectives and analyses the relevance of its programming. Since 2009, the MOPAN Common Approach was derived from seven existing bilateral assessment tools. MOPAN members use the MOPAN assessments either as a major source of evidence, or as a replacement, for their own assessments. As such MOPAN demonstrates members commitments to the Paris Principles on Aid Effectiveness, specifically through the 1 Australia, Austria, Belgium, Canada, Denmark, Finland, France, Germany, Ireland, The Netherlands, Norway, Republic of Korea, Spain, Sweden, Switzerland, the United Kingdom and the United States of America. 1

5 harmonisation of existing approaches and the avoidance of duplication of activities. MOPAN s approach also supports members more recent commitments and principles articulated in the Global Partnership for Effective Development Cooperation in Although some Governments still require their own bilateral assessments, these are increasingly being based on using the available information from MOPAN and other sources to minimise the duplication of demands on the Multilateral Partners. Indeed, UNDP observed during its assessment that there is still scope for further harmonizing of bilateral assessments, considering that UNDP itself had undertaken twelve such assessments over An overview of bilateral assessments of multilateral assessments carried out by MOPAN member states between is provided below (Annex D). With the increasing attention on MOPAN s role and future development, together with the finalisation of negotiations in 2012 to establish a permanent, independent Secretariat for MOPAN within the Development Cooperation Division of the OECD, MOPAN members are now well-positioned to consider how to strategically develop the Network to ensure that the information produced remains relevant and that the findings from assessments are used as readily and widely as possible. In this respect, the evaluation of MOPAN, which will be carried out in 2013, will provide further insight as to how MOPAN can respond to the changing needs of donors in the future. MOPAN assessments are intended to: Generate relevant, credible and robust information on multilateral organisations effectiveness which MOPAN members can use to fulfil their responsibilities and obligations as bilateral donors and based on which MOPAN members, multilateral organisations and direct partners can discuss multilateral effectiveness in order to build better understanding and improve performance. Support dialogue between MOPAN members, multilateral organisations and their partners, with a specific focus on improving multilateral effectiveness over time, both at country level and at headquarters level. 2 Global Partnership for Effective Development Cooperation website: 2

6 MOPAN s Methodological Approach Ten Years of Continuous Refinement The first MOPAN survey was conducted in From 2004 to 2008, three multilateral organisations in each of eight to ten developing countries were surveyed. Until 2008 MOPAN s main focus was on conducting an annual perception-based survey. The assessments traditionally focused on whether multilateral organisations had in place four strategic dimensions of organisational effectiveness strategic, operational, relationship and knowledge management. In 2009 MOPAN introduced the Common Approach broadening and extending the methodology. The Common Approach continues to collect data through a survey but additionally includes a review of the documents published by the multilateral organisations under review. The Common Approach also extends the survey to direct partners with all respondents being required to demonstrate experience and expertise related to the multilateral organisations being assessed (i.e. expert sampling ). These developments bring about a better balance of factual and perception-based findings. In 2009, four multilateral organisations in nine developing countries were assessed. In 2010, four multilateral organisations were reviewed in ten developing countries. Five multilateral organisations in twelve developing countries were assessed in 2011 and MOPAN adapted the methodology to match the specific circumstances of humanitarian organisations (UNHCR). In 2012, MOPAN again adapted the methodology to examine the GAVI Alliance, the first time that a Global Fund type organisation has been assessed. The 2013 cycle will apply an expanded assessment framework tested in 2012 which goes beyond organisational effectiveness to examine the results achieved by the multilateral organisations. Before 2012, the Common Approach has not examined development effectiveness or the achievement of development results but rather focused on assessing whether the multilateral organisations had in place the necessary behaviours, systems and processes to help achieve those results. In 2012, MOPAN piloted an expanded methodological framework which provided an assessment of organisational effectiveness as well as the results 3 See Annex C for a complete list of MOPAN assessments since

7 achieved by multilateral organisations 4. The assessment of results focused on the degree to which progress is being made towards the organisation s stated objectives and analysed the relevance of its programming. It was piloted with AfDB, UNDP, UNICEF and the World Bank, all previously assessed in A key assumption in the new Common Approach assessment is that organisational effectiveness has an influence on an organisation s ability to achieve results. Feedback on the achievement of expected results can in turn provide insights for further improvements in organisational practices. By adding a component that analyses results, MOPAN members can use the existing dialogue process with the multilateral organisations to understand the way that organisational practices are facilitating or hindering the organisations results on the ground. A second assumption is that organisations provide or are moving towards evidencebased reporting on results. Thus, the assessment should also provide input for the discussions between donors and multilateral organisations on reporting. Figure 1: MOPAN Common Approach: Assessing Performance of Multilateral Organisations in The term results will be used in accordance with OECD/DAC s definition: Results are the output, outcome, or impact (intended or unintended, positive and negative) of a development intervention. 4

8 Table 1: Methodological Development of MOPAN CONCEPTUAL FOCUS THEMATIC FOCUS DATA COLLECTION Organisational effectiveness Results Development Humanitarian Assistance Environment Gobal Funds Survey Document Review Consultations with staff of the organisations MOPAN Survey X MOPAN members only 2009 Introduction of the Common Approach (CA) focusing systematically on four strategic dimensions of organisational effectiveness X MOPAN members & direct partners of the organisations 2010 CA X X 2011 CA X X X X 2012 CA X X X X X 2013 CA X X X X X

9 The MOPAN Assessment in 2012 In 2012 MOPAN assessed the following six multilateral organisations: African Development Bank (AfDB) Global Alliance for Vaccines and Immunization (GAVI) Joint United Nations Programme on HIV/AIDS (UNAIDS) United Nations Development Programme (UNDP) United Nations Children s Fund (UNICEF) World Bank (IBRD/IDA) The countries surveyed included Cambodia, Democratic Republic of Congo, Ghana, Honduras, Morocco, Nigeria, Niger, Philippines, and Zimbabwe. Table 2: Multilateral Organisations assessed in the 2012 Common Approach Countries IBRD/IDA* AfDB* UNDP* UNICEF* UNAIDS GAVI Cambodia x x x x Dem. Rep. Congo x x x x x x Ghana x x x x x x Honduras x x x x x Morocco x x x x x Nigeria x x x x x x Niger x x x x x x Philippines x x x x Zimbabwe x x x x x x * An expanded assessment framework will be piloted which will provide an assessment of organisational effectiveness as well as the results achieved. The assessment of results will focus on the degree to which progress is being made towards the organisation s stated objectives.

10 Data was collected through a survey of key stakeholders, which was tailored to the operations of the different types of multilateral organisations, and a document review process. The list of survey respondents working with the multilateral organisations included: MOPAN staff from Embassies in survey countries; from donor headquarters; and from donor missions at the headquarters of the multilateral organisations Government partners of the reviewed multilateral organisations in the survey countries Civil society partners of the reviewed multilateral organisations in the survey countries Respondents from international or multilateral peer organisations of the reviewed multilateral organisations in the survey countries. Building on preliminary work undertaken in 2010, MOPAN for the first time specifically adapted the methodology to assess a global fund (GAVI). Throughout the assessment process, the six multilateral organisations were given various opportunities to input to the MOPAN assessment process, particularly in adapting the methodology as required to better fit the mandates of the particular organisations, identifying appropriate survey respondents and providing feedback during the drafting stages of the overall Institutional Reports. The MOPAN Organisational Effectiveness Assessments of the relevant multilateral organisations together with the respective Management Responses are available for download at: A brief profile of each multilateral organisation, an overview of the respective survey response rates and a summary of key findings for each organisation assessed in 2012 is provided below. In addition, information is provided on the presentations of key findings contained in the Institutional Reports (usually taking place at the multilateral organisation s headquarters) to the key stakeholders involved in the assessment. The presentation of findings, prior to the Institutional Report s publication, allows a further opportunity for discussion and exchange on the methodology and outcomes of the assessment, to share ideas on how to ensure the findings are integrated into work practises and planning and to suggest any minor adjustments for the final Institutional Report. 7

11 African Development Bank (AfDB) The African Development Bank's objective is to spur sustainable economic development and social progress by mobilising and allocating resources and by providing policy advice and technical assistance. In 2012, MOPAN assessed the organisational effectiveness of the AfDB and evidence of its progress towards key development results based on information collected through a survey of key stakeholders, including donors and direct partners ( clients ) of AfDB in six countries 5, a document review of AfDB's publicly available corporate and country programming documents and consultations with relevant staff of the bank. The response rates 6 to the MOPAN survey of the AfDB are shown in the table below. Table 3: AfDB survey response rates 5 While there are variations in the response rates by category and location of respondents, the African Development Bank survey results reflect the views of 152 respondents. 6 MOPAN aims to achieve a 70 per cent response rate from donors at headquarters and a 50 per cent response rate among the population of respondents in each of the survey countries (i.e., donors in-country and clients). The number of respondents targeted in each category (i.e., the total population) and the actual response rates are presented in the figure above. 8

12 Key findings contained in the Institutional Report for AfDB were presented at AfDB headquarters in Tunis to its Senior Management team and the Board of Directors (in two separate presentations) in November The assessment, excluding the pilot component on results, was accepted as broadly accurate, including the finding that the Bank needs to improve communication of its outcomes. The Bank, furthermore, provided detailed and valuable comments on the Common Approach methodology overall and suggestions on how to improve the pilot results component in the future (outlined in next section below under challenges ). The official AfDB management response is available on the MOPAN website. The following conclusions of the assessment provide some major messages that can contribute to dialogue between MOPAN, AfDB and its partners: Organisational Effectiveness The AfDB remains strongly committed to a results-focused reform agenda and is improving its capacity to manage for development results. The Bank is recognised for aligning its country strategies with national development priorities but the extent to which the AfDB contributes to country level outcomes remains difficult to assess. The AfDB is seen to have sound policies and processes for financial accountability. AfDB s human resource management is the focus of an on-going reform process. The Bank has increased its country presence significantly in recent years, but its delegation of decision making authority to the country level remains a work in progress. The Bank has made progress on most of its Paris Declaration commitments but there is room for improvement. Key Development Results The Bank s progress towards organisation-wide outcomes is not clear. The extent to which the Bank s operations contribute to the achievement of country-level goals or relevant Millennium Development Goals is not always clearly evident in its reporting. For further information including detailed conclusions, ratings and analysis for key performance indicators, the full 2012 MOPAN Assessment of the Africa Development Bank is available at: 9

13 Global Alliance for Vaccines and Immunization (GAVI) The GAVI Alliance is a public-private partnership which brings together multilateral organisations, civil society organisations, public health institutes, the pharmaceutical industry, donor and implementing country governments, and non-profit organisations with a common focus on increasing access to immunisation in developing countries. The year 2012 marked the first occasion for MOPAN to involve a global fund in its Common Approach assessments. The assessment is based on information collected at the organisation s headquarters, from MOPAN members, and from stakeholders in fifteen GAVIsupported countries and a document review of publicly available corporate and country programming documents. 7 The response rates to the MOPAN survey of the GAVI Alliance are shown in the table below. Table 4: GAVI Survey Response Rates Key findings contained in the Report for GAVI were presented to MOPAN Members and GAVI s Executive team in October 2012 at GAVI headquarters Geneva. Overall, the positive 7 While there are variations in the response rates by category and location of respondents, the GAVI survey results reflect the views of 102 respondents. Given the small number of responses by MOPAN donors incountry, implementing partners, government, and civil society groups, caution should be used in interpreting survey results from these respondent groups. 10

14 results of the assessment were noted, and particularly the fact that there was little discrepancy between the document review and survey findings, indicating that GAVI does what it says (results focus). GAVI furthermore, provided detailed and valuable comments on the Common Approach methodology overall (outlined in next section below under challenges ). The official GAVI management response is available on the MOPAN website. The following conclusions of the assessment provide some major messages that can contribute to dialogue between MOPAN, GAVI and its partners: GAVI is well-positioned within the international aid architecture. As both a public-private partnership and an Alliance makes meeting the expectations of its constituent groups challenging GAVI s mandate and comparative advantage provide a good foundation for its focus on results, but could strengthen this focus. GAVI is recognised for ensuring transparency and accountability in many its operational practices GAVI is undertaking reforms to strengthen its knowledge management practices. GAVI successfully manages a complex set of relationships at the country level, but adapting to the needs and expectations of partners in so many countries is a slow, delicate, and costly balancing act. Progress on some specific Paris Declaration indicators is not clear. For further information including detailed conclusions, ratings and analysis for key performance indicators, the full 2012 MOPAN Assessment of GAVI is available at: 11

15 Joint United Nations Programme on HIV/AIDS (UNAIDS) UNAIDS is an innovative partnership that aims to achieve universal access to HIV treatment, prevention, care and support. UNAIDS is one of the main advocates for accelerated, comprehensive and coordinated global action on the HIV/AIDS epidemic. In 2012, MOPAN assessed the organisational effectiveness of UNAIDS. The assessment is based on information collected through a survey of key stakeholders, including donors and direct partners of UNAIDS in nine countries and at headquarters and a document review of publically available corporate and country programming documents at the organisation s headquarters and in nine countries. The response rates to the MOPAN survey of UNAIDs are shown in the table below. Table 5: UNAIDS Survey Response Rates Key findings contained in the Report for UNAIDS were presented to MOPAN members and UNAIDS staff members at UNAIDS headquarters, Geneva in November Overall the findings were well received by UNAIDS and they saw the assessment as a valuable opportunity to engage in open discussion. UNAIDS, furthermore, provided detailed and valuable comments on the Common Approach methodology overall (outlined in next section below under challenges ). The official UNAIDS management response is available on the MOPAN website. 12

16 The following conclusions of the assessment provide some major messages that can contribute to dialogue between MOPAN, UNAIDS and its partners: UNAIDS commitment to organisational development has brought positive changes. Substantial progress made in becoming a more performance-oriented and accountable organisation but its ability to measure its own performance needs improvement The UNAIDS Secretariat s technical expertise, evidence-based advocacy, and influence in policy setting is valued UNAIDS highly consultative approach is crucial to the achievement of its mandate and its Getting to Zero strategy. UNAIDS effectiveness in building partnerships is highly valued and recognised by stakeholders as one of its strengths. UNAIDS unique organisational structure presents both opportunities and challenges. For further information including detailed conclusions, ratings and analysis for key performance indicators, the full 2012 MOPAN Assessment of UNAIDS is available at: 13

17 United Nations Development Programme (UNDP) UNDP s mission is to assist countries to achieve sustainable human development. Its work is demand-driven and focused on country priorities. UNDP have four focus areas based on the highest levels of demand: achieving the MDGs and reducing human poverty, fostering democratic governance, supporting crisis prevention and recovery, and managing energy and the environment for sustainable development. In 2012 MOPAN assessed the organizational effectiveness of UNDP and evidence of its progress towards key development results based on information collected through a survey of key stakeholders, including donors and direct partners of UNDP in nine countries, interviews with UNDP staff and a document review of publicly available corporate documents and country programming documents. The response rates to the MOPAN survey of UNDP are shown in the table below. Table 6: UNDP Survey Response Rates Key findings contained in the Institutional Report for UNDP were presented to UNDP management, representatives of MOPAN Member States and one country of the Common Approach assessment 2012, Zimbabwe, in New York (hosted by the Permanent Representation of Norway), in November UNDP emphasized that they were very pleased by the many strong results in the assessment and the positive development compared to 2009 and observed that many recommendations were already integrated in the 14

18 preparations of the next strategic plan. UNDP, furthermore, provided detailed and valuable comments on the Common Approach methodology overall and suggestions on how to improve the pilot results component in the future (outlined in next section below under challenges ). The official UNDP management response is available on the MOPAN website. The following conclusions of the assessment provide some major messages that can contribute to dialogue between MOPAN, UNDP and its partners: Organisational Effectiveness UNDP is recognised for its coordination role in the United Nations system. Direct partners continued to identify UNDP s bureaucracy and administrative inefficiencies as a key area for improvement. While considered strong in mainstreaming gender, UNDP s integration of other cross-cutting priorities received mixed ratings. UNDP has sound financial accountability systems. UNDP s Evaluation Office works independently and has strong mechanisms to ensure the quality of evaluations. UNDP s commitment to management for results has not yet translated into perceived or documented changes in the practices assessed by MOPAN. UNDP faces the challenge of developing robust results Key Development Results Surveyed stakeholders generally hold positive views of UNDP s achievement of results. UNDP s reporting on results achieved needs improvement. For further information including detailed conclusions, ratings and analysis for key performance indicators, the full 2012 MOPAN Assessment of UNDP is available at: 15

19 United Nations Children s Fund (UNICEF) UNICEF works for the protection of children's rights, to help meet their basic needs and to expand their opportunities to reach their full potential. In 2012 MOPAN assessed the organizational effectiveness of UNICEF and evidence of its progress towards key development results based on information collected through a survey of key stakeholders, including donors and direct partners of UNICEF in nine countries and a document review of publically available corporate documents and country programming documents. The response rates to the MOPAN survey of the UNICEF are shown in the table below. Table 7: UNICEF Survey Response Rates Key findings contained in the Institutional Report for UNICEF were presented to UNICEF management and representatives of MOPAN Members States in November 2012 at UNICEF headquarters in New York. Since UNICEF s previous assessment in 2009, the component whereby staff input at country level into the Institutional Report had been omitted from the assessment methodology. UNICEF reported that the MOPAN report will have country specific follow up and will feed into UNICEF s Mid Term Review (MTR). UNICEF, furthermore, provided detailed and valuable comments on the Common Approach methodology overall and suggestions on how to improve the pilot results component in the future (outlined in next section below under challenges ). The official UNICEF management response is available on the MOPAN website. 16

20 The following conclusions of the assessment provide some major messages that can contribute to dialogue between MOPAN, UNICEF and its partners: Organisational Effectiveness UNICEF s organisational development processes are likely to improve its effectiveness and efficiency. UNICEF remains strongly committed to a results-focused reform agenda and is improving its capacity to manage for development results. UNICEF s country programming processes have improved but planning at the project level received low ratings. UNICEF s has proven particularly effective in mainstreaming gender equality and emergency responses/humanitarian action. UNICEF is valued for its field presence and engagement with governmental and non-governmental partners. UNICEF is highly valued by its direct partners. UNICEF allocates resources in a transparent manner, but has not yet fully implemented results-based budgeting. In contexts where it has significant humanitarian programming, UNICEF is fulfilling its responsibilities as a Cluster Lead and is recognised for its respect for humanitarian principles. UNICEF has strengthened its evaluation function but could improve further. 17

21 Key Development Results Although UNICEF has provided strong direction for resultsbased management in recent years, limitations in its frameworks and systems to report on organisation-wide results make it challenging to identify the difference being made in the lives of children and women. UNICEF reports do not yet adequately document UNICEF s contributions to its Country Programme results and to partner country efforts to achieve the MDGs For further information including detailed conclusions, ratings and analysis for key performance indicators, the full 2012 MOPAN Assessment of UNICEF is available at: 18

22 World Bank (IBRD/IDA) To assist developing countries worldwide in reducing poverty and achieving sustainable growth, the World Bank provides loans to middle income countries, and credits and grants to low-income countries, and has recently committed to substantially increase its support to fragile and conflict-affected states. The allocated resources are invested by governments in a wide array of development sectors. The World Bank also provides analytical and advisory activities and technical assistance to client countries. In 2012 MOPAN assessed the organizational effectiveness of the World Bank and evidence of its progress towards key development results based on information collected through a survey of key stakeholders, including donors and direct partners of the World Bank in eight countries and a document review of publically available corporate and country programming documents. The response rates to the MOPAN survey of the World Bank are shown in the table below. Table 8: World Bank Survey Response Rates Key findings contained in the Institutional Report for the Bank were presented to the World Bank s management and representatives of MOPAN Members States in December 2012 at World Bank headquarters in Washington. The Bank was pleased with the results overall across the Key Performance Indicators (KPIs). 19

23 The World Bank, furthermore, provided detailed and valuable comments on the Common Approach methodology overall and suggestions on how to improve the pilot results component in the future (outlined in next section below under challenges ). The official World Bank management response is available on the MOPAN website. The following conclusions of the assessment provide some major messages that can contribute to dialogue between MOPAN, the World Bank and its partners: Organisational Effectiveness The World Bank has demonstrated strategic leadership in managing for and reporting on development results. Some shortcomings were noted in the frameworks and systems used to measure and report on organisation-wide results, such as linking country-level outputs and outcomes with the Bank s organisation-wide development outcomes and a lack of a theory of change of how the organisation conceives its contributions to development change. The World Bank is committed to transparency as seen by the launch of its Open Data Initiative in 2010 and Open Access Policy in Resource Allocation is done transparently based on country performance, but the Bank s administrative budget is not yet allocated to expected results The World Bank is recognised for its strong financial and fiduciary practices. Human resources management is considered strong overall but it recognises it needs to improve its staff compensation system, particularly in terms of performance-based incentives for staff. Surveyed stakeholders rated the Bank adequate for its use of country systems and adjustment of procedures but expressed concerns about the Bank s bureaucratic procedures. The Bank is considered strong in its evaluation of results and dissemination of knowledge. 20

24 Key Development Results The Bank is making progress towards organisation-wide outcomes and in reporting on these. The Bank is seen to be contributing to country-level objectives and national Millennium Development Goals. For further information including detailed conclusions, ratings and analysis for key performance indicators, the full 2012 MOPAN Assessment of UNICEF is available at: 21

25 Learning from the Assessment Process 2012 Achievements Assessment 2012 There has been generally constructive dialogue with multilateral organisations over the course of the 2012 assessment, including discussion of the methodology before the assessment began, in identifying the necessary methodological adaptations required to assess particular organisations and through feedback received during the drafting of reports. Many of the multilateral partners expressed appreciation for the consultative manner in which MOPAN engaged with them to ensure that the survey indicators were appropriately adapted to their specific circumstances. MOPAN methodology Valuable feedback has been obtained from the four multilateral organisations that had the pilot results component as part of their assessment. This new component has provided an additional dimension to the Common Approach and will be integrated as part of the methodology in all assessments in For the first time the Common Approach methodology was adapted to assess a Global Fund, and GAVI was included in the 2012 assessment process. First attempts were made to increase efficiencies according to the decision to aim for a more intensive, well-managed, high-quality process in a smaller country sample for four organisations in Developing MOPAN as a network In 2012, an internal M&E framework was developed for MOPAN to monitor the outcomes of MOPAN s activities, utilisation of the assessment findings and mapping the development of bilateral assessments of multilateral organisations. It is intended that the new permanent Secretariat, to begin work in early 2013, will be given the task of collecting and reporting information about MOPAN s performance on basis of the performance monitoring framework. Over the course of 2012, preparation of a comprehensive evaluation of MOPAN in 2013 was undertaken. Terms of Reference were developed by an evaluation steering committee with input from MOPAN members and the selection of the consultants will happen early in the New Year. It is expected that the final report of the evaluation will be available in early autumn. Over 2012, members increasingly recognised the critical stage of the MOPAN network in terms of its future methodological development and ensuring it positions itself as a leading 22

26 source for multilateral assessments. To this end, members agreed to establish a Strategic Working Group (SWG) which shall, by liaising with key stakeholders such as the MOPAN assessment consultants, the future evaluators of the MOPAN Common Approach and the OECD EvalNet, develop different ideas and options to explore which direction MOPAN could go in, both with view to its products, i.e. data and the reports, as well as with view to its actions and communications to spread the knowledge of its work (dialogue in partner countries, with MOs etc.) and its position in the broader multilateral effectiveness landscape. The latter shall also include ideas about the organizational set up of the network and secretariat. Challenges MOPAN methodology Further attention on how to include indicators related to the Survey on Monitoring the Paris Declaration will be required to ensure that all MOPAN stakeholders are clear on the nature of the indicator and that sufficient data is actually available to provide meaningful findings. There were a number of commonalities in the feedback and suggestions received from the multilateral organisations assessed in 2012 on the Common Approach methodology and its pilot results component which will need careful consideration. In some cases, multilateral organisations believed that their practices in the following areas were stronger than the ratings of the assessment indicated: organisation/corporate strategies being resultsbased; budget allocations linked to expected development results and in the presentation of performance information on its effectiveness. Following a review of such feedback, the MOPAN has made a number of changes to streamline the assessment in 2013, such as merging selected Key Performance Indicators (B and C), revision of the indicators lists and enhancing clarity on how the Common Approach has changed over time in the methodology papers shared with organisations prior to their assessment. Developing MOPAN as a network The peer review of the UNHCR assessment in 2011, completed in early 2012, established a baseline to measure MOPAN s own performance in assessing humanitarian organisations and enhances MOPAN s credibility by regularly evaluating its own work. However, the review also showed, amongst other findings, that there is a tension between doing a more detailed assessment in light of resource/financing constraints, the need for further consideration of what organizational effectiveness means in the humanitarian context and also highlighted the need for a more analytical focus within the reports. 23

27 The importance of engaging donors and partners at the country level was noted as a leasson learned during the 2012 process. It was observed that the hosting arrangement with OECD should allow for better streamlining of processes and follow up especially at country level. 24

28 Other Processes and Major Developments in 2012 Country level dialogue 2011 A dialogue process for the 2011 assessments took place in six out of eight countries in the first half of These countries were Bolivia, Brazil, Bangladesh, Ecuador, Nepal and Peru. It had been agreed previously that no country dialogue would take place in Jordan, Lebanon, Palestinian Territories or Syria. Overall, members agreed that the country dialogue process needs further improvement to ensure engagement, especially by MOPAN members at country level, and that country dialogue quality varies. More incentives for country leads are necessary; the possibility of collecting data at country level to feed into institutional reports and the necessity of prioritising multilateral effectiveness in bilateral programmes should be explored. However, the dialogue process in Ecuador can be noted as best practice where areas for action and follow-up were identified with each MO and for each KPI. Suggestions on how to enhance the dialogue process will be a feature of the evaluation of MOPAN, expected to report in autumn New members At the MOPAN meeting in Berlin September 2012, the USA became a full member of MOPAN, ending its observer status. Furthermore, a number of other countries have made informal expressions of interest in the work of MOPAN during the year, including Italy, Japan, Luxembourg and New Zealand but no firm commitment had been made. Further information to those interested in becoming observers/members was circulated by the Chair and Secretariat. Permanent Secretariat With an increase in membership and assessments; the development of a more complex methodology and the need to consolidate institutional memory, the MOPAN members decided that a permanent secretariat was required as soon as possible saw the finalisation of negotiations on the Memorandum of Understanding (MoU) with the OECD which establishes a trial hosting arrangement for a permanent Secretariat for MOPAN in the Development Cooperation Division (DCD) for a three year period. It is envisaged that the new permanent Secretariat will be set up and begin its work early in MOPAN will remain as an independent network at the OECD. The annual work programme for the secretariat and the budget will continue to be set by the MOPAN members, the Steering Committee includes only MOPAN members and MOPAN will retain full ownership of any work led by or financed by MOPAN. However, it is envisaged that the hosting arrangement will bring synergies between MOPAN and the Development Assistance Committee (DAC), where, for example, there has already been some cooperation with the Evalnet task team on multilateral effectiveness. 25

29 ANNEXES A. Roles and responsibilities of members within the network in 2012 Secretariat MOPAN is managed by a Secretariat which until now has been hosted by MOPAN members on a rotational basis. 8 In 2012, the MOPAN Secretariat was led jointly by Ireland and Germany. The Secretariat is the focal point for members and all players involved in the assessment process as well as other interested parties for all MOPAN activities. It is responsible for implementing, coordinating and delivering the MOPAN Common Approach. In 2012, it has been supported in doing so by the following working groups: Technical Working Group, Communications Working Group, Contractual Working Group and a Task Team dealing with the future development of MOPAN. MOPAN Working Groups in 2012 Communications Working Group Chair: Members: The Netherlands, Barbara Rietbroek and Blanche Voorneman Australia, France, Spain, Finland, Republic of Korea, the Netherlands, the United Kingdom and the Secretariat Contracts Working Group Chair: Members: The United Kingdom, Elaine Herd and Louise McLeish Norway, Denmark, the Netherlands, the United Kingdom and the Secretariat Technical Working Group Chair: Members: Norway, Thomas Dam Belgium, Denmark, Germany, Norway, Spain, Switzerland, the United Kingdom and the Secretariat Strategic Working Group Chair: Germany, Julia Lehman 8 From 2013 on the MOPAN Secretariat will be hosted by the Development Cooperation Directorate of the Organisation for Economic Cooperation and Development (OECD) for a three year trial period. 26

30 Members: Australia, Belgium, Denmark, Canada, Denmark, Switzerland, Finland, the Netherlands and Germany Coordination of the assessment at institutional and at country level MOPAN Institutional Leads and Co-Leads are the key interface between MOPAN and the multilateral organisations under review and are responsible for all communication with, and liaison between, MOPAN and the multilateral organisations. Multilateral organisation MOPAN Institutional Leads World Bank (IBRD/IDA) Australia and The Netherlands African Development Bank (AfDB) United Nations Development Programme (UNDP) United Nations Children s Fund (UNICEF) Joint United Nations Programme on HIV/AIDS (UNAIDS) Global Alliance for Vaccines and Immunization (GAVI) Canada, Switzerland and the United Kingdom Norway, Switzerland and Sweden Austria and Spain Finland and France France, Spain and Sweden MOPAN Country Leads and Co-Leads represent MOPAN in each survey country and are responsible for coordinating the assessment in-country. Survey countries MOPAN Country Leads Cambodia Democratic Republic of Congo Ghana Honduras Morocco Nigeria Niger Philippines Germany and Spain France and the Republic of Korea Canada and Denmark Switzerland France and Belgium United Kingdom and Finland Switzerland and France Australia and Spain 27

31 Zimbabwe Sweden and France B. Expenditure 2012 Managed by DFID on behalf of MOPAN Details Income '000 Opening Balance as at 01/01/ Actual Income received for Accrued Income received in Total Income Details Expenditure '000 Implementing Consultancy Sub Total 805 Other costs 50 Total 2012 expenditure Accrued Expenditure 154 Unallocated Carry forward from Total Expenditure Balance (remaining income after all expenditure deducted) 21 28

32 A. C. Previous and current assessments * WB x x x x x AfDB x x x x x ADB x x x x IDB x x WHO x x x x UNDP x x x x FAO x x UNFPA x x x UNAIDS x x UNICEF x x x ILO EC IFAD x x UNEP x UNHCR** x UNRWA** x GAVI*** x WFP x x x * 2003 pilot of the annual MOPAN Survey (after a methodological review in 2008 which brought significant changes from 2009 on the MOPAN Common Approach ) ** 2011 pilot to broaden the assessment approach to include humanitarian organisations *** 2012 pilot to broaden the assessment approach to include global funds African Development Bank (AfDB) Asian Development Bank (ADB) European Commission (EC) Global Alliance for Vaccines and Immunization (GAVI) Inter-American Development Bank (IDB) International Fund for Agricultural Development (IFAD) International Labour Organization (ILO) United Nations Children s Fund (UNICEF) United Nations Development Programme (UNDP) United Nations Environment Programme (UNEP) United Nations Food and Agriculture Organization (FAO) United Nations High Commissioner for Refugees (UNHCR) Nations Joint Programme on HIV/AIDS (UNAIDS) United Nations Population Fund (UNFPA) United Nations Relief and Works Agency for Palestine Refugees in the Near East (UNRWA) United Nations World Food Programme (WFP) World Bank (WB) World Health Organization (WHO) 29

33 D. Overview of bilateral assessments by MOPAN members 2012 BILATERAL ASSESSMENTS Multilateral Organisations The Australian Multilateral Assessment (AMA) 42 organisations were assessed: ADB, AF, AfDB, CGIAR, CIFs, ComSec, FAO, GAVI, GCDT, GEF, GFATM, GFDRR, GPE, ICRC, IADB, IFAD, IFC, ILO, IMF, IOM, LDCF, MLF, OHCHR, UNPBF, PIDG, UNAIDS, UNCDF, UNDP, UNEP, UNESCO, UNFPA, UN-Habitat, UNHCR, UNICEF, UNISDR, UNMAS, UNOCHA, UNODC, UNRWA, WFP, WHO, WB UNESCO, OHCHR, UNODC, GEF Swedish assessments of multilateral organisations Denmark The Netherlands evaluation of the World Bank The Netherlands evaluation of ILO 17 multilateral organizations with a development and/or humanitarian mandate: UNDP, UNICEF, UNFPA, IFAD, WFP, UNHCR, GFATM, UNAIDS, WHO, ICRC, OHCHR, UNWRA, WB/IDA, AsDB/AsDF, AfDB/AfDF, OCHA, UNEP World Bank ILO (Partnership Programme 2010/2013). Objective The Australian Multilateral Assessment (AMA) delivered on a commitment in An Effective Aid Program for Australia: Making a real difference - delivering real results to assess the effectiveness of Australia's key multilateral partners. The assessment is supposed to be a guiding instrument for funding decisions, it should also be a starting point for dialogue to help strengthen the relevance and effectiveness of the organisation in question. To strengthen the strategic orientation and coherence of Denmark's multilateral cooperation Draw lessons from the evaluation of the Netherlands World Bank policies and funding Focus on accountability. Limited review to guide decision making on Dutch funding of ILO s Regular Budget Supplementary Account (RBSA) and Better Work Programme (BWG) in the second biennium (2012/2013) of the Partnership. Main focus Key criteria The main focus of the assessment is to judge the organisations relevance and effectiveness (internal and external). The assessment it is also supposed to give an The AMA focused on two key areas of overview of trends within the organisation multilateral effectiveness: results & relevance and to suggest measures based on the and organisational behaviour assessment.(new for 2012) The AMA included seven components for assessment: (1) delivering results on poverty and sustainable development in line with mandate (2) alignment with Australia's aid priorities and national interests (3) contribution to wider multilateral development system (4) strategic management and performance (5) cost and value consciousness (6) partnership behaviour (7) transparency and accountability The key criteria s are relevance and effectiveness (internal and external). Relevance refers to the mandate and the area of activity of the organisation and the alignment of this to objectives of Swedish development cooperation. Effectiveness refers to how the organisation is rigged to deliver good results, and if it actually delivers good results. Examination of trends in int'l policy and financial environment and assessment of key multilateral partners' performance and alignment to Danish development policy priorities. In 2012, special thematic focus on changing growth patterns and shifts in global decision-making capacity, trends in multilateral funding, conflict-affected and fragile states, and Rio+20 with focus on green economy Evaluation of the Netherlands policies and funding vis-àvis the World Bank. A review of the policy intentions of the Netherlands, the way these intentions were implemented and the effects that were achieved in working with the World Bank over the past decade. Organizational effectiveness (based on MOPAN findings) and strategic alignment with Danish development policy priorities (based on new set of Effectiveness in the priority areas of the Netherlands indicators, see below) World Bank policies. To assess whether funds are being spent in keeping with the goals of the RBSA and BWG and the achievements. The review has also taken into account the overall performance of ILO, in particular in the context of RBSA and BWG. As for ILO s performance, the review paid attention to, amongst others: accountability, transparancy, efficiency, performance management (outcome based work plans), ILO in the UN system (including DaO and UNDAF) and evaluations and their follow-up within the organisation. Effectiveness of RBSA, to a limited extent linked to outcomes that are relevant for the Netherlands assistance: allocation of RBSA support to Decent Work Country Programmes (DWCP). Is management based on RBM principles? Progress and effectiveness of the BWG are assessed on the basis of an independent evaluation of a third party. Use of scores Comparison of MOs Use of MOPAN assessments Rating scale - Very strong, Strong, Satisfactory, Weak, Not Applicable No comparison of MOs The most recent MOPAN reports were used as evidence in the assessments Yes, Effectiveness: Very low, low, acceptable, high, very high. Relevance: Very relevant, relevant, not relevant. Sweden does not rank the organisations but it is possible to compare the organisations by comparing the scores. Use of MOPAN assessments: No, as MOPAN has not yet made assessments of these organisations Brief qualitative summing-up of individual MOs' strengths and challenges Assessment of organizational effectiveness based partly on MOPAN findings (supplemented by information from UK's MAR) Yes Additional workload for MOs Yes (minor) MOs were provided with copies of draft assessment and asked for factual corrections prior to publication No No. MOs invited to comment on and provide follow-up to final report, but not involved in preparation/production phases (for instance, no interviews with MO staff) Yes ILO s Department for Partnerships and Development Cooperation assisted in collecting data and scheduled interviews for the external reviewers. Comments Link to assessments The findings of the AMA will be updated on an annual basis through an Annual Multilateral Performance Scorecard. The first scorecards will be published in October a-submissions.aspx Assessments of 2012 are not available yet Annual updates planned installment to focus on alignment with Danish policy priorities as expressed in Denmark's new development policy "The Right to a Better Life" (June, 2012); that is, 1) human rights and democracy, 2) green growth, 3) Special attention is given to IDA and World Bank Trust social progress, and 4) stability and protection Funds. The report will be made public beginning of Multilateral%20Analysis%202012%20- %20English%20Version.ashx