Introduction. Background

Transcription

1 Measuring Governance for Development Effectiveness Summary Note from the GPF Window 1 Workshop Implementing Effective Country Level Governance Programs September 13-16, 2010, Cape Town, South Africa Introduction In September 2010, the Governance Partnership Facility (GPF) 1 organized a workshop in Cape Town, South Africa to examine the progress of comprehensive country-level governance programs implemented with GPF support since the spring of Eighteen countries (also referred to as Window 1 countries 2 ) were selected to pilot broad-spectrum governance approaches in World Bank operations toward enhanced development effectiveness. In the lead up to the workshop, Vivek Srivastava, Senior Public Sector Specialist, Poverty Reduction and Economic Management (PREM) Network, was asked to produce an issue paper on measuring results from Governance and Anti- Corruption (GAC) initiatives. During a breakout session in Cape Town, participants discussed and debated recent experience, current issues, and possible next steps on measuring GAC results in operations. This summary note brings together the issue paper and discussion notes and seeks to share lessons learned with colleagues in the broader development community. Background The objective of this session was to initiate discussion and obtain feedback on measuring GAC results during the second phase ( ; Phase II, hereafter) of GAC implementation. The first phase of GAC implementation (Phase I, hereafter) focused primarily on changes in the way we use the GAC agenda to think about development effectiveness and strengthening World Bank processes and systems for analysis, planning, design and implementation (e.g. implementation arrangements, GAC in Country Assistance Strategies [CASs], country-level GAC work, political economy analysis); and on developing the appropriate tools, instruments and incentives for staff (e.g., GAC online portal, toolkits). While lessons learned from Phase I will be deepened and broadened in Phase II, the latter will be designed to prioritize results and development impact. 1 2 The Governance Partnership Facility (GPF) is a trust fund set up in the World Bank by the United Kingdom, the Netherlands, and Norway.

2 1. Change processes, systems, and mindsets 2. Analysis 3. Revise approach to upstream design (CASs) 4. Revise downstream design 5. Approval and implementation Development Effectiveness Organizational Effectiveness (Internal) Environment and Outcomes (External) Country 6. Quality of governance environment/equilibrium 7. Quality of institutions: (i) Core Public Sector Management (PSM) Systems (ii) Check and balance systems (Formal and informal) (iii) Accountability relationships 8. Quality of policy and implementation capacity 9. Service and infrastructure outcomes [Note: The processes are neither linear nor necessarily sequential] The Results Chain is long and complex and the full pass through to final outcomes (e.g., improved services, elimination of poverty) could take several years 3. Some sectoral experts estimate that the full pass through could take between 10 to 20 years. 3 It is difficult to measure success against the achievement of final outcomes (better services and infrastructure) as (i) these are subject to many influences that are beyond the scope of Bank support; (ii) there is a long chain that links PSM systems to final outcomes and even a four year period is insufficient for the effects of interventions to pass through; (iii) the Bank can only focus on a small sub-set of the PSM sub-systems in a country during any period; and (iv) due to the variety of factors that influence final outcomes, attribution will always be difficult.

3 It takes two to tango: The quality of final outcomes depends on both the Bank and Country Clients (Clients, hereafter) performance and, thus, all the elements need to be working together toward improved outcomes. It is therefore important to identify steps (outcomes) that are on the critical path to better services and poverty reduction and measure achievements against these. (The GAC Phase I results capture several of these on the upstream parts of the chain). GAC focus has been on: (i) Doing the right things (analysis understanding political economy, GAC constraints); (ii) Doing them right (improved project design, Operational Risk Assessment Framework [ORAF]); (iii) Improving the country governance environment (CGAC pilots; Public Sector Management [PSM] interventions) with a focus on enhancing accountability; (iv) Learning to work better within the existing environment making GAC-cognizant strategic choices (GAC in CASs; GAC in sectors) and project design (GAC in projects) for better outcomes. GAC Results going forward. Even if we do the right things and do them right, a key link in the results chain is the governance and institutional environment in the country. Recognizing that best fit rather than best practice is appropriate for evaluating these contextual factors, it is widely accepted that a set of well functioning institutions is required for attaining desired outcomes. These intermediate outcomes are final outcomes for the Project Status Reports (PSR) and the governance agenda and critical for development effectiveness. Institutional quality is more measurable and can be influenced more on some dimensions rather than others: The availability of Actionable Governance Indicators (AGIs) enable the measurement of quality for Core PSM Institutions. At the sector level the quality of the accountability relationships can be evaluated (e.g., using the World Development Report 2004 framework). The measurement of check and balance institutions is harder, although there are good examples of where these have been carried out to good effect in project design (see cases on the GAC Knowledge Platform; see also the vast literature on randomized evaluations). The broader governance environment is harder to measure along any acceptable metric and is also more difficult to influence. Question: What changes did the Bank s actions lead to? What should we be tracking/measuring and how? What changes can we reasonably expect within this time frame?

4 Slicing the results chain. Instead of working on the entire results chain, the focus needs to be on segments of the chain the relative importance of which would vary in different contexts and for different purposes. For example: (i) the link from changes in the Bank s processes and system to better design and implementation (1-3 4,5) ; (ii) the link from the Bank s changed approach to changes in intermediate/final outcomes (ref: annex in GAC Year 1 report) ( ); or (iii) from changes in project design to changes intermediate/final outcomes (4,5 6-9). A related set of questions revolves around what aspects of the results chain are relevant for GAC at the country level, GAC in sectors and GAC in projects? Question: From which GAC Action to which Result? Towards a Results Framework. It might be useful to think of objectives at two levels: Objective 1 (External Development effectiveness): Improved development outcomes intermediate outcomes (i.e., a basic set of wellfunctioning institutions) and final outcomes (e.g., growth, equity, poverty, better services/infrastructure) Objective 2 (Internal Organizational effectiveness): Internal (re) organization for GAC processes, systems, staffing leading to changed approaches being designed, approved and implemented. The former could include: (i) core PSM systems; (ii) check and balance systems; and (iii) service and infrastructure outcomes. (The Annex lays out a generic framework for discussion) Discussion notes from Cape Town As mentioned above, the primary objective of this session was to initiate discussion and obtain feedback on measuring GAC results for Phase II ( ) implementation. Given the session s high attendance, the organizers decided to provide a quick summary of the background paper and leave the floor open for a plenary discussion around the following questions:

5 Is the logic of the suggested GAC results chain convincing? What should we be tracking/measuring and how? To what extent are intermediate outcomes intended to be final outcomes? What changes can we reasonably expect within this time frame? What aspects of the results chain are relevant for GAC at the country level, GAC in sectors and GAC in projects? The questions generated a lively debate and open discussion among participants. The presence and active engagement of PREM Vice President Otaviano Canuto, Managing Director Sri Mulyani Indrawati, Country & Sector Managers, as well as GPF donors (United Kingdom Department for International Development [DfID], Norwegian Agency for International Development [NORAD]) was critical at various point of the discussion. Key points emerging from discussion There was general agreement among the participants that the suggested need to move beyond internal measures of success and the idea of starting to focus on external outcomes makes sense and should become a key concern for GAC Phase II. Consensus also emerged with regard to the difficulty of linking intermediate outcomes to final outcomes (i.e., service delivery). In this context, however, several challenges and measurement problems have been discussed: We need to be realistic about outcomes and consider that we start from different levels and different contexts. The baseline is different; hence a general model might fail to give adequate attention to such differences. We also need to consider the need for a simple/parsimonious reporting structure in light of the understanding that different results frameworks are required for different environments. It is hard to think about a one-size fits all solution; i.e., indicators should reflect specific contexts. A good starting point could be to think what success looks like in a given country context (build narratives of institutional change) and then think about what/how to aggregate the stories. Be aware that the Governance Agenda still needs to be fully mainstreamed within the Bank. Organizational change is ongoing but a new governance environment has not yet been fully achieved. This often includes limited awareness of governance constraints by staff and country teams. Therefore, while moving towards more external objectives, GAC Phase II should continue to keep track of progress against internal objectives. We need to be realistic about what we can achieve within 3 to 5 years and make sure we do not set ourselves up for failure. In difficult environments (low-income and fragile states) progress in governance is slow, as the enabling environment has yet to be created.

6 We need to be clear about what we can take credit for -- as results of our GAC interventions. Carefully designing the GAC Results Framework is crucial to this end. In particular, service delivery [box 9] is the ultimate goal, but additional factors (other than just 6-7-8; intermediate outcomes) account for the final outcome. This generates an attribution problem with regard to GAC interventions. Given the short time-frame to assess results (3-5 years), the link between GAC interventions and intermediate outcomes is more direct and feasible. Representatives of donors shared views that it is hard to claim success in final outcomes [box 9], meaning that measures of success pertaining to boxes would NOT be considered a failure. Challenges remain, however, as it is difficult to assess performance in intermediate outcomes (especially at institutional level) without linkage to final outcomes (functional approaches required). Senior management and corporate level actions are required to ensure that budget planning and priorities reflect the growing role of governance within country/sector portfolios. In this regard, the traditional cash-based one year budget cycle might not work properly, as a 2-3 year planning cycle within the Bank s budget process is required (consider, for example, the GPF trust fund). Another challenge pertains to the types of data used to assess progress in governance. Often, these data are perception based (household and individual surveys), and can therefore be misleading. The experience of Afghanistan is indicative: sector-level indicators are improving, yet perceptions regarding performance in governance have deteriorated. This suggests that it is difficult to link inputs to outcomes from the perspective of members of the general population, as it takes time for them to perceive changes in the environment. The learning aspects should be a measure of success in the GAC Phase II agenda. This also includes learning from mistakes. Feedback loops and other evaluation tools might be required in the results chain to take this issue into account. How do we measure good institutions? Shall we look at (i) best practices and measure country performance against that? Or shall we look at (ii) best fit, which takes into account country-context but makes measurable outcomes harder? Resources/Contacts (THESE NEED TO BE ADDED, NOT IN DOCUMENTS)

7 Annex Generic Results Framework (for discussion) Objective Intermediate objective Indicators Baseline Target Inputs Internal Staffing and objective: management Doing the right things and doing them better ( Organizational effectiveness ) External objective: Improved development outcomes ( Development effectiveness ) 1. Better analysis (including PE analysis) of GAC Training, toolkit, constraints knowledge sharing 2. Better identification and management of risks ORAF 3. GAC cognizant CASs Knowledge and guidance: GAC portal, toolkits 4. GAC cognizant project design Knowledge and guidance: GAC portal, toolkits 5. Bank s reputation/values (spell out) 6. Better evaluation and feedback Doing the right things and doing them better 7. Improvements in Public Sector Country Policy Investments

8 Management/Country systems and Institutional Assessment Global Integrity (?) Others (?) o Public Financial Management (PFM) Public Expenditure and Financial Accountability (PEFA) Procurement OECD DAC o Civil Service Actionable Governance Indicators for Human Resource Management (AGI HRM) o Judiciary? o Business climate Doing Business Survey (DBS) o Enhanced TAP (value/faith based?) Projects promote Transparency, Accountability, Participation (TAP) 8. Quality of PSM portfolio Quality Assurance Group (QAG) 9. Improvements in delivery of services/infrastructure Technical Assistance (TA) Dialogue Smarter project design Doing the right things and doing them better

9 o Health o Education o Water/sanitation o Power o Roads o Enhanced TAP (value/faith based?) Projects promote TAP 10. Quality of sector portfolio QAG Quality of Bank-wide portfolio (including QAG 2008 fiduciary dimensions) % of projects with xxxx (TBD) % of projects with yyyy (TBD) % of projects with zzzz (TBD) Better functioning institutions Investments TA Dialogue Smarter project design