Indicators of Change: Measuring Impact in Voice and Accountability Work

Transcription

1 Indicators of Change: Measuring Impact in Voice and Accountability Work By Jeremy Holland and Allyson Thirkell, Social Development Direct 1 Citizen s voice and accountability (V&A) work has emerged as a priority in the international development agenda and is part of the broader debate about the importance of governance in improving poor people s prospects. Yet the evidence of its impact is fairly limited. Measuring change and demonstrating results in all areas of development policy is critical for donors to understand the impact of their work, ensuring that approaches are effective and provide value for money. Measuring Change and Results in Voice and Accountability Work 2 was commissioned by the UK s Department for International Development s (DFID) Evaluation Department, in response to a demand for practical tools to measure more effectively the results of V&A work throughout its programmes. It examines existing ways of measuring V&A, and proposes accessible and practical indicators to evaluate V&A. It is currently profiled and the full paper is available on DFID s website as a DFID Evaluation Working Paper 3. Support is growing for initiatives that raise people s voice in developing countries and improve awareness of their rights so that they are able to talk with governments, and hold officials to account. Governments need to listen to the communities that they serve in order to understand issues, develop appropriate policies and deliver effective services. Knowing whether these initiatives are working is problematic as voice and accountability work (V&A) is complex. In addition there is considerable investment needed in raising people s awareness of their rights or current policies. In addition communities and organisations need to organise, form networks and understand local and central governance structures and how to approach them. The effects of V&A work are often intangible and demonstrating changes in policy or the behaviour of policy makers can take a long time. Furthermore any change depends very much on where you are in the world posing significant challenges to developing common indicators and evaluating this area of work. Whilst measuring change may be complicated, donors are determined to demonstrate the impact that they are having in all areas of their work. Based on DFID s Capability, Accountability and Responsiveness (CAR) framework 4, the paper presents a practical approach to identifying appropriate indicators and mapping V&A programmes. It separates out horizontal and vertical accountability and responsiveness components. Vertical accountability is the direct engagement that individuals and groups have with governments and other duty bearers, while the extent to which governments and duty bearers respond to 1 Social Development Direct is a leading international provider of social development consultancy and research services. SDDirect is based in London and was established as an organisation in 1999 with a network of leading independent consultants and researchers committed to working with government and non-governmental organisations to improve the quality and impact of development assistance. See: for more information. 2 Source: Holland, J and Thirkell, A. with Emmanuel Trepanier and Lucy Earle. Social Development Direct (December 2009): Measuring Change and Results in Voice and Accountability Work, DFID; Working Paper DFID, Making Governance Work for the Poor, White Paper 3; London, DFID. 1

2 Accountability Capability their voices marks their degree of responsiveness. Horizontal accountability involves state institutions engaging in mutual scrutiny to be held accountable, for example, a public audit function which reviews probity in public spending. Equally the changes they adopt as a result of these measures signify their responsiveness to such accountability mechanisms. The working paper reviewed a range of DFID V&A interventions as case studies and mapped them and their V&A output indicators on to the CAR framework. A fuller list of indicators is available in the Working Paper. Mapping the case studies on to the CAR framework 5 Type of change Intervention V&A Output Indicators Case study Formal institutional change (formal rules) Informal institutional change (cultural norms) vertical accountability National legislation establishing and protecting social, economic and political freedoms/rights Justice law reform democratic accountability institutions Increasing representation of excluded groups in positions of authority Community sensitisation and mobilisation Public awareness campaigns direct collective or individual civic engagement with the state civil society and media oversight/watchdog role Level and quality of institutional change Level and quality of behavioural change Level and quality of participation/ membership Level and quality of networks Level and quality of interaction Level of citizen monitoring/ oversight Level and quality of monitoring evidence generated (for example, for media quality of published investigative journalism pieces) DFID ETRM DFID Media DRC DFID CSUP DFID ETRM DFID CSUP DFID Media DRC WB ME initiative DFID RGCF DFID PPAs LAm WB PNPM DFID PACS DFID SAVI vertical accountability participation in electoral processes Level and quality of political participation DFID ETRM DFID PPAs Latin America horizontal Supporting use of formal (legislative, judicial, executive) redress/oversight mechanisms Incidence and effectiveness of sanctions/ adjudication Quality of behaviour (for example, rent seeking) DFID FMRP 5 Source: Holland, J and Thirkell, A. Social Development Direct (December 2009): Measuring Change and Results in Voice and Accountability Work, DFID; Working Paper 34: p.8. 2

3 Responsiveness accountability vertical responsiveness Building awareness and capacity to respond to citizens Removing barriers and improving direct access to decision makers Supporting systematic and transparent budget and policy processes Perceptions of quality of responsiveness (for example, corruption, trust, fairness) Predictable and transparent budget process # and frequency of invited spaces/forums Frequency of access to MPs UNIFEM GDG DFID PACS horizontal responsiveness Changing incentives Supporting performance measures and review Supporting systematic and transparent budget and policy processes Incidence and quality of information provision and responses DFID PRBS DFID FMRP Source: Authors analysis The Working Paper goes on to present a simple tool or framework - with which it is possible to design initiatives using the anticipated programme outputs along these lines, identifying any gaps and designing programmes accordingly. The tool allows programme outputs and their corresponding indicators to be mapped onto a capability, accountability and responsiveness axis (see figure below). 3

4 A Graphic representation of V&A in the CAR Framework 6 The paper graphically shows how outputs and indicators can be mapped onto the framework either during the design stage of the programme or during its review or evaluation. For example, DFID s Bangladesh Financial Management Reform Programmme (FMRP) is mapped onto the framework as an example, revealing that the programme deals almost exclusively with horizontal accountability and responsiveness relations within government itself. An additional set of outputs may be suggested strengthening vertical responsiveness and accountability (illustrated in purple) which may be taken up in the next round of the project or in another project, perhaps, in conjunction with another donor. 6 Source: Holland, J and Thirkell, A. Social Development Direct (December 2009): Measuring Change and Results in Voice and Accountability Work, DFID; Working Paper 34: p.26. 4

5 Mapping project outputs on to the CAR framework: FMRP Bangladesh 7 This framework enables development practitioners to understand how the separate components of their programme fit together to bring about change in voice and accountability. For example, developing accountability mechanisms without measuring the responsiveness of governments or duty bearers in reacting to such measures, may leave the programme unbalanced and unable to track what changes are actually being made as a result of better accountability systems or structures. Mapping outputs using such a tool with accompanying indicators tests the underlying assumptions about change and allows any gaps to be identified early. Voice and accountability is an extremely broad development area and no one programme can cover the extent and breadth of change potentially needed in governance relationships to ensure better accountability. However this tool provides a good opportunity for joined-up development as donors can collaborate on V&A work, overlaying their individual programmes, understanding their approaches jointly and the potential synergies between them. It offers an opportunity for indicators to be identified and measured from the bottom up, facilitating local action and encouraging donors to ask whose indicators?. It also offers possible opportunities for multi-donor evaluation and aggregation of results by agreeing and using common indicators across a range of V&A programmes. 7 The 5 yellow output areas mapped on to the framework are actually at purpose level, which is consistent with a large programme. 5

6 DFID s Working Paper 34 sets out a list of detailed indicators that can be used to measure change in voice and accountability work. It also elaborates on accompanying data collection instruments and provides some guidance on methodology for collecting evidence of change. Evaluating V&A initiatives is notoriously difficult, as the range of approaches and methods is highly diverse and there is often a lack of understanding as to what is being achieved across the board. Using a set of commonly understood indicators within a simple framework, across a range of activities and actors means that consistent data across programmes can be collected and the overall impact of such work can be measured and understood. 6