Monitoring and Evaluation of Policy Influencing. A presentation for UNDP Vietnam By Arnaldo Pellini and Ajoy Datta, November 2010

Transcription

1 Monitoring and Evaluation of Policy Influencing A presentation for UNDP Vietnam By Arnaldo Pellini and Ajoy Datta, November 2010

2 Agenda 1. ODI and RAPID 2. What is policy change? 3. Policy influencing approaches 4. Challenges of M&E of policy influence 5. Theory of change 6. How to do M&E of influence 7. Conclusion

3 The Overseas Development Institute (ODI) UK s leading think tank on international development and humanitarian issues Staff: 160 staff including 75 researchers Functions: research, advice and public debate Issues: Rural, Humanitarian, Poverty & Aid, Economics, Policy Processes For more information see

4 RAPID

5 RAPID s double focus Understanding the research/policy/practice nexus Political context Social and cultural factors Type of evidence/ knowledge Mapping stakeholders Self-positioning Facilitating organisational engagement Developing strategies Managing knowledge Collaboration and innovation Communication Monitoring, evaluation and learning

6 RAPID s work on M&E of policy influence Young, J. and Mendizabal, E. (2009) Helping researchers become policy entrepreneurs ODI Briefing Paper 53. London: ODI, available at Jones, H. (2010 forthcoming) The monitoring and evaluation of policy influence: A scoping review for the Department for International Development ODI Mendizabal, E. and Clarke, J. (2010) Monitoring and Evaluation of Policy influencing interventions: How to Note. Report for DFID. ODI. Mendizabal, E. (2010) Monitoring and Evaluation of Research Based Influence Prezi Presentation available at

7 2. What is policy change Attitudinal change Discursive commitment Procedural change Policy content Behaviour change

8 3. Policy influencing approaches Evidence / science based Advising Advocacy ODI DFID DFID Jubilee 2000 Inside track / cooperation Companies Governments DFID Greenpeace Sea Shepherd Conservation Society Outside track / pressure Lobbying Activism Interest / values based Start and Hovland (2004)

9 3. Policy influencing approaches Advising / evidence Through what channels? National policy debates Formal/informal meetings How? Research / analysis / good practices / piloting Advocacy Public and policy debates / public meetings / media Public campaign / public education / advocacy Lobbying Formal meeting / informal meetings / working groups Face-to-face / trust / diplomacy Activism Public campaign / media Direct activism

10 4. Challenges of M&E of policy influencing Conceptual and technical challenges Policy processes are not linear Project impact different from programme impact Nature of policy influencing Policy context and actors change Networks and alliances Practical constraints Work load of staff Capacity Incentives Turnover

11 5. Theory of change A basic Theory of Change explains how a group short and medium term actions sets the stage for producing long-term results

12 5. Theory of change Causal chain e.g. Logframe Types External factors Political/ Policy context Dimensions of influence Networks Evidence e.g. RAPID framework

13 5. Theory of change Types Behaviour Actor-centred theories e.g. Outcome Mapping Now Future IDRC Outcome Mapping

14 5. Theory of change Advising / evidence Research / advice / seminars / policy briefs / face-to-face meetings ToC Activities & research outputs Uptake / response to research Evidence in policy change M&E Peer reviews paper / web / briefs / reports Uptake logs Citation analysis User surveys In depth studies: Most Significant Change Episode studies After Action Review

15 5. Theory of change Advocacy Public campaign, public messaging, media ToC Messaging / campaigning Increased public awareness Policy change M&E Analysis of target groups (surveys / FGD/ interviews) Review Media tracking Interviews / surveys In depth case studies

16 5. Theory of change Lobbying Direct interaction with policy makers ToC Engaging with key stakeholders Influence Policy change M&E Map key stakeholders Learn interest, perspective and links Keep track of meetings Review links Keep track of perceptive change In depth case studies After Action Review

17 How to do M&E of policy influence Influencing efforts are diverse Based on previous work for DFID Help to monitor, evaluate interventions, communicate, measure VfM Approach depends on type and scale of intervention

18 Influencing plan as a basis for M&E Start by defining your policy objectives constantly review them during the process 21

19 The steps Start by defining your policy objectives constantly review them during the process 22

20 How to do M&E of influence 1. Develop a pathway of change 2. Decide which indicators to measure 3. Choose the most relevant tools 4. Reflect and/or evaluate 5. Decide what resources to allocate to M&E?

21 Developing a pathway of change Designing ToC requires following steps: Understanding how change happens What are long term objectives What needs to change, and WHO needs to do things differently to bring about objectives What approaches could you take to contribute to those changes

22 Developing a pathway of change Goal (Beneficiaries) Obj ecti ves Purpose (Policy Influencing Objectives) Output 1 DFID Output 2 Target or audience Output 3 Target or audience Output N Actions of actors not targeted by DFID Stra tegy Activities A-Z from other actors Activities A-Z from other actors Approaches or Activities 1 Activities 2 Management, inputs Project Management (Budget, HR, Organisational Practices)

23 Developing a pathway of change Pathway of Change: should be based on theories of how policy changes happen should be plausible should be doable should be testable

24 Some theories of change

25 How to do M&E of policy influence Ensure that risks are routinely tracked and monitored Assumptions do not always materialise A statement of the counterfactual No policy change would otherwise happen Alternative policies adopted Other actors would fill the gap

26 Decide which indicators to measure Progress indicators Skills and inputs Activities Changes in behaviour of the intervention s primary audience Changes in policies socio-economic indicators of the final beneficiaries (e.g. child mortality, poverty levels, etc.)

27 Decide which indicators to measure Value for Money and performance Indicators Value for money Economy (cost of input) Efficiency (cost of activities) Effectiveness (cost of achieving the intended influencing outputs or purpose) Relevance (validity of theory of change) Contribution (of the intervention) Sustainability (influencing activity will continue) Coherence (fit with organisational strategy)

28 Choose the most relevant monitoring tools Learning (about our actions and other s reactions) Back to Office Reports, After Action Reviews, Weekly Team Meetings, online discussions, retreats and workshops and trainings Monitoring Log Records what happened and what changes if any took place, expected and unexpected Reviews Periodic meetings to reflect on actions, draw lessons, consider changes to the strategy, and how to implement

29 Reflection and evaluation Focuses on lessons and recommendations for future interventions as well as serving an important accountability function The evaluation should reflect the right balance between accountability and learning purposes Test the Theory of Change Assess changes in the context (expected and unexpected) Assess relevance and value of intervention

30 Reflection and evaluation Level of M&E effort required What type of final assessment? Who should do it? Low effort A final Review The team or a facilitator Medium effort (short and medium term)* Evaluation An evaluator directly commissioned by the team (or their department) Medium effort (long term) Evaluation An evaluator commissioned by EvD High effort Evaluation An evaluator commissioned by EvD

31 6. Conclusion M&E of policy influence presents unique challenges But there are ways of generating useful information recognise the value of M&E Some recommendations Ensure information has multiple uses Develop a theory of change during the planning phase of the project Some well known tools that can be used

32 Some other useful resources Start, D. and Hovland, I. (2004) Tools for policy impact: a handbook for researchers, London: ODI Hearn S. And Jones, H. (2009). Outcome Mapping: A realistic alternative for planning, monitoring and evaluation, ODI Background Notes, October, London?:ODI outcome-mapping-realistic-planning-monitoring-evaluation

33 Discussion What approach does UN / UNDP take to policy attribution? How does UN / UNDP see change happening in Vietnam policy circles? What tools does it use to measure policy attribution?