OXFAM GB S GLOBAL PERFORMANCE FRAMEWORK: CAPTURING AND COMMUNICATING OUR EFFECTIVENESS Claire Hutchings
Oxfam Great Britain Who we are Part of the Oxfam family an international confederation of 17 organizations (affiliates) networked together in over 100 countries. Page 2
THE DRIVERS shifts in external environment - need to be able to capture and communicate organisational impact for strategic financing / accountability Diverse portfolio of rights based programming learning account -ability Mixed quality of evaluations - limited institutional knowledge of what works, how, for whom, under which conditions, and for what cost time and resource constraints require a practical, proportional approach to access credible intervention effectiveness feedback Page 3
GLOBAL PERFORMANCE FRAMEWORK Global Output Reporting (GOR) Project Effectiveness Reviews Demonstrating the scale and much of the diversity of what we do (all relevant projects) The Global Performance Framework (GPF) Demonstrating our effectiveness reliable, credible evidence (random sample of mature projects) Page 4
THEMATIC AREAS Policy Influencing Humanitarian Assistance Citizen s Voice Livelihoods Support Women s Empowerment Resilience Accountability Page 5
OUTPUT INDICATORS Humanitarian Support Total number of people provided with appropriate humanitarian assistance, disaggregated by sex Adaptation & Risk Reduction # of people supported to understand current and likely future hazards, reduce risk, and/or adapt to climatic changes and uncertainty, disaggregated Livelihoods # of women and men directly supported to increase income via enhancing production and/or market access Women's Empowerment # of people reached to enable women to gain increased control over factors affecting their own priorities and interests Citizen s Voice Degree to which interventions have contributed to enabling citizens to hold duty bearers to account Policy Influencing Degree to which interventions have contributed to influencing pro-poor policy change OUTCOME INDICATORS Degree to which humanitarian responses meet agreed quality standards for humanitarian % of supported households demonstrating greater ability to minimise risk from shocks and adapt to emerging trends & uncertainty % of supported households demonstrating greater income, measured by daily consumption &expenditure per capita % of women demonstrating greater empowerment at household, community, and institutional levels # supported to engage with state / other relevant actors; and b) duty bearers benefiting from capacity support Number of campaign actions directly undertaken or supported Accountability (cross-cutting) Degree to which programmes meet Oxfam s Programme Standards Page 6
Only one part of Oxfam S MEAL system OXFAM REFLECTS Regional Reviews Audits Country Reviews Programme Monitoring Reviews BASELINES Learning feeds into planning 7 Page 7
A closer look at the working parts... Global Output reporting Effectiveness Reviews Page 8
GLOBAL OUTPUT REPORTING
What is Global Output Reporting? Annual beneficiary/output information from all active projects Demonstrates the scale and much of the diversity of what we do 36 indicators across six thematic areas, in addition to total direct / indirect beneficiaries Started in 2010/11 now going into its fourth year Page 10
EXAMPLE OUTPUTS: PRACTICALITIES Page 11
Page 12
EFFECTIVENESS REVIEWS
EFFECTIVENESS REVIEW PROJECT SELECTION c. 1000 LIVE PROJECTS c. 300 OVER > 2.5 YEARS OLD c. 100 > 250k c. 20 SELECTED FOR EFFECTIVENESS REVIEWS Page 14
MEASURING IMPACT? The Conventional NGO approach: Endline indicator reading Baseline indicator reading Change achieved Feedback OK but not reliable for most interventions Page 15
CONTRIBUTION/ ATTRIBUTION CHALLENGE Oxfam Intervention Positive Change in desired outcomes Association does not necessarily imply causation Contribution of Other Factors? Page 16
Large N Interventions OUTPUT INDICATORS Humanitarian Support Total number of people provided with appropriate humanitarian assistance, disaggregated by sex Adaptation & Risk Reduction Resilience # of people supported to understand current and likely future hazards, reduce risk, and/or adapt to climatic changes and uncertainty, disaggregated by sex Livelihoods # of women and men directly supported to increase income via enhancing production and/or market Livelihoods access Women's Empowerment # of people reached to enable women to gain increased control over factors affecting their own priorities and interests Empowerment Citizen s Voice # of a) citizens, CBO members and CSO staff supported to engage with state institutions/ other relevant actors; and b) duty bearers benefiting from capacity support OUTCOME INDICATORS Degree to which humanitarian responses meet agreed quality standards for humanitarian programming (e.g. Sphere guidelines) % of supported households demonstrating greater Mimic ability randomisation to minimise risk from through shocks and adapt to emerging trends & uncertainty use of comparison groups combined with techniques such % of supported households demonstrating greater as propensity income, as measured score matching by daily consumption and expenditure per capita to control for hopefully most of the relevant differences % of supported women demonstrating greater empowerment between at household, them community, and institutional levels Degree to which interventions have contributed to enabling citizens to hold duty bearers to account Policy Influencing Number of campaign actions directly undertaken or supported, e.g. contacts made with policy targets, online and offline actions taken, media coverage, publications, and specific events held Accountability (cross-cutting) Degree to which programmes meet Oxfam s Programme Standards Degree to which interventions have contributed to influencing pro-poor policy change Page 17
Small N Interventions OUTPUT INDICATORS Humanitarian Support Total number of people provided with appropriate humanitarian assistance, disaggregated by sex Adaptation & Risk Reduction # of people supported to understand current and likely future hazards, reduce risk, and/or adapt to climatic changes and uncertainty, disaggregated by sex Livelihoods # of women and men directly supported to increase income via enhancing production and/or market access Women's Empowerment # of people reached to enable women to gain increased control over factors affecting their own priorities and interests Citizen s Voice # of a) citizens, CBO members and CSO staff supported to engage with state institutions/ other relevant actors; and b) Citizen Voice duty bearers benefiting from capacity support Policy Influencing Number of campaign actions directly undertaken or supported, e.g. contacts made with Policy policy Influencing targets, online and offline actions taken, media coverage, publications, and specific events held Accountability (cross-cutting) Degree to which programmes meet Oxfam s Programme Standards OUTCOME INDICATORS Degree to which humanitarian responses meet agreed quality standards for humanitarian programming (e.g. Sphere guidelines) % of supported households demonstrating greater ability to minimise risk from shocks and adapt to emerging trends & uncertainty % of supported households demonstrating greater income, as measured by daily consumption and expenditure per capita % of supported women demonstrating greater empowerment at household, community, and institutional levels Degree to which interventions have contributed to enabling Non-counterfactual citizens to hold duty approach bearers to to account causal inference (process tracing) to explore significance of interventions contribution to observed outcomes. Degree to which interventions have contributed to influencing pro-poor policy change Page 18
MEASURING & EVALUATING RESILIENCE
RESILIENCE MEASUREMENT: THE CHALLENGE Complex, multi-dimensional concept Contextually dependent Need to evaluate resilience-building work even in the absence of a shock. Page 20
Examples of Indicators Dimensions RESILIENCE FRAMEWORK Livelihood viability Innovation potential Access to contingency resources & support Integrity of the natural & built environment Social & institutional capability Ownership of productive assets Diversification: Crop types Livestock types Other livelihoods Access to markets Use of improved crop & livestock management techniques Access to vaccination Access to veterinary care Access to credit Attitudes towards new livelihood practices Adoption of innovative practices Awareness of climate change Access to a grain bank Savings Access to remittances or formal earnings Ownership of small livestock Social support networks Access to medical care Access to irrigation Damage to soil Access to desalination pump Access to safe drinking water source Access to agricultural & livestock extension services Risk-reduction activities in community Possession of CNIC card Confidence in local leaders accessing support from government/ngos Confidence in government structures to deal with crises Social cohesion in community Page 21
PROCESS Identify appropriate characteristics of resilience. Define what it means to score positively in terms of each characteristic. Collect data from intervention and comparison group. Aggregation: count the number of characteristics in which each household scores positively. Compare outcomes for intervention and comparison group (regression and/or propensity-score matching) Page 22
REMAINING CHALLENGES Assumptions about resilience characteristics and thresholds not tested Reliance on perception-based indicators Intra-household level Evaluating against resilience overall v. evaluating against project objectives Using PSM or regression modeling without a baseline is not ideal. Page 23
MEASURING POLICY INFLUENCE PROCESS TRACING PROTOCOL
PROCESS TRACING Investigating causal inference: 1. Understand whether and to what extent the outcome has materialised 2. Consider the evidence supporting different causal stories or explanatory hypotheses to build an understanding of "how change happened", with the intervention just one possible contributing factor 3. Where the evidence supports a causal relationship between the intervention and the outcome - determine the significance of the intervention s contribution in light of other contributing factors Page 25
ADAPTATION OF PROCESS TRACING (MAIN STEPS FOR RESEARCHERS) 1. Specify outcomes the intervention is seeking to achieve (explicit theory of change). 2. Assess and document what was done to achieve the targeted outcomes. 3. Examine the extent the targeted outcomes have actually materialised, as well as any unintended outcomes. 4. Undertake process induction to identify all salient, plausible causal explanations ( causal stories ) for each materialised outcome. 5. Use process verification to assess the extent each causal story is supported or not by the available evidence signatures, footprints. Page 26
Gender Mainstreamed into the Departmental Statute of Autonomy The Women s Platform (Oxfam and its IFFI) advocacy efforts Approval of the National Political Constitution (incorporation of gender equity) Departmental Federation of Indigenous and Rural Women of Bolivia Page 27
PROBATIVE TESTS Conducting probative tests on available evidence/ causal-process observations (aka clues ) can help to support or overturn explanatory hypotheses Sufficient to Establish Causation Necessary to Establish Causation No No Straw in the Wind May increase the plausibility of a given hypothesis, or raise doubts about it, but are not decisive by themselves Yes Smoking Gun Passing confirm the hypothesis, but does not eliminate others Yes Hoop tests cannot provide direct support for a hypothesis, but can eliminate a hypothesis Doubly decisive confirm one hypothesis and eliminate others Bennett (2010); Van Evera (1997) Page 28
WHERE ARE WE NOW? WHAT ARE SOME OF THE OUTSTANDING CHALLENGES?
WHERE HAVE WE GOT TO Clear set of Global Indicators (outputs and outcomes, 6 thematic areas, 1 cross-cutting) Piloted and refined measurement approaches, and developed 4 distinct methodologies humanitarian, accountability, development, campaigns/voice - robust, but practical and adaptable Completed 72 evaluations (effectiveness reviews) of randomly selected mature projects & 4 Accountability Reviews Transparent in communicating findings and reflections, the good, the bad and the ugly Developed a management responses system Learning, learning, learning! Page 30
PUBLISHING THE GOOD, THE BAD & THE UGLY Page 31
LEARNING, LEARNING, LEARNING PERCEIVED USEFULNESS OF EFFECTIVENESS REVIEWS - % of Country/Regional staff seeing as effective for the following purposes Page 32
LEARNING, LEARNING, LEARNING CHANGES MADE FOLLOWING EFFECTIVENESS REVIEWS % identifying changes in each category Page 33
BALANCING ACCOUNTABILITY & LEARNING learning account -ability Page 34
SUMMING IT UP (EXAMPLE FROM LIVELIHOODS GLOBAL INDICATOR: % of supported households demonstrating greater income) Year Global indicator (percentage points) 2011/12 aggregate 5.3 2012/13 aggregate 2.1 2013/14 aggregate 14.3 Year Country Weight (sample frame) Global indicator (percentage points) 2011/12 Zimbabwe 70 13.0 2011/12 Uganda 427-11.2 2011/12 DRC 586 6.8 2011/12 Haiti 2243 7.6 2011/12 Nicaragua 127 8.3 2012/13 Georgia 134 10.7 2012/13 Liberia 855-3.9 2012/13 Vietnam 271-1.0 2012/13 Sri Lanka 9950 3.0 2013/14 Armenia 284 9.3 2013/14 Colombia 751 4.8 2013/14 Honduras 380 2.1 2013/14 Zimbabwe 2115 19.9 Page 35
SUMMING IT UP... CHALLENGE % of supported households demonstrating greater income, measured by daily consumption &expenditure per capita Agricultural Inputs Enterprise Development Micro-Credit Page 36
Key Emerging Issues Priority Uses of Performance Reporting - balance between evaluation, accountability, and learning Rigour vs breadth, intervention vs institutional learning, random selection vs learning priorities, standardisation/ aggregation vs contextualisation More coherent approach to evaluation - link between rigorous, centrally managed &funded Effectiveness Reviews, and wider portfolio of country led evaluations Funding of Effectiveness Reviews - could project planning and budgeting processes be adapted to enable Effectiveness Reviews be funded from project funding Enhancing Institutional Learning strengths/ weaknesses, prioritisation Transition to Oxfam 2020 - How should GPF processes evolve to align with emerging OI-wide processes Page 39