Oxfam GB s Attempt to Measure Resilience. Karl Hughes Oxfam GB AEA October 2012

Size: px

Start display at page:

Download "Oxfam GB s Attempt to Measure Resilience. Karl Hughes Oxfam GB AEA October 2012"

Marjory Summers
5 years ago
Views:

1 Oxfam GB s Attempt to Measure Resilience Karl Hughes Oxfam GB AEA October 2012

2 Presentation Outline: Part A: Conceptual Framework Part B: Application to Assess Impact Part C: Improving Measurement with AF Method Part D: Strengths, Limitations & Further Work Why, What, and How Impact Assessment Design Results from 4 Case Studies Alkire-Foster (AF) Method Application to Indonesia data Strengths and Limitations Going Forward

3 Part A: Conceptual Framework

4 Motivation/Rationale In search of a reliable but practical way of measuring the effectiveness of community level DRR/CCA (resilience promotion) work How well has the intervention worked to reduce risk and promote future adaptation success?

5 Inherent Challenge: We can really only assess effectiveness after the fact Was community X able to cope and/or adapt more effectively than if we had never intervened? Can we do anything before the fact?

adaptive communities and households possess particular

6 Inspiration taken from John Twigg s work in Characteristics of Disaster Resilient Communities Let s assume that resilient and/or adaptive communities and households possess particular characteristics What does a resilient and/or adaptive household or community look like?

7 But what characteristics should be examined? Twigg s work starting point Also looked at ACCRA and other relevant literature, e.g. Doman et. al Building Resilience (2009) But realisation that one set of characteristics not applicable to all contexts Nevertheless, useful to have a framework to look at various dimensions

8 Key Dimensions of Resilience What is needed to reduce risk and support positive adaptation to emerging trends and uncertainty? Extent livelihood strategies can thrive in spite of shocks, stresses, and uncertainty Ability to take appropriate risks and positively adjust to change Access to back up resources and appropriate assistance in times of crisis Health of local ecosystems, soundness of natural resource management practices, and robustness of essential physical infrastructure Extent formal & informal institutions are able to reduce risk, support positive adaptation, and ensure equitable access to essential services in times of shock/stress.

9 EXAMPLES: Livelihood Diversification Motivation Savings Farming practices Existence of committee & plan Crop portfolio Beliefs about climate change Social support system Status of local soils Linkages to external support Access to early warning information Access to climate trend information Insurance Livestock grazing practice Community awareness and participation

activities Higher scores = Wider portfolio of

10 Characteristic scoring example Lower scores = Dependency on small number of vulnerable livelihood activities Higher scores = Wider portfolio of livelihood activities, including those that are still viable in times of shock

11 Part B: Application to Assess Impact

12 Impact Assessment Design Map out intervention communities Identify similar communities not supported (purposive selection) Administer surveys to randomly selected households in both intervention and comparison communities Use statistical methods (e.g. propensity score matching and regression) to control for observable differences between the groups

13 Comparison can be made in terms of the overall score Sample Size: 2,542 (intervention: 994; comparison: 1,638) = statistically significant difference

14 Or by specific dimension Sample Size: 605 (intervention: 242; comparison: 362) = statistically significant difference

15 Or by specific characteristic Sample Size: 841 (intervention: 341; comparison: 500) = statistically significant difference

16 Part C: Improving Measurement with Alkire-Foster (AF) Method

17 New Approach to Data Analysis Key Issue: How to meaningfully aggregate the scores? Resilience is a multidimensional construct The Oxford Poverty & Human Development Initiative (OPHI) has done good work on measuring multidimensional poverty Adopt the OPHI approach the Alkire-Foster (AF) Method

18 AF Method applied to data collected from three districts in Indonesia s Nusa Tenggara Barat (NTB) province. Sample of 242 HHs from 23 intervention villages and 363 HHs from 23 purposively matched comparison villages

19 Step 1: Define Weights for Dimensions Weights Assigned to Dimensions for Indonesia Case Study Social Capability 20% Livelihood Viability 25% Natural & built environment 10% Access to Contingency Resources and Support 20% Livelihood Innovation Potential 25%

20 Step 2: Define Binary Cut-offs & Indicator Weights Dimension Livelihood Viability (weight = 25%) Cut-off Characteristic/Indicator a HH is non-deprived if... Household wealth status HH owns at least 3 small assets (TV, sat dish, fridge, fan, mobile phone, radio/cd play, mattress, iron, gold jewellery, smart floor (tiles/carpet) OR 1 big asset (motorbike, car, power generator, tractor) Basic needs HH reports being in a position to meet basic needs or greater Livelihood diversification dependent on > 4 disaster susceptible livelihood activities (LA) OR > 1 disaster tolerant LA OR 3-4 disaster susceptible LA Farming extension support Raw Headcount (2012) Indicator weights / with 1 disaster tolerant LA reports having access to service support, at least sometimes uses it, finds it at least somewhat useful, and found timing at least to be fair Access to marking information & support as above Access to seasonal forecast information as above Access to disaster preparedness information as above /7 0.25/7 0.25/7 0.25/7 0.25/7 0.25/7

21 Density Step 3: Weight indicators and construct counting vector Counting vector = % of weighted indicators each household is deprived in Counting Vector - Indonesia % of weighted indicators HH is deprived in

22 Step 4: Select vulnerability cut-off and compute indices Head count ratio (H) e.g. cutoff 40% Average deprivation share among vulnerable (A) Adjusted Headcount Ratio (M0)

23 Deprivation gap indices (only when characteristics are continuous or interval) Head count ratio (H) Head count ratio (H) Adjusted Deprivation Gap (M1) Adjusted Squared Deprivation Gap (M2) Normalized Gap (A1) Normalized Gap Squared (A2)

24 Step 5: Compare intervention and comparison groups on indices Comparison of Intervention and Comparison Groups on Vulnerability Measures Computed with AF Method: Indonesia Counting Vector Headcount Ratio (H) Average deprivation share (A) Adjusted Headcount Ratio (M0) Unadjusted: Sample mean Intervention mean Comparison mean Unadjusted difference *** *** *** *** (-8.17) (-6.77) (-4.43) (-8.13) Observations PSM Kernel (ATT): Post-matching difference: *** *** * *** (-4.44) (-3.93) (-2.09) (-4.25) Observations t statistics in parentheses * p < 0.05, ** p < 0.01, *** p < PSM estimates bootstrapped 1000 repetitions

25 M0 Step 6: Perform Robustness Checks 0.8 Example: M0 at Different Deprivation Cut-offs for Intervention and Comparison Groups Indonesia (unadjusted) Int. lower bound Int. mean Int. upper bound Comp. lower bound Comp. mean Comp. upper bound k=10 k=20 k=30 k=40 k=50 k=60 k=70 k=80 k=90 k=100

26 Step 7: Decompose data to assess what is driving the difference Comparison of Intervention & Comparison Groups by Characteristic with PSM Kernel Characteristic /Indicator Difference t-statistic Obs. Household wealth status Basic needs Livelihood diversification Farming extension support Access to marking information & support Access to seasonal forecast information ** Access to disaster preparedness information *** Motivation to pursue alternative livelihood strategies Attitudes about climate change Credit access Access to climate trend information Access to livelihood innovation support Social support system Gov. Support 0.136* Savings Convertable assets Natural resource management practices ** Protection of assets ** Knowledge of disaster management plan *** Participation in disaster preparation meetings *** * p < 0.05, ** p < 0.01, *** p < PSM estimates bootstrapped 1000 repetitions

27 Comparison of Intervention and Comparison Groups on Relative Contributions of Characteristics to M0 - Indonesia (unadjusted) Householod wealth status Basic needs Livelihood diversification Farming extension support Access to marking information & support Access to seasonal forecast information Access to disaster preparedness information Motivation to pursue alternative livelihood strategies Attitudes about climate change Credit access (formal and informal) Access to climate trend information Access to livelihood innovation support Social support system Gov. Support Savings Convertable assets Natural resource management practices Protection of assets Knowledge of disaster management plan Participation in flood preparation meetings Social Capability 20% Livelihood Viability 25% 0 Intervention Comparison Enviro. 10% Contin. Resources & Support 20% Innovation 25%

28 Part D: Strengths, Limitations, and Further Work

29 Benefits of Approach Doable assess resilience before the fact Provides direction for program development identifies gaps Can be used as part of situational analysis to inform intervention design

30 Limitations/Challenges Many of the characteristics difficult to measure e.g., many depend on perception data How can we be sure we are looking at the right package of characteristics given the context? More research needed to inform the characteristics So far, very strong emphasis on household level data limitations in getting at community capacity and beyond

31 Further Work Seek to improve measurement of specific characteristics Continuously update characteristics based on new research Take forward framework, approach, and learning to inform programme design

32 Optional Slide

33 Are the characteristics really successful predictors of shock coping and/or adaptation success? Took advantage of 2010 floods in Pakistan study Number of interesting results: Intervention communities reported significantly more advanced warning 48 versus 24 hours Also reported less loss of livestock, grain, and tools Intervention group poorer at baseline but now better off (asset wealth measure)

A Multidimensional Approach for Measuring Resilience. Oxfam GB Working Paper

A Multidimensional Approach for Measuring Resilience Oxfam GB Working Paper Oxfam GB August, 2013 Abstract: There is currently a wave of enthusiasm for building resilience in the international humanitarian