Oxfam GB s Attempt to Measure Resilience Karl Hughes Oxfam GB AEA October 2012
Presentation Outline: Part A: Conceptual Framework Part B: Application to Assess Impact Part C: Improving Measurement with AF Method Part D: Strengths, Limitations & Further Work Why, What, and How Impact Assessment Design Results from 4 Case Studies Alkire-Foster (AF) Method Application to Indonesia data Strengths and Limitations Going Forward
Part A: Conceptual Framework
Motivation/Rationale In search of a reliable but practical way of measuring the effectiveness of community level DRR/CCA (resilience promotion) work How well has the intervention worked to reduce risk and promote future adaptation success?
Inherent Challenge: We can really only assess effectiveness after the fact Was community X able to cope and/or adapt more effectively than if we had never intervened? Can we do anything before the fact?
Inspiration taken from John Twigg s work in Characteristics of Disaster Resilient Communities Let s assume that resilient and/or adaptive communities and households possess particular characteristics What does a resilient and/or adaptive household or community look like?
But what characteristics should be examined? Twigg s work starting point Also looked at ACCRA and other relevant literature, e.g. Doman et. al Building Resilience (2009) But realisation that one set of characteristics not applicable to all contexts Nevertheless, useful to have a framework to look at various dimensions
Key Dimensions of Resilience What is needed to reduce risk and support positive adaptation to emerging trends and uncertainty? Extent livelihood strategies can thrive in spite of shocks, stresses, and uncertainty Ability to take appropriate risks and positively adjust to change Access to back up resources and appropriate assistance in times of crisis Health of local ecosystems, soundness of natural resource management practices, and robustness of essential physical infrastructure Extent formal & informal institutions are able to reduce risk, support positive adaptation, and ensure equitable access to essential services in times of shock/stress.
EXAMPLES: Livelihood Diversification Motivation Savings Farming practices Existence of committee & plan Crop portfolio Beliefs about climate change Social support system Status of local soils Linkages to external support Access to early warning information Access to climate trend information Insurance Livestock grazing practice Community awareness and participation
Characteristic scoring example Lower scores = Dependency on small number of vulnerable livelihood activities Higher scores = Wider portfolio of livelihood activities, including those that are still viable in times of shock
Part B: Application to Assess Impact
Impact Assessment Design Map out intervention communities Identify similar communities not supported (purposive selection) Administer surveys to randomly selected households in both intervention and comparison communities Use statistical methods (e.g. propensity score matching and regression) to control for observable differences between the groups
Comparison can be made in terms of the overall score Sample Size: 2,542 (intervention: 994; comparison: 1,638) = statistically significant difference
Or by specific dimension Sample Size: 605 (intervention: 242; comparison: 362) = statistically significant difference
Or by specific characteristic Sample Size: 841 (intervention: 341; comparison: 500) = statistically significant difference
Part C: Improving Measurement with Alkire-Foster (AF) Method
New Approach to Data Analysis Key Issue: How to meaningfully aggregate the scores? Resilience is a multidimensional construct The Oxford Poverty & Human Development Initiative (OPHI) has done good work on measuring multidimensional poverty Adopt the OPHI approach the Alkire-Foster (AF) Method
AF Method applied to data collected from three districts in Indonesia s Nusa Tenggara Barat (NTB) province. Sample of 242 HHs from 23 intervention villages and 363 HHs from 23 purposively matched comparison villages
Step 1: Define Weights for Dimensions Weights Assigned to Dimensions for Indonesia Case Study Social Capability 20% Livelihood Viability 25% Natural & built environment 10% Access to Contingency Resources and Support 20% Livelihood Innovation Potential 25%
Step 2: Define Binary Cut-offs & Indicator Weights Dimension Livelihood Viability (weight = 25%) Cut-off Characteristic/Indicator a HH is non-deprived if... Household wealth status HH owns at least 3 small assets (TV, sat dish, fridge, fan, mobile phone, radio/cd play, mattress, iron, gold jewellery, smart floor (tiles/carpet) OR 1 big asset (motorbike, car, power generator, tractor) Basic needs HH reports being in a position to meet basic needs or greater Livelihood diversification dependent on > 4 disaster susceptible livelihood activities (LA) OR > 1 disaster tolerant LA OR 3-4 disaster susceptible LA Farming extension support Raw Headcount (2012) Indicator weights 0.2280992 0.25/7 0.4297521 with 1 disaster tolerant LA 0.4561983 reports having access to service support, at least sometimes uses it, finds it at least somewhat useful, and found timing at least to be fair 0.6942149 Access to marking information & support as above 0.7471074 Access to seasonal forecast information as above 0.5752066 Access to disaster preparedness information as above 0.6991736 0.25/7 0.25/7 0.25/7 0.25/7 0.25/7 0.25/7
Density 0 1 2 3 Step 3: Weight indicators and construct counting vector Counting vector = % of weighted indicators each household is deprived in Counting Vector - Indonesia.2.4.6.8 1 % of weighted indicators HH is deprived in
Step 4: Select vulnerability cut-off and compute indices Head count ratio (H) e.g. cutoff 40% Average deprivation share among vulnerable (A) Adjusted Headcount Ratio (M0)
Deprivation gap indices (only when characteristics are continuous or interval) Head count ratio (H) Head count ratio (H) Adjusted Deprivation Gap (M1) Adjusted Squared Deprivation Gap (M2) Normalized Gap (A1) Normalized Gap Squared (A2)
Step 5: Compare intervention and comparison groups on indices Comparison of Intervention and Comparison Groups on Vulnerability Measures Computed with AF Method: Indonesia Counting Vector Headcount Ratio (H) Average deprivation share (A) Adjusted Headcount Ratio (M0) Unadjusted: Sample mean 0.64 0.89 0.69 0.61 Intervention mean 0.57 0.77 0.65 0.50 Comparison mean 0.69 0.96 0.71 0.68 Unadjusted difference -0.117 *** -0.1886 *** -0.0542 *** -0.175 *** (-8.17) (-6.77) (-4.43) (-8.13) Observations 604 604 535 604 PSM Kernel (ATT): Post-matching difference: -0.0887 *** -0.152 *** -0.0316 * -0.134 *** (-4.44) (-3.93) (-2.09) (-4.25) Observations 549 549 494 549 t statistics in parentheses * p < 0.05, ** p < 0.01, *** p < 0.001 PSM estimates bootstrapped 1000 repetitions
M0 Step 6: Perform Robustness Checks 0.8 Example: M0 at Different Deprivation Cut-offs for Intervention and Comparison Groups Indonesia (unadjusted) 0.7 0.6 0.5 0.4 0.3 0.2 Int. lower bound Int. mean Int. upper bound Comp. lower bound Comp. mean Comp. upper bound 0.1 0 k=10 k=20 k=30 k=40 k=50 k=60 k=70 k=80 k=90 k=100
Step 7: Decompose data to assess what is driving the difference Comparison of Intervention & Comparison Groups by Characteristic with PSM Kernel Characteristic /Indicator Difference t-statistic Obs. Household wealth status 0.0627 1.27 549 Basic needs 0.0725 1.24 549 Livelihood diversification 0.0678 1.24 549 Farming extension support -0.108-1.91 549 Access to marking information & support 0.0553 1.13 549 Access to seasonal forecast information -0.169** -2.97 549 Access to disaster preparedness information -0.354*** -7.13 549 Motivation to pursue alternative livelihood strategies -0.0624-1.05 549 Attitudes about climate change -0.0616-1.03 549 Credit access 0.0372 0.70 549 Access to climate trend information 0.0235 0.51 549 Access to livelihood innovation support -0.0179-0.75 549 Social support system 0.0304 0.51 549 Gov. Support 0.136* 2.31 549 Savings -0.00246-0.08 549 Convertable assets -0.0437-0.84 549 Natural resource management practices -0.151** -2.61 549 Protection of assets -0.110** -2.83 549 Knowledge of disaster management plan -0.417*** -8.96 549 Participation in disaster preparation meetings -0.225*** -5.39 549 * p < 0.05, ** p < 0.01, *** p < 0.001 PSM estimates bootstrapped 1000 repetitions
Comparison of Intervention and Comparison Groups on Relative Contributions of Characteristics to M0 - Indonesia (unadjusted) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 Householod wealth status Basic needs Livelihood diversification Farming extension support Access to marking information & support Access to seasonal forecast information Access to disaster preparedness information Motivation to pursue alternative livelihood strategies Attitudes about climate change Credit access (formal and informal) Access to climate trend information Access to livelihood innovation support Social support system Gov. Support Savings Convertable assets Natural resource management practices Protection of assets Knowledge of disaster management plan Participation in flood preparation meetings 0.2 0.1 Social Capability 20% Livelihood Viability 25% 0 Intervention Comparison Enviro. 10% Contin. Resources & Support 20% Innovation 25%
Part D: Strengths, Limitations, and Further Work
Benefits of Approach Doable assess resilience before the fact Provides direction for program development identifies gaps Can be used as part of situational analysis to inform intervention design
Limitations/Challenges Many of the characteristics difficult to measure e.g., many depend on perception data How can we be sure we are looking at the right package of characteristics given the context? More research needed to inform the characteristics So far, very strong emphasis on household level data limitations in getting at community capacity and beyond
Further Work Seek to improve measurement of specific characteristics Continuously update characteristics based on new research Take forward framework, approach, and learning to inform programme design
Optional Slide
Are the characteristics really successful predictors of shock coping and/or adaptation success? Took advantage of 2010 floods in Pakistan study Number of interesting results: Intervention communities reported significantly more advanced warning 48 versus 24 hours Also reported less loss of livestock, grain, and tools Intervention group poorer at baseline but now better off (asset wealth measure)