Measuring Impact of Food Assistance Programmes Insights from WFP s Experience

Transcription

1 Measuring Impact of Food Assistance Programmes Insights from WFP s Experience Susanna Sandström Policy, Planning and Strategy Division Workshop on Impact Evaluation of Food Security Related Programming FAO, Rome 6 7 December 2010

2 Outline WFP IFPRI collaboration: measuring impact of food assistance programmes What do we want to learn? Common features of the evaluations IFPRI survey instruments: main modules & food security module Measuring food security with different intervention objectives Sample size requirements for different outcome variables

3 WFP IFPRI collaboration: rigorous evaluations of food assistance programmes Five innovative pilots will be rolled out in a wide range of contexts: Ecuador Cash/vouchers/ food for urban refugees Niger Cash/food for chronically food insecure Timor Leste Cash for assets to disaster prone communities Uganda Cash/food for early childhood development Yemen Cash/food for chronically food insecure The pilots will be evaluated under the leadership of the International Food Policy Research Institute The objective of the evaluation component is to provide rigorous evidence of the benefits and costs of the use of cash and vouchers as opposed to food transfers Baseline data collected in Uganda and Timor Leste, interventions in 2011, final report by March 2012

4 Why is this important to WFP? The strategic plan expands WFP s toolbox to include cash& voucher transfers Important question: when to deploy different types of transfer modalities? Given WFP s mandate, do we obtain food security objectives with cash and voucher transfers? Number of planned cash and voucher projects

5 What are some of the learnings so far? Beneficiaries do not always use the expanded choice given by cash & vouchers > sometimes limited nutritional impact in Afghanistan beneficiaries used their vouchers for on average three different food items > no evidende of nutritional impact as measured by the Food Consumption Score in Zimbabwe people bought mainly wheat with their cash... but in Malawi cash was by far the most cost effective transfer moldality... and in Sri Lanka cash beneficiaries showed both increased dietary diversity and larger spending on non food items Gender preferences do not always work in the expected way In Sri Lanka preference for transfer type (food/cash) varied more by access to markets than by gender In Afghanistan men and women unanimoulsy preferred vouchers to food

6 Common features of the five pilots Design Comparison of different treatment groups where the treatment differ in transfer modality received (cash food vouchers) Control group included in all evaluations Method Programme placement randomized on cluster level (village, barrio etc) In some cases nonrandom control group Questions Impact on food security relates outcomes: nutrition, livelihoods, intra household resource allocation when different transfer modalities are used Distribution of benefits across and within households Why are differences in impacts observed?

7 IFPRI survey instruments: modules Module Level of collection Variations Household roster and demographics Individual Standardized Education Individual Standardized Dwelling characteristics Household Standardized Health and food security Individual By intensity and country Livelihood information Household/individual By intensity and country Expenditures Household/individual By intensity and country Budgeting behavior Household By intensity Gender dynamics Household/individual By intensity Market conditions Household Standardized Transfer process and program co responsibilities Household By country Nutritional biomarkers Individual By intensity and country

8 Survey instruments: standard food security module Household coping strategies Diet diversity scales Household caloric acquisition Nutrition knowledge number of meals per day hunger levels or formal food insecurity access scales number of days category of food eaten in last week type of food consumed amount of food consumed source of food consumed (bought, traded, gift, produced) diet diversity, vitamins, minerals hygiene young child feeding supplementation source of acquired information

9 Measuring food security with different intervention objectives Uganda (1) Intervention Cash or food transfers to households conditional on participation of young children (3 5) in Early Childhood Development Centres Take home ration 1000 cal per child per day: nutrient rich soyacorn blend, oil and sugar given to mothers Nutrition sensitization of parent groups part of intervention Main objectives: to increase attendance and attention of children and improve their nutritional status Evaluation The evaluation has a randomized prospective design including 109 Early Childhood Development (ECD) Centres and 3270 households Point of randomization at ECD centre level: a third selected for cash, a third for take home rations and a third for no transfer For each ECD centre, sample of 25 households with children in the centre and 5 households with eligible children but no enrolled

10 Measuring food security with different intervention objectives Uganda (2) Context and assumptions Karamoja is the region with the worst human development indicators in Uganda Poor nutritional intake due to restricted household budgets > risk that especially cash is used for other purposes than child feeding Poor child feeding practices > risk that cash transfers won t improve nutritional status

11 Measuring food security with different intervention objectives Uganda (3) Special focus in the survey instrument: Key outcomes of interest: food security and dietary diversity of especially children physical and cognitive development of children education outcomes (not discussed here) consumption of different food groups during past seven days of index children health/nutrition knowledge related to children: breastfeeding, knowledge of micronutrients health and development of index children: immunization, cognitive skills, activities + anthropometric assessment of children: weight for height z scores, weight for age and height for age

12 Measuring food security with different intervention objectives Ecuador (1) Intervention Unconditional food/cash/vouchers to Colombian refugees and poor host Ecuadorians Nutrition campaign part of the intervention Main objectives: improve food consumption of beneficiaries and increase the role of women in household decision making related to food consumption Evaluation The evaluation will have a randomized design, details still under development Sample to include 3080 households of which 660 belong to a control group

13 Measuring food security with different intervention objectives Ecuador (2) Context and assumptions Poverty and malnutrition high in targeted provinces Many refugees excluded from basic services Poor food consumption practices and dietary diversity Low decision making power of women regarding food purchase in refugee communities

14 Measuring food security with different intervention objectives Ecuador (3) Key outcomes of interest: food security and dietary diversity anaemia prevalence in children and adolescent women involvement of women in food and nutrition related decisions Special focus in the survey instrument: slightly expanded list of food items since programme operating in an urban area household decision making food frequency and dietary diversity of women and adolescent girls + nutritional biomarker: haemoglobin of children under 5 and adolescent women years old

15 Sample size requirements for different outcome variables in cluster randomized studies Sample size requirements are driven by: 1. n of households per cluster in each treatment arm 2. n of clusters in each treatment arm 3. intracluster correlation for any positive intracluster correlation, increasing the number of clusters allows detecting smaller changes in outcome than increasing households within a cluster the more similar households within clusters are, the more clusters are required to detect a change in outcome 4. desired minimum detectable effect in a particular outcome given the variability in the outcome high inherent variability of the outcome requires large sample size for several food security related variables, for example food consumption, there is high variability even in the absence of treatment 5. the desired power in detecting the effect at a desired significance level (set at 0.80 and 0.05 respectively)

16 Sample size requirements for different outcome variables: example from Uganda Outcome Food consumption per capita, in value terms Data source Suggested feasibility, per data source Intracluster correlation Standard deviation of outcome, as a proportion of mean Uganda Harvest+ Infeasible Clusters/gro up (g) HH/clust er (m) HH/group (m*g) Minimum detected effect, as a proportion of mean Uganda SFP 2005 Feasible Mean height forage z score Uganda DHS 2006 Tight even if feasible Proportion of stunted children Uganda DHS 2006 Possibly feasible Proportion of women making decisions re: household purchases Uganda DHS 2006 Possibly feasible