Summary report of the P4P Instrument Review workshop,

Transcription

1 Summary report of the P4P Instrument Review workshop, Nairobi, 4-5 February 2013 hosted by the African Economic Research Consortium Introduction In September 2008, WFP launched an innovative agricultural and market development support programme known as Purchase for Progress (P4P). Through P4P, WFP is testing ways to extend a part of its procurement demand to more directly benefit smallholder farmers with the capacity to supply to the quality standards required by WFP. The pilot is being implemented in 20 countries over a period of five years from 2009 to The learning emerging from this pilot programme about the most effective pro-smallholder procurement approaches and their impact on the productivity and income of smallholder farmers is potentially of interest and relevance not only to WFP, but to a wide variety of stakeholders engaged in agricultural and market development support to smallholder farmers. In March 2011, WFP signed a long term partnership agreement with the African Economic Research Consortium (AERC). AERC represents a network of 40 collaborating African universities and has a solid reputation in policy research. AERC supports WFP in the analysis and reporting of the quantitative data being collected by WFP to measure the impact of the P4P programme, and is also helping WFP to globally build stakeholder buy in to the emerging results from programme. In August 2012, they hosted the first global validation of P4P results on behalf of WFP. A range of academics and development experts were invited to explore the results and recommend improvements in the analysis and presentation of the data. This workshop was preceded by a meeting with WFP s Technical Review Panel 1 in June 2012, during which members similarly offered recommendations on how to strengthen the quantitative data analysis to tell a credible story of P4P s impact over the 5 years of implementation. In February 2013, AERC and WFP met in Nairobi, Kenya, to discuss the recommendations offered by the validation workshop participants and the TRP. The two day meeting attended by AERC staff and network members, members of WFP s P4P coordination unit and representatives from two of P4P s key partners (AGRA, Bill and Melinda Gates Foundation) also provided an opportunity for joint reflection on the essence of the story that the P4P experience will be able to tell. In particular, the participants were able to exchange on P4P s expected theory of change, discuss key modelling options to test it and review the change indicators as well as the data collection instruments. 1 The TRP is an independent, unremunerated group of experts that meets annually to discuss the progress of P4P implementation and offer WFP their guidance and advice on a range of implementation and M&E issues presented to them for input. TRP members are drawn from the following organizations: African Union, Catholic Relief Services, Food and Agriculture Organization (FAO), Inter-American Institute for Cooperation on Agriculture (IICA), International Food Policy Research Institute (IFPRI), Intermon-Oxfam (Spain), International Fund for Agriculture Development (IFAD), Michigan State University and World Bank.

2 Context P4P has just entered its final year of implementation and the focus of the project has now moved towards documenting the qualitative and quantitative impact of the project on farmer livelihoods and farmer organisation capacities. Over the course of the pilot, WFP has tested alternative ways to purchase food from smallholder farmers for use in its mainstream programmes. WFP assures steady demand implemented through pro-smallholder procurement modalities while partners with the relevant expertise support farmers on the supply side, which includes capacity-building activities. By committing to provide an assured, substantial market for a higher-quality product over a specific period, WFP expects to catalyse and deepen the impact of the activities of technical partners to build up farmers production and marketing capacities. Farmers and their organizations have an incentive and sufficient security to invest in their capacity to produce and market sufficient quantities of products of acceptable quality so that they can sell to buyers such as WFP and increase their profits. P4P aims to test the relevance of its development hypothesis: P4P development hypothesis Productivity increased + Profitable market access f(organization, markets, policy) = Improved livelihoods The global P4P experience is extremely complex and even though all the elements of this development hypothesis are present in each of the country pilots, the countries vary widely in their application of the different aspects, due to factors such as : The level of FO with which WFP engages (first tier, second tier, third tier 2 ) ; The emphasis on the different types of FO and household-level capacities that are being built 3 ; The approaches adopted by the country pilot (different types of WFP procurement modalities, direct vs indirect procurement from FOs e.g. through use of commodity exchanges, Warehouse Receipt Systems, engagement with traders and/or local processors etc.) 2 In P4P the FO is the channel to our ultimate target group, the smallholder farmer. This unit represents farmers and can enter into contracts with WFP on behalf of its members. The first tier FO is the simplest arrangement in which WFP has the most direct connection to the smallholder farmer. A second arrangement is where an umbrella group represents a number of first level organisations. The individual FOs gain access to capacity building and WFP contracts through their representative body. However, in a few countries, a third tier arrangement exists in which a type of super umbrella body represents a number of second level umbrella bodies. 3 Under P4P, seven core capacities are being built at the household level (capacity to increase production and minimize post-harvest food losses) as well as at the FO level in terms of FO strengthening (expanding business through increased capacity to strategically plan production and achieve the best return; supporting the financial and physical assets of the FO; improving the capacity to build relationships both within the FO as with other value chain actors) and in terms of FO market engagement (supporting the capacity to aggregate sufficient quantities of staples; improving the capacity to meet quality standards; and improving the capacity to negotiate with market actors). All of these capacities are being built to some extent in each of the P4P country pilots, but the emphasis on the different capacities may vary significantly across the countries.

3 National or regional context (this includes aspects such as policy, type of supply side support available to farmer organisations etc.). This diversity has to be taken into account in the analysis and interpretation of the quantitative outcomes. The learning and documentation process that is currently taking place will also feed into the final evaluation of P4P, which will be carried out in The evaluation will constitute a critical milestone for P4P as it will draw independent lessons and will test whether the key P4P assumptions are relevant and whether the programme objectives have been met. The evaluation will be greatly helped by the ready availability of quality of the information and therefore WFP and AERC are seeking to complete as much of the analysis and documentation as possible by December AERC has a key role to play in documenting the quantitative analysis. Data quality In each P4P country, data has been collected through extensive surveys both at household level and at the level of the FO. When relevant, a survey was also administered within a sample of traders. A baseline survey was carried out for each of these actors at the start of the project, whilst follow-up surveys have subsequently been carried out on a regular basis (in principle yearly for FOs and biannually for households). AERC noted the importance of the quality of data available for analysis. Considerable time has been spent by AERC in cleaning the available data sets and making them available for analysis. It has been a big challenge, given the scope of the project, to get clean and comparable datasets for all countries in the expected timeframe 4. Most of the inconsistencies have arisen for the household datasets, whilst problems with the FO and trader datasets have been generally easier to resolve. AERC noted that in some cases they had been required to reconstruct and re-enter data to maintain a uniform structure of the data across the different country datasets. However, some general limitations remain, such as attrition, missing cases and variables as well as poorly matched treatment and comparison groups. Further, specific issues have been observed with some key indicators, including: Overestimated income and expenditure Information on prices High variation in the household s agricultural production data for some IA and no-ia countries. Inaccurate (estimated) cultivated area and production measures Low values for post-harvest losses (which may result from a different definition of PHL) Missing Geographical Information System (GIS) codes. 4 At this stage, clean follow-up data is available for Tanzania, Ethiopia, Ghana, El Salvador, Kenya, Burkina Faso and Mali.

4 Different technical measures were proposed as possible means to mitigate the effect of such limitations on impact assessment. An important one is to check collected data with existing secondary sources. AERC concluded that data cleaning and re-construction is a continuous process and it will be pursued into the impact analysis phase. Impact pathway Out of the 20 pilot countries, four have been identified as candidate countries for thorough impact assessment (IA). In these countries, an experimental (Ghana) or quasi-experimental (El Salvador, Ethiopia and Tanzania) design has been set up, with the aim of comparing outcomes for P4P FOs and farmers with outcomes for similar non-p4p groups. For the non-ia countries, the analysis is expected to focus on the study of trends with the aim of confirming or contradicting the results from the IA countries. This quantitative analysis will be complemented with relevant qualitative information from case studies in both IA and non-ia countries. It is expected that P4P will have impacts both on farmers and their households as on their FOs. Specific datasets have indeed been collected for these two different groups of actors. The participants of the meeting were divided into two groups to reflect on the specific impact pathway that would need to be tested for both FOs and households given the available data in IA and non-ia countries. With this impact pathway, it is expected that a thorough P4P theory of change can be developed both for FOs and for the households, confirming or contradicting the key assumptions of the P4P project. The group discussions were guided by the following questions: What intermediate outcomes should we expect at this stage of implementation? How can these outcomes be modelled given available data? What, if any, changes to the data collection instruments or procedures would strengthen the impact assessment story? In doing so, the participants looked in detail in the recommendations made during the validation workshop and the TRP to discuss their applicability and how to move forward on them. The summarised results of the group discussions are detailed below: Livelihood theory of change: At the household level, the sample sizes allow for in-depth modelling, certainly in the IA countries. The break-out group managed to discuss anticipated outcomes and come up with modelling proposals in terms of indicators and potential explanatory variables. The summary of the group deliberations is provided in the table below:

5 Anticipated Outcome Indicator Explanatory Variables Timing of Outcome Production Use of improved inputs (improved/certified seed, fertilizer) Change in cultivated area Quantity produced Quantity of surpluses produced for staples Total expenditure/area Cultivated area by crop and total cultivated area Quantity produced (staples and aggregate) Percentage of HH producing surpluses and average size of surplus Access to credit; sex, age, and education of household head; HH income, training on improved inputs (treatment); FO provision of inputs (treatment); supply-side partner promoting fertilizer/seed use; access to fertilizer/seed services (question A2) Family size; relative gross margins of different crops (as explanatory variable in decisions about allocating land among crops); total cultivated area; assets; using credit to invest in agriculture; remittance income; characteristics of household head (education, sex); measures of market access; mechanization; lagged price Cultivated area; use of improved inputs; family size; use of credit to invest in agriculture; access to productivity enhancing services/inputs; use of hired labor; mechanization; lagged price Same as previous Post-harvest losses Post-harvest losses FO provides storage to members; training in post-harvest losses; use of storage practices (B12) Yield Yield Use of improved inputs; baseline yields; use of improved inputs; use of credit to invest in agriculture; access to productivity enhancing services/inputs; mechanization; lagged price Sales/Marketing Total quantity sold quantity sold; quantity Everything that affects quantity produced; Intermediate outcome but context important, e.g., availability, culture, etc. Might expect changes in allocation of existing area among crops initially with increase in total area later Might expect increase in staples production first with increase in aggregate production later (as total area increased) Tracks change in production of staples. Longer term outcomes as farmers change storage practices/behavior Intermediate outcome associated with improved inputs and practices

6 Anticipated Outcome Indicator Explanatory Variables Timing of Outcome sold/quantity harvested; quantity sold/quantity of surplus; % of households with sales % of sales through FO % of sales through FO Total sales by FO (per member); prices received by FO (and transmitted to farmers) relative to average price received by farmer; access to credit (production and consumption); services provided by FO (specific services?) Welfare Income (per capita) Expenditure farm income; farm income as % of total income (cash) Expenditure (per capita) Disaggregate by food. Other, etc.? Cultivated area; family size (# of employed members) dependency ratio); characteristics of household head (education, age, sex, employment status); lagged capital (transform assets into income) Same as previous Intermediate outcome. Interpretation not immediately obvious. Intermediate outcome Intermediate outcome Asset score Asset score Same as previous Longer term outcome (because most assets on which we have data are large) Livestock Livestock (asset value) Same as previous Outcome within one year Percentage of schoolaged children enrolled Food Consumption Score Percentage of schoolaged children enrolled Income, income from agriculture (?) Outcome within one year Food Consumption Score Income, same as previous Intermediate outcome

7 Household theory of change At the FO level, given the sample size, the break-out group concluded that for most of the countries, and definitely all the IA countries, the sample size is too small to enable regression modelling. The analysis is likely to focus on trend analysis and comparison of means between control and treatment groups for IA countries. As a result, no specific modelling options were discussed for each anticipated outcome. The summary of the group deliberations is provided in the table below:

8 Anticipated outcome Indicator Comments Collective Action Increase in FO aggregation Increased diversification in marketing channels Equity of access to FO collective sales Increased ability of FOs to aggregate on behalf of members Decrease in rejections due to quality requirements Governance Aggregated volumes (per capita) Disaggregated by farm size and gender Number and diversity of buyers/channels Individual contributions to sales Could be disaggregated by gender, farm size, Percentage FOs marketing on behalf of their members Percentage of production defaulted because of quality requirements Indicators available, but change to Season instead of last 5 sales. Indicators available. Could be complemented with a value chain analysis Available from FO records Indicator available from FO survey (A19, to be moved to B4) Indicator available from WFP procurement records. Extension or stabilisation of membership Female participation in leadership positions Increased FO planning for production and marketing Increased capacity to comply with contracts Capacity building No of members in the FO Percentage of men and women in leadership position Percentage of FOs engaged in Planning for production/marketing Compliance with contracts (timeliness, full/partial delivery) Indicator available Indicator available Indicator available in FO survey Need to check reliability of data Information available in WFP procurement records Increase in trained capacities Number of different types of trainings Indicator available in FO survey. Not an outcome indicator. Increased capacity to increase quality of production Percentage of FOs carrying out quality improvement activities Indicator available in FO survey (C8)

9 Increased access to credit for production/ marketing Access to credit for production/marketing Access, accepted amount and acceptance rate are available in FO survey However, need to add additional question on usage of credit (How, for whom, for what?). This should enable to make a link with credit use at the household level. Increased access to market Methods of price information Indicator available prices Increased records at FO level Percentage of FOs that keep records Available for those FOs sharing their FO records with WFP Need to add additional question in FO survey for more consistency (Does your FO keep records? If so, since when?) Increased amount of services provided to members of the FO Assets Increased access to storage Increased access to production and post-harvest infrastructure and equipment Number of services provided to the members Percentage of FOs with storage Storage capacity (volume) Could be disaggregated per type of ownership production and post-harvest infrastructure and equipment Indicator available (B3) Indicator available in FO survey (B1) Indicator available in FO survey

10 The break-out group also discussed P4P s key assumption that collective action is an effective way of addressing market failure in rural agricultural markets. Given its engagement with FOs, P4P is hopeful to provide evidence that selling collectively through FOs is an effective approach that actually reduces transaction costs, among others by improving the bargaining capacity of FOs and reducing market information asymmetry. Therefore, the measurement of market failures and transaction costs in general should be part of the key research topics for P4P. However, it is important to note that this is a very complex issue and certain data that would allow for a more robust analysis of this issue is not available e.g. reliable information on farm-gate prices. In general, the break-out groups were able to make significant progress regarding the way the recommendations from the validation workshop and the TRP should be taken on board. In particular, the participants have developed a clearer view of the P4P theory of change, identifying the key impact pathways. Specific modelling options were discussed for IA countries and broad modelling guidelines were identified for non-ia countries. At this late stage in the project, it was not deemed appropriate to extend the size of the samples, but some additional indicators were proposed to be included in the questionnaires, including retrospective questions. Finally, the importance to complement the quantitative analysis with qualitative information was reiterated. Management aspect of data cleaning and reporting Following a review of the current status of datasets at the hub and with country offices, it was realized that the follow-up data for a significant number of non-ia countries was not yet available. Some countries have not submitted clean follow-up datasets or some of the datasets have not been reconstructed to a usable standard as yet. Given the need to move forward on the analysis as quickly as possible, it was agreed that all the usable IA follow-up datasets should be available by the time of the next TRP end February. In terms of reporting, the need to facilitate the drafting process for the baseline and follow-up reports following the difficult experience of the first baseline reports that have been produced recently. Some quality issues have emerged, and the process has been considerably delayed compared to the initial schedule. Part of the problem has resulted from the large number of actors (DAKMAH, AERC regional coordinators, the P4P Country Offices and the P4P Coordination Unit) involved in the drafting process. The process and the role of each of these actors has to be defined more clearly in order to ensure the production of high quality and contextualised follow-up reports.