Executive Summary: Understanding the impact of rural and agricultural finance on clients. Learning Lab Technical Report No. 2.1.

Transcription

1 Executive Summary: Understanding the impact of rural and agricultural finance on clients Learning Lab Technical Report No.. December 05 EXECUTIVE SUMMARY: -

2 Introduction The purpose of the Rural and Agricultural Finance Learning Lab s report, Understanding the impact of rural and agricultural finance on clients, is to: Articulate a theory of impact for rural and agricultural finance Assess the current evidence base for the impact of financial products on smallholder and rural households, particularly in Africa Highlight current approaches and challenges faced in measuring the impact of rural and agricultural finance Recommend a path forward for addressing current evidence gaps Rural and agricultural finance refers to financial solutions for poor agricultural and rural households. These financial solutions can include credit, savings, insurance, or payment products, with providers often bundling two or more of these products into a more comprehensive offering for their clients. There is great diversity within RAF offerings; some products and services are customized specifically to agricultural needs, such as input loans or weather-index insurance, while others, such as savings accounts, are suitable for more general customer needs. Figure lays out a theory of impact (ToI) that captures different pathways to impact for RAF. The ToI starts with inputs on the left in this case, financial solutions (and associated products and supporting services) and progresses through outputs and outcomes to impacts. Vertically, the ToI is partially divided into two parts the top focuses on agriculture-specific financial solutions while the bottom captures non-agriculture specific solutions. This is meant to highlight the different pathways, though in actuality there is both overlap and convergence, especially at the impact level (as depicted). In practice, a complex range of other factors influence the realization of any impact pathway. In particular, the inputs (or financial products) alone do not determine impact. As illustrated in Figure below, developing a complete perspective on the impact of RAF also requires assessing the ways in which product experience, customer characteristics, and the enabling environment, among other factors, contribute to impact. Section I: Theory of impact We recognize that clients of financial solutions may not think in terms of these product categories, but it is useful when assessing impact to use a product characterization recognizable to providers, even while noting the need for new perspectives about the offering. We include in the Annex the theories of change of several key stakeholders in RAF for reference purposes. EXECUTIVE SUMMARY: -

3 Figure : Rural and agricultural finance theory of impact Inputs Outputs Outcomes Impact Agriculture- specific financial (and complementary) solu7ons Ag credit (working and investment capital) Ag- specific savings* Insurance (e.g., crop, livestock, price) Payment services in ag value chains Technical assistance / agronomy training Upstream and downstream market linkages General financial solu7ons Broader (non- agriculture specific) financial solu5ons: Non- ag credit Savings Insurance (life, health, etc.) Payment solueons Less likely to be complemented with a non- financial solu7on EffecEve use of more and beher inputs (incl. labor) Purchase/rent of produceve assets (incl. land, irrigaeon equip.) InstallaEon of storage/ processing faciliees Formal proteceon of farm revenues Efficient, safe access to input and output markets Increased financial liquidity Investment in off- farm income- generaeng aceviees Purchase of savings- generaeng or producevity- enhancing assets (e.g., solar lamp) AccumulaEon of savings Formal proteceon against shocks Access to remihances Higher yields / surplus produceon Less volaelity in output Reduced postharvest losses Control over Eming of surplus sales Reduced distressed sales of farm assets Increased sense of security Lower transaceon costs, losses BeHer pricing beyond farm gate Access to higher quality inputs Ability to make bulk payments (e.g., school fees) DiversificaEon of income sources Reduced household expenses / enhanced producevity Financial buffer against shocks Reduced distressed sales of assets Increased sense of security / improved psychological state Improved nutrieon and food security Higher and more stable net agricultural income and consumpeon Recurring investment in farm producevity Wealth accumulaeon Improved household well- being (educaeon, health) Strengthened household resilience Ability to contribute to community Figure : Factors influencing impact outside of the standard results chain Enabling environment: Ecosystem and contextual factors such as policy and culture INPUTS OUTPUTS + OUTCOMES IMPACT Customer: Client characteristics, e.g., age, gender, initial wealth EXECUTIVE SUMMARY: -

4 THE CUSTOMER S PERSPECTIVE To understand more what a theory of impact might look like coming from actual beneficiaries smallholder clients the Dalberg team employed a human-centered design approach in conducting in-depth interviews with 8 farmers in Ghana and Kenya. The sample was, of course, not meant to be representative, as the objective of this qualitative research was to try to surface new farmer-generated insights that might inform future approaches to measuring impact. The farmers we interviewed including financial service clients and non-clients cited measures of success similar to the impact goals noted by providers, funders, and researchers; however, the farmers expressed a distinct order of priorities. The farmers interviewed had a very clear prioritization of end goals, typically starting with meeting an essential need, such as putting enough food on the table, before progressing through a series of steps perceived as representing increased success. The figure below shows how a smallholder finance client might start out at a certain point along an axis of priorities and his/her priorities evolve over time. Understanding the priorities of farmers is important not just for the design of financial solutions which is beyond the scope of this report but also for the measurement of impact. By understanding where a particular customer is along the curve of household needs or outcome priorities, finance providers or researchers can design more effective evaluations of impact and improve the accuracy of impact assessment efforts. Figure : Evolution of smallholder measures of success EXECUTIVE SUMMARY: - 4

5 Section II: Evidence to date A literature review conducted by EPAR offers a starting point for assessing the evidence base. The Learning Lab commissioned this report to identify all the methodologically rigorous studies on the impact of rural and agricultural finance in Africa on four outcome areas of interest: ) production, ) income and wealth, ) consumption and food security, and 4) resilience. FINDINGS FROM THE ACADEMIC LITERATURE A literature review conducted by EPAR offers a starting point for assessing the evidence base. The Learning Lab commissioned this report to identify all the methodologically rigorous studies on the impact of rural and agricultural finance in Africa on four outcome areas of interest: ) production, ) income and wealth, ) consumption and food security, and 4) resilience. 4 The empirical evidence base for rural and agricultural financial inclusion is limited. Ultimately, only 8 studies qualified for inclusion in the literature review based on EPAR s filter criteria; 5 ultimately, Production includes measures of total agricultural production as well as measures of farmer productivity such as yield per hectare or technical efficiency. Resilience is defined as the ability to cope with shocks and/or smooth consumption. Production includes measures of total agricultural production as well as measures of farmer productivity such as yield per hectare or technical efficiency. 4 Resilience is defined as the ability to cope with shocks and/or smooth consumption. 5 The researchers conducted a systematic search of academic and gray literature databases to identify a total of 08 studies. They then carefully filtered these studies based on these considerations: i) a focus on Sub-Saharan Africa as the continent s unique agricultural context makes it difficult to generalize the findings of studies from other regions eliminated over 60% of the studies; ii) measuring a dependent variable significantly outside the four outcome areas of interest (e.g., measuring the impact of an intervention on access to finance, which is not a question of ultimate impact on the customer) eliminated a further 45 or nearly 90% of the remaining studies; and iii) application of rigorous methodologies experimental, quasi-experimental, and econometric methodologies with reasonable sample sizes and careful implementation eliminated a final 4 studies. the Dalberg team included 9 studies in our review. 6 Twenty of those studies focus on credit, while other product categories remain relatively understudied. It is notably difficult to aggregate lessons from the available studies because definitions of impact and product types vary significantly. Nevertheless, the available studies suggest that financial products can make a real impact on smallholder and other poor rural clients, but that this impact is far from guaranteed. Of 9 studies, record only positive, statistically significant impact and nine find a mix of significant and not significant, but still positive, results. Six papers fail to find any evidence of significant impact (either positive or negative). Only two studies both of them focused on group-liability business loans find any evidence of a negative impact. 7 The remaining studies find a mix of positive and not significant results. RESULTS FROM PRACTITIONER EVALUATION EFFORTS Practitioner research generally shows positive effects but also highlights areas to improve; this research is iterative and has been expanding in scope. Even where RAF providers internal measurement, evaluation, and learning (MEL) efforts do not meet the same standards of rigor as academic research, findings from these efforts are important and complementary to the academic research base. We reviewed the impact measurement results of five organizations whose work was deemed relevant to this study. 6 Dalberg excluded one study as being of limited relevance but added a further two studies based on recommendations from interviews and a supplementary consultation of the literature. 7 The 9th study offers an interesting insight about the interaction between different financial products mainly insurance and credit but does not directly measure impact on the household (Gine and Young, 007). EXECUTIVE SUMMARY: - 5

6 Table : Mapping evidence to impact framework IMPACTS OUTPUTS AND OUTCOMES Total number of studies Consump>on and food security Income Non- material household well- being Households assets and resilience Produc>on Use of inputs Ag credit 5 Crop Insurance Broader Rural Finance Non- Ag. Credit 5** Savings 7 Payments 4 Insurance 5 Exclusively posi.ve results Mix of posi.ve and non- significant results Mix of posi.ve and nega.ve results Exclusively non- significant results Exclusively nega.ve results EXECUTIVE SUMMARY: - 6

7 Practitioners primarily measure impact on agricultural production/yields and net income; they generally find that their programs have a positive effect on these measures. However, not all impacts are positive, and providers can use disappointing findings to improve their programs. EVIDENCE GAPS Significant research gaps remain across the board. First, although the bulk of the literature has focused on credit, the evidence around agricultural lending is inconclusive and merits additional investigation. In addition, there are general gaps in evidence around the impact of savings, payments, and insurance products. As some of these products continue to gain traction with customers e.g., mobile money developing a better understanding of their impact will be critical. Finally, there is little quantitative evidence around the influence of such factors as delivery method, customer characteristics, or larger contextual considerations (e.g., ecosystem, policy, or culture). Table : Summary of practitioner evaluation results Total number of RAF providers with studies reviewed* Consump)on and food security Income IMPACTS Non- material household well- being Households assets and resilience OUTPUTS AND OUTCOMES Produc)on Use of inputs Ag credit** Ag savings Other*** Broader Rural Finance Exclusively posi.ve results Mix of posi.ve and nega.ve results Mix of posi.ve and non- significant results Exclusively non- significant results EXECUTIVE SUMMARY: - 7

8 Section III: Measurement approaches PRACTITIONER APPROACHES TO MEASURE- MENT AND EVALUATION Practitioners and their funders are recognizing the importance of and increasing investment in measuring the impact of their programs. For some, this has meant collecting more data or identifying comparison groups. For others, internal impact assessments are being supplemented with externally-led impact evaluations (IEs). Qualitative approaches to evaluation and learning are also valuable for practitioners, providing a more in-depth customer understanding and often guiding the design of future quantitative research efforts. For example, interviewees highlight that case studies serve multiple purposes. First, anecdotes can provide inspiration for and inform the design of quantitative research: many organizations have success stories that they feel capture their impact, but may not always have the data to back them up. By illuminating how a program is changing outcomes, these stories can help organizations formulate specific hypotheses to test in future quantitative research. Individual stories also offer a more holistic perspective on program impact, adding color to quantitative data. They can also raise flags in what is largely an average-dominated field; as one interviewee described it, they allow the evaluation officer to double click on phenome- na they see reflected in the data. Finally, case studies play an important role in fundraising and awareness building, as they offer very tangible, personal examples of success. Participatory monitoring and evaluation is a qualitative approach that makes MEL more client-centric and can make research designs more relevant. As the name implies, in participatory evaluation, a broader set of stakeholders actively participate in developing and implementing the evaluation. Practitioners/providers, program beneficiaries (or clients in the case of RAF), and funders all play a part, and participation occurs throughout the evaluation process. Dalberg utilized participatory methods in-depth, one-on-one farmer interviews to inform this research effort. Taking care to identify a diversified sample of target interviewees based on characteristics such as gender, age, experience with RAF, type of crop farmed, etc., we spent about two hours with each interviewee on his/her farm, learning about his/her families, daily activities, experiences with financial products, and hopes and aspirations. The degree of comfort cultivated by embedding ourselves in the farmers settings allowed for an unparalleled level of depth and honesty in sharing experiences. This allowed us to garner a wide array of insights into their lives and circumstances and lent additional nuance to how we think about measuring impact by seeking to capture the multiple dimensions of the farmer. Another key benefit was the sense of reward that farmers felt in contributing to the solution generation process. Generally, practitioners seek to balance different approaches in a MEL framework that is just right for their organi- EXECUTIVE SUMMARY: - 8

9 zation. The goal is to balance the need for information on impact, i.e., how is your program affecting your client, with operational data to inform management decisions, i.e., data on program rollout, efficiency, financial sustainability, etc. To do this effectively, RAF providers must also take into consideration ) the target audience, ) the available operational and financial resources, and ) the potential burden on customers. RAF sponsors and providers interviewed noted that monitoring, evaluation and learning plans typically undergo frequent revisions to remain relevant to the organization and build on lessons from early rounds of data gathering. OVERCOMING CHALLENGES TO MEASUREMENT Rigorous measurement by practitioners is challenging, and while many of the difficulties exist in other fields as well, the rural/smallholder farmer context often makes them even more acute. Practitioners and researchers describe a number of key challenges across the MEL process, including: Design: Challenges related to defining metrics for capturing impact, setting up control groups, and aligning research agendas between researchers and providers. Implementation: Low accuracy of self-reported metrics (e.g., income), high costs of data collection, respondent fatigue, operational constraints when implementing randomized control trials (RCTs), and competing demands on scarce human resources. Results: Differences in ideal time horizons for data to inform operations and impact evaluations, errors in data cleaning and/or analysis, misinterpretation of results contributing to inaccurate recommendations, and low generalizability of findings. An important theme that emerges from the long list of challenges is the complexity of implementing and generating meaningful insights from rigorous impact evaluations, including randomized control trials. All of the stakeholders we interviewed acknowledge that IEs can be expensive, difficult to implement, and disruptive to operations. However, they diverge in their opinions as to whether navigating these difficulties is worthwhile. Some note that IEs offer a valuable, unbiased measure of their impact. This is useful both internally for example, in helping to calibrate in-house MEL or determine whether or not to scale up and in justifying future investments by funders and donors. Others consider the design and implementation challenges insurmountable and the results not sufficiently generalizable in many cases. Views on IEs are further complicated by the fact that these studies vary significantly in terms of time, cost and operational requirements to implement, as well as in the technical rigor achieved. The experiences of practitioners and researchers highlight possible solutions to measurement challenges. In some cases, these solutions focus on the adoption of different evaluation techniques or data collection tools. In other cases, solutions involve increased communication or new models of collaboration between practitioners, researchers, and/or funders. For example, rather than having an external researcher own the end-to-end RCT process, RAF providers could consider conducting RCTs in-house and engaging ex- EXECUTIVE SUMMARY: - 9

10 ternal researchers to consult on up-front study design or interpretation of results. This type of approach allows the RAF provider to maintain greater control over the RCT implementation, while drawing on research providers expertise in the design and/or analysis stages. Our team s field work also highlights lessons around incorporating qualitative methods in evaluation and learning. In demonstrating a participatory evaluation technique, we note six key insights for undertaking qualitative field research or client-centered design processes. Use a proven methodology/technique to promote rigor of research and generate more valid results. Speak to a purposive sample of beneficiaries, a non-representative sample focused on client characteristics of interest to the research aim. Utilize local experts to mitigate cultural barriers arising from different social norms. Preserve transparency of the respondent s voice by paying attention to factors that may prevent farmers from presenting their views honestly and transparently, including the presence of community leaders or partner organizations. Mitigate distractions to the participant s narrative, including their past experience interviewing or respondent unease from use of video to record interviews. Revise approaches on an ongoing basis to ensure that the interview process and content reflect emerging findings. Section IV: A path forward As we have seen from the evidence, financial products certainly can but do not always generate impact for smallholders and other rural clients; there is still much to learn about what works, for whom, and in what setting. In particular, future research should focus on understanding understudied solutions (e.g., digital transactions) and the key factors influencing the impact of RAF (e.g., customer profiles). External researchers, internal MEL teams, and funders can contribute to filling these gaps in evidence. Much of this measurement can be done in-house by qualified MEL staff and used to fine tune program design. In some cases for esample, for more complex evaluations and/ or when an additional degree of evaluator neutrality is useful - practitioners will benefit from partnering with external researchers. Funders also have an important role to play by dedicating more funds to RAF impact measurement. To be relevant and effective, this additional research should take into account emerging best practices from measurement efforts to date. In particular:. Maintain a balance of rigor and relevance measure what matters, rather than measuring for the sake of measurement, and do not let measurement supersede high quality service delivery;. Be flexible consider what you can learn given tradeoffs involved and organizational constraints, rather than simply focusing on what you want to learn; EXECUTIVE SUMMARY: - 0

11 . Create value for all parties involved clients should understand how the data they are asked to provide will be used to help them, field staff need to have a vision for how measurement will improve operations, etc.; 4. Reflect client priorities identify the customer segments you are targeting, consult with clients to understand what success looks like from their perspective, and incorporate this into the MEL approach; 5. Look beyond the short-term impact look to understand how intended and unintended effects unfold and endure over time. The rural and agricultural finance community will benefit from greater collaboration around a common set of research goals going forward. RAF stakeholders should explore formal and informal mechanisms for ongoing sharing and collaboration that will help to disseminate research findings and improve MEL practices across the sector. EXECUTIVE SUMMARY: -

12 About this study The Rural and Agricultural Finance Learning Lab is a MasterCard Foundation initia tive to foster learning and collaboration that leads to better financial solutions provided to more smallholder farmers and other rural clients. The Lab was launched in 05 and is run jointly by Dalberg and the Global Development Incubator. Among other activities in support of the Foundation s Rural and Agricultural Finance (RAF) portfolio and the broader RAF community, the Learning Lab researches key questions that comprise its learning agenda. These questions are centered on impact and measurement, client demand, financial solution provision and ecosystem development. At the impact level, the Learning Lab is interested in how fi nancial solutions contribute to poverty reduction and improved livelihoods for smallholder and rural households. For this, the Lab s first major study, a Dalberg team looked at this question from theoretical, empirical, and methodological angles; i.e., how do we think about impact, what do we currently know about impact, and how do we measure impact. The study builds upon the literature review of RAF impact in Africa commissioned by the Lab and conducted by the Evans School Policy Analysis and Research Group (EPAR). The EPAR literature review is an important companion to this study, providing additional details on the evidence presented in Section, and can be downloaded here: The Lab is committed to actionable and collaborative learning, and we invite the engagement of our readers, including feedback on this report, contributions of additional data/information, or questions/suggestions about future areas of study. At our website, users can contact the Lab directly or comment on this or any other publication. We are on or the Rural and Agricultural Finance professional group on LinkedIn.