Impact Assessment & Social Performance Management

Size: px

Start display at page:

Download "Impact Assessment & Social Performance Management"

Ralph O’Neal’
6 years ago
Views:

1 Impact Assessment & Social Performance Management by Friederike Lenel and Katja Kirchstein Project co- financed by the European Commission

2 IMPACT ASSESSMENT 2

3 Impact Assessment A. INTRODUCTION 3

4 A. Introduction: What is an Impact Assessment? What? Study of the influence and impact of a (microfinance-) programme on the target group Have a concrete vision of the outcome rather than of the process Who? Donors and consultants MFIs and consultants Academics Why? Interest in the effective granting of funds Improvement of the financial product and/or process in terms of target group Interest in the influence of MF as a contributor to poverty reduction

5 A. Introduction: What is an Impact Assessment? Have the clients experienced social Have the clients experienced social and economic improvements? and economic improvements? Can those improvements be attributed to institutional activities? Intent and Design Internal systems/ activities Outputs Outcomes Impacts Social Performance Measurement and Management Impact Assessment Source: CGAP

6 Impact Assessment B. TYPES OF IMPACT ASSESSMENT 6

7 B. Types of Impact, Unit of Assessment and Indicators Economic impact Vs. Units of assessment Individuals Enterprises Households Community (costly) Social impact Income Patterns of expenditure Assets Health Litteracy Empowerment

8 Impact Assessment C. VARIOUS METHODOLOGIES 8

9 C. Various Methodolgies of Impact Assessments 3 different models of impact assessment Scientific model Humanities Participatory Learning and Action Who Who Who Academics Donors and consultants MFIs and consultants How quantitative methods Analysis of pre and post situation by using multivariable regression (average treatment effect) Control group approach Existing data How Qualitative methods Also geographical, sociological and anthropological Data collection and analysis undertaken by the same actors How Group discussions Multiple realities Importance of the perception of the target group

10 C. Various Methodolgies of Impact Assessments Problems: -Quantitative data is hard to obtain and costly -Little representativeness -Only actual clients assessed

11 C. Various Methodolgies of Impact Assessments Solution: Randomnized Control Trials!? Practitioners, donors and academics are turning to randomnized experiments to address the fundamental challenge in impact evaluations the need to clearly establish that the intervention caused the impact FAI

12 Impact Assessment D. RANDOMIZED CONTROL TRIALS 12

13 D. Randomized Control Trials Randomized Control Trials quantitative method experiments similar to the onse realized by medical researchers Problems to be solved: the counterfactual: seperate the effect of environmental factors control goups Selection bias within countrol groups: isolate the effect of unobservable characteristics such as motivation (measurable characteristicse eg. age, education) Solution: two groups are (eg. borrowers and non-borrowers) are selected at random With sufficiently large samples: certainty about two groups are similar in both observable and unobservable characteristics intervention is the only systemic difference between groups 13

14 D. Randomized Control Trials Challanges of Randomized Control Trials in Practice: Sample size: large samples are expensive an time consuming but are more reliable Level of randomnization: individual or group level (eg. villages, village banks etc.); group level randomization requires a larger sample. Enforcement of initial random assignment: Non-compliance, attrition, spillovers Ethical concerns: is temporal exclusion of possible clients in line with institutions mission? 14

15 D. Randomized Control Trials Limitations of Randomized Control Trials : Limited generalization: experiments implemented in a specific organisation in a particular setting No insights into distribution of impacts 15

16 D. Randomized Control Trials Examples of Randomized Control Trials : Banerjee et al Karlan and Zinman 2009 Where? Slums of Hyderabad, INDIA Manila What? Effects on the creation and profitability on small business, investment and consumption Effect of credit expansion to microentrpreneurs Sample Size: 104 Slums 1, 601 Individuals Duration: 18 month 21 month Findings: HETEROGENOUS -No effect on avarage monthly expenditure -Durable expenditures (investments) and profits increased in case of existing businesses -Decrease in non-durable consumption in case of high propensity to become business owner (otherwise increase in consumption) -No impact on health, education of women s decision making HETEROGENOUS -Increased access to informal finance (risk management) -Substitution for formal insurance -Increase in profits for males but not for females -No increase in subjective well-being, rather decline -Males use profits to send children to school -Decrease in houshold members employed outside the familiy business 16

17 Impact Assessment E. PERCEPTION OF IMPACT ASSESSMENT 17

18 E. Perception of Impact Assessments (April 2010) Big Banks Draw Profits From Microloans to Poor >>Underlying the issue is a fierce debate over whether microloans actually lift people out of poverty, as their promoters so often claim. The recent conclusion of some researchers is that not every poor person is an entrepreneur waiting to be discovered, but that the loans do help cushion some of the worst blows of poverty. The lesson is simply that it didn t save the world, Dean S. Karlan, a professor of economics at Yale University said about microlending. It is not the single transformative tool that proponents have been selling it as, but there are positive benefits. << 18

19 E. Perception of Impact Assessments >> As microfinance practitioners, we have witnessed the positive impact of microfinance first-hand...the media s interpretations of several recent studies on the impact of microfinance, however, have questioned whether microfinance has made a quantitative improvement in the lives of the borrowers, or has had any effect on poverty alleviation on a systemic basis...such studies face two fundamental challenges: their ability to capture and analyze all the benefits of microfinance, and the duration of the study itself << 19

20 Impact Assessment F. STAKEHOLDERS 20

21 F. Important Stakeholders for Impact Assessments Jonathan Morduch Managing Director, Lead Researcher Dean Karlan Director, Lead Researcher Sendhil Mullainathan Director, Lead Researcher 21

22 F. Important Stakeholders for Impact Assessments + >>TRANSLATING RESEARCH INTO ACTION<< Network of 44 affiliated professors Randomitzed Evaluations for Poverty-Fighting Policies MIT: Esther Duflo, Abhijit Banerjee 22

23 Transition Impact Assessment Costly & time-consuming For MFIs not affordable to assess on an ongoing basis Methodologies often theoretical & unsuitable for practitioners Reasons for impact/no impact clouded what s behind the scene? Outcomes setting specific, not comparable across MFI s Social Performance Assessment Once a SP tracking system is implemented, SP can be assessed on an ongoing basis and by MFIs themselves SPM tools practitioner-friendly Since whole process assessed, the roots of specific problems can be identified Outcomes transparent and comparable with SP Assessments of other MFIs Social Performance focuses on the entire Impact-Creation-Process: Intent and Design Internal systems/ activities Outputs Outcomes Impacts Social Performance Management Impact Assessment 23

24 SOCIAL PERFORMANCE MANAGEMENT 24

25 Social Performance Management A. DEFINITION 25

A. Definition Social Performance is the effective translation of an institution s social mission goals into practice in line with accepted social values; these include: sustainably serving increasing

26 A. Definition Social Performance is the effective translation of an institution s social mission goals into practice in line with accepted social values; these include: sustainably serving increasing numbers of poor and excluded people improving the quality and appropriateness of financial services, improving the economic and social conditions of clients, and ensuring social responsibility to clients, employees and the community by Social Performance Task Force, CGAP (2007)) USAID (2006) 26

27 A. Definition SP effective translation of MFI s social mission into practice SPM the system that MFI uses to track, understand & improve social performance (Source: SPM in MF Imp-Act Guidelines, IDS, 2005) 27

28 A. Definition Social Performance SP effective translation of MFI s social mission into practice Social Performance is related to the entire Impact-Creation-Process: Intent and Design Internal systems/ activities Outputs Outcomes Impacts mission of institution? clear social objectives? what activities to achieve mission? systems in place and designed to achieve those objectives? progress routinely tracked? reaching large number of intended clients (poor and very poor people)? products designed to meet their needs? have clients experienced social and economic improvements? can improvements be attributed to institutional activities? 28 IFAD (2006)

29 A. Definition SPM the system that MFI uses to track, understand & improve social performance The instutionalization process of translating mission into practice, which includes setting clear social objectives, tracking social performance and using this information to improve practice. 29

30 Social Performance Management B. SPM in Practice B. MAIN PLAYERS 30

31 B. Main Players In the last years many different tools and frameworks (most of them publicly available) were developed with initiators coming from different kind of backgrounds; e.g.: Governmental Agencies Research Centers Rating Agencies Foundations MFI Networks 31

32 Social Performance Management SPM in Practice C. TOOLS 32

33 C. Tools SPM is a process institutionalizing the tracking and monitoring of Social Performance. Not a single tool!!! however, several tools have been developed to facilitate this process, focusing on different aspects of the SP process.

34 C.I. SPM Resources The available tools focus on different stages of the SP process Intent and Design Internal systems/ activities Outputs Outcomes Impacts CERISE SPI ACCION Social Audits Social Rating Tools USAID SPA Tool PPI, FCAT, PAT, SEEP/AIMS Tool, Microsave Tools, SIM, ILS, QUIP

35 C.I. SPM Resources Checking if intentions are aligned with actions, if mission is actually reflected by activities? Assessment of the Institutional Process Analyzing the actual conditions of clients and if any change occurred? Assessment of Client Condition and Poverty Overall assessment of Social Performance? SP Assessment internally externally

36 C.I. SPM Resources Specific Assessment Tools Institutional Process a) SPI (CERISE) b) SPA (USAID) c) QAT (MFC) Client Conditions and Poverty a)ppi (CGAP) b)fcat (FINCA) c)pat (CGAP) SP Assessment Internally a)accion b) MIX/SPTF SP Indicators Externally a) Social Rating 36

37 C.II.1. Institutional Process Specific Assessment Tools Institutional Process SPI (CERISE) Client Conditions and Poverty PPI (CGAP) SP Assessment Internally MIX/SPTF SP Indicators Externally Social Rating Planet Rating

38 C.II.1. Institutional Process a) CERISE Social Performance Indicators (SPI) CERISE (2008)

39 C.II.2. Client Conditions and Poverty Specific Assessment Tools Institutional Process SPI (CERISE) Client Conditions and Poverty PPI (CGAP) SP Assessment Internally MIX/SPTF SP Indicators Externally Social Rating Planet Rating

40 C.II.2. Client Conditions and Poverty a) CGAP/Grameen/Ford Progress out of Poverty Index (PPI) Simple Poverty Scorecard for Bolivia Schreiner (2009) 40

41 C.II.3. SP Assessment - Internally Specific Assessment Tools Institutional Process SPI (CERISE) Client Conditions and Poverty PPI (CGAP) SP Assessment Internally MIX/SPTF SP Indicators Externally Social Rating Planet Rating

42 C.II.3. SP Assessment b) Social Performance Task Force Social Performance Indicator Framework Downloadable at MIX

43 C.II.3. SP Assessment - Externally Specific Assessment Tools Institutional Process SPI (CERISE) Client Conditions and Poverty PPI (CGAP) SP Assessment Internally MIX/SPTF SP Indicators Externally Social Rating Planet Rating

44 C.II.3. SP Assessment - Externally Planet Rating Planet Rating (2010) 44

45 Social Performance Management D. BENEFITS, CHALLENGES AND OUTLOOK 45

46 D. Benefit, Challenges and Outlook D.I. Benefits Uses of SPM: Baseline information for future studies Tracking performance against targets (MFI, Branch, Loan officer level) Identifying problems Portfolio segmentation Monitor product/service use; product development Staff and client learning USAID (2006)

47 ACCION(2007) 47

48 D.II. Challenges Among many others: - Buy-in -Compatibility of Social and Financial Performance (Mission Drift vs. Synergies) - Choosing the right indicators (universal, comparable, time-sensitive, measurable,inexpensive to collect AND representing social objectives) 48

49 D.III. Outlook Indicator definition Standardization even possible, or desirable? Development of norms Setting right incentives for social performance Role of governmental organizations (e.g. Monitoring...)

Thank you for your attention! University Meets Microfinance umm@planetfinance.org www.universitymeetsmicrofinance.

50 Thank you for your attention! University Meets Microfinance PlaNet Finance 13 rue Dieumegard Saint-Ouen Paris FRANCE (0)

51 Informative Websites SPM Resourcecenter: Microfinance Gateway: 260/ microlinks: Social Performance Task Force: Imp-Act: CERISE:

52 References ACCION (2007) Guidelines to Evaluate Social Performance, Insight No.24, ACCION CERISE (2008) Operational Guide to the SPI questionnaire Version 3.0, CERISE Bédécarrats et al. (2009) Is Social Performance Profitable?, MicroBanking Bulletin, Issue 19, p CGAP (2007) Beyond good Intentions: Measuring the Social Performance of Microfinance Institutions, Focus Note No.41, CGAP Copestake (2007) Mainstreaming Microfinance: Social Performance Management or Mission Drift, World Development, p Cull et al. (2008) Microfinance Meets the Market, Journal of Economic Perspectives, vol. 23(1), p IFAD (2006) Assessing and Managing Social Performance in Microfinance, IFAD Imp-Act (2005) Social Performance Management in Microfinance Guidelines, Imp- Act Microfinance Centre for CEE and MIS - Ramm, Penh (2009) Social Performance Measurement and Reporting in Microfinance (Powerpoint), Workshop of Cambodian Microfinance Association Schreiner (2009) A Simple Poverty Scorecard for Bolivia Social Performance Task Force (2009a) Overview of Social Performance (Powerpoint), Social Performance Task Force, Social Performance Task Force (2009b) Overview of Social Rating (Powerpoint), Social Performance Task Force, USAID (2006) Pilot Test of the new Social Performance Audit Tool, MicroReport No. 51 USAID

Social Performance, Impact and Governance: Framework for Evaluating and Supporting MFIs Reach Their Social Objectives

Social Performance, Impact and Governance: Framework for Evaluating and Supporting MFIs Reach Their Social Objectives Cécile Lapenu (Cerise) CERMi Seminar March 5th, 2008 CERISE Network for exchanging