A post-lunch parable. *Adapted from speech by Vivek Maru, CEO of Namati Legal Empowerment

Transcription

1 A post-lunch parable *Adapted from speech by Vivek Maru, CEO of Namati Legal Empowerment

2 What do you see as the primary purpose of impact evaluations? A. Reporting requirement for donors B. As a fundraising tool to show the world how effective your enterprise is C. To guide internal organizational decisionmaking D. As a research tool to improve our general understanding of development E. Other

3 What is the primary obstacle to running a rigorous impact evaluation in your organization? A. Technical expertise to design and analyze the evaluation B. Expense C. Long timeline D. Not useful for guiding organizational strategy E. Intrudes on normal operations/at odds with business approaches F. None G. Other

4 Rigorous impact evaluation as a practical decision-making tool for social enterprises Neil Buddy Shah, MD, MPA/ID New Delhi, India

5 Bad evaluation can lead to poor decision-making % of households owning cell phone, Millennium Villages Project, Kenya 60% 61% 53% 40% 40% 20% 0 13% 6% 4% 10%

6 Rigorous impact measurement requires a counterfactual Impact is defined as a comparison between: 1. the outcome some time after the program has been introduced 2. the outcome at that same point in time had the program not been introduced (the counterfactual ) The key is to select a comparison group that is exactly like the group of participants in all ways except one: their exposure to the program being evaluated Source: 6

7 Rigorous impact evaluation can be a practical tool for social enterprise Demand-driven Questions sourced from social enterprise. Experimental design optimized keeping in mind ground-level realities of client Timely & affordable - Proximate outcome indicators - Short, tailored surveys - Smaller, policy relevant sample size Rigorous Most rigorous field experiment approaches (including, but not limited to, randomized control trials) Context-specific Goal is to provide actionable results for a specific organization in a specific geography and specific timeframe Rigorous impact evaluations can be practical decision making tools for social enterprises

8 IE as a decision-making tool: case #1 RCT of latrine financing in Cambodia Social Enterprise: ide Cambodia Operational question: What is the most cost-effective way to increase sales & use of ide-designed latrines in rural Cambodia? Evaluation method: Randomized evaluation Outcomes: % of non-latrine owners in each village who purchase a latrine Self-reported latrine usage Timeline: January March 2013 (plus follow-up usage survey) Cost to client: $54,000 ($10 million total project size)

9 The Challenge 80% of Cambodians lack hygienic sanitation

10 The Decision-Makers ide Country Director and Gates Foundation WASH lead:

11 Potential Solution 1 Marketing campaign, sell latrines at market price

12 Potential Solution 2 Marketing campaign + offer loans for latrines

13 Randomization and sampling strategy Maximizing internal and external validity subject to operational constraints Sampling strategy: 1. Restrict to 1 province because of MFI staffing constraints 2. Randomly select 30 villages in Kampong Thom (similar to other provinces in northern Cambodia) 3. Randomly choose 15 villages to receive financing and 15 to be non-financing 4. Randomly select 50 non-latrine owners per village, stratified by ID Poor status: 35 non-id Poor HHs 15 ID Poor HHs

14 Treatment and control groups balanced at baseline indicates that randomization produced a good counterfactual Median baseline latrine coverage* Marketing-only Villages Marketing+Financing Villages % ID Poor* % 23.1% 30.0% 28.4% Average market price* USD USD Market price variation* (42.50 USD, 57.5 USD) (40.00 USD, USD) Payment At delivery 12-month MFI loan # of Villages * Differences are not statistically significantly different 1 ID Poor is the official poverty line in Cambodia

15 Sales 50% 40% 30% 20% 10% 12% 0% Just marketing

16 Sales 50% 50% 40% 30% 20% 10% 12% 0% Just marketing + financing

17 Financing causes a FOUR FOLD increase in sales at market price Difference is statistically significant at the 1% level Dotted bars indicate 95% confidence intervals

18 Financing reduces sales and marketing cost per latrine sold by 70%* * Reflects some assumptions about component costs.

19 Impact evaluation as a practical decision-making tool Decision-oriented and Demand-driven ide committed before evaluation to act on findings Responsive to decision timeline (fast) 3 months, start to finish Proximate outcomes; Analyze & report immediately; No publishing delays Affordable Minimize sample - decision-relevant effect size Short, focused surveys Rigorous Randomized evaluation Did not skew normal business operations Strong case for generalizability to northern Cambodia

20 Case Study #2: Kepler Background Social Enterprise: Kepler University, Rwanda Project: Does a low-cost model of online course content and teaching assistants increase student learning outcomes compared to normal university? Evaluation methodology: Propensity score matching (with difference-in-difference) analysis Outcomes: Test scores (critical thinking, problem solving, quantitative reasoning) Computer skills (typing, finding information online, Excel) Timeline: Baseline: Oct. 13; Midline: Jan. 14; Endline: Jun. 14 Cost to client: $43,000 + $8,000 for testing company

21 Case Study #2: Kepler In the client s words

22 Case Study #2: Kepler Evaluation design Baseline Treatment group: Comparison group: 50 Kepler students 200 students from 4 other universities match 100 from this pool

23 Case Study #2: Kepler Evaluation design: Matching Baseline Treatment group: Comparison group: 100 Kepler students 200 students from 4 other universities match 100 from this pool Matching: Gender: Male Age: 19 Poverty level: 3 High school GPA: 3.4 National exam score: 82 CLA+ score: 45 Grit score: 4.5 Expected salary: 700,000 Gender: Male Age: 20 Poverty level: 3 High school GPA: 3.3 National exam score: 85 CLA+ score: 43 Grit score: 4.6 Expected salary: 500,000 Gender: Male Age: 19 Poverty level: 3 High school GPA: 3.5 National exam score: 79 CLA+ score: 46 Grit score: 4.2 Expected salary: 900,000

24 Case Study #2: Kepler Evaluation design Test score Test score For illustration only Not the actual test scores 63% 68% 42% 43% 49% 56% Kepler Comparison Baseline Kepler Kepler Comparison Comparison Endline Endline

25 Case Study #2: Kepler Shortcomings of non-randomized experiment Unobserved characteristics (what could some be?) may differ between Kepler and non-kepler students and could affect test scores Matching design requires a larger sample size than randomized evaluation design Analysis can be more complicated than with randomized evaluation Matching design tends to be more expensive, all else being equal

26 Guiding principles of decisionoriented impact evaluation 1. What specific decision will the impact evaluation inform? What actions will you take if you find a negative effect? Zero effect? Positive effect? 2. Can you get results on the timeframe you need before you have to make a decision? Are there proximate outcomes that you can quickly measure that are closely linked to the ultimate outcome you care about? 3. Is the cost of the evaluation justified given the size/importance of the decision it will inform? 4. What financial and human resources do you need to act upon the findings of the impact evaluation? 5. Can you create a comparison group that is very similar (on observable and un-observables) to the group that receives your program?

27 Faster evaluation allows for more experimentation & iteration Product/ser vice design Process pilot Impact evaluation Scale up support INNOVATE LEARN IMPROVE