Global Youth Economic Opportunities Summit. Meghan Mahoney Justin Loiseau

Transcription

1 INFORMING SCALE: USING RIGOROUS EVIDENCE TO EXPAND YOUTH LABOR PROGRAMS Global Youth Economic Opportunities Summit Meghan Mahoney Justin Loiseau

2 OUTLINE Introduction Developing a theory of change Considering spillover Considering capacity 2

3 USING EVIDENCE Today s focus: how to use rigorous evidence, not if/when to conduct randomized evaluations 3

4 INTRODUCTION 4

5 WHAT IS J-PAL? J-PAL s mission is to ensure that policy is informed by evidence and research is translated into action. 5

6 WHAT IS J-PAL? J-PAL s network of 120+ professors use randomized evaluations to inform policy. 6

7 GLOBAL EVIDENCE BASE We have more than 680 ongoing and completed projects across eight sectors in 60+ countries. 7

8 SCALING UP J-PAL EVIDENCE 250 People reached by the scale up of evidence-based programs evaluated by J-PAL affiliates (millions)

9 J-PAL SCALE-UPS BY PROGRAM Program Raskin: ID cards for distributing in-kind transfer Remedial education School-based deworming Generasi: Conditional community block grants Chlorine dispensers for safe water TOTAL People Reached million million million 6.7 million 1.86 million million 9

10 BEYOND SCALE: INFORMING WITH EVIDENCE Commissions and government partnerships Training courses for policymakers Policy publications Policy Influence examples Police skills training Free insecticidal bednets Third party pollution audits 10

11 DEVELOPING A THEORY OF CHANGE 11

12 INCREASING EVIDENCE, BUT WHAT NOW? Dramatic rise in impact evaluations, but: Unlikely to be rigorous evaluation of precisely the program policy makers wants to introduce in exactly same location How should we respond? Wait for more evidence? New rigorous evaluation? Use less rigorous local evidence? Use results from study in another context? Use evidence from other countries if at least [insert arbitrary number here] replications or if replicated in a similar enough context?

13 A (BAYESIAN) APPROACH TO EVIDENCE Evidence Probabilistic Theorem Priors Reference: Kremer and Glennerster, 2012

14 YOUTH (UN)EMPLOYMENT IN INDIA Understanding national issues can be a first step to understanding how similar contexts could suggest similar solutions. World s largest youth population Youth unemployment rate is nearly 20% (2013, most recent data) One in three youth with college degree are unemployed Skills gap or mismatch with employers References: Govt. of India, 2013; Ernst and Young, 2013; Mitra and Verik, 2013

15 THE ORGANIZATIONS Understanding who partner organizations are and how they function can help inform how policy makers consider similarities and differences in implementation. National Employability Through Apprenticeship Program (NETAP) Government program Goal: Become largest employability scheme in India ACME Employment Largest private sector service provider under NETAP

16 EVIDENCE OF ON-THE-JOB TRAINING In Colombia, a vocational training program that combined classroom and on-the-job training improved employment and earnings for female participants. Ongoing research testing apprenticeship programs in France (Crépon et al.), India (Bertrand et al.), and Ghana (Hardy et al.) References: Attanasio et al., 2011; J-PAL Policy Briefcase, 2012; Bertrand et al.; Hardy et al. 16

17 THE BLACK BOX APPROACH TO EVIDENCE Knowing this, should we encourage ACME to implement an apprenticeship program in India? What is our evidence of the following relationship? Problem statement: youth unemployment Program: apprenticeship incentives and matching Intended impact/program goal: higher youth employment Only a few randomized evaluations on apprenticeships Positive non-experimental evidence from Europe

18 THEORY OF CHANGE: APPRENTICESHIPS IN INDIA Youth want to be employed Youth lack relevant skills Employers want additional labor Youth know about/can access program Local Basic conditions descriptive evidence suggest basic conditions hold Appropriate youth selected for apprenticeships Training content is relevant and sufficient to master skills Youth accept apprenticeships Youth complete apprenticeship training Process Emerging Behavioral RCT evidence of behavioral conditions Youth master relevant skills Employers value apprentices work Youth gain permanent employment Youth have higher wages Emerging Impact RCT evidence of impact

19 THEORY OF CHANGE KEY TAKEAWAYS Delineating a theory of change, and dissecting the links can provide insight when direct evidence is lacking Design research for generalizability Policy making requires drawing on different kinds of evidence, but that does not mean all evidence is equal Question for the end: How could you use this in your work? 19

20 CONSIDERING SPILLOVER 20

21 GENERAL EQUILIBRIUM EFFECTS AND SPILLOVERS Considering impact at scale: How might the average impact change at scale? What is the impact of implementing my program on those who do not participate? Particularly important to consider in certain labor market interventions 21

22 POSITIVE SPILLOVER EXAMPLE: BEDNETS 22

23 NEGATIVE SPILLOVER: DISPLACEMENT Those who don t participate in my program are made worse off by my program Employed Unemployed 23

24 HOW DOES SEARCH COUNSELING AFFECT EMPLOYMENT? In 2007, more than one quarter of recent college grads were out of work. Youth s lack of experience and unfamiliarity with workplace environments could make youth unattractive candidates. Solution: Two-stage counseling to help youth find and maintain longterm employment References: Crépon et al, 2013; J-PAL Policy Briefcase,

25 COUNSELING HAD SHORT-TERM EFFECT, DISAPPEARED IN LONG TERM Individuals in the treatment group found work more quickly than peers in comparison group But gains disappeared after 12 months Rates of Long-term Employment, Over Time References: Crépon et al, 2013; J-PAL Policy Briefcase,

26 RESEARCHERS VARIED PROPORTION OF TREATED JOB SEEKERS. Variation in Proportion of People Treated, Across Communities References: Crépon et al, 2013; J-PAL Policy Briefcase,

27 POSITIVE IMPACTS DO NOT NECESSARILY TRANSLATE TO THE WHOLE POPULATION Surprising results: counseling created a displacement effect, lowering employment rates for comparison group individuals living in treatment areas Displacement effects larger in tighter job markets Employment Rates among Treated and Untreated Job Seekers References: Crépon et al, 2013; J-PAL Policy Briefcase,

28 KEY TAKEAWAYS Important to think about spillovers When considering scale, also important to think about how effect size may change Size of the effect Duration of effects Distribution of effects among population Question for discussion at the end: How does your organization consider this?

29 CONSIDERING CAPACITY 29

30 THE EVIDENCE ELSEWHERE 30

31 EVIDENCE ELSEWHERE Country Goal Incentive Value (USD) Daily wage equivalent Effect India Immunization Lentils $ % Final immunization Metal plates $ % 6% to 18% (immunization camp) to 39% (adding incentive) Immunization Sugar $ % India Final immunization Mustard oil $ % (results forthcoming) Malawi Collect HIV results Collect HIV results Cash $ % 35% to 75% Cash $ % 35% to 85% Pakistan DTP3 immunization Food coupon $ % 22% to 52% Kenya Immunization Immunization Mobile money Mobile money $ % (results forthcoming) $ % (results forthcoming) 31

32 PROVEN THEORY OF CHANGE Basic conditions Incentive program Parents want to vaccinate Can access clinic Provider presence sufficient Behavioral Process Parents procrastinate or are deterred Completed schedule salient Incentives delivered to clinic Incentives given to parents Impact Small incentives offset bias Parents repeatedly immunize Full immunization rises Improved health 32

33 ASSESSING BASIC CONDITIONS Do they want the program? (demand) Institutional Perceived need to increase immunization rates Government approves Partner(s) motivated Programmatic Parents want to vaccinate Incentive type and amount salient Can they incorporate it? (supply) Institutional Reliable health infrastructure Partners have capacity to implement program Funding available Programmatic Incentive procured and distributed Households can access clinic Note: non-exhaustive 33

34 DISCUSSION 34

35 SAMPLE DISCUSSION QUESTIONS What are the challenges you face in determining if and how to use evidence to inform your program? What is your process for deciding whether or not to scale? Have you used a theory of change to inform your program? What is an example? What are the main ways in which your program might change as a scaled program? What will need to change about your current process? How might the impact of your program differ at scale? What are the largest implementation challenges (expected or experienced) for scaling your program? 35

36 THANK YOU 36