Econ 1629: Applied Research Methods Assignment 9: Assessing Empirical Studies

Transcription

1 Econ 1629: Applied Research Methods Assignment 9: Assessing Empirical Studies Part I: Case Study on Jamaica s PATH program 1 For most of this course, you have been asked to either conduct or interpret statistical analyses based on existing data. But some of you may end up occupying positions in which you will have the opportunity of designing or commissioning a study from the beginning (i.e. before the data are collected). The objective of this problem set question is to help you think about how you may do this by putting yourself in the position of the Director of the Social and Manpower Planning Division of the Ministry of Labour and Social Security of the Government of Jamaica. Read the case study Designing Impact Evaluations Assessing Jamaica s PATH Program that follows this question. The class on Monday November 24 (the day the problem set is due) will be focused on this case, and you are expected to come prepared to defend your choice of evaluation design. After reading the case, answer the following two discussion questions: 1. Imagine yourself in the role of the Director of the Social and Manpower Planning Division. Create a table or list in which you assess the strengths and weaknesses of each design. This table can be written in a technical language. Be sure to consider: a. The scientific quality of the design, keeping in mind the concept of validity discussed earlier in the course b. The political feasibility of implementing the design c. The logistical implications of the design, in terms of ensuring that findings from the evaluation are available in a timely manner for policymakers d. The financial implications of the design, in particular if it involves more resources than those already budgeted. 2. Write a one-page single-spaced memo to the Minister of Labour and Social Security recommending which design should be selected to evaluate PATH. Justify your recommendation using the strengths and weaknesses you identified above, but write the memo in non-technical language. Attach your table of strengths and weaknesses to the memo. 1 Created by Dan Levy. 1

2 Case Study Designing Impact Evaluations Assessing Jamaica s PATH Program 2 1. Introduction It is March 2003, and the Government of Jamaica s Ministry of Labour and Social Security is in the midst of a major reform of its social safety net. The World Bank helped finance this reform and has required the government to evaluate the cornerstone of the social safety net reform, PATH (Programme Advancement Through Health and Education). The government has selected a firm to evaluate the program, and is now in discussions with this firm about how best to evaluate the impact of the program. The firm has presented three possible evaluation designs, and the Minister of Labour and Social Security has assigned you, as the Director of the Social and Manpower Planning Division, to select a design. 2. Background on PATH In early 2000, the Government of Jamaica undertook a reform of its social safety net system, refocusing the system around PATH, a conditional cash transfer program. Through the conditional cash transfer program, eligible families received cash assistance conditional on regular attendance at school and regular checkups at health centers. This meant that once a family started receiving cash transfers, they would continue to receive them as long as they would continue to meet the program s conditions. 3 The primary objective of the program was to link social assistance with human capital accumulation. Another aim was to improve the targeting of welfare benefits over previous social assistance programs in Jamaica. With nearly 20% of the Jamaican population under the poverty line, targeting of previous social assistance programs was seen as inadequate. For example, only 36% of the benefits of the Food Stamps program were received by poor households. The PATH program sought to improve targeting of benefits to poor households by gathering detailed information about applicants and ranking them by level of poverty. Each household interested in the program would fill out an application form, reporting information on socio-economic and demographic characteristics. This information was entered into a computer system that used an algorithm to analyze each household s responses and calculate a household eligibility score. If the score was below a predetermined threshold, then the household was considered eligible, and invited to register in the program. If the score exceeded the threshold, then the household was not eligible for the program. Applicants were generally informed of their eligibility status 2-5 months after applying. The program was piloted in the parish of St. Catherine in 2001, and was expanded Island-wide in The Ministry of Labour and Social Security received most applications in the summer of 2002, and continued to receive applications until the summer of The number of individual beneficiaries who received PATH payments was about 180,000 of the 245,000 individuals who registered. The cash 2 This case was written by Dan Levy, Lecturer in Public Policy, as well as Michael McCreless and Daniel Björkegren. It was written for use in the Reasoning from Evidence course. Funding from the Kennedy School s SLATE Initiative is gratefully acknowledged. It is based on an evaluation conducted by Mathematica Policy Research (Dan Levy and Jim Ohls) for Jamaica s Ministry of Labour and Social Security and funded by The World Bank. The case does not necessarily reflect the views of any of these institutions. 3 The conditions for receiving benefits are as follows: Children 0-6 years old need to visit a health clinic every two months during the first year and twice a year thereafter. Children 7-17 years old need to attend school at least 85 percent of school days. 2

3 transfer amount given per eligible member of a household was about J$4004 per month (later increased to J$530) and the average amount received per household was J$1, Is PATH Effective? Even though many in the ministry were certain that PATH would help the poor, there were many unanswered questions. Would children in participating households attend school more often? Would participating households become healthier? In order to answer these questions, the Minister asked the firm to evaluate the causal impact of the program, using the most rigorous and credible evaluation design possible. In particular, this involves choosing an evaluation design that meets the following methodological criteria: The design should compare a participant group representative of some population of policyinterest, to a comparison group similar to PATH participants in: o Observable characteristics such as demographics and socio-economic status o Unobservable characteristics such as motivation or need for PATH (This is crucial to ensure the differences in outcomes between participant and comparison groups can be attributed to PATH and not to something else.) It is important to use large enough sample sizes to be able to reliably detect policy-relevant program impacts. The sample sizes needed to reliably detect PATH s impacts were 2,500 participant group households and 2,500 comparison group households. The evaluation should measure the effect of PATH 1-2 years after the participant group enrolled in the program, on at least two key outcomes of interest: o School attendance in the last 4 weeks o Attendance at a health center for preventive reasons in the last 6 months 4 In 2002, one Jamaican dollar was worth approximately two US cents. 3

4 4. Data The following data were available for use in the evaluation: PATH data on socio-economic and demographic characteristics for all applicants at the time they applied to the program Jamaica Survey of Living Conditions (SLC): Household survey on a representative sample of Jamaican households. Total Sample Size: 8,000 households. Conducted in the summer of 2002 (just before the program started in the non-pilot areas). Survey of PATH participants: household survey similar to SLC 2002, but on a representative sample of households who had been declared eligible for the PATH program. Total Sample Size: 1,200 households. Conducted in November In addition, there are resources available to conduct two more household surveys (each surveying a total of 5,000 households). 5. Evaluating Design Proposals The firm selected to evaluate PATH has proposed the following three evaluation design options, from which the Director of the Social and Manpower Planning Division must choose one: Design 1 Design 2 Design 3 Participant Group PATH participants from PATH Participant Survey Applicants who scored just below the eligibility threshold Randomly selected group of applicants who scored just above the threshold (normally ineligible) who will be offered to participate in the program Comparison Group Subset of households in the Summer 2002 SLC who are poor and are similar to the participant group on a set of observable characteristics Applicants who scored just above the eligibility threshold Randomly selected group of applicants who scored just above the threshold and who will not be offered to participate in the program Impact Estimation Using regression analysis, compare change in outcomes for participants to change in outcomes for comparison group and control statistically for a host of household and individual characteristics Using regression analysis, compare outcomes for participants to outcomes for comparison group and control statistically for a host of household and individual characteristics including the eligibility score Using regression analysis, compare outcomes for participants to outcomes for control group and control statistically for a host of household and individual characteristics 4

5 Special instructions for Part II: You may complete this assignment in groups. It is designed to help you move forward in completing the final exercise. As indicated in the final exercise document, each group should consist of no more than 3 students. Each group should submit only one copy of the assignment. Groups may not communicate with each other. Part II: Systematically and Critically Reading Empirical Articles: Final Exercise 5 Read the three articles assigned for the final exercise, and for each article, assess its strengths and weaknesses. To do so, complete the form in Appendix below (one sheet per article). Your response will be useful as you complete the final exercise, so I recommend that you keep a copy of your answers. 5 Created by Dan Levy. 5

6 Appendix: Assessing Empirical Studies Article/Author: Population and Treatment 1) What is the main goal of the paper? (e.g., assess the effect of class size reduction on students performance, assess the effect of a job training program on earnings, etc.) 2) What outcome measures are used? 3) What population is being studied? Methodology 4) What method is being used for the evaluation (Randomized Controlled Trial, simple OLS, Differences-in-Differences, Fixed Effects, etc.)? 5) What are the key threats to the internal validity of the study? 6) How would these threats affect the estimate (direction of bias)? Results 7) What is the range of the effects estimated by the study? Their statistical significance? Is there a preferred estimate designated by the author? 8) How big are these effects, from a policy or practical perspective? 9) What have you learned from this study and what are you unable to learn? 10) What are important issues for assessing the external validity of the study? 6