Application: Effects of Job Training Program (Data are the Dehejia and Wahba (1999) version of Lalonde (1986).)

Transcription

1 Application: Effects of Job Training Program (Data are the Dehejia and Wahba (1999) version of Lalonde (1986).) There are two data sets; each as the same treatment group of 185 men. JTRAIN2 includes 260 men as a control group. Assignment to the training program was random. N 445 in total. Simple comparison of means should be sufficient. JTRAIN3 uses a control group drawn from the Current Population Survey (CPS). Here, there are 2, 490 men in the control group, and many of them look nothing like the treated group or the control group from the experiment. In particular, overlap is very poor in JTRAIN3. 1

2 . * Use the experimental data first.. use jtrain2. des Contains data from jtrain2.dta obs: 445 storage display value variable name type format label variable label train byte %9.0g 1 if assigned to job training age byte %9.0g age in 1977 educ byte %9.0g years of education black byte %9.0g 1 if black hisp byte %9.0g 1 if Hispanic married byte %9.0g 1 if married nodegree byte %9.0g 1 if no high school degree mosinex byte %9.0g # mnths prior to 1/78 in expmnt re74 float %9.0g real earns., 1974, $1000s re75 float %9.0g real earns., 1975, $1000s re78 float %9.0g real earns., 1978, $1000s unem74 byte %9.0g 1 if unem. all of 1974 unem75 byte %9.0g 1 if unem. all of 1975 unem78 byte %9.0g 1 if unem. all of

3 . tab train 1 if assigned to job training Freq. Percent Cum Total sum re78 Variable Obs Mean Std. Dev. Min Max re count if re

4 . sum unem74 unem75 re74 re75 educ if train Variable Obs Mean Std. Dev. Min Max unem unem re re educ sum unem74 unem75 re74 re75 educ if ~train Variable Obs Mean Std. Dev. Min Max unem unem re re educ di ( )/sqrt(3.219^ ^2) * So the normalized difference for re75 is much less that the Imbens-Rubin. * ROT,.25. For other variables, even smaller. So overlap seems fine, as. * it should with random assignment. 4

5 . reg re78 train, robust Linear regression Number of obs 445 F( 1, 443) 7. Prob F R-squared Root MSE Robust re78 Coef. Std. Err. t P t [95% Conf. Interval train _cons

6 . * Difference of means estimate is $1,794, and statistically significant.. reg re78 train age educ black hisp re74 re75, robust Linear regression Number of obs 445 F( 7, 437) 3. Prob F R-squared Root MSE Robust re78 Coef. Std. Err. t P t [95% Conf. Interval train age educ black hisp re re _cons * Slightly smaller estimate, but could just be sampling error. 6

7 . gen avgre (re74 re75)/2. reg re78 train age educ black hisp re74 re75 if avgre 15, robust Linear regression Number of obs 433 F( 7, 425) 3. Prob F R-squared Root MSE Robust re78 Coef. Std. Err. t P t [95% Conf. Interval train age educ black hisp re re _cons

8 . logit train age educ black hisp re74 re75 Logistic regression Number of obs 445 LR chi2(6) 8. Prob chi Log likelihood Pseudo R train Coef. Std. Err. z P z [95% Conf. Interval age educ black hisp re re _cons predict phat (option p assumed; Pr(train)) 8

9 . gen kate (train - phat)*re78/(phat*(1-phat)). * Average the kate to get ATE:. sum kate Variable Obs Mean Std. Dev. Min Max kate * Estimate is 1.63, which is pretty close to the regression adjustment. * estimate of reg kate Source SS df MS Number of obs F( 0, 444) 0. Model 0 0. Prob F Residual R-squared Adj R-squared Total Root MSE kate Coef. Std. Err. t P t [95% Conf. Interval _cons

10 10

11 . * The unadjusted standard error is.844. Theory tells us this is conservative. * Generate score from first-stage logit.. gen uh train - phat. gen ageuh age*uh. gen educuh educ*uh. gen blackuh black*uh. gen hispuh hisp*uh. gen re74uh re74*uh. gen re75uh re75*uh 11

12 . reg kate uh-re75uh Source SS df MS Number of obs F( 7, 437) 44. Model Prob F Residual R-squared Adj R-squared Total Root MSE kate Coef. Std. Err. t P t [95% Conf. Interval uh ageuh educuh blackuh hispuh re74uh re75uh _cons

13 . predict ehat, resid. * The residuals are kate with the score netted out.. sum ehat Variable Obs Mean Std. Dev. Min Max ehat e di 13.58/sqrt(445) *.644 is the adjusted standard error, compared with * New t statistic is. di 1.63/ * Much closer to the regression adjustment t statistic. 13

14 . * What about regression on the propensity score". reg re78 train phat, robust Linear regression Number of obs 445 F( 2, 442) 4. Prob F R-squared Root MSE Robust re78 Coef. Std. Err. t P t [95% Conf. Interval train phat _cons sum phat Variable Obs Mean Std. Dev. Min Max phat

15 . gen train_phat train*(phat -.456). reg re78 train phat train_phat Source SS df MS Number of obs F( 3, 441) 4. Model Prob F Residual R-squared Adj R-squared Total Root MSE re78 Coef. Std. Err. t P t [95% Conf. Interval train phat train_phat _cons

16 . * Now nearest neighor matching. Just one nearest neighbor, ATE.. * Actually, the variance is computed as if we want the sample. * ATE (which is the same estimate as for population ATE, but. * calcuation of standard error is easier).. nnmatch re78 train age educ black hisp re74 re75 Matching estimator: Average Treatment Effect Weighting matrix: inverse variance Number of obs 445 Number of matches (m) re78 Coef. Std. Err. z P z [95% Conf. Interval SATE Matching variables: age educ black hisp re74 re75. * Almost identical to other estimates; somewhat less precise. 16

17 .* Nonexperimental data: control group is drawn from CPS.. use jtrain3. tab train 1 if in job training Freq. Percent Cum , Total 2,

18 . sum re78 Variable Obs Mean Std. Dev. Min Max re sum unem74 unem75 re74 re75 educ if train Variable Obs Mean Std. Dev. Min Max unem unem re re educ sum unem74 unem75 re74 re75 educ if ~train Variable Obs Mean Std. Dev. Min Max unem unem re re educ

19 . * Lack of overlap is clearly a problem.. di ( )/sqrt(3.22^ ^2) * The normalized difference for re75 is more than one in absolute. * value.. * Can further see the problem if we ask: What if we try to match on. * the propensity score? 19

20 train = 0 Density Pr(train) 20

21 train = 1 Density Pr(train) 21

22 . * Comparison of means gives very different result now.. reg re78 train, robust Linear regression Number of obs 2675 F( 1, 2673) 537. Prob F R-squared Root MSE Robust re78 Coef. Std. Err. t P t [95% Conf. Interval train _cons

23 . * Regression adjustment, also controlling for marital status (which matters. * now):. reg re78 train age educ black hisp married re74 re75, robust Linear regression Number of obs 2675 F( 8, 2666) 253. Prob F R-squared Root MSE 10. Robust re78 Coef. Std. Err. t P t [95% Conf. Interval train age educ black hisp married re re _cons

24 . * Interactions with continuous variables don t help much:. sum age educ re74 re75 Variable Obs Mean Std. Dev. Min Max age educ re re gen train_age train*(age ). gen train_educ train*(educ - 12). gen train_re74 train*(re ). gen train_re75 train*(re ) 24

25 . reg re78 train age educ black hisp married re74 re75 train_age train_educ train_re74 train_re75, robust Linear regression Number of obs 2675 F( 12, 2662) 184. Prob F R-squared Root MSE Robust re78 Coef. Std. Err. t P t [95% Conf. Interval train age educ black hisp married re re train_age train_educ train_re train_re _cons

26 26

27 . * What if we drop all observations with (re74 re75)/2 15? Comparison. * of means still does not "work":. reg re78 train if avgre 15 Source SS df MS Number of obs F( 1, 1160) 37. Model Prob F Residual R-squared Adj R-squared Total Root MSE re78 Coef. Std. Err. t P t [95% Conf. Interval train _cons

28 . * But regression adjustment does.. reg re78 train age educ black hisp married re74 re75 if avgre 15, robust Linear regression Number of obs 1162 F( 8, 1153) 82. Prob F R-squared Root MSE Robust re78 Coef. Std. Err. t P t [95% Conf. Interval train age educ black hisp married re re _cons

29 . * But all is not well: matching using the restricted dta set does not. * produce a positive effect:. nnmatch re78 train age educ black hisp married re74 re75 if avgre 15 Matching estimator: Average Treatment Effect Weighting matrix: inverse variance Number of obs 1162 Number of matches (m) re78 Coef. Std. Err. z P z [95% Conf. Interval SATE Matching variables: age educ black hisp married re74 re75 29

30 . * Now estimate the propensity score by logit using all data:. logit train age educ black hisp married re74 re75 Logistic regression Number of obs 2675 LR chi2(7) 872. Prob chi Log likelihood Pseudo R train Coef. Std. Err. z P z [95% Conf. Interval age educ black hisp married re re _cons Note: 158 failures and 0 successes completely determined.. * It is not good to perfectly predict failures or successes completely.. * In effect, p(x) 0 for some values of x. 30

31 . predict phat (option pr assumed; Pr(train)). sum phat Variable Obs Mean Std. Dev. Min Max phat e count if phat

32 . reg re78 train age educ black hisp married re74 re75 if phat.1 & phat.9, robust Linear regression Number of obs 309 F( 8, 300) 6. Prob F R-squared Root MSE Robust re78 Coef. Std. Err. t P t [95% Conf. Interval train age educ black hisp married re re _cons

33 . gen kate (train - phat)*re78/(phat*(1 - phat)). sum train Variable Obs Mean Std. Dev. Min Max train gen katt (train - phat)*re78/(.06916*(1 - phat)). reg kate Source SS df MS Number of obs F( 0, 2674) 0. Model 0 0. Prob F Residual e R-squared Adj R-squared Total e Root MSE 991. kate Coef. Std. Err. t P t [95% Conf. Interval _cons

34 . reg katt if train Source SS df MS Number of obs F( 0, 184) 0. Model 0 0. Prob F Residual R-squared Adj R-squared Total Root MSE 113. katt Coef. Std. Err. t P t [95% Conf. Interval _cons * The PS weighted estimate of ATT seems unbelievably large and. * much too significant.. * Should redo the analysis with observations having phat.1 & phat.. * but can get some idea what restricting the sample will do: 34

35 . reg kate if phat.1 & phat.9 Source SS df MS Number of obs F( 0, 308) 0. Model 0 0. Prob F Residual R-squared Adj R-squared Total Root MSE kiate Coef. Std. Err. t P t [95% Conf. Interval _cons * This is remarkably similar to the regression estimate when restricted. * to.1 phat.9.. * General point: when the sample has been balanced, method of. * estimation is much less important.. * Note that many fewer observations are lost by choosing sample based on. * avgre, rather than the Imbens/Rubin propensity score rule. 35