RESIDUAL FILES IN HLM OVERVIEW

Transcription

1 HLML09_ RESIDUAL FILES IN HLM Ralph B. Taylor All materials copyright (c) by Ralph B. Taylor OVERVIEW Residual files in multilevel models where people are grouped into some type of cluster (neighborhoods, organizations, ect.) are useful for several general purposes including: looking at results for an individual or a group. For example, in an ANOVA model you might want to see how far above or below the grand mean (G00) various individual neighborhoods are scoring (EBINTRCP), or you might want to look at the predicted score in a model for an individual. The residual files contain diagnostic information about how well your model is performing. You want to look at some of these. This is analogous to looking at the structure of your residuals in an OLS regression model. Describing the relationship between different model parameters. For example, in a model with varying slopes, in which you also are trying to predict group means, you can look at the relationship between a neighborhood s (residual or predicted) score on the outcome, and its predicted slope for a variable. They also are useful models of variation over time, of course. Perhaps most of interest is using the level 2 residual file to capture predicted scores of individuals for whom you have repeated observations over time. HOW TO CREATE You create a residual file by issuing the command for one. This is under the basic settings menu. specifications. If you are doing a 3 level model, you can request level 1 and/or level 2 and/or 3 level residual files. With a 2 level model you can request level 1 and/or level 2 residual files. NOTE in box at bottom you need to NAME the residual file. Tip. To keep these straight, I always end my residual file name with blabla_r.sav. I know that every file ending in _r is a residual file. It is a good idea if blabla corresponds to the name of the hlm run itself HLML_RESIDUALS Page 1

2 Tip. If you are going to be generating both level 1 and level 2 residual files, suggest you develop a strategy for keeping these straight like blabla_l2r.sav and blabla_l1r.sav. By default it is going to save these as SPSS type files. WHAT IS IN A LEVEL 2 FILE FOR A GROUPING HLM PROBLEM [NOTE the specifics of this illustration are geared to a data file with respondents in 45 Philadelphia neighborhoods.] First, there is a row of data for each Level 2 unit. So if you have 45 neighborhoods you should have 45 records. THE VARIABLES IN THE LEVEL 2 RESIDUAL FILE Tip. You will not see all these variables calculated for every problem, depending on things like weights, whether you are running a generalized model, and things of that nature. Do not be alarmed if it tells you it cannot calculate certain things. It looks like if analyses are weighted, you cannot get some of these diagnostics. ID$ NJ ID number for the L-2 unit number of L-1 cases in corresponding L-2 unit Variation across Level 2 Units CHIPCT "expected values of the order statistics for a sample size J selected from a population that is distributed Chi square(v)" where v=number of random effects per unit (HLM v. 4 manual, p. 32) These are expected values for lack of fit, for the overall model, for each Jth unit. These can be tested against the chi square distribution with one degree of freedom. In other words if you have a significant (> = p <.05) chi square, it is telling you that the observed values did not match up with the expected values, across a range of parameters in the model. If you have a significant chi squared value, or maybe even two, do not be alarmed. If you had 100 level 2 groups in your model, just by chance how many of these chi squared statistics do you think would be significant? HLML_RESIDUALS Page 2

3 MDIST These are Mahalanobis distances (D squared).d 2 is used in a wide variety of contexts, for example, as a measure of psychometric distance in a multi-variable hyperspace, or as a measure of effect size when there are multiple variables of interest. 1 In HLM what it is doing for each varying parameter is going into each group; seeing how sizable the discrepancy or distance of the residuals in a multivariate space are relative to the variation (distances) of the predicted scores; 2 such that a larger score means a larger lack of fit, as with the chi squared; and adding up across all the random parameters. The question about how do you find the statistical significance of D 2 complicated according to some. See Hess et al. (2007) for background. Raudenbush & Bryk (Hierarchical Linear Models, 2 nd edition, 2002: 274), however, suggest that with sufficiently large samples at level 1, this statistic will have a chi squared distribution with Q+1 degrees of freedom when the data are normal where Q are the random effects in the model. So if you have just varying intercepts, and no varying slopes, Q+1 = 2. The question, of course, is what is sufficiently large. Tip. D 2 can be plotted, in a P-P or Q-Q plot, to see if it is relatively normal. It should be, especially if there are a lot of level 2 units. In short, with D 2 the individual variables tell you about specific groups that are not fitting the model well (discrepancies between observed scores on a random effect like the intercept, and the predicted scores on a random effect like the intercept, and about the overall distribution of the Level 2 units. Remaining (Residual) Variation Within Each Level 2 Unit In a level 2 residual file these next three measures are measures of within group remaining residual variation. LNTOTVAR is the "natural log of total standard deviation within each L-2 unit" on residual Y scores. (V 4 manual, 32). 1 Hess, M. R., Hogarty, K. Y., Ferron, J. M., & Kromrey, J. D. (2007). Interval Estimates of Multivariate Effect Sizes: Coverage and Interval Width Estimates Under Variance Heterogeneity and Nonnormality. Educational and Psychological Measurement, 67(1), Raudenbush & Bryk (2002: 274) HLML_RESIDUALS Page 3

4 Tip. These can be negative, suggesting standard deviations of less than one. OLSRSVAR If you do an OLS regression model for each group, and log the standard deviation of the residuals, you get OLSRSVAR: the "natural log of residual standard deviation within each unit based on its least squares regression." (v 4. manual, p. 33) In other words, the standard deviation of raw (unstandardized) OLS residuals in each level 2 unit, logged. If you did OLS, this is your measure of residual variance for each level 2 unit. MDRSVAR When HLM has fitted its best model, you can get the same residual variation, logged. This is MDRSVAR: the "natural logarithm of the residual standard deviation from the final fitted fixed effects model." (v 4. manual, p. 33) In other words, the log of the standard deviation of the residuals after HLM has fit its best model. Tip. MDRSVAR should be plotted in a Q-Q plot or a P-P plot to see if it is relatively normal (Raudenbush & Bryk 2002: ). You should do this. Background on P-P / Q-Q plots: if you want more background Hamilton, Chapter 1. 3 Tip. Raudenbush & Bryk (2002: 264) recommend looking at a stem and leaf plot of MDRSVAR. In their HSB data they found some schools where respondents were particularly homogeneous (see Figure 9.2 on p. 266). They suggest adding additional Level 1 variables to try and remove at least some of the residual heterogeneity. (264). Predicted and Residual Intercepts and Slopes The Example The following parameter come from an example problem completed with unweighted PHMC 2006 survey data for Philadelphia only. Files _01.hlm _01.txt _01_l2_r.sav hlm problem setup hlm output file level 2 (neighborhoods) residual file 3 Hamilton, L. C. (1992). Regression with graphics. Pacific Grove, Calif.: Brooks/Cole. HLML_RESIDUALS Page 4

5 _01_l1r.sav level 1 (individuals) residual file Outcome = distrust; improve (2 = not work together, 1 = yes work together) has varying slope Level-1 Model: Y = B0 + B1*(RESPAGE) + B2*(IMPROVE) + B3*(FEMALE_D) + B4*(HS_LESS) + B5*(BLACK_D) + B6*(LATINO_D) + B7*(LT2XPOV) + R Level-2 Model B0 = G00 + G01*(STATUS) + U0 B1 = G10 B2 = G20 + U2 B3 = G30 B4 = G40 B5 = G50 B6 = G60 B7 = G70 In this example there is one set of residuals of interest: the residuals of the intercept. There is another random effect of interest: the variations in the slope of the IMPROVE variable. You can see the value of each of these for each level 2 unit. EBINTRCP Empirical Bayes best estimate of the residual L-2 value. These are your estimates of U0J. These are the discrepancies between the predicted score for each neighborhood and the EB estimate of that score: (EB estimate of intercept predicted intercept) G00 Tip. You probably want to check and be sure that your level 2 residuals (EBINTRCP) are uncorrelated with the level 2 predictors already entered in your model. (B&R p. 269). This is analogous to looking in OLS to be sure that e does not correlate with X. We expect homoscedastic residuals dispersed randomly around zero. (p. 269). You also can see if any of the level 2 residuals correlate with a level 2 predictor not yet entered. ECINTRCP These are the same as above except that the grand mean (G00) has been added in, so this equals (EB estimate of intercept predicted intercept) OLINTRCP OLS estimates of L-2 residual value: U0J. Note these have an average of zero, as HLML_RESIDUALS Page 5

6 they must. Note these have an average of zero, as residuals must axiomatically. Note the standard deviation of EBINTRCP is probably smaller than the standard deviation of OLINTCRP; the estimates are somewhat shrunken. Tip. If you graph EBINTRCP against OLINTRCP you will find that the latter is smaller relative to the former. How much the EBs shrink is a function of the remaining reliability of the U0js are people in the group still agreeing a lot with each other about the remaining discrepancies from the predicted values. FVINTRCP. In the Level 2 file these are predicted group scores. Very very useful. Tip. Sort the file based on these predicted scores and think about what places are high and what are low, and the implications. In this model a higher predicted score means more distrust. Tip. Look at the scatterplot of FVINTRCP (x) with EBINTRCP (y). There should not be too much of a correlation between the two; positive and negative residuals should be equally likely to appear all along the various predicted values, more or less. The residuals may bulge more in some places than others, but ignore unless extreme. If you allowed your slopes to vary, as we did in this model, then you will get some interesting information. Here we allowed IMPROVE (1 = yes worked together with neighbors, 2 = no did not) to have varying impacts across neighborhoods. There are three parameters of interest. FVIMPROV. EBIMPROV. OLIMPROV. This is the average slope and corresponds to the value you see in the output, G20 =.356. This is each neighborhood s Empirical Bayes estimated deviation around from the average slope, each neighborhood s U2j. The variance of these slopes adds up to T22. This is the OLS estimate of the deviation between a neighborhood s slope and the average slope across all neighborhoods. Tip. It is not a bad idea to take a look at the OLS slope deviations compared to the EB slope deviations, to see how much they have shrunken, and to see, overall, what kinds of adjustments HLM has made here. HLML_RESIDUALS Page 6

7 ECIMPROV This is each neighborhood s predicted slope (G20 + U20) on improve Tip. Not a bad idea to take a look at the relationship between a neighborhood s predicted slope and it s predicted score. Useful information. Tip. Also extremely useful to look at each neighborhood s deviation from the grand intercept and deviation from the grand slope. Gives you a sense of how hard HLM is working! EBZAGE Empirical Bayes estimate of the discrepancy between the average slope for z scored age, and the slope in each L-2 unit. This is: U*1J Will be dramatically shrunken when compared to OLS estimate of same which is: OLZAGE OLS estimate of same: U1J HLML_RESIDUALS Page 7